arXiv Computation and Language
The Geometry of Low-Resource Language Representations
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Jul 23
The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
arXiv:2607. 20301v1 Announce Type: new Abstract: Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks.
arXiv AI
Aug 20
Latent Space Refusal Anchoring for Low-Resource African Languages: Mechanistic Safety Recovery Without Retraining
The paper introduces Latent Space Refusal Anchoring (LSR‑Anchoring), a training‑free technique that extracts a refusal direction from English prompts and applies it to the residual stream of instruction‑tuned models at inference time. The primary variant, Mean‑Activation Steering (MAS), works across several architectures (Llama‑3‑8B, Llama‑3.1‑70B, Mistral‑7B‑Instruct, Qwen2.5‑7B), restoring safety for low‑resource African languages with minimal performance loss, while a refined SAE‑Derived Steering (SDS) further reduces KL divergence without degrading legitimate prompt performance. The method shows positive transfer for Yoruba, Igbo, Igala, and Hausa, but fails for Arabic, suggesting a geometric mismatch rather than a data scarcity issue.
arXiv AI
Oct 12
Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
The paper introduces Representation Geometry Regularization (RGR) to address capability degradation observed in On-Policy Self-distillation (OPSD) for large language models. OPSD can cause previously correct responses to become incorrect even as overall performance rises, a problem linked to deteriorating hidden representation geometry—specifically, less compact intra-cluster and weaker inter-cluster separation. RGR preserves this geometry by encouraging cluster compactness and separation during post‑training, and can be added to existing OPSD methods without changing their core optimization, with experiments showing it mitigates degradation while maintaining improvements.
arXiv Computation and Language
Aug 27
One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
The study evaluates how different input representations—orthographic text, IPA transcription, and romanization—affect cross‑lingual transfer in autoregressive multilingual language models. Across three model sizes and eight languages grouped into typologically motivated pairs, romanized pretraining consistently outperforms native orthography and IPA, especially as model scale increases. Fine‑tuning a text‑pretrained model on romanized data can harm performance on languages already covered by the base model, suggesting romanization should be integrated at pretraining rather than applied later.
arXiv Computation and Language
3d ago
Scaling Multilingual Pretraining of Pixel Language Models
Pixel language models treat text as rendered images, enabling strong cross‑lingual transfer. This study trains MPIXEL models varying languages, scripts, and capacity while keeping per‑language compute constant, and evaluates them on five tasks across 83 languages. Results show that scaling improves performance on added languages and scripts, mitigates the curse of multilinguality, and outperforms subword and character baselines on typologically distant low‑resource languages, though transfer to unseen scripts does not benefit further from scaling.
arXiv AI
Aug 10
Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages
arXiv:2605. 02608v2 Announce Type: replace-cross Abstract: Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-resource settings remains poorly understood.