← Back to all news
arXiv AI By Mingyu Jin, Yutong Yin, Jingcheng Niu, Qingcheng Zeng, Wujiang Xu, Mengnan Du, Wei Cheng, Zhaoran Wang, Tianlong Chen, Dimitris N. Metaxas

Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs

Read the original on arXiv AI →

The paper examines how Large Language Models’ internal representations become sparser as task difficulty rises, finding that harder inputs lead to more concentrated activation patterns in the last hidden state. This sparsity–difficulty relationship holds across various models and domains, indicating that models learn dense representations for familiar patterns but default to sparse activations for unfamiliar ones. Leveraging this insight, the authors propose Sparsity‑Guided Curriculum In‑Context Learning (SG‑ICL), which selects few‑shot demonstrations based on sparsity and outperforms standard CoT and Auto‑CoT baselines on the MATH‑500 benchmark.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jul 16

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

arXiv:2510. 11686v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.

By Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, Jordan T. Ash
llmsreinforcement-learning
More like this →
arXiv Computation and Language
Sep 4

LLMs Learn Better In-Context from Rules than from Examples

The paper investigates how large language models learn new tasks in-context, comparing rule-based instruction following to example-based few-shot prompting across five diverse tasks. Results show that models generally learn more reliably from rule descriptions than from examples alone, and adding more examples does not consistently improve performance. Instruction tuning further enhances rule-based learning while preserving example-based capabilities, with rule advantages being strongest for algebraic tasks and weaker for tasks requiring distributional sensitivity or parametric knowledge.

By Xiang Fu, Seungmin Cho, Yukyung Lee, Najoung Kim
llmsfine-tuning
More like this →
arXiv Computation and Language
Aug 28

Selective State-Space Adaptation and Retrieval for Language Model Reasoning

The paper introduces a family of adapters that enhance language model reasoning by adding selective state-space control at token and context levels. The token-level MaLoRA makes the adapter’s scaling factor dynamic and recurrent, improving over static low‑rank adaptation. The context-level MaRA tracks cross‑segment reasoning state and retrieves relevant segments, outperforming an eight‑billion‑parameter dense retriever and boosting reasoning accuracy by an average of +6.4 F1 over LoRA.

By Atahan Dokme, Larry Heck
llmsfine-tuningbenchmarks
More like this →