arXiv Machine Learning
Theory on Attention Dynamics for Out-of-Distribution In-Context Learning
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 8
How Can Mamba Learn In Context with Outliers and Generalize Provably?
arXiv:2510. 00399v2 Announce Type: replace Abstract: The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks.
arXiv Machine Learning
Jul 7
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
arXiv:2607. 03660v1 Announce Type: cross Abstract: Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt.
arXiv Machine Learning
Jul 2
Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.
arXiv Machine Learning
Jun 4
Activation-Based Active Learning for In-Context Learning: Challenges and Insights
arXiv:2606. 05134v1 Announce Type: cross Abstract: Deep active learning has previously been explored for LLM in-context sample selection, but not with methods that utilise recent advances in understanding of transformer activations.
arXiv Machine Learning
Jun 18
Decomposing Prediction Mechanisms for In-Context Recall
arXiv:2507. 01414v2 Announce Type: replace Abstract: We introduce a new family of toy problems that combine features of linear-regression-style continuous in-context learning (ICL) with discrete associative recall.
arXiv Machine Learning
Jul 21
Bigger Is Safer: Provable Robustness in In-Context Learning Scales with Capacity
arXiv:2602. 17743v2 Announce Type: replace Abstract: In-context learning (ICL) allows large language models to adapt to new tasks from a few examples without updating their parameters.