arXiv Machine Learning September 17, 2026 By Ziniu Zhang, Zhenshuo Zhang, Ruoxuan Xiong, Gene Cooperman, Hongyang R. Zhang

Long-Context Demonstration Selection Using State Space Models

Read the original on arXiv Machine Learning →

The paper addresses the challenge of selecting demonstrations for long-context language model queries, where transformer inference costs grow quadratically with sequence length. It proposes two algorithms that distill transformer behavior into state space models (SSMs) with linear inference time, partitioning transformer layers into groups and estimating separate SSMs for each. The distilled SSMs achieve less than 0.7% approximation error, and in downstream tasks they reduce FLOPs by 14.2× while improving accuracy by 6.48% compared to baseline methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Sep 15

Long-Context Demonstration Selection Using State Space Models

We study the problem of demonstration selection, which involves selecting a subset of examples for prepending to a query to a language model. This problem is closely related to in-context learning and...

llms rag nlp efficiency

arXiv AI

Sep 1

DIP: Dynamic In-Context Planner For Diffusion Language Models

arXiv:2601.03199v2 Announce Type: replace-cross Abstract: Diffusion language models (DLMs) have shown strong potential for general natural language tasks with in-context examples. Existing In-Context...

By Yang Li, Han Meng, Chenan Wang, Zhenyu Bi, Xuan Wang, Haipeng Chen

llms diffusion benchmarks

arXiv Computation and Language

Sep 30

Pretraining Latent Information Feedback Transformers with Teacher Supervision

arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informa...

By Dor Tirosh, Ido Amos, Mor Geva

llms

Hugging Face Trending Papers

Sep 23

Distilling Sequential Computation in Transformer Language Models

Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adjacent token spans are highly predictable or freque...

llms rag nlp efficiency

Hugging Face Trending Papers

Sep 15

Large Language Models Develop Belief State Geometry In-Context

Large language models (LLMs) trained on next-token prediction exhibit remarkable in-context learning (ICL) abilities, yet the representations that support ICL remain poorly understood. We consider suc...

llms

arXiv AI

Jul 28

Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models

arXiv:2607. 22646v1 Announce Type: new Abstract: Large language models (LLMs) display a striking ability to predict next observations from Hidden Markov Models (HMMs) via in-context learning (ICL), but the algorithm underlying this capability remains undetermined: prior work has proposed several candidates without consensus, and none has been grounded in the model's internal activations.

By Yijia Dai, Zhaolin Gao, Yahya Sattar, Jennifer J. Sun, Sarah Dean

llms

One email a morning, machine-written

Related stories

Long-Context Demonstration Selection Using State Space Models

DIP: Dynamic In-Context Planner For Diffusion Language Models

Pretraining Latent Information Feedback Transformers with Teacher Supervision

Distilling Sequential Computation in Transformer Language Models

Large Language Models Develop Belief State Geometry In-Context

Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models