The paper introduces F-ICL, a benchmark that measures in‑context algorithmic reasoning in language models by exhaustively enumerating 86 million valid programs of length ≤13 on a Turing‑complete machine and computing the exact posterior under a bounded Levin–Solomonoff prior. Unlike typical benchmarks, F‑ICL provides a distributional reference rather than just answers, allowing the evaluation of models’ inductive priors. Across 105 configurations of models ranging from 0.8 B to 675 B parameters, models achieve up to 92 % accuracy, yet many still deviate from the Bayes‑optimal reference, and the study derives theoretical bounds on cumulative loss for predictors with positive prior weight on the reference.
Whether large language models perform genuine algorithmic reasoning or mere pattern completion is hard to test, because most benchmarks lack a ground truth for correct inductive inference. We introduce F-ICL, an in-context-learning benchmark that supplies one exactly.
arXiv:2608. 01575v1 Announce Type: new Abstract: Whether large language models perform genuine algorithmic reasoning or mere pattern completion is hard to test, because most benchmarks lack a ground truth for correct inductive inference.
arXiv:2610.14007v1 Announce Type: cross Abstract: A familiar line of reasoning runs as follows: language models are flexible approximators trained under a strictly proper loss, so scale brings them c...
Large Language Bayes (LLB) samples probabilistic programs from a language model, runs approximate inference on each, and averages them weighted by an exponentiated evidence bound. The authors demonstrate that this weighting is not invariant to reparameterisation, unlike the log marginal likelihood, leading to significant discrepancies in weights across different program formulations. These discrepancies can reach up to 31.9×, affect Bayes factors, and introduce controlled errors in posterior estimates.
arXiv:2606. 29278v1 Announce Type: new Abstract: We introduce the Complexity Ceiling Benchmark (CCB), a controlled evaluation of how language-model reasoning decays as the number of required sequential steps grows.
arXiv:2601.03199v2 Announce Type: replace-cross Abstract: Diffusion language models (DLMs) have shown strong potential for general natural language tasks with in-context examples. Existing In-Context...
arXiv:2610.36169v1 Announce Type: new Abstract: A retrieval-augmented language model answers from two sources, the retrieved context and its own knowledge, and the two disagree whenever the context i...
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
arXiv:2609.00759v1 Announce Type: new Abstract: Large language models (LLMs) increasingly handle in-context learning (ICL) tasks where a long, novel context defines the rules, knowledge, and output s...
The paper introduces SFT-as-context, a training‑free method that lets a parent large language model (LLM) use the response of a supervised fine‑tuned (SFT) model as context to answer queries. This approach preserves the parent model’s general capabilities while acquiring the SFT model’s specialized skills, achieving performance close to the SFT model on fine‑tuned benchmarks and near the parent model on general tasks. Experiments across 19 parent‑SFT pairs and 11 benchmarks show minimal gaps in performance, and the method can solve problems requiring both specialized and general knowledge, even when neither model alone succeeds.
arXiv:2606. 15877v1 Announce Type: cross Abstract: Chain-of-thought (CoT) improves large language models' performance in math and symbolic reasoning.
arXiv:2601.05280v4 Announce Type: replace-cross Abstract: On the one hand, the question of whether Large Language Models (LLMs) are Solomonoff induction estimators has become an explicit question at...