arXiv:2601.05280v4 Announce Type: replace-cross
Abstract: On the one hand, the question of whether Large Language Models (LLMs) are Solomonoff induction estimators has become an explicit question at...
By Hector Zenil, Abicumaran Uthamacumaran, Luan Ozelim
arXiv:2601. 05280v3 Announce Type: replace-cross Abstract: On the one hand, the question of whether large language models (LLMs) are Solomonoff induction estimators has become an explicit question at the intersection of Algorithmic Information Theory (AIT) and Machine Learning (ML) of great interest.
By Hector Zenil
arXiv:2610.14007v1 Announce Type: cross
Abstract: A familiar line of reasoning runs as follows: language models are flexible approximators trained under a strictly proper loss, so scale brings them c...
By Luan Ozelim, Abicumaran Uthamacumaran, Hector Zenil
The paper investigates how well practical machine‑learning algorithms approximate Solomonoff’s universal predictor when trained by minimizing log‑loss over diverse datasets. It establishes tight bounds on the worst‑case increase in prediction error caused by approximating the Bayesian mixture predictor, showing that while the excess error can be large in low‑prior environments, it remains bounded on average and under distribution shifts. The authors also extend their analysis to finite‑sample settings relevant to neural networks and Transformers.
By Qian Li, Shang-Hua Teng
The paper investigates the relationship between large language models (LLMs) and compression, showing that LLMs, modeled as next‑token predictors, are equivalent to monotone (order‑preserving) compression algorithms with an error gap of at most 2. It further proves that this monotonicity is necessary for the equivalence only if cryptographic one‑way functions exist, and derives a cryptographic corollary linking next‑bit pseudoentropy to monotone incompressibility.
By Noam Mazor, Andrew Morgan, Rafael Pass
arXiv:2607. 28667v1 Announce Type: cross Abstract: Recent work has shown that classifying large language models (LLMs)' responses can be distinguished by modeling token embeddings as trajectories of a black-box dynamical system (DS) and comparing prediction residuals of two DSs.
By Mohamed Akrout, Dan Wilson
The paper introduces mentored decoding, a formal framework for lossy speculative decoding that can accelerate inference of autoregressive language models while potentially improving output quality. It connects this inference technique to boosting theory and extends it to all f‑divergences, revealing geometric insights for total variation, simple approximations tied to boosting compliance, and a divergence‑independent data structure enabling efficient optimal parameter queries and mentored distribution construction.
By Vivien Tran-Thien, Richard Nock
arXiv:2608. 01005v1 Announce Type: new Abstract: Solomonoff Induction, or SolInd, provides an ideal unbounded model of a priori sequence prediction but cannot naturally describe extrapolation from a given training dataset, as performed by Large Language Models.
By Nathan Young
The paper introduces F-ICL, a benchmark that measures in‑context algorithmic reasoning in language models by exhaustively enumerating 86 million valid programs of length ≤13 on a Turing‑complete machine and computing the exact posterior under a bounded Levin–Solomonoff prior. Unlike typical benchmarks, F‑ICL provides a distributional reference rather than just answers, allowing the evaluation of models’ inductive priors. Across 105 configurations of models ranging from 0.8 B to 675 B parameters, models achieve up to 92 % accuracy, yet many still deviate from the Bayes‑optimal reference, and the study derives theoretical bounds on cumulative loss for predictors with positive prior weight on the reference.
By Luan Ozelim, Hector Zenil
The paper investigates how Chain-of-Thought (CoT) reasoning enables Large Language Models to generalize to out-of-distribution (OOD) tasks by composing reusable in-distribution (ID) atomic steps. It provides a theoretical framework showing that OOD risk depends on the sum of ID sub-task risks, scaled by a factor independent of the number of reasoning steps, and that the estimation error grows sub-linearly with the number of steps. Experiments on Atom-Task and GSM8K datasets confirm that CoT supports short-to-long and compositional generalization, and that explicit state tracking is essential.
By Nuo-Jing Liang, Xiao-Tong Yuan, Pan Zhou
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
By Chiwun Yang
arXiv:2606. 30372v1 Announce Type: new Abstract: Quantitative research across the social and behavioral sciences depends on human subject experiments that are expensive, slow, and subject to sampling bias.
By Haobo Yang