The paper argues that Large Language Models (LLMs) do not function as Solomonoff induction estimators because their training objectives—cross‑entropy, negative log‑likelihood, and next‑token prediction—optimize fit to a supplied conditional distribution rather than a program‑weighted universal mixture. It further contends that additional computation alone does not transform these models into optimal predictors without external hyper‑parameter or architectural changes. The authors suggest that neurosymbolic machine learning, exemplified by models such as Fable and Astra, represents a shift toward symbolic model synthesis, moving beyond purely statistical LLMs.
By Hector Zenil, Abicumaran Uthamacumaran, Luan Ozelim
arXiv:2601.05280v4 Announce Type: replace-cross
Abstract: On the one hand, the question of whether Large Language Models (LLMs) are Solomonoff induction estimators has become an explicit question at...
By Hector Zenil, Abicumaran Uthamacumaran, Luan Ozelim
The paper introduces OAttention, a token‑level attention mechanism that assigns each token a presence coefficient based on its hidden representation. This coefficient both gates the token’s output and weights its contribution to other tokens, making zero‑vector tokens behave as true zeros and enabling exact null‑receiver, null‑source, and empty‑support properties. The authors extend this idea to local O‑components and an O‑Transformer, and demonstrate small performance changes when retrofitting a pretrained TabPFN model.
By Heyang Gong
arXiv:2608. 10420v1 Announce Type: new Abstract: Reasoning shortcuts are solutions of a neurosymbolic system's rules that produce correct predictions through unintended concepts.
By Xin Xu
The paper introduces Coupled Scaling, a framework that links neural scaling laws to the relationship between task structure and the geometry that an architecture‑optimization system can access. It shows that finite‑budget scaling depends on how well the system’s representational support aligns with the task’s energy distribution, deriving residual exponents that vary with architectural coverage and tail decay. The authors propose tests to verify whether static task‑relevant geometry tracks loss and whether multiscale geometry follows coupling‑specific exponent ordering, suggesting a factorial audit of emergence trajectories to isolate geometry from scaling fits.
By Jie Wang
arXiv:2607. 18305v1 Announce Type: cross Abstract: Some limits on what language models know are not gaps in data coverage but structural properties of learning from text.
By Priyansh Srivastava, Romit Chatterjee
arXiv:2601. 05280v3 Announce Type: replace-cross Abstract: On the one hand, the question of whether large language models (LLMs) are Solomonoff induction estimators has become an explicit question at the intersection of Algorithmic Information Theory (AIT) and Machine Learning (ML) of great interest.
By Hector Zenil
The paper demonstrates that a prompt’s influence is not inherent to the prompt itself but depends on the model, as prompts optimized for one model degrade on another and rankings shift under neutral reformatting. By examining a task‑free structural readout—specifically the fixed‑point behavior of a short‑window argmax map—the authors show that nine tokens of conditioning can move the fixed‑point fraction across most of its range, altering structural classes and model rankings, while instruction tuning has no effect. Attempts to explain this phenomenon through prefix length, content type, bidirectionality, or attention‑sink dominance all fail, indicating that the prompt‑model pair is the fundamental unit of explanation.
whyItMatters":"The study reveals that prompt effectiveness is model‑specific and that simple structural readouts can capture this interaction, challenging assumptions about prompt generality and guiding future prompt‑engineering efforts."
By Nicol\'as Vera Z\'u\~niga
arXiv:2605. 04893v2 Announce Type: replace Abstract: When a language model processes a hallucinated response, its attention routing tends to fail in one of two shapes: over-concentrating on a narrow set of positions, or spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal.
By Dominik Dahlem, Diego Maniloff, Mac Misiura
The paper introduces MACCHIATO, a training algorithm that builds a ReLU‑MLP from partial truth‑table data while simultaneously constructing an explicit Boolean circuit over AND, OR, and XOR gates that certifies the network’s computation. The method iteratively projects residuals onto low‑dimensional Boolean classes, compiles the resulting circuit into a ReLU‑MLP, and uses logic minimization and influence‑based variable selection to achieve a six‑layer network with provable truth‑table error bounds. Experiments on synthetic random‑junta tasks show that these certified networks outperform Adam‑trained MLPs in data‑sparse or projection‑aligned regimes and complete faster than flat ESPRESSO in certain settings.
By Hrad Ghoukasian, Anastasis Kratsios
arXiv:2605. 25085v2 Announce Type: replace-cross Abstract: We study the rate-distortion limits of online KV cache compression in autoregressive language models, formulating it as sequential Wyner-Ziv source coding on the filtration induced by the model, with the next-step query as decoder side information.
By Munsik Kim
arXiv:2606. 16541v1 Announce Type: new Abstract: Autoformalization, translating natural-language mathematics into formal proof assistants, is bottlenecked not by translation fluency but by \emph{faithfulness}: a formal statement can typecheck and be provable, yet still encode a different theorem than the source intended.
By Noor Islam S. Mohammad, Tamim Sheikh