arXiv:2606. 03780v1 Announce Type: cross Abstract: Causal tracing of factual recall has been studied predominantly in dense transformer language models, where interventions localize information flow to layers or feed-forward modules.
arXiv:2607. 20426v1 Announce Type: cross Abstract: Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models'internal knowledge or have poor cross-domain generalization.
arXiv:2606. 10338v1 Announce Type: cross Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored.
arXiv:2606. 10703v1 Announce Type: new Abstract: Interpretability methods routinely use population-level summary statistics over observed model behaviour to license claims about the effects of targeted interventions on specific computations; in Pearl's terms, they treat rung-1 associational evidence as if it supported rung-2 interventional conclusions, a move whose validity is rarely tested.
arXiv:2609.05801v1 Announce Type: new Abstract: A document modeled as a discrete sequence of tokens can be thought of as being generated from a composition of texts from different domains; a README f...
arXiv:2607. 21692v2 Announce Type: replace Abstract: Sparse attention prunes a long context to the blocks a model needs, and the usual selector is distilled from a dense teacher's attention.
arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
arXiv:2605. 09692v3 Announce Type: replace Abstract: Autonomous language agents increasingly expose traces, memories, plans and constraints, but existing evaluations rarely test whether these state variables are bound to final actions.
The paper introduces a two‑level readout for mixture‑of‑experts reasoning models. First, it compresses the model’s internal reasoning states into a 64‑dimensional semantic frame (J64) that reveals process dynamics beyond the emitted trace. Second, it reconstructs this frame from native expert‑routing statistics (R64), achieving high correlation and preserving most predictive gains while enabling low‑overhead, test‑time decision making.
arXiv:2606. 25092v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) models route each token to a few of many experts, inviting the hypothesis that experts form functional modules tied to capabilities or languages.
The paper introduces "Trace as State," a method that places collected reasoning traces before a long-context block in a transformer, allowing earlier derived information to guide rereading. Experiments across three models and datasets show that this approach consistently outperforms a control where traces are appended after the context, achieving significant accuracy gains on tasks such as GraphWalks Parents and GLM-5.2. The results demonstrate that positioning traces before the context can enhance long-context reasoning while preserving the causal transformer architecture.
The paper introduces a two‑level internal readout for mixture‑of‑experts reasoning models. First, it compresses the model’s reasoning states into a 64‑axis semantic frame (J64) that reveals process states not captured by the emitted trace, improving held‑out AUC by 0.096–0.135. Second, it reconstructs this frame from native expert‑routing statistics (R64), achieving high correlation with J64 and preserving most of its predictive gain while enabling low‑overhead, test‑time decision making and improved routing policies.
arXiv:2608. 07911v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard.