The paper reports a data‑efficient language modeling study conducted by Qiushi Engine on the BabyLM 2026 Strict‑Small benchmark, using only 10 million corpus words and 100 million cumulative presentations. It describes a three‑stage research program: Stage I built a frontier model via compact restatements and incremental learning; Stage II identified that exact repetition versus aligned restatement affect context use and proposed a principle for organizing experience around contextual dependencies; Stage III applied selective supervision and preservation techniques, achieving a modest overall score increase from 42.02 to 42.25 and the highest public Strict‑Small result as of 8 September 2026. The work also discusses further studies on compression, relational anchors, shared representations, and measurement, and makes models and code publicly available.
By Shuxing Yang, Kaihao Zhu, Junjie Yang, Rui Zhao, Junyao Wu, Yize Wang, Wenhao Li, Fujia Chen, Taowen Deng, Shenzhan Hong, Yaqi Li, Zichen Li, Jincheng Mi, Yuang Pan, Hongsheng Chen, Yihao Yang
The paper extends mechanistic interpretability of large language models by modeling concepts as low‑dimensional non‑linear manifolds rather than linear subspaces. It introduces a concept‑based alignment (CBA) score to compare these manifolds across layers and models, revealing block structures in intermediate layers, a shift from syntax‑dominated to mixed syntactic‑semantic concepts, and training‑dependent multilingual sharing. The study also shows that alignment patterns differ across model families and training stages, with adjacent stages aligning more closely than distant ones.
By Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff
Posted by Amirkeivan Mohtashami, Research Intern, and Florian Hartmann, Software Engineer, Google Research Large language models (LLMs) have significantly improved the state of the art for solving tasks specified using natural language, often reaching performance close to that of people. As these models increasingly enable assistive agents, it could be beneficial for them to learn effectively from each other, much like people do in social settings, which would allow LLM-based agents to improve each other’s performance.
By Google AI
--> Understanding the behavior of complex machine learning systems, particularly Large Language Models (LLMs), is a critical challenge in modern artificial intelligence. Interpretability research aims to make the decision-making process more transparent to model builders and impacted humans, a step toward safer and more trustworthy AI.
The paper introduces NinaXander, a method for composing frozen language models from different architecture families by inserting a trained shared‑latent adapter between their layers. By running the initial layers of one model, converting the intermediate representation with the adapter, and then continuing with the remaining layers of another model, multiple composed models can be created without retraining. Experiments with RWKV and Pythia show that while some compositions preserve syntactic quality and reduce memory usage, none match the parent model’s accuracy and language‑modeling performance drops on out‑of‑domain data.
By Takanori Kotama, Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri
Posted by Yun Zhu and Lijuan Liu, Software Engineers, Google Research Large language model (LLM) advancements have led to a new paradigm that unifies various natural language processing (NLP) tasks within an instruction-following framework. This paradigm is exemplified by recent multi-task LLMs, such as T0 , FLAN , and OPT-IML .
By Google AI
Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels.
arXiv:2607. 00004v1 Announce Type: cross Abstract: While advanced foundation models like ModernBERT significantly outperform older architectures in dense retrieval, they surprisingly lag behind the aging BERT-base baseline in learned sparse retrieval (LSR).
By Zhichao Geng, Yang Yang
In this post, I’ll introduce a reinforcement learning (RL) algorithm based on an “alternative” paradigm: divide and conquer . Unlike traditional methods, this algorithm is not based on temporal difference (TD) learning (which has scalability challenges ), and scales well to long-horizon tasks.
arXiv:2608. 05783v1 Announce Type: cross Abstract: Machine unlearning has become a critical capability for safely removing specific, sensitive knowledge from large language models (LLMs).
By Pawe{\l} Batorski, Przemys{\l}aw Spurek, Paul Swoboda
arXiv:2606. 26749v1 Announce Type: new Abstract: Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs.
By Yize Zhao, Isabel Papadimitriou, Christos Thrampoulidis
The paper examines a multimodal approach that combines a self‑supervised GNN encoder with an alternating optimization scheme involving a language‑model teacher. Despite the expectation that this joint strategy would enhance predictive performance, the authors find that the combined model fails to deliver significant gains. They identify six key factors—ranging from anchor strength trade‑offs to misaligned representation spaces—that explain why the integration of text knowledge does not fully benefit graph learning.
By Fumiaki Kimino (SOKENDAI), Ryoma Sato (SOKENDAI, National Institute of Informatics)