MAGIC introduces a dense‑reward reinforcement learning framework for generating mixed‑granularity agent graphs in large‑language‑model based multi‑agent systems. The method sequentially selects functional roles, instantiates them as either single agents or reusable groups, and connects them to existing units, optimizing the construction policy with intermediate feedback from probe‑based utility and structural signals. Experiments show MAGIC outperforms state‑of‑the‑art baselines on eight benchmarks and achieves strong inference efficiency.
By Kairui Yang, Ziheng Yi, Xunkai Li, Minghao An, Zhanke Liu, Zekai Chen, Rong-Hua Li
The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.
By Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands
The paper introduces the Proximity Trap, a problem where long-context language models fail to attend to distant evidence because of competition from abundant, irrelevant proximal background. To mitigate this, the authors propose LYRA, a t‑distributed directional matching mechanism that reshapes the retrieval distribution to focus more on task‑relevant evidence while preserving positional information. Experiments on LongBench‑v2, RULER, and LongBench show consistent performance gains, and a new benchmark, ProxBench, is presented to evaluate distant evidence usage under varying proximal interference.
By Xiaoyu Yang, Jie Lu, Wei Duan, En Yu
The paper investigates how different 4‑bit quantization techniques affect privacy when fine‑tuned small language models are deployed. It finds that methods using a calibration corpus, such as Activation‑aware Weight Quantization (AWQ) and Gradient‑based Post‑Training Quantization (GPTQ), prevent the reproduction of planted private records, whereas a calibration‑free format (GGUF Q4_K_M) leaks 5.3% of them. Across models ranging from 0.5 to 7 billion parameters, AWQ consistently leaks the least while maintaining minimal accuracy loss, indicating that the choice of 4‑bit method is a privacy decision as well as a performance one.
By Cristhian Kapelinski, Diego Kreutz
SSP-Bench is a dynamic benchmarking framework designed to evaluate large language models on safety, security, and privacy (SSP) by generating evaluation instances on demand while maintaining domain consistency. It ensures label validity through external sources, enforces scope with service-specific validation, and calibrates difficulty using a multi-model steering panel, framing benchmark construction as a multi-objective optimization over difficulty, separability, novelty, and diversity. Across 24 models and four SSP services, SSP-Bench exposes systematic failures of static benchmarks, such as near-zero correlation in safety rankings, strong safety–over-refusal coupling, and hidden within-family regressions.
By Fatih Deniz, Yazan Boshmaf, Issa Khalil
Matryoshka Attribution (MAttr) is a mask‑learning method that identifies nested subsets of a language model’s internal components by minimizing downstream loss. It uses a differentiable sigmoid top‑k operator and randomizes sparsity during training to produce an attribution ordering of components. MAttr tops the Mechanistic Interpretability Benchmark leaderboard and can be applied via reinforcement learning to pinpoint weight changes that control behaviors such as refusal in Llama 3.1 8B Instruct, where restoring just 1% of weights removes refusals while preserving capabilities.
By Aryaman Arora, Kirill Acharya, Nathan Hu, Yanzhe Zhang, Noah Goodman, Dan Jurafsky, Christopher Potts
The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.
By Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young, Hyesoon Kim
Ladders-of-Thought (LoT) is a framework that enhances reasoning in small- to mid-scale large language models by automatically generating easier variants of reasoning problems and organizing them into difficulty buckets. It uses a self‑evolving bandit scheduler to adaptively allocate training, improving performance across math and multi‑hop reasoning tasks on 1–8 B models. LoT achieves significant gains (e.g., +32 pp on AddSub, +16 pp on QASC) and converges faster than staged curricula.
By Minghui Liu, Thomas Magelinski, Dehao Yuan, Qi Yu, Furong Huang
arXiv:2609. 25710v1 Announce Type: cross Abstract: The statistical accuracy of neural networks depends on both their approximation power and the complexity of the class fitted from data.
By Baicheng Li, Zuowei Shen, Haizhao Yang, Shijun Zhang
ToolCompass is a post‑training framework that improves how large language model agents explore new tools by organizing tool‑call representations according to shared functions. It models each function class as a von Mises–Fisher distribution, reducing variation within a function while increasing separation between different functions, thereby guiding exploration toward functionally similar unseen tools. Experiments on AppWorld and FTRL show consistent gains, with up to a 10.71‑percentage‑point improvement in out‑of‑distribution task success over vanilla post‑training and outperforming competitive baselines.
By Junlin Fang, Chong Zhang, Do Nguyen-Thanh, Xiaogang Xu, Zhen Fang, Sean Du
The paper investigates whether a looped estimator—one that repeatedly applies a single fitted operator with shared parameters—can enhance statistical accuracy while staying within a fixed parameter budget. It establishes upper and lower bounds on squared Hellinger risk for looped sieve maximum likelihood and compares them to the untied counterpart, revealing a tradeoff between parameter sharing, iteration count, and accuracy. For models with known H"older smoothness, looped residual feedforward networks and a post‑layer‑normalized Transformer achieve minimax polynomial rates with a fixed number of bounded real parameters, and under certain conditions the looped estimator’s worst‑case risk vanishes as sample size grows, outperforming the untied approach.
By Xinyu Tian, Xiaotong Shen
The paper introduces a lightweight evolutionary heuristic scheduler for Diffusion Large Language Models (dLLMs) that optimizes denoising trajectories using CMA-ES. It addresses failure modes of existing confidence-based schedulers—EOS Overflow and Proximal Bias—by integrating multiple heuristic features with a contextual mean-field embedding, requiring only 393 trainable parameters. Evaluations on LLaDA and Dream across reasoning and planning benchmarks show consistent outperformance over strong baselines and recent state‑of‑the‑art methods.
By Zijian Zhao, Dian Jin, Xialiang Tong, Sen Li, Mingxuan Yuan
The paper introduces MGRL-RSCC, a multi‑granularity reward reinforcement learning framework for Remote Sensing Change Captioning (RSCC). It uses a CNN with hierarchical self‑attention to extract visual features, a Transformer decoder for visual‑to‑linguistic translation, and a dual‑decoding strategy combined with token‑level supervised learning and self‑critical reinforcement learning. Three reward functions—linguistic fluency, change state consistency, and structural‑semantic relevance—are employed to improve caption quality and reduce exposure bias and conservative generation.
By Futian Wang, Mengqi Wang, Xiao Wang, Wentao Wu, Haowen Wang, Zhicheng Zhao, Jin Tang
Magnitude Profile Pruning introduces a training‑free, calibration‑free method for removing attention heads in Transformer models by statistically detecting outliers in weight row norms. Heads whose projection weights fall within the bulk of the distribution are pruned, while outlier heads are retained. Across several models, the MP‑G variant achieves superior perplexity at various sparsity levels and yields significant parameter and FLOP reductions without requiring forward passes, calibration data, or gradient computations.
By Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara, Suranadi De Silva
The paper investigates whether domain‑matched calibration data is necessary when compressing large language models for financial tasks. It finds that if compression causes little task damage, the choice of calibration corpus has minimal impact, whereas significant damage—especially from pruning—can be mitigated by using a finance‑specific calibration set (FinMix). The study tests this across multiple models, compression settings, and financial tasks, consistently supporting the link between damage and recovery.
By Junyi Ye, Mengjia Yu, Debapriya Hazra, Guiling Wang
CliffCompaction is an autocompaction technique that reduces cost by up to 50% while maintaining or improving performance on benchmarks such as Terminal‑Bench and KernelBench. It achieves this by truncating or dropping content without rephrasing, ensuring compacted information remains faithful and preventing context drift. The method enables efficient test‑time scaling, matching or surpassing higher‑cost models like Opus 4.7 and GPT‑5.3 Codex, and delivers significant CUDA kernel speedups on KernelBench.
By Trang Nguyen, Eulrang Cho, Bingqing Chen, Tim Dettmers
STAR-VAE is a Transformer-based variational autoencoder that uses SELFIES encoding and a bidirectional encoder with an autoregressive decoder pretrained on 79 million PubChem molecules. It incorporates a property signal to jointly condition the prior, posterior, and decoder, and employs LoRA adapters for fine‑tuning on small datasets without altering the backbone. The model achieves 100 % validity and near‑perfect novelty in MOSES sampling, low KL divergence on several GuacaMol descriptors, strong synthetic‑accessibility conditioning, and effective docking‑score control across multiple protein targets, while also enabling scaffold recovery and diverse label‑conditioned generation on ChEMBL targets.
By Bum Chul Kwon, Ben Shapira, Moshiko Raboh, Shreyans Sethi, Shruti Murarka, Joseph A Morrone, Leili Zhang, Wendy Cornell, Jianying Hu, Parthasarathy Suryanarayanan
The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.
By Christopher M. Bryant, Hao Liu
MONA is a new optimizer that extends the Muon optimizer by adding a Nesterov‑style acceleration term derived from an exponential moving average of gradient differences. The paper provides a convergence analysis showing that this term offers curvature‑aware corrections while maintaining Muon’s spectral‑norm regularization. Empirical results demonstrate that MONA outperforms both Muon and AdamW on Mixture‑of‑Experts pretraining across models ranging from 1 B to 68 B parameters, and achieves state‑of‑the‑art performance on downstream benchmarks after fine‑tuning the largest model.
By Jiacheng Li, Jianchao Tan, Hongtao Xu, Jiaqi Zhang, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai
The paper investigates how unbiased stochastic gradients affect stability in one‑layer linear self‑attention models used for in‑context linear regression. It shows that while full‑batch training preserves an elliptic stability region, mini‑batch sampling introduces batch‑dependent boundaries that drift outward, leading to an exact one‑step crossing criterion and adaptive concentration bounds. The authors also derive drift identities, prove almost‑sure escape in a balanced switching model, and analyze mode leakage and effective context length for softmax attention, corroborated by experiments on causal Transformers.
By Krishnakumar Balasubramanian