The paper introduces an in‑span adaptation technique for reduced‑order models, where the reduced subspace is continually updated using the model’s own predictions via an incremental singular‑value decomposition with a forgetting factor. This creates a trajectory‑informed spectral preconditioner that reweights and realigns the basis without changing the subspace, enabling the model to better absorb future out‑of‑span corrections. The authors demonstrate the method on a 3‑D spiral example and nonlinear PDEs such as viscous Burgers and Fisher–KPP, and relate the approach to in‑context learning in dynamical systems.
By Amirpasha Hedayat, Laura Balzano, Karthik Duraisamy
SafeTune is a source‑available library that consolidates four safety‑intervention paradigms—post‑hoc weight recovery, safety‑constrained fine‑tuning, gradient‑based unlearning, and inference‑time steering—into a single, configuration‑driven workflow. It offers shared interpretability, evaluation, and deployment tools, and its modular registry allows easy addition of new methods, benchmarks, judges, models, and fine‑tuning domains. The authors demonstrate SafeTune with controlled comparisons and case studies in finance and medical deployments, showing how it characterizes safety drift, evaluates interventions on refusal‑behavior and capability metrics, and supports calibrated or layered mitigation.
By Pratinav Seth, Saisab Sadhu, Anshul Kaushal, Vinay Kumar Sankarapu
The paper investigates how post‑training modifies the weights of Large Language Models relative to their pretrained state. By expressing weight updates in the pretrained matrix’s singular value decomposition, the authors separate changes into three geometric components: diagonal (reshaping singular values), off‑diagonal (rotating input‑output coupling), and null‑space (routing outside the original SVD core). Experiments on a math evaluation suite show that removing the diagonal component largely preserves post‑training gains, indicating that improvements stem mainly from reconfiguring and extending pretrained pathways rather than altering singular values.
By Jianing Qi, Hao Tang, Zhigang Zhu
StepKV introduces a step-aware approach to compressing the key-value cache used during large language model inference, treating reasoning steps as primary units of retention rather than individual tokens. By linking cache entries to the steps that generated them and estimating each step’s utility from trajectory signals, StepKV assigns a combined token‑ and step‑level score to guide pruning. Experiments on multi‑hop question answering and long‑horizon web reasoning show that StepKV maintains accuracy even under tight cache budgets, outperforming token‑level baselines that suffer sharp performance drops.
By Boyu Feng, Jiahong Liu, Yifan Li, Wenhao Yu, Zexuan Qiu, Yuliang Sun, Ming Shen, Xiang Li, Quanyu Dai, Irwin King
The paper introduces CS-MoE, a Transformer architecture that shares experts across layers to reduce inter‑layer parameter redundancy. By combining layer‑independent experts with a globally shared expert pool, CS‑MoE allows elastic control over token‑level parameter activation and computational cost. Experiments show that CS‑MoE achieves lower perplexity than equal‑scale dense Transformers while activating only 55% of parameters, and its performance scales with the number of activated experts, approaching MoE performance within a fixed FLOPs budget.
By Dian Jiao, Jiaxin Duan, Shuai Zhao, Jiabing Leng, Yiran Zhang, Feng Huang
Toollery is a training‑free framework that compresses candidate lists for large language model agents, enabling efficient selection from thousands of skills and tools. It generates user‑intent queries from each skill or tool specification, builds a retrieval index, and limits online selection to a compact top‑k set before the LLM makes its final decision. Evaluations on the SkillRouter benchmark, BFCL‑V4, and a proprietary smart‑cockpit dataset show that Toollery improves recall and end‑to‑end selection while keeping selection costs bounded.
By Xiangxi Tian, Ran Guan
The paper investigates why large language models often miscount items in bracketed lists. It finds that different instruct models exhibit distinct error patterns—Qwen and Gemma 27B tend to round odd lengths to nearby even numbers, OLMo focuses errors on mid-sized integers, and Llama under‑counts—challenging the idea that a single cause explains all failures. The study also shows that a linear probe can recover the correct count from the residual stream and that fixes effective for one model may not transfer to others.
By Iyad Ait Hou, Saad Mankarious, Aya Zirikly, Rebecca Hwa
CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.
By Jingang Zhou, Haiyang Guo, Yuan Ma, Han Zhu, Xu-Yao Zhang
RS-Claw-Evolution is an environment-feedback-driven framework designed to enhance lightweight remote sensing agents for long-horizon tasks. It improves agents through three stages—interaction evolution, experience evolution, and decision evolution—using executable code, failure-aware trajectory generation, and reinforcement learning with multi-dimensional rewards. On Earth-Bench, a Qwen3-4B agent trained with this framework reaches 65.9% accuracy, surpassing larger baselines and approaching GPT-5 performance.
By Kai Ouyang, Dongyang Hou, Liangtian Liu, Zeyuan Wang, Ziyu Li, Chengfu Liu, Zichao Tang, Xuezhi Cui, Shengwu Ouyang, Wentao Yang, Hanwen Yu, Haifeng Li
The paper studies how the design of Mixture-of-Experts (MoE) routers affects inference speed when combined with Speculative Decoding (SD). It shows that routers promoting high expert coactivation reduce memory transfer costs and improve runtime. By integrating a global load‑balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism, the authors achieve a 21% throughput gain over baseline MoEs while preserving accuracy.
By Kumari Nishu, Han-Byul Kim, Santosh Chilkunda, Maxwell Horton, Arnav Kundu, Mohammad Samragh, Lauren Hannah, Mohammad Sekhavat, Nikhil Bhendawade, Manuel Ciosici, Iman Mirzadeh, Keivan Alizadeh Vahid, David Harrison, Irina Belousova, Mehrdad Farajtabar, Minsik Cho
The paper introduces COREM, a Cosine-Relation Momentum Reshaping method that exploits relational structure within matrix‑valued optimizer states. COREM partitions the momentum state into update units, computes cosine relations among them, and reshapes the momentum before writing it back, thereby influencing both current and future optimization dynamics. Experiments on CIFAR‑10 and enwik8 show that COREM improves mid‑to‑late training performance and enhances spectral properties while using fewer FLOPs than the Muon baseline.
By Yan Wang, Xiaochuan Wang, Yuxiang Sun
The paper introduces Physics‑SIMS‑TS, a conditional diffusion model designed for long‑horizon oil and gas production forecasting. It enforces monotone decline through negative guidance, decline‑curve constraints, and isotonic projection during sampling, and incorporates spatial training augmentation and an ensembled stochastic sampler to produce calibrated predictive distributions. Evaluated on over 35,000 wells across three jurisdictions, Physics‑SIMS‑TS achieves the highest accuracy among diffusion forecasters and matches transformer ensembles, with only a 0.5% increase in mean squared error for monotonicity.
By Temesgen Mikael Abraha, Yves Lucet
The paper investigates whether properties of a Self‑Attention Encoder (SAE) that can be computed before any forward pass can predict how difficult it is to steer individual SAE features. It finds that decoder‑space geometry—specifically neighbor density and maximum cosine similarity to nearby decoder directions—partially predicts feature steerability, with significant correlations across multiple model scales, widths, and architectures. The relationship holds up to a certain layer depth, suggesting a boundary where steering becomes too costly.
By Muhammad Khan, Shlok Channawar, Akshaj Gurugubelli, Girish Gupta, Aditya Shah
The paper introduces CSC, a calibrated-simplicity framework for detecting social bots in the era of large language models. CSC combines a simplified prototype-guided graph expert, calibrated simplex-constrained fusion, and a lightweight inconsistency expert to address modality conflict between semantic and structural signals. Experiments on TwiBot-22, TwiBot-20, and MGStBot-large demonstrate that CSC improves calibrated decision quality while maintaining competitive performance across benchmarks.
By Yipeng Qian, Pengjie Zhao, Chaoxi Niu
arXiv:2609.23659v1 Announce Type: new
Abstract: Ethereum decentralized finance (DeFi) provides a public, time-stamped record of transaction-level event streams, but the same public symbols can create...
By Kemal Kirtac, Carsten Maple
arXiv:2609.23686v1 Announce Type: new
Abstract: Patch-based autoregressive time-series forecasting often ties input representation, learned transitions, and recursive execution to one patch length. W...
By Ziang Li, Yue Huang, Guoxu Zhou, Na Han, Jie Wen, Lunke Fei, Xiaozhao Fang
arXiv:2609.23900v1 Announce Type: new
Abstract: Tree speculative decoding verifies multiple candidate continuations in one target forward pass. For attention-only transformers, the verifier mainly ne...
By Zhiyuan Ma
arXiv:2609.24144v1 Announce Type: new
Abstract: Group-relative reinforcement learning compares rollouts of the same prompt, but independent environment noise can obscure these comparisons. We study p...
By Nazmus Sakib
arXiv:2609.24150v1 Announce Type: new
Abstract: Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are...
By Tianhua Xia, Mugilan Ganesan, Yifei Feng, Haiyu Wang, Maximilian Egger, Sai Qian Zhang
arXiv:2609.24202v1 Announce Type: new
Abstract: Sparse attention reduces the quadratic cost of global self-attention while retaining strong empirical performance, but how its restricted interactions...
By Jingkun Liu, Yue Song