The paper introduces CS-MoE, a Transformer architecture that shares experts across layers to reduce inter‑layer parameter redundancy. By combining layer‑independent experts with a globally shared expert pool, CS‑MoE allows elastic control over token‑level parameter activation and computational cost. Experiments show that CS‑MoE achieves lower perplexity than equal‑scale dense Transformers while activating only 55% of parameters, and its performance scales with the number of activated experts, approaching MoE performance within a fixed FLOPs budget.
By Dian Jiao, Jiaxin Duan, Shuai Zhao, Jiabing Leng, Yiran Zhang, Feng Huang
Toollery is a training‑free framework that compresses candidate lists for large language model agents, enabling efficient selection from thousands of skills and tools. It generates user‑intent queries from each skill or tool specification, builds a retrieval index, and limits online selection to a compact top‑k set before the LLM makes its final decision. Evaluations on the SkillRouter benchmark, BFCL‑V4, and a proprietary smart‑cockpit dataset show that Toollery improves recall and end‑to‑end selection while keeping selection costs bounded.
By Xiangxi Tian, Ran Guan
The paper investigates why large language models often miscount items in bracketed lists. It finds that different instruct models exhibit distinct error patterns—Qwen and Gemma 27B tend to round odd lengths to nearby even numbers, OLMo focuses errors on mid-sized integers, and Llama under‑counts—challenging the idea that a single cause explains all failures. The study also shows that a linear probe can recover the correct count from the residual stream and that fixes effective for one model may not transfer to others.
By Iyad Ait Hou, Saad Mankarious, Aya Zirikly, Rebecca Hwa
CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.
By Jingang Zhou, Haiyang Guo, Yuan Ma, Han Zhu, Xu-Yao Zhang
RS-Claw-Evolution is an environment-feedback-driven framework designed to enhance lightweight remote sensing agents for long-horizon tasks. It improves agents through three stages—interaction evolution, experience evolution, and decision evolution—using executable code, failure-aware trajectory generation, and reinforcement learning with multi-dimensional rewards. On Earth-Bench, a Qwen3-4B agent trained with this framework reaches 65.9% accuracy, surpassing larger baselines and approaching GPT-5 performance.
By Kai Ouyang, Dongyang Hou, Liangtian Liu, Zeyuan Wang, Ziyu Li, Chengfu Liu, Zichao Tang, Xuezhi Cui, Shengwu Ouyang, Wentao Yang, Hanwen Yu, Haifeng Li
The paper studies how the design of Mixture-of-Experts (MoE) routers affects inference speed when combined with Speculative Decoding (SD). It shows that routers promoting high expert coactivation reduce memory transfer costs and improve runtime. By integrating a global load‑balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism, the authors achieve a 21% throughput gain over baseline MoEs while preserving accuracy.
By Kumari Nishu, Han-Byul Kim, Santosh Chilkunda, Maxwell Horton, Arnav Kundu, Mohammad Samragh, Lauren Hannah, Mohammad Sekhavat, Nikhil Bhendawade, Manuel Ciosici, Iman Mirzadeh, Keivan Alizadeh Vahid, David Harrison, Irina Belousova, Mehrdad Farajtabar, Minsik Cho
The paper introduces COREM, a Cosine-Relation Momentum Reshaping method that exploits relational structure within matrix‑valued optimizer states. COREM partitions the momentum state into update units, computes cosine relations among them, and reshapes the momentum before writing it back, thereby influencing both current and future optimization dynamics. Experiments on CIFAR‑10 and enwik8 show that COREM improves mid‑to‑late training performance and enhances spectral properties while using fewer FLOPs than the Muon baseline.
By Yan Wang, Xiaochuan Wang, Yuxiang Sun
The paper introduces Physics‑SIMS‑TS, a conditional diffusion model designed for long‑horizon oil and gas production forecasting. It enforces monotone decline through negative guidance, decline‑curve constraints, and isotonic projection during sampling, and incorporates spatial training augmentation and an ensembled stochastic sampler to produce calibrated predictive distributions. Evaluated on over 35,000 wells across three jurisdictions, Physics‑SIMS‑TS achieves the highest accuracy among diffusion forecasters and matches transformer ensembles, with only a 0.5% increase in mean squared error for monotonicity.
By Temesgen Mikael Abraha, Yves Lucet
The paper investigates whether properties of a Self‑Attention Encoder (SAE) that can be computed before any forward pass can predict how difficult it is to steer individual SAE features. It finds that decoder‑space geometry—specifically neighbor density and maximum cosine similarity to nearby decoder directions—partially predicts feature steerability, with significant correlations across multiple model scales, widths, and architectures. The relationship holds up to a certain layer depth, suggesting a boundary where steering becomes too costly.
By Muhammad Khan, Shlok Channawar, Akshaj Gurugubelli, Girish Gupta, Aditya Shah
The paper introduces CSC, a calibrated-simplicity framework for detecting social bots in the era of large language models. CSC combines a simplified prototype-guided graph expert, calibrated simplex-constrained fusion, and a lightweight inconsistency expert to address modality conflict between semantic and structural signals. Experiments on TwiBot-22, TwiBot-20, and MGStBot-large demonstrate that CSC improves calibrated decision quality while maintaining competitive performance across benchmarks.
By Yipeng Qian, Pengjie Zhao, Chaoxi Niu
arXiv:2609.23659v1 Announce Type: new
Abstract: Ethereum decentralized finance (DeFi) provides a public, time-stamped record of transaction-level event streams, but the same public symbols can create...
By Kemal Kirtac, Carsten Maple
arXiv:2609.23686v1 Announce Type: new
Abstract: Patch-based autoregressive time-series forecasting often ties input representation, learned transitions, and recursive execution to one patch length. W...
By Ziang Li, Yue Huang, Guoxu Zhou, Na Han, Jie Wen, Lunke Fei, Xiaozhao Fang
arXiv:2609.23900v1 Announce Type: new
Abstract: Tree speculative decoding verifies multiple candidate continuations in one target forward pass. For attention-only transformers, the verifier mainly ne...
By Zhiyuan Ma
arXiv:2609.24144v1 Announce Type: new
Abstract: Group-relative reinforcement learning compares rollouts of the same prompt, but independent environment noise can obscure these comparisons. We study p...
By Nazmus Sakib
arXiv:2609.24150v1 Announce Type: new
Abstract: Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to generate multiple candidate tokens that are...
By Tianhua Xia, Mugilan Ganesan, Yifei Feng, Haiyu Wang, Maximilian Egger, Sai Qian Zhang
arXiv:2609.24202v1 Announce Type: new
Abstract: Sparse attention reduces the quadratic cost of global self-attention while retaining strong empirical performance, but how its restricted interactions...
By Jingkun Liu, Yue Song
arXiv:2609.24289v1 Announce Type: new
Abstract: As Large Language Model (LLM) agents are applied in continuously interactive environments, driving the evolution of their own capabilities becomes a co...
By Ruimin Pei, Yongkang Wu, Shangyi Zheng, Yaqing Zhang, Deyang Li, Jianjun Tao, Xinyu Zhang, Xiang Zhang
arXiv:2609.24464v1 Announce Type: new
Abstract: Using a Large Language Model (LLM) as the clusterer at production scale is hard: prompts cannot hold the entire label space, and per-document serial pr...
By Armin Oliya, Aleksandra Sawczuk, Rados{\l}aw Bia{\l}obrzeski
arXiv:2609.24629v1 Announce Type: new
Abstract: A/B testing requires large sample sizes, long timelines, and significant costs. When auxiliary predictions of experimental outcomes are available from...
By Ziyad Benomar, Aymen Al Marjani, Paul Missault, Saab Mansour
arXiv:2609.24797v1 Announce Type: new
Abstract: Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivit...
By Julien Siems, Riccardo Grazzi, Korbinian P\"oppel, Jaisidh Singh, Arber Zela, Timur Carstensen, Jenia Jitsev, Frank Hutter, Volkan Cevher, Antonio Orvieto, Aaron Klein