RBS-Attention introduces a training‑free, radius‑bounded sparse prefill strategy for long‑context large language models, addressing the mean dilution problem where a block centroid can miss highly relevant tokens. The method employs two complementary selection branches: a centroid base branch that captures average relevance and a rescue branch that uses the maximum key‑block radius to flag under‑estimated blocks. Experiments on Qwen3 models demonstrate significant speedups—over 20× in standalone prefill‑attention and nearly 6× in end‑to‑first‑token time—while maintaining competitive accuracy compared to dense attention.
By Chuxu Song, Jiuqi Wei, Zhencan Peng
Attention-Aware Routing (AAR) augments the router in Mixture-of-Experts language models with temporal and spectral features derived from a sliding window of attention weights, thereby separating contextual information from the token’s hidden state. By keeping the base transformer frozen and training only routing parameters, AAR achieves a +3.37‑point improvement on GSM8K over a routing‑only baseline and demonstrates that routing changes propagate through the residual stream to reshape attention without directly updating the attention mechanism. The method also reduces long diverging generations, shows depth‑sensitivity affecting retrieval versus reasoning, and offers a controlled probe of routing‑relevant information across layers.
By Despoina Kosmopoulou, Anastasios Tsetsilas, Efthymios Georgiou, Giannis Karamanolakis, Swastik Roy, Alexandros Potamianos
The paper investigates how the topology of attention graphs can differentiate hallucinated from non-hallucinated responses in large language models. By analyzing Forman-Ricci curvature, the authors identify structural bottlenecks and develop a method that captures both semi-local and global information-flow characteristics of attention heads associated with hallucinations. Extensive evaluation across multiple LLMs and benchmarks shows that this single-pass approach consistently outperforms existing attention-based and multi-response baselines, while also revealing that impaired context sharing—such as over-reliance on self-attention and information over-squashing—correlates strongly with hallucination occurrences.
By Amir Jalilifard, Anderson Rocha, Eric Wong, Marcos Medeiros Raimundo
TinyCeNN-LM proposes a quality‑gated post‑training conversion framework that replaces attention in pretrained language models with CeNN‑inspired cellular‑recurrent layers. The method includes bounded local processing, compact recurrent memory, routing, fusion, and an accept‑or‑rollback validation step, with three specific implementations studied. Experiments on SmolLM2‑135M and Qwen3.5‑0.8B show that the conversion can accept certain layers while rejecting others based on representation fidelity and NLL thresholds, achieving minimal perplexity changes and modest downstream accuracy retention.
By Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya
arXiv:2609.21208v1 Announce Type: new
Abstract: Self-play methods that co-train a single language model as both coder and test author promise to move code-generation RL beyond fixed test suites, but...
By Ana Nunez, Peyman Najafirad
The paper explores offline large language models as decision‑support tools for air operations, where analysts must work with doctrine and imagery under limited connectivity. It presents a modular retrieval‑augmented architecture that handles both text and image inputs from technical manuals and can operate without Internet access. A pilot study with four Brazilian Air Force image analysts shows the system matches human performance on doctrinal knowledge tests while completing the task in a fraction of the time, and it highlights the high cognitive load of manual target reporting.
By Joao P. A. Dantas, Jelton A. Cunha, Gabriel Dietzsch
arXiv:2609.21432v1 Announce Type: new
Abstract: Post-training plays a pivotal role in enhancing the reasoning capabilities and task-specific expertise of large language models (LLMs). Despite recent...
By Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong
arXiv:2609.21493v1 Announce Type: new
Abstract: Multimodal large language models, or MLLMs, perform well at visual understanding and structured generation, yet these capabilities do not establish whe...
By Zicheng Zhao, Dongyin Chen, Rui Xu, Yinghui Xu
The paper introduces DSRec, a dual‑interest sequential recommendation model that separates item representations into long‑term and short‑term semantic contexts. Long‑term embeddings capture stable preferences through historical aggregation, while short‑term embeddings focus on local session intent modulated by inter‑click time intervals. Each branch is processed by a distinct State Space Model— a full‑sequence Mamba for long‑term modeling and a time‑modulated SSM for short‑term dynamics— and a residual cross‑fusion mechanism aligns the two granularities while preserving their independence. Experiments on public benchmarks show that DSRec outperforms state‑of‑the‑art methods.
By Shuiying Liao, P. Y. Mok
arXiv:2609.21626v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly deployed for enterprise information extraction (IE), where the same document must be reorganized differen...
By Hongliang Li, Lu Wang, Yong Xu, Hanyang Chen, Zhitao Hou, Xiaoting Qin, Song Ge, Qingwei Lin, Dongmei Zhang
The paper investigates how transformers can possess a world model despite exhibiting behavioral failures. Using TaxiGPT, a transformer trained on random Manhattan walks, the authors show that the model internally represents intersections, streets, and its position, and uses a goal compass for navigation. They attribute failures to interference between overlapping intersection features and demonstrate that affordance packing mitigates these errors, concluding that world‑modeling abilities emerge at distinct training stages and should be studied mechanistically rather than merely observed behaviorally.
By Pierre Beckmann, Matthieu Queloz, Andre Freitas
arXiv:2609.21801v1 Announce Type: new
Abstract: We study how far a simple statistical pipeline can go on univariate time series anomaly detection under a strict selection protocol. The method extract...
By Youssef Attia El Hili, Malik Tiomoko, Corinne Ancourt
arXiv:2609.21841v1 Announce Type: new
Abstract: Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically val...
By Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid
AutoRecLab is a Python-based autonomous laboratory for recommender‑system research that transforms natural‑language prompts into executable experiments. It automatically derives experiment requirements, builds and validates prototypes, and iteratively expands them into full studies using retrieval‑augmented generation, static type verification, and execution‑steered tree search. In a demonstration, AutoRecLab autonomously conducted an explicit‑to‑implicit feedback conversion study, achieving 8 of 9 successful runs at an average cost of about $1 per run with GPT‑5.4‑mini.
By Moritz Baumgart, Philipp Meister, Justus Krell, Michael Schmidt, Bela Gipp, Joeran Beel
Designer‑RSI presents a continual adaptation framework that lets a frozen frontier model operate professional design software while an external procedural memory learns natural‑language design skills from user traffic. Over five rounds on 1,406 real briefs and 1,869 graded trajectories, the memory grew from 76 to 139 skills, boosting execution success from 72.7% to 99.3% and improving win rates on four design benchmarks. The study shows that widening and deepening the memory, especially together, significantly outperforms a no‑skill baseline.
By Hongyang Du, Lan Yan, Christian Flores, Asim Kadav
The paper introduces the COmmunity-centered Peer Engaged Support (COPES) dataset and a three‑axis evaluation framework to gauge how well Large Language Models (LLMs) align with community perspectives on mental‑health support queries. Experiments show that fine‑tuning LLMs on COPES improves strategy alignment and emotion‑tone alignment by over 50% for general‑purpose models, yet these gains are uneven across subreddits and coping strategies. The study also finds that post‑training shifts the model’s recommendations toward problem‑focused advice while reducing emotion‑focused responses, indicating persistent disparities in performance across different communities and needs.
By Mohit Chandra, Nabin Kim, Eli Min, Aamogh Sawant, Tanmay Sutar, Munmun De Choudhury
arXiv:2609.21133v1 Announce Type: cross
Abstract: SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We...
By Tarfah Alrashed, Fatma Ozcan, Per Jacobsson, Tal Neiman, Xianshun Chen
arXiv:2609.21216v1 Announce Type: cross
Abstract: Vision-language-action (VLA) policies based on flow matching generate action chunks through repeated evaluations of an action expert. Increasing the...
By Zhipeng Tang, Xinda Chen, Weining Rao, Xiao Li, Wenting Tan, Yuning Wang, Xiao Shi, Xiaofang Zhao
FOCAL‑VLA is a framework that improves vision‑language‑action models by combining subtask‑guided geometry distillation with implicit world modeling. It transfers geometric knowledge from VGGT to focus on subtask‑relevant image regions and uses Track4World features to capture future 3D evolution, guiding action generation without running these models at inference time. Experiments demonstrate that FOCAL‑VLA outperforms baselines on both simulation benchmarks and real‑world manipulation tasks.
By Zhiyuan Gao, Di Wen, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Sch\"afer, Kunyu Peng, Michael Beetz
Draft-OPD introduces an on‑policy distillation method for speculative draft models, addressing the mismatch between supervised fine‑tuning and inference by letting the target model supervise the drafter on draft‑induced states. The approach uses target‑assisted rollouts for stable continuations and replays drafting from error positions exposed during verification, enabling the drafter to learn from both accepted and rejected proposals. Experiments demonstrate that Draft‑OPD achieves more than five‑fold lossless acceleration across diverse tasks, outperforming prior draft models such as EAGLE‑3 and DFlash by 23 % and 13 % respectively.
By Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng