The paper investigates how the local landscape geometry of language model pre‑training evolves, identifying two distinct phases. In Phase I, the landscape starts sharp, causing instability and loss plateaus at high learning rates, which explains the need for learning‑rate warmup and suggests longer warmups for larger peak rates. In Phase II, the geometry is governed by gradient noise scale, revealing a depth‑flatness trade‑off that motivates a dynamic batch‑size scheduler that starts small and grows later in training.
By Zhanpeng Zhou, Yuhan Sun, Bingrui Li, Jinbo Wang, Huaijin Wu, Lei Wu, Junchi Yan
arXiv:2609.38612v1 Announce Type: new
Abstract: As natural language drives more applications, language models increasingly run inside programs as decision components: the program sends them the curre...
By Jhen-Ke Lin, Chung Chun Wang
The study evaluates synthetic pre‑pretraining (PPT) across a wide range of models (500 M–7 B parameters) and training budgets (up to 100 B tokens). Results show that PPT consistently improves downstream performance and token efficiency, saving at least 21 B tokens at the 3 B scale, but these gains do not appear to stem from a grammatical prior. Instead, PPT benefits arise from tasks that enhance long‑range retrieval, and the improvements remain robust across diverse data mixtures, diminishing only when web text is omitted.
By Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus, Michal \v{S}tef\'anik, Aline Villavicencio, Nikolaos Aletras
SYNCR is a synthetic benchmark designed to evaluate multimodal large language models on cross‑video reasoning. It contains 4,000 question‑answer pairs across 4,827 unique videos, covering tasks in temporal alignment, spatial tracking, comparative reasoning, and holistic synthesis. The benchmark reveals a significant performance gap between current models and humans, with models excelling at temporal ordering but struggling with precise physical and spatial reasoning.
By Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami
The paper investigates whether adding Abstract Meaning Representation (AMR) data to large language models (LLMs) improves performance on downstream tasks. By reproducing recent studies and applying a consistent hyperparameter protocol, the authors find that text-only baselines match or surpass AMR-augmented models. A perplexity-based probe shows that AMR does not provide LLMs with additional relational knowledge, suggesting no clear benefit from AMR augmentation.
By Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan
The paper introduces a framework that uses large language models (LLMs) to generate natural‑language narratives explaining cross‑sectional stock return predictions. It combines temporal Shapley additive explanations (SHAP) from an XGBoost model with historical regime analogs to provide context. A controlled study shows that progressively externalizing numerical and relational reasoning improves evidence faithfulness and accuracy, while historical analogs boost human‑rated usefulness.
By Sujung Kim, Seung Hwan Cho, Sangjin Park, Young-Min Kim
arXiv:2609.38480v1 Announce Type: cross
Abstract: Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients presen...
By Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati, Vallijah Subasri, Caihua Shan
AREX-2 is a new approach that enhances the self‑improving ability of large language model agents by combining reflection—producing better solutions—and long‑horizon execution—maintaining effectiveness over many iterations. The method synthesizes improvement trajectories from machine‑learning and algorithmic programming tasks, providing verifiable feedback and sustained iteration. Trained on this data, an agent based on Qwen3.8‑27B achieves strong performance across multiple benchmarks and continues to improve as more iterative rounds are allowed.
By Hongjin Qian, Chaofan Li, Kun Luo, Wenqing Wei, Jianlyu Chen, Shuqi Lu, Yuyang Hu, Hongwang Xiao, Hui Wang, Chaozhuo Li, Qiwei Ye, Zhicheng Dou, Defu Lian, Zheng Liu
The study analyzes 20,462 student turns from 1,260 sessions with a guided LLM chemistry tutor, identifying 6,630 impasse turns categorized as conceptual errors, expressed uncertainty, or help‑seeking. Three tutoring conditions—baseline, no‑direct‑answer, and guided—were simulated, revealing that the baseline tutor often gave direct answers, the no‑direct‑answer tutor always asked follow‑up questions, and the guided tutor varied its responses based on context. Impasse trajectories showed that each additional impasse turn reduced the likelihood of recovery, while addressing errors became increasingly beneficial compared to repeated scripted questioning.
By Bakhtawar Ahtisham, Kirk Vanacore, Alessandra Napoli, Josh Arens, Ksenia Ionova, Clayton Cohn, Shima Salehi, Rene Kizilcec
The paper introduces NAQD‑Env, a synthetic benchmark designed to test language agents’ ability to selectively withdraw and resume actions when new evidence, permissions, or stop instructions arise. It evaluates models against a deterministic reference policy across eleven dependency families, measuring policy agreement, task value, withdrawal, resumption, and event reporting. Experiments on 350 scenarios show low withdrawal recall, no valid resumption, and limited policy alignment, highlighting the need to treat selective withdrawal as a distinct reliability component.
By Mohamed Abouzahra
PrivMeSA is a privacy‑aware, self‑evolving multi‑agent system that enables local clinical LLM agents to consult remote specialists while controlling patient data disclosure. It uses reinforcement learning to balance task accuracy with privacy risk, evaluates risk over entire outbound transcripts, and employs a local lesson memory to distill and reuse expertise without further remote exchanges. On an emergency‑department benchmark, PrivMeSA boosts task accuracy by up to 15.8 percentage points and virtually eliminates patient re‑identification risk, reducing personal detail disclosure from 98.0% to 0.2% of cases.
By Dannong Wang, Yuran Zhang, Bian Sun, Alex Stinard, Yuzhang Shang, Song Wang, Yu Tian
The paper introduces a benchmark of over 6,000 clinical triage scenarios, 7,000 physician annotations, and 225,000 large language model (LLM) responses to assess how LLMs perform under realistic variations in clinical text. The study finds that LLMs tend to recommend unnecessary care more often than physicians, especially when the input text is perturbed, and that LLM recommendations are more sensitive to gender and tone changes than human recommendations. These findings underscore the importance of deployment‑oriented evaluations that reflect expert physician behavior.
By Abinitha Gourabathina, Haoran Zhang, Yuexing Hao, Walter Gerych, Marzyeh Ghassemi
The paper presents a method for generating creative chess puzzles using masked diffusion models that can be conditioned on tactical themes and partial board positions. It introduces an auxiliary best‑move prediction task that boosts solution uniqueness by 11.6% and theme‑conditioning accuracy by 2.5%. A reinforcement learning framework further increases the yield of unique, theme‑matching puzzles by 89.1%, and the authors release open‑weights models for the community.
By Aatu Selkee, Severi Rissanen, Xidong Feng, Tom Zahavy, Eric Malmi
The study examines how language models decide whether two scientific findings conflict by translating an unsatisfiable XOR constraint system into reports from different laboratories. When constraints are explicit, GPT‑5.6 Sol and Claude Opus 5 correctly select the best-supported assignment in 90% and 96% of cases, respectively. In scientific prose, Claude Opus 5 tends to favor biologically expected assignments, but this bias can be mitigated by formalization requests and explicit paired‑design cues, raising correct recovery to 92%.
By Tal Zeevi, Trey W. Jensen, Maxwell Strome
EvoSteer introduces an online self‑evolving graph orchestration framework that continuously builds and repairs a team of tool‑using agents during execution. It employs Anchored Trajectory Balance (AnchorTB), a regression‑style loss that assigns credit to each orchestration action by comparing subtrajectories to a frozen reference, and Validated Skill Admission, which tests candidate skills before promotion. Experiments on twelve datasets demonstrate that EvoSteer outperforms existing baselines in question answering, mathematical reasoning, code generation, and interactive decision making.
By Mingda Zhang, Hanwen Zhang, Qiang Huang, Zijia Wang, Pengfei Guo, Yuchen Zhang, Jionghao Zhu, Xiaoying Tang
The paper treats multi‑path large language model reasoning as a diversity‑combining problem, analogous to noisy channel observations in wireless communications. It shows that the optimal symmetric linear combiner of latent embeddings is uniform, justifying majority vote in standard self‑consistency while allowing for weighting or pruning when prompt‑template branches are heterogeneous. By reducing path correlation through prompt‑template diversity, the authors propose an Adaptive‑K rule that selects an optimal number of reasoning paths, preserving most of the accuracy achieved with a fixed large number of paths across multiple models and benchmarks.
By Guangsheng Yu, Litianyi Zhang, Qin Wang, Xu Wang, Mingyuan Li, Shaoxiong Ji, Ren Ping Liu, Massimo Piccardi
The paper introduces Consistent Plan-Act (ConPAct), a method that addresses coordination failures between high-level planners and low-level actors in long-horizon agentic tasks. By prompting both agents to produce structured state assertions and programmatically detecting contradictions, the authors identify a systematic planner-actor state mismatch. ConPAct feeds these detected contradictions back to both agents, fine‑tunes them on consistent interactions, and achieves notable performance gains, such as raising MiniGrid success rates from 38.6% to 54.4% with GPT‑5.6‑sol/terra.
By Heng-Zhuang Li, Yi-Kai Zhang, Yu Wang, Yueqing Sun, Jiayuan Zhang, Qi Gu, Han-Jia Ye
The paper evaluates 22 state‑of‑the‑art large language models on 12 molecular regression datasets in a zero‑shot setting, comparing their predictions to a molecule‑blind reference derived from the datasets’ labels. It finds that many models retrieve published values rather than truly predicting properties, with significant retrieval concentrated on five datasets and additional flagged instances elsewhere. Increasing the reasoning setting doubles the number of flagged model–dataset pairs, and an in‑context blinding experiment reduces but does not eliminate retrieval, also altering model rankings and increasing errors.
By Matthias Busch, Marius Tacke, Sviatlana V. Lamaka, Mikhail L. Zheludkevich, Christian J. Cyron, Roland C. Aydin, Christian Feiler
SkillFM is a generative framework that creates task‑conditioned textual skills for large language model agents without relying on manual skill banks or reinforcement learning. It encodes skills into a continuous latent space using a codec and trains a conditional flow model with improved MeanFlow, allowing single‑step latent sampling at inference. The sampled latent is decoded by an LLM into textual guidance, and the method outperforms other vector‑based skill approaches on ALFWorld, Search‑QA, and other tasks.
By Zuming Zhang, Jie He, Yizhe Zhang, Jeff Z. Pan
The paper investigates whether computation from earlier problems can aid large language models (LLMs) in solving subsequent ones. Preliminary experiments show that retained conversation history can both improve and degrade later-turn accuracy. To address this, the authors propose STAIR, a lightweight module that stores key-value pairs from previous responses and redirects new queries to this bank, improving later-turn accuracy by up to 11.67 percentage points across several benchmarks.
By Jipei He, Wenhui Tan, Xiaoyi Yu, Enver Sangineto, Fiorenzo Parascandolo, Rita Cucchiara, Ruihua Song