The study evaluates 26,999 pull requests that modify both application code and deployment infrastructure, using two large datasets and six AI agent architectures. It finds that such dual‑scope changes occur in only about 4.5% of agent‑generated PRs but lead to two distinct integration outcomes: either significant delays due to CI/CD and provisioning checks, or a high rate of silent merges with reduced human review. The results highlight that current AI agents lack deployment awareness, causing unverified dependencies to slip into production pipelines.
By Mohammad Wathiq Soualhi, Basem Suleiman
The paper introduces CIG‑SFT, a supervised fine‑tuning method that uses conditional information gain to evaluate how much each demonstrated token reduces uncertainty about the task. By comparing token likelihoods from a frozen reference model with and without task input, the method derives a KL‑regularized projection of the current policy, yielding a scalar‑weighted SFT objective whose gradients match the projected target. Experiments on mathematical reasoning and code‑generation benchmarks show that CIG‑SFT outperforms standard SFT and adaptive fine‑tuning baselines, especially in generalizing to challenging evaluation distributions, without requiring extra data or online rollouts.
By Yuxuan Gu, Kequan Niu, Xiaocheng Feng, Yun Li, Yiqiao Yang, Bing Qin
The paper introduces ReqClarifyBench, an interactive benchmark for evaluating how well large language models (LLMs) can clarify vague user intent into clear requirements in repository-level software development. It includes 341 tasks across game, miniapp, and web development, each with a vague intent, a human-validated requirement document, and code evaluation artifacts, and uses LLM-based user simulators with diverse personas. The study evaluates six LLMs on clarification effectiveness and requirement quality, finding that stronger clarification does not always yield better documents, performance varies by persona, and coding-agent harnesses do not improve document quality over base LLMs.
By Mingyu Chen, Zhiqing Zhong, Manyi Wang, Linxi Liang, Rouyu Chen, Pinjia He
CoBe is a benchmark that evaluates a model’s ability to perform conversational counterfactual text editing, requiring the model to revise a given scenario based on a hypothetical change while preserving unaffected facts. The dataset includes 2,000 questions across domains such as healthcare, science, engineering, business, and everyday life. Experiments show that current models often default to associational generation rather than maintaining causal structure, and performance does not reliably improve with larger model size or longer outputs.
By Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar, Kai-Zhan Lee, Aurghya Maiti, Yushu Pan, Jeffrey Wu, Mingxuan Li, Elias Bareinboim
The paper introduces Synthetic Self-Distillation (SynthSelf), a method that enables continual post‑training for small language models by outsourcing in‑distribution learning to a slightly larger aligner. SynthSelf combines off‑policy supervision and rejection sampling to provide high‑quality supervision while preserving the model’s original distribution. Experiments on four acquisition datasets and fourteen retention benchmarks show SynthSelf consistently outperforms six baseline methods across three small‑model scales, demonstrating its effectiveness for low‑latency, on‑device continual learning.
By Woosung Koh, Hyunsoo Lee, Kyungjae Lee, Haeju Park, Dahyun Lee, Dasol Hwang, Moontae Lee
The paper proposes a bilevel Stackelberg formulation that trains standard post‑hoc calibrators end‑to‑end with the neural network backbone, eliminating the need for a separate calibration dataset or dedicated loss. By jointly optimizing accuracy and calibration with cross‑entropy and a single calibrator learning‑rate hyperparameter, the method can use any calibrator type, but shows that the final classification layer alone suffices to produce accurate, well‑calibrated models. This approach outperforms state‑of‑the‑art calibrators while adding minimal complexity and avoiding the overhead of separate calibration components, especially beneficial in multi‑task learning settings.
By Nektarios Kalampalikis, Pablo M. Olmos, Isabel Valera
MC‑EvoBench is a controlled Minecraft benchmark designed to audit experience‑driven self‑improvement in agents. It features 19 evolution chains (12 primary, 7 calibration/probe) and three graph‑aligned probe families, each task represented as a task‑state graph with stage‑level verifiers and strict source‑target boundaries. The benchmark evaluates probes under no‑prior, related, unrelated, and shuffled source conditions, reporting metrics such as TSR, WGP, and VMP while preserving provenance records.
By Jingwei Song, Zhisheng Chen, Bingfan Zeng, Jing Tang, Yubo Hou, Jinhao Jing, Zhengwei Xie, Linfeng Zhang
The paper introduces MAGI, an open modular agent that orchestrates molecular‑design tools by setting objectives, launching optimization, interpreting structure–activity relationships, and adjusting strategy. In nine retrospective lead‑optimization campaigns from three pharmaceutical companies, MAGI produced valid molecules via either direct LLM generation or delegation to REINVENT 4; LLM proposals stayed closer to local chemistry and achieved comparable or higher activity with fewer operations, while REINVENT explored broader chemical space. The success of a campaign depended on the accuracy of predictive models rather than the generation route, and a blinded evaluation showed chemists could not distinguish MAGI’s outputs from expert‑generated compounds, though the SAR reasoning was deemed plausible yet incomplete. The study positions MAGI as a coordination layer that can be plugged into existing workflows, but notes that the current ceiling for real projects is set by scorer applicability rather than tool orchestration.
By Pierre Llompart, Levent Guner, Helen Lai, Alessandro Tibo, Yijie Xu
WoundAmbit is a framework that evaluates state‑of‑the‑art deep learning models for wound segmentation, benchmarking general‑purpose vision, medical imaging, and public wound challenge methods under standardized training, augmentation, and cross‑validation. The study finds transformer‑based TransNeXt offers the best generalizability, while all models run at least one image per second on a CPU and show strong interpretability and expert‑approved mask quality. The framework also introduces a reference‑object approach to convert masks into clinically relevant wound size estimates, demonstrating close alignment with expert annotations and integration into a custom telehealth system.
By Vanessa Borst, Timo Dittus, Tassilo Dege, Astrid Schmieder, Samuel Kounev
VITAL introduces a visual‑semantic dual supervision framework for latent reasoning in medical multimodal large language models. It uses an auxiliary text decoder to reconstruct reasoning chains from latent states and a visual projector to regress ROI features from a frozen medical vision encoder, both discarded during inference to avoid overhead. The approach yields a 61K dataset across nine imaging modalities and achieves state‑of‑the‑art performance on seven benchmarks, outperforming existing latent reasoning baselines and large medical MLLMs.
By Qiaoru Li, Shaotian Liang, Jintao Chen, Haoran Sun, Yuxiang Cai, Jianwei Yin, Yankai Jiang
STAPLE is a method that integrates static sparse memory into pretrained language models by adding token‑indexed embeddings as a residual pathway while preserving the original feed‑forward network. On the OLMo‑2 1B and 7B models, STAPLE outperforms dense training and other static sparse baselines, improving benchmark scores by up to 4.27 points and achieving faster decoding with less GPU memory and energy usage. The approach demonstrates that large pretrained models can be expanded efficiently without costly sparse pretraining from scratch.
By Ranajoy Sadhukhan, Euijin Hong, Sheng Cao, Zechun Liu, Shuowei Jin, Sha Li, Changlong Yu, Xin Liu, Qin Lu, Zhenyu Shi, Priyanka Nigam, Beidi Chen
The paper introduces HART, a Hybrid Attention-RWKV Time-series Foundation Model that alternates attention and RWKV blocks to improve long-horizon forecasting. It identifies a problem in RWKV-7 called Placeholder Idling Decay, where repeated state transitions degrade historical information during non-autoregressive multi-step forecasting. Experiments on GIFT-Eval and TIME-Benchmark show that HART outperforms pure RWKV on medium and long horizons while matching short-horizon performance, and achieves comparable long-horizon MASE to a much larger Transformer with only 36 M parameters.
By Qingwen Lin, Boyan Xu, Shifeng Xie, Zhifeng Hao, Keli Zhang, Ruichu Cai
CB-Orchestrator is an adaptive framework for optimizing large language model (LLM) agent workflows. It separates workflow generation—via evolutionary search—from workflow selection, which is treated as a contextual bandit problem to enable sample‑efficient learning. The approach reduces training‑phase API token usage by up to 40.55% and overall token overhead by at least 29.73%, while outperforming baselines on five benchmarks, including the MATH dataset.
By Jiahang Sun, Zhiwei Shang, Zhipiao Liu, Hongwei Yang, Guoqing Xie, Shuang Qiu, Zhongxiang Dai
MolP2SBench is a new benchmark that tests whether large language models can reason from molecular properties to structure. Each task presents a masked scaffold and a set of physical properties, asking the model to pick the correct structure from plausible candidates. By shuffling or replacing properties, the benchmark evaluates whether models truly rely on correct property‑structure pairings and how they respond to counterfactual changes, using a Property‑Grounded Recovery metric to compare performance across models.
By Yuhang Xie, Zhiyi Shi, Zeyi Zhang, Zhisheng Qi, Yushun Dong, Yu Wang, Xiongye Xiao
The paper presents an in-context learning framework for system identification that replaces traditional two-step model estimation with a meta model mapping training data and query inputs directly to outputs. It introduces a probabilistic Transformer-based architecture capable of handling long training sequences and producing calibrated predictions over extended horizons. Additionally, the authors demonstrate that a pre-trained meta model can generate uncertainty-aware synthetic data to aid classical identification methods and validate their approach on standard nonlinear benchmarks and challenging out-of-distribution scenarios, achieving competitive accuracy and strong generalization.
By Matteo Rufolo, Dario Piga, Filippo Pura, Marco Forgione
NucleoNet is a neural‑network variational Monte Carlo ansatz that treats electrons and nuclei as a unified particle set, allowing a single model to be optimized across multiple geometries. The approach uses a shared electron–nucleus backbone to evolve one‑particle and two‑particle representations, with nuclear streams learning representations that condition orbital construction while antisymmetry is enforced only by electronic determinants. On small‑molecule benchmarks, NucleoNet matches the accuracy of established neural‑wavefunction baselines with far fewer parameters and successfully describes dissociation curves and minimum‑energy paths for several systems, demonstrating that a compact wavefunction can capture molecular geometries with a single set of parameters.
By Qiao Gao, Yang Huang, Enze Hou, Yuzhi Liu, Jingrun Chen, Han Wang
DEAF (Diagnostic Evaluation of Acoustic Faithfulness) is a benchmark comprising over 2,700 conflict stimuli that test Audio Multimodal Large Language Models (Audio MLLMs) across three acoustic dimensions: emotional prosody, background sounds, and speaker identity. The benchmark uses a controlled multi-level evaluation framework that gradually increases textual influence, from semantic conflicts to misleading prompts, to separate content-driven bias from prompt-induced sycophancy. Diagnostic metrics are introduced to measure how much models rely on textual cues versus acoustic signals, and an evaluation of seven Audio MLLMs shows that while models respond to acoustic variations, their predictions are largely driven by textual inputs, highlighting a gap between benchmark performance and true acoustic understanding.
By Jiaqi Xiong, Yunjia Qi, Qi Cao, Yu Zheng, Yutong Zhang, Ziteng Wang, Ruofan Liao, Weisheng Xu, Sichen Liu
The paper introduces a lightweight AI framework for vibration-based condition monitoring of consumer rotating devices. It transforms 1‑D vibration signals into 2‑D time‑frequency maps using continuous wavelet transform and Hilbert‑Huang transform, then applies compact deep learning models such as MobileNetV3‑Small, ConvNeXt‑Tiny, and YOLOv11‑cls. Experiments on public bearing datasets show that CWT with YOLOv11‑cls delivers high diagnostic accuracy, while MobileNetV3‑Small offers a low‑latency alternative suitable for edge deployment, and the study emphasizes the need to avoid data leakage in model evaluation.
By Po-Heng Chou, Jen-Yu Chiu, Chun-Yu Yeh, Wei-Lung Mao, Walid Saad
The paper introduces TAP-JEPA, a joint‑embedding predictive architecture trained on over 113,000 CT volumes, which achieves performance comparable to leading encoders in scan‑level classification. It demonstrates a predictability bias: while anatomy is largely predictable from context, pathology is not, leading to significant degradation in pathology detection when predictions are applied. The study audits the model’s predictive pathway, showing that pathology representations shift toward control regions and that removing pathology tokens minimally affects downstream decisions, underscoring that contextual predictability alone is insufficient for clinical relevance.
By Fabian Drexel, Bernhard Kainz, Jakob N. Kather, Daniel Rueckert, Lisa C. Adams, Keno K. Bressem, Cosmin I. Bercea
The paper introduces GraphTok, a GPU-based tokenizer that executes the entire tokenization pipeline as a single CUDA graph, matching reference token IDs for models such as GPT‑2, Llama 3, Qwen 2.5, DeepSeek‑V3, and Gemma 3. GraphTok achieves significant speedups, encoding one million GPT‑2 tokens in 1.7–3.2 ms and outperforming the leading CPU tokenizer gigatoken by 4.7–6.0× on Llama 3. In a vLLM deployment with cached prefixes, it reduces mean time‑to‑first‑token by 2.1–2.4× and dramatically increases request throughput, sustaining 518 req/s of 121k‑token traffic versus 296 with gigatoken.
By Mohammad Siavashi, Pascal Schwinte, Ali Banaei, Gerald Q. Maguire Jr., Marco Chiesa, Dejan Kosti\'c