Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 21

Evolving Self-tuning Benchmarks for Long-Horizon Tool Use

The paper introduces TURTLE, an Unsupervised Environment Design framework that evolves tool‑use tasks for AI assistants by using a proposer, solver, and feedback generator powered by large language models. TURTLE automatically generates self‑tuning, long‑horizon benchmarks that increase task difficulty, diversity, and failure modes across multiple applications. Experiments on AppWorld, τ‑Bench, and AutomationBench demonstrate significant gains in hard task percentages and broader reasoning challenges.

By Hyunji Alex Nam, Michael Dennis, Natasha Jaques, Minsuk Chang
arXiv AI
Oct 21

Recoverability and Verifier Strength: Two Measured Preconditions for Unattended Agent Self-Improvement

The paper introduces two measurable preconditions—Verifier Strength and Recoverability—to determine when an agent platform can safely allow unattended self‑improvement. Verifier Strength quantifies how closely an automated verifier (often an LLM judge) aligns with a trusted reference, limiting the extent of permissible changes. Recoverability measures how easily a detrimental change can be undone, requiring that the original version be retained and that rollback restores the prior behavior. The authors implemented these concepts on a platform, demonstrating that unattended edits are viable only when both conditions are met, and highlighted failures in verifier consistency and undo reliability through controlled experiments.

By De Wang, Peng Qi
arXiv AI
Oct 21

Does Multi-Agent Collaboration Pay When Agents Serve Users? Testing Agent Scaling in Dual-Control Settings

The study evaluates whether multi‑agent systems (MAS) outperform single‑agent systems (SAS) when agents interact with users under a written policy on the dual‑control benchmark. Across 7,950 simulated conversations in airline, retail, and telecom domains, the MAS performed worse in 12 of 13 valid cells and tied only once, with a negative average gain of 28 points. Even in scenarios where prior theory predicted a benefit, the MAS lost performance and incurred higher token costs.

By Ethan Yang, Xiangcheng Shi
arXiv AI
Oct 21

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

ACE-Compression introduces an ability‑centric evaluation framework for compressed large language models, moving beyond traditional task‑performance metrics. It probes four deployment‑relevant dimensions—procedural reasoning, knowledge grounding, social cognition, and persona fidelity—to determine which core abilities survive compression. The study finds that compression redistributes ability profiles, with coarse signals persisting while fine‑grained abilities become fragile, and that quantization generally retains more abilities than pruning.

By Jiahui Ning, Liang Chen, Yuchen Liang, Yi Zhang, Fenglei Fan
arXiv AI
Oct 21

Learning Inter-Graph Interactions Between Heterogeneous Substructures of Chemical Systems

The paper introduces a data descriptor and a graph neural network designed to learn inter‑graph interactions between heterogeneous substructures in complex chemical systems, specifically between organic and inorganic compounds. It addresses the limitation of prior methods that only considered homogeneous graph interactions. The proposed approach was applied to predict band gaps for 1,682 hybrid solar cell materials, achieving state‑of‑the‑art accuracy.

By Gyoung S. Na
arXiv AI
Oct 21

Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

Dynamic Cross-Modal Prompt Generation (DRAPE) is a prompt-learning framework designed for Multimodal Continual Instruction Tuning (MCIT). It generates continuous, instance-specific soft prompts by cross-attending textual instructions with visual patch features, rather than selecting from a fixed prompt pool. DRAPE also incorporates null‑space gradient projection to reduce forgetting and uses CLIP‑based prototype routing for task‑label‑free inference, achieving state‑of‑the‑art results on MCIT benchmarks.

By Tao Hu, Da-Wei Zhou
arXiv AI
Oct 21

What a One-Hour Interview Does Not Tell an LLM About You: Interviews Elicit Traits, Not Decision Policies

The study examines whether a one‑hour adaptive interview can reveal a person’s value‑laden decision policies to a large language model (LLM). Forty‑six adults underwent the interview and later answered 25 dilemma questions; the researchers compared predictions from various transcript segments, adaptive versus fixed follow‑ups, and a baseline model with no personal data. Results show that no part of the interview reliably predicts the participants’ answers better than the baseline, and adaptive follow‑ups offer no advantage over fixed ones, though the transcripts do help recover the participants’ MBTI types.

By Ruoxi Su, Yuhan Liu, Jingyu Hu
arXiv AI
Oct 21

Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs

Omni-Woodpecker (OWP) is a training‑free framework designed to reduce cross‑modal hallucinations in Omni LLMs by rebalancing fused representations. It identifies conflicts between modality‑specific views, separates grounding evidence from competing priors via orthogonal decomposition, and adaptively suppresses misleading priors while restoring ignored evidence before decoding. Experiments show OWP improves performance on CMM, AVHBench, and several general reasoning benchmarks, and generalizes across different Omni LLM architectures.

By Zijian Liu, Chaoyou Fu, Yan Yang, Caifeng Shan, Ran He
arXiv AI
Oct 21

LSA-PAST: Layer-Specific Adaptation and Prototype Transport for Compositional Incremental Learning

The paper introduces LSA-PAST, a parameter‑efficient approach for compositional incremental learning that addresses the challenge of learning new state‑object compositions while preserving knowledge of previously seen compositions. It combines Layer‑Specific Adaptation using Low‑Rank Adaptation (LoRA) with depth‑specialized adapters for states, objects, and compositions, and employs Prototype‑Anchored Adapter‑Space Transport (PAST) to map past prototypes into the current adapter space without rehearsal. Experiments show that LSA-PAST achieves state‑of‑the‑art accuracy‑forgetting trade‑off performance in this setting.

By Kyle Stein, Andrew Arash Mahyari, Guillermo Francia III, Eman El-Sheikh, Jiangpeng He
arXiv AI
Oct 21

Sparse RL Relaxation for LLM Reasoning

The paper introduces SpaRLax, a sparse relaxation method for reinforcement learning that enhances soft Chain-of-Thought reasoning in large language models. By treating soft tokens as differentiable intermediate steps and applying an adaptive entmax sparsity technique, SpaRLax stabilizes training and improves performance across nine reasoning benchmarks. The method retains multiple plausible candidates when uncertain while truncating noise at confident steps.

By Jiabin Fan, Matthias Lindemann, Zita Marinho, Andr\'e FT Martins, Lili Mou