Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 31

DecompGym: Benchmarking LLM Agents on Designing Decomposition Algorithms

DecompGym is a benchmark that tests large language model agents on designing Benders decomposition and column generation algorithms. The benchmark contains 81 instances across nine problem families, requiring agents to produce solvers that run within 60 seconds, pass a method audit by an LLM judge, and achieve an objective within 1% of the optimal value. Evaluation of 12 GPT, Claude, and Gemini models shows that while some models solve all instances on the first try, many fail due to missing or suboptimal solutions, and reflection-based debugging improves success rates compared to simple regeneration.

By Sikai Cheng, Tinghan Ye, Arnaud Deza, Xiaonan Qi, Miao Li, Litong Liu, Changkun Guan
arXiv AI
Oct 31

Proving and Improving the Platonic Representation Hypothesis with JEPAs

The paper proves the Platonic Representation Hypothesis by showing that Joint-Embedding Predictive Architectures (JEPAs) trained on different modalities converge to the same representation up to a rotation when their positive pairs are good views of the same scene. It demonstrates that next‑token prediction is a JEPA with bad views, explaining why language models only align with vision through large scale. Experiments on millions of captions and web images confirm a scaling law of alignment, revealing that good views require far fewer samples than next‑token prediction and that a 109M‑parameter text JEPA outperforms larger language models in aligning with image encoders, even enabling a simple rotation to produce an image captioner with high COCO CIDEr scores.

By Randall Balestriero, Yann LeCun
arXiv AI
Oct 31

Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution

The paper presents a post‑training method for dynamically routed mixture‑of‑experts (MoE) models that integrates independently optimized specialists efficiently. It introduces calibrated multi‑teacher on‑policy distillation (C‑MOPD) with teacher‑wise loss normalization and clipping, combined with a post‑capacity bounded‑width dispatch, final‑route replay, and asynchronous rollout–training overlap. On the Turing‑20B‑A2B model, the approach achieves a high integration score (0.9988), retains most specialist gains, outperforms shared SFT and Qwen3‑8B on reasoning‑enabled benchmarks, and reduces consolidation time by 2.86×.

By Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Shuang Su, Siqi Liu, Kai Yang, Weichao Huang, Hang Zhang, Honggou Yang, Xianming Liu
arXiv AI
Oct 31

A-RSI: Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement

A-RSI is a method for zero‑data multimodal self‑improvement that separates the decision of when to refresh training experience from the generation of new tasks. It monitors reward variation and accuracy in training logs to trigger task pool refreshes without relying on validation or test scores. The approach uses a static library of model‑authored programs to generate multimodal questions and verify answers, achieving a performance increase from 57.77 to 61.05 on an eleven‑benchmark suite with 800 fresh tasks.

By Yu Liu, Youxing Li, Yuedong Tan, Zhiwei Yang, Zihan Su, Hanqi Feng, Shaojie Zhang, Libin Liao, Zihao Wan, Chenxu Wu, Cong Cao, Fangfang Yuan, Yanbing Liu, Ziheng Chen, Pei Fu, Radu Timofte, Zongwei Wu, Nicu Sebe, Barnabas Poczos, Jian Luan
arXiv AI
Oct 31

CALM: Conflict-Aware Lifecycle Memory Governance for Long-Horizon Language Agents

CALM is a conflict‑aware lifecycle memory governance framework designed for long‑horizon language agents. It transforms heterogeneous records into evidence‑grounded memory cells, updates their validity and lifecycle states using support, conflict, and dependency signals, and admits query‑appropriate evidence through training‑free or learned gating based on temporal applicability. Experiments on LongMemEval‑S and LoCoMo with Qwen and GPT backbones show that CALM outperforms state‑of‑the‑art baselines, improves evidence selection, suppresses erroneous memories, and preserves superseded records for accurate earlier‑state reasoning.

By Chang Li, Xianhao Ou, Xinyan Ma, Hao Li, Haoyuan Xu, Jiafeng Liang, Xiaohan Wang, Jiajun Chai, Guojun Yin, Ming Liu
arXiv AI
Oct 31

Leaving, Staying, or Living There? Duration Commonsense Reasoning with Human-LLM Co-annotations

The paper introduces a new corpus of geotagged tweets that are co‑annotated by humans and large language models to test duration commonsense reasoning in vision‑language models. It shows that current state‑of‑the‑art models struggle to infer how long a person stays at a location, even with advanced prompting techniques. Error analysis indicates that models often miss contextual clues and over‑rely on visual details, highlighting the difficulty of context‑aware multimodal understanding.

By Chenyang Yu, Zhaomin Xiao, Yan Huang, Eduardo Blanco
arXiv AI
Oct 31

DynaRev: Benchmarking Web Agents under Dynamic Goal Revision

DynaRev is a new benchmark comprising 181 tasks designed to evaluate web agents when users revise their goals during interaction. It introduces semantic milestones to trigger additive, unconditional, and runtime-conditioned revisions, and assesses agents on correctness, responsiveness, and efficiency via trajectory‑grounded evaluation. Experiments with five models show that agents often only partially satisfy revised tasks, with performance varying by task source, revision type, and model, and highlight the benefits of instruction rewriting versus browser resets.

By Shiqi Zhang, Guochen Zhou, Zesheng Wei, Hai Zhao, Zhisong Zhang
arXiv AI
Oct 31

OpenFrontiers: A Collection of Open Research Problems and a Benchmark for Scientific Rediscovery

OpenFrontiers is a cross‑disciplinary resource that compiles 12,151 open research problems and a benchmark of 200 recently resolved mathematics problems, including 110 solved in 2026 and 90 still open. Each benchmark problem is formally stated in Lean 4, verified by automated probes, and reviewed independently. The evaluation protocol tests LLM agents under closed‑book, frozen‑retrieval, and live‑web conditions, revealing that closed‑book rediscovery is rare while live‑web success largely reflects retrieval of published solutions.

By Jiarui Yao, Ruida Wang, Tong Zhang
arXiv AI
Oct 31

Seq@K: Evaluating Language Models Under Sequential Feedback

Seq@K is a new evaluation protocol that measures a language model’s ability to solve a problem within K sequential attempts, receiving feedback after each failure. It isolates the core capability of iterative refinement from feedback, contrasting with existing metrics like Pass@K that do not capture this behavior. Experiments on seven benchmarks and sixteen frontier models show that Seq@K can change model rankings, reveal more efficient configurations, and provide insights into how models use feedback.

By Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen, Daksh Mittal, Fangyu Wu, Hongseok Namkoong
arXiv AI
Oct 31

Beyond Low-Pass Dynamics: Frequency-Selective Spiking Reservoirs with Resonant Neurons

The paper introduces HRF-Res, a spiking reservoir computing model that replaces traditional Leaky Integrate-and-Fire neurons with a heterogeneous mix of Harmonic Resonate-and-Fire neurons. These HRF neurons possess intrinsic oscillatory dynamics that selectively respond to specific input frequencies, allowing the reservoir to capture frequency‑specific temporal patterns. As a result, HRF-Res delivers more discriminative representations and higher classification accuracy across ten diverse benchmarks while maintaining the energy efficiency of spiking computation, achieving up to 19× lower energy consumption than comparable non‑spiking oscillatory reservoirs.

By Aurora Micheli, Claudio Gallicchio, Nergis Tomen
arXiv AI
Oct 31

GARDEN: Generalist Autonomous Reward Design Through Evidence Navigation for Cooperative Multi-Agent Reinforcement Learning

The paper introduces GARDEN, a framework that uses large language models to autonomously design a single reward program for cooperative multi‑agent reinforcement learning tasks. By navigating evidence from environment code, training logs, and agent behavior, the reward is tested and finalized without altering the learning algorithm. Experiments on Overcooked and SMAX benchmarks show that the autonomously engineered reward improves task returns by up to 58.2% and win rates by up to 89.9% compared to default rewards, outperforming prior methods.

By Dogan Urgun
arXiv AI
Oct 31

SkillMem: Compact External Skill Memory for Latent Multi-Agent Reasoning

SkillMem introduces a compact external latent memory for reusable skills in latent multi‑agent systems, enabling role‑conditioned cross‑attention to guide agents without storing large per‑skill parameters. Trained via behavioral distillation, SkillMem matches the effect of textual skills while keeping the backbone frozen. Across six benchmarks and Qwen3 models from 4B to 14B, it delivers strong gains for smaller backbones, reduces skill storage by over 38×, and achieves up to 4.23× inference speedup.

By Yuxuan Zhu, Mohammad Mohammadi Amiri, Pin-Yu Chen, Keerthiram Murugesan
arXiv AI
Oct 31

Advancing Clinical Genomic Analysis with a Skill-Centred Agent

The paper introduces ClinGeX, a large‑language‑model agent that uses 105 curated executable skills to automate clinical genomic analysis. In benchmark tests, ClinGeX achieved 71.7% macro‑averaged task‑level concordance, outperforming the best comparator at 45.7%. Human‑AI collaboration studies showed ClinGeX significantly improved case‑level concordance across variant analysis, rare disease interpretation, and oncology decision‑making, while also reducing interpretation time.

By Qiaoyu Zheng, Weike Zhao, Yanjie Fan, Ye Cao, Mengqi Jiang, Xiao Zhou, Xiaoman Zhang, Pengcheng Qiu, Yongguo Yu, Yanfeng Wang, Ya Zhang, Chaoyi Wu, Weidi Xie
arXiv AI
Oct 31

FVH-Bench: Evaluating Image Generator Safety Beyond Overtly Harmful Imagery

The paper introduces FVH-Bench, a benchmark designed to evaluate image generators for functional visual harm (FVH)—the risk posed by images that enable dangerous actions or deception rather than merely depicting overtly harmful content. FVH-Bench contains 669 prompts across five harmful visual functions and ten risk domains, and tests 11 image generators, many of which rarely refuse these prompts and often produce artifacts that fulfill the targeted harmful functions. The study also shows that existing safety detectors provide limited coverage of FVH, highlighting gaps in current safety alignment assessments.

By Zihao Zhu, Yuhan Liu, Siwei Lyu, Baoyuan Wu
arXiv AI
Oct 31

Predicting What Answer Engines Cite: Open Decision Models, Rerankers and a Distilled 4B Student on Ten Engines under Controlled Retrieval

The paper introduces a new scorer for generative answer engines that ranks candidate pages by the likelihood of being cited. It evaluates pretrained rerankers, a hosted typed decision model (Jev), eight open decision models, and a distilled 4B student on GEO‑Bench queries answered by ten engines. The distilled student, fine‑tuned with low‑rank adapters, achieves comparable performance to larger models and the authors release the model, benchmark, predictions, and code.

By Elisha Bajemon, Andre-Louis Rochet
arXiv AI
Oct 31

Convex-SKM-Net: Low-Latency Block Halfspace Corrections for Neural Convex Constraint Satisfaction

Convex‑SKM‑Net is a neural layer that enforces convex constraints by constructing linearization halfspaces from residual and gradient oracles, avoiding constraint‑specific projections. It uses fixed‑depth dual projected‑gradient descent on GPU‑friendly matrix operations to solve minimum‑displacement block problems, jointly learning a predictor and a positive‑definite mobility matrix to shape corrections. The method converges theoretically, achieves sub‑$10^{-3}$ feasibility on SOC‑MPC and power‑system relaxations, and outperforms state‑of‑the‑art constraint‑satisfaction techniques with up to 144× speedups and near‑optimal objective gaps.

By Haoyu Zhu, Xiaochen Zhang, Qingchun Hou
arXiv AI
Oct 31

MUSE: Multi-Level Memory with Selective Runtime Injection for Scientific Coding Agents

MUSE is a memory management framework for scientific coding agents that combines multi-level memory—global, approach-level, and implementation-level—with selective runtime injection. The agent starts with global and problem-matched approach-level memory, then during execution it retrieves implementation-level memory and an LLM gate decides whether to inject relevant experience as review advice. Experiments on four benchmarks in numerical problem solving, kernel optimization, and scientific software repair show that MUSE outperforms retrieval and memory baselines, and ablations confirm the benefits of the multi-level design and gated injection.

By Haoran Jin, Defu Lian, Xiting Wang, Hong Xie
arXiv AI
Oct 31

HERA: Assessing an Autonomous Agent for Quantum Device Measurement

HERA is an autonomous agent that performs cryogenic transport measurements on quantum devices using a curated knowledge base of protocols, software, and literature. In a blinded trial, HERA matched an expert’s conclusions on the stacking order of multilayer graphene devices, and ablation and replay studies demonstrated that the knowledge base improves measurement design and reasoning across models. When tasked with discovering superconductivity in a new material, HERA autonomously developed and executed the experiment after minimal expert input, illustrating its capability for sound experimental science.

By Zachary W. Gomez, Nathan Bittner, Jordan T. McCourt, Qingrui Cao, Sandesh S. Kalantre, Benjamin H. Alexander, Zhiren Zheng, Kenji Watanabe, Takashi Taniguchi, Aaron Sharpe
arXiv AI
Oct 31

Tactus: Tactile & Pose Sensing Glove for Natural Dexterity Data Collection

Tactus is a textile‑integrated glove that combines a thin, conformal 3D‑knit design with strategically placed capacitive tactile sensors and IMU‑fused pose tracking to capture natural human manipulation. It offers a calibrated, cross‑glove consistency pipeline and per‑session subject correction, achieving a minimum detectable force of 0.59 N and fingertip pose error of 11.0 mm even under heavy occlusion. The authors benchmark Tactus against two commercial gloves, presenting the first quantitative cross‑platform evaluation of tactile force regression and hand pose estimation.

By Li Guan, Rachael Granberry, Sahar Aseeri, Sahar Rostami, Taylor Niehues, Sai Dhanooshi Marri, Mia Huang, Dingtian Zhang, Ziyi Kou, Jason Pursel, Eric Whitmire, Vatsal Mehta, Ashish Deshpande, Tristan Trutna, Reed Belcher, Emelia Black, Julian Blake, Roger Boldu, Sean Bonaventure, Einar Boyovich, Jordan Brennan, Elizabeth Casillas, David Christensen, Tyler Christensen, Nicholas Colonnese, Stuart Dealey, Gaige DeHaven, Katie Doran, Chris Doughty, Robert Drury, Silvia Espino, Gabriel Geyne, Beth Goodrich, Kyra Gould, Joe Grisso, Susana Guerrero, Andrew Gwozdziowski, Monica Ha, Ehsan Hajiesmaili, Kate Healy, Rose Hope, Michael Hurst, Claire Ikemiya, Daisy Isibor, Rick James, Yashasvi Joginapally, Samuel Johnson, Tyler Jorgensen, Kristy Jost, Ben Kamath, Slava Karulin, Alexander Keener, Woo Kim, Alison Krauskopf, Ankit Kumar, Joshua Lang, Lizzy Li, Cameron Lynch, Daniel Ma, Hector Martinon, Christina Matos, Keagan McCurdy, Lee Mincey, Stuart Nezlek, Tiffany Nguyen, James O'Hara, Jason Osborne, Jacky Pai, Roham Pardakhtim, David Perek, Alexandre Poulin, Nick Raines, Salwa Rasheed, Leah Resneck, Kevin Rhodes, Ergys Ristani, Douglas Roberts, Anik Sarker, Harrison Scarborough, Jonas Schmidtler, Brian Shieh, Benjamin Shuman, Garrett Simard, Forrest Smith, Udara Somarathna, Eliot Sosin, Jonathan Stewart, Kristine Strom, Adam Swett, Hal Talbot-Mantooth, Julia Tanner, Sai Sharan Thirunagari, Curtis Torrey, Simge Uzun, Eileen Wachs, Rylan Wasson, Henry Wei, Jesse White, Beau Woodbyrne, Antonio Woods, Noah Young, Chandler Yount, Joseph Zhang, Sean Keller
arXiv AI
Oct 31

EmbodiExp: Compositional Experience Transfer for Embodied Agents

EmbodiExp is a framework that enables embodied agents to transfer and compose experiences across tasks by focusing on subgoals. It introduces outcome‑grounded subgoal experiences that include a subgoal definition, applicability conditions, an adaptable policy, and a verifiable outcome. During task execution, applicable experiences guide action generation, while verified outcomes help set conditions for subsequent experiences, allowing online composition and even leveraging failures for local success insights.

By Zhifeng Gu, Ying Chen, Rui Jiang, Lei Yi, Bing Wang