Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv AI

Jun 30

ReactiveBFM: Reactive Closed-Loop Motion Planning Towards Universal Humanoid Whole-Body Control

arXiv:2606. 30362v1 Announce Type: cross Abstract: While current Behavior Foundation Models (BFMs) provide robust control priors for humanoids, they only execute pre-defined reference motions.

By Xiao Chen, Weishuai Zeng, Xiaojie Niu, Zirui Wang, Jianan Li, Huayi Wang, Furui Xu, Jiahe Chen, Weixiang Zhong, Lihe Ding, Kailin Li, Jiangmiao Pang, Tai Wang, Tianfan Xue, Jingbo Wang

rag benchmarks safety

arXiv AI

Jun 30

Mixture of Debaters: Learn to Debate at Architectural Level in Multi-Agent Reasoning

arXiv:2606. 29425v1 Announce Type: new Abstract: Existing multi-agent debate frameworks suffer from two critical limitations: they rely on static architectures where agent roles and coordination patterns are fixed at design time, and they require instantiating multiple model copies, incurring substantial computational overhead.

By Dayong Liang, Kaisong Gong, Yi Cai, Changmeng Zheng, Xiao-Yong Wei

agents multimodal benchmarks

arXiv AI

Jun 30

Diversity is the Strength of the AI Crowd

arXiv:2606. 29661v1 Announce Type: new Abstract: Top AI forecasting systems are approaching superforecaster-level accuracy on future world events, but still rely primarily on off-the-shelf LLMs combined with forecasting-specific context gathering and scaffolding.

By Matthew Aitchison, Scott Jeen, Toby Shevlane, Ben Day

llms benchmarks

arXiv AI

Jun 30

When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

arXiv:2606. 28661v1 Announce Type: cross Abstract: People overthink; language models over-sample, and the extra effort can talk both into a worse answer.

By Yong Yi Bay, Kathleen A. Yearick

llms benchmarks

arXiv AI

Jun 30

One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models

arXiv:2606. 29600v1 Announce Type: cross Abstract: A faithful 3D world representation should account for layered geometry, where a single camera ray may contain multiple visible and geometrically valid surfaces.

By Xiaohao Xu, Feng Xue, Xiang Li, Haowei Li, Shusheng Yang, Tianyi Zhang, Matthew Johnson-Roberson, Xiaonan Huang

llms computer-vision benchmarks

arXiv AI

Jun 30

An Agentic AI Pipeline for Appliance-Level Energy Anomaly Detection and LLM-Driven Recommendations

arXiv:2606. 28467v1 Announce Type: cross Abstract: Appliance-level energy monitoring in office buildings produces noisy alerts that non-expert facility managers struggle to use.

By Dihia Falouz, Aida Douaibia, Amine Bechar, Youssef Elmir, Abbes Amira, Adel Oulefki

llms rag agents benchmarks

arXiv AI

Jun 30

Reported Confidence in LLMs Tracks Commitment More Than Correctness

arXiv:2606. 29490v1 Announce Type: cross Abstract: Confidence is an estimate of the probability that a chosen answer is correct.

By Dharshan Kumaran

llms benchmarks

arXiv AI

Jun 30

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

arXiv:2606. 29464v1 Announce Type: cross Abstract: Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-language models under strict data and compute budgets.

By Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon

llms rag efficiency multimodal benchmarks safety

arXiv AI

Jun 30

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation

arXiv:2606. 28410v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) enables text-guided segmentation of unseen objects, breaking fixed-class limitations to achieve open-world understanding.

By Shanwen Wang, Xin Sun, Sirui Wang, Xiao Xiang Zhu

llms computer-vision benchmarks safety

arXiv AI

Jun 30

Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study

arXiv:2601. 16172v3 Announce Type: replace Abstract: RL-trained Lean theorem provers mode-collapse at inference time: on miniF2F-test with DeepSeek-Prover-V1.

By Zachary Burton

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.

By Yuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng, Yuan Tian, Dazhen Deng, Yingcai Wu

llms multimodal benchmarks

arXiv AI

Jun 30

Adaptive Financial Transformer with Regime-Gated Attention for Stock Return Prediction

arXiv:2606. 29347v1 Announce Type: cross Abstract: Adaptive Financial Transformer (AFT) is proposed for stock return prediction under non-stationary financial markets.

By Dishan Sarkar

llms benchmarks safety

arXiv AI

Jun 30

FlatLands: Generative Floormap Completion From a Single Egocentric View

arXiv:2603. 16016v2 Announce Type: replace-cross Abstract: A single egocentric image typically captures only a small portion of the floor, yet a complete metric traversability map of the surroundings would better serve applications such as indoor navigation.

By Subhransu S. Bhattacharjee, Dylan Campbell, Rahul Shome

robotics benchmarks

arXiv AI

Jun 30

Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

arXiv:2606. 29159v1 Announce Type: new Abstract: Offline root-cause-analysis (RCA) benchmarks commonly rank methods by a single pooled top-1 accuracy across multiple subsystems, and engineers often read the pooled winner as a recommendation for their own subsystem.

By Lining Hu, Ting Liu, Yuzhuo Fu

benchmarks

arXiv AI

Jun 30

ARMOR: Adaptive Retriever Optimization for Low-Resource Telecom Question Answering

arXiv:2606. 29706v1 Announce Type: cross Abstract: Telecom question answering (QA) is a challenging setting for retrieval-augmented generation (RAG): evidence is fragmented across standards, papers, encyclopedic resources, and web documents, and answers often hinge on technical tables, equations, and specialized protocol language.

By Heshan Fernando, Quan Xiao, Yan Xin, Tianyi Chen

rag nlp fine-tuning benchmarks

arXiv AI

Jun 30

The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling

arXiv:2606. 29278v1 Announce Type: new Abstract: We introduce the Complexity Ceiling Benchmark (CCB), a controlled evaluation of how language-model reasoning decays as the number of required sequential steps grows.

By Shubh Chapra, Dhruv Kumar, Murari Mandal, Yash Sinha

llms robotics benchmarks

arXiv AI

Jun 30

Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation

arXiv:2606. 29908v1 Announce Type: cross Abstract: Existing world model-based planners for visual navigation typically follow a verification-centric paradigm, decoupling goal intent from trajectory synthesis.

By Hong Chen, Daqi Liu, Zehan Zhang, Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Yihua Tan

robotics benchmarks safety

arXiv AI

Jun 30

Invariant Reasoning Directions in Latent Trajectories of Language Models

arXiv:2606. 29164v1 Announce Type: cross Abstract: Latent reasoning models perform multi-step inference directly in hidden-state space, yet the structure of these latent reasoning trajectories remains poorly understood.

By Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu

llms diffusion benchmarks safety

arXiv AI

Jun 30

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

arXiv:2411. 10109v3 Announce Type: replace Abstract: Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes.

By Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, Michael S. Bernstein

llms agents benchmarks

arXiv AI

Jun 30

CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

arXiv:2511. 02734v3 Announce Type: replace Abstract: Current evaluations of Large Language Model (LLM) agents primarily emphasize task completion, often overlooking resource efficiency and adaptability.

By Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, Yi R. Fung

llms agents benchmarks

Get the daily briefing