Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv AI

Jun 30

Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering

arXiv:2606. 30294v1 Announce Type: new Abstract: Live product demonstrations are a recurring, high-cost activity in software organizations: a human presenter must select features, dispatch the corresponding interactions on a running application, narrate them coherently, and answer questions in real time.

By Rahul Khedar, Mayank Malhotra, Avinash Karn, Mouli V, Prakhar Mehrotra

agents nlp multimodal benchmarks

arXiv AI

Jun 30

HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning

arXiv:2606. 28349v1 Announce Type: cross Abstract: Long-context reasoning requires models to access, retrieve, and integrate evidence scattered across documents, dialogues, and accumulated interaction histories.

By Zeju Li, Ziyang Zheng, Yizhou Zhou, Qiang Xu

rag agents benchmarks

arXiv AI

Jun 30

Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study

arXiv:2601. 16172v3 Announce Type: replace Abstract: RL-trained Lean theorem provers mode-collapse at inference time: on miniF2F-test with DeepSeek-Prover-V1.

By Zachary Burton

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.

By Yuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng, Yuan Tian, Dazhen Deng, Yingcai Wu

llms multimodal benchmarks

arXiv AI

Jun 30

FlatLands: Generative Floormap Completion From a Single Egocentric View

arXiv:2603. 16016v2 Announce Type: replace-cross Abstract: A single egocentric image typically captures only a small portion of the floor, yet a complete metric traversability map of the surroundings would better serve applications such as indoor navigation.

By Subhransu S. Bhattacharjee, Dylan Campbell, Rahul Shome

robotics benchmarks

arXiv AI

Jun 30

Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

arXiv:2606. 29159v1 Announce Type: new Abstract: Offline root-cause-analysis (RCA) benchmarks commonly rank methods by a single pooled top-1 accuracy across multiple subsystems, and engineers often read the pooled winner as a recommendation for their own subsystem.

By Lining Hu, Ting Liu, Yuzhuo Fu

benchmarks

arXiv AI

Jun 30

The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling

arXiv:2606. 29278v1 Announce Type: new Abstract: We introduce the Complexity Ceiling Benchmark (CCB), a controlled evaluation of how language-model reasoning decays as the number of required sequential steps grows.

By Shubh Chapra, Dhruv Kumar, Murari Mandal, Yash Sinha

llms robotics benchmarks

arXiv AI

Jun 30

The Human Creativity Benchmark

arXiv:2606. 30561v1 Announce Type: new Abstract: Modern AI evaluation frameworks treat evaluator disagreement as noise to be resolved.

By Aspen Hopkins, Allison Nulty, Alexandria Minetti, Anoop Pakki, Angad Singh

benchmarks

arXiv AI

Jun 30

FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars

arXiv:2606. 30145v1 Announce Type: new Abstract: Natural face-to-face conversation requires real-time speech generation together with synchronized facial motion.

By Habin Lim, Jae-Ho Lee, Hah Min Lew, Ji-Su Kang, Gyeong-Moon Park

diffusion benchmarks

arXiv AI

Jun 30

EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots

arXiv:2606. 30256v1 Announce Type: new Abstract: Safety benchmarks often buy scalability by fixing the prompt, the language, and the turn structure.

By Camilo Chac\'on Sartori

benchmarks safety

arXiv AI

Jun 30

CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigation

arXiv:2606. 28397v1 Announce Type: cross Abstract: Vision-language navigation (VLN) has recently advanced with large language and multimodal models, enabling agents to follow natural-language instructions in unseen environments without training a task-specific navigation policy.

By Shaoxuan Li, Xiangyu Dong, Xiaoguang Ma, Junfeng Chen, Haoran Zhao, Yaoming Zhou

rag agents multimodal benchmarks

arXiv AI

Jun 30

Invariant Reasoning Directions in Latent Trajectories of Language Models

arXiv:2606. 29164v1 Announce Type: cross Abstract: Latent reasoning models perform multi-step inference directly in hidden-state space, yet the structure of these latent reasoning trajectories remains poorly understood.

By Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu

llms diffusion benchmarks safety

arXiv Machine Learning

Jun 30

fev-bench: A Realistic Benchmark for Time Series Forecasting

arXiv:2509. 26468v3 Announce Type: replace Abstract: Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained models.

By Oleksandr Shchur, Abdul Fatir Ansari, Caner Turkmen, Lorenzo Stella, Nick Erickson, Pablo Guerron, Michael Bohlke-Schneider, Yuyang Wang

benchmarks

arXiv AI

Jun 30

SUMO: Segment and Track Any Motion with Nonlinear State Space Models

arXiv:2606. 29861v1 Announce Type: cross Abstract: Visual Object Tracking (VOT) and Moving Object Segmentation (MOS) are two fundamental tasks in computer vision that involve both spatial and temporal object dynamics.

By Kexin Tian, Sixu Li, Keshu Wu, Yang Zhou, Zhengzhong Tu

computer-vision robotics benchmarks

arXiv Machine Learning

Jun 30

Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

arXiv:2606. 29975v1 Announce Type: new Abstract: Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts.

By Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt

efficiency benchmarks

arXiv AI

Jun 30

Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

arXiv:2601. 05366v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls.

By Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani, Wanpeng Xu, Hua Wei, Xiyang Hu

llms agents benchmarks

arXiv Machine Learning

Jun 30

MALOQ: Massively Accelerated Learning of Operators for Quantum Transport

arXiv:2606. 28911v1 Announce Type: new Abstract: Machine-learned (ML) operator models can be trained to predict density functional theory (DFT) Hamiltonian/density matrices at significantly reduced computational cost, thus extending electronic-structure calculations to previously unfeasible scales.

By Manasa Kaniselvan, Alexander Maeder, Denghui Lu, Alexandros Nikolaos Ziogas, Mathieu Luisier

benchmarks

arXiv AI

Jun 30

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

arXiv:2605. 13087v2 Announce Type: replace-cross Abstract: Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance.

By Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon, Arghya Bhattacharya, Kumarmanas Nethil

fine-tuning multimodal benchmarks

arXiv AI

Jun 30

Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees

arXiv:2606. 29681v1 Announce Type: new Abstract: Probabilistic model checking for Markov decision processes (MDPs) provides quantitative guarantees, but often offers limited insight into why undesired outcomes occur.

By Ryohei Oura, Georgios Fainekos, Hideki Okamoto, Bardh Hoxha

reinforcement-learning benchmarks

arXiv AI

Jun 30

Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

arXiv:2606. 29700v1 Announce Type: new Abstract: Planning often requires symbolic specifications that are both executable and verifiable.

By Jiamei Jiang, Jiajing Zhang, Feifei Mo, Linjing Li, Daniel Zeng

llms agents fine-tuning benchmarks safety

Get the daily briefing