Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv AI

Jun 30

CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigation

arXiv:2606. 28397v1 Announce Type: cross Abstract: Vision-language navigation (VLN) has recently advanced with large language and multimodal models, enabling agents to follow natural-language instructions in unseen environments without training a task-specific navigation policy.

By Shaoxuan Li, Xiangyu Dong, Xiaoguang Ma, Junfeng Chen, Haoran Zhao, Yaoming Zhou

rag agents multimodal benchmarks

arXiv AI

Jun 30

Invariant Reasoning Directions in Latent Trajectories of Language Models

arXiv:2606. 29164v1 Announce Type: cross Abstract: Latent reasoning models perform multi-step inference directly in hidden-state space, yet the structure of these latent reasoning trajectories remains poorly understood.

By Arun Vignesh Malarkkan, Manan Roy Choudhury, Utkarsh Byahut, Yash Ravindra Charde, Vivek Gupta, Yanjie Fu

llms diffusion benchmarks safety

arXiv AI

Jun 30

RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

arXiv:2606. 29867v1 Announce Type: cross Abstract: Deep Reinforcement Learning (DRL) has achieved significant success in robotics and autonomous systems, yet remains vulnerable to adversarial perturbations that can severely degrade performance.

By Adithya Mohan, Daniel Kriegl, Torsten Sch\"on

agents reinforcement-learning robotics benchmarks safety

arXiv AI

Jun 30

Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers

arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.

By Rabin Adhikari

llms rag benchmarks safety

arXiv AI

Jun 30

How to Train Your Long-Context Visual Document Model

arXiv:2602. 15257v3 Announce Type: replace-cross Abstract: We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text.

By Austin Veselka

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Jun 30

KrishokChat: A Citation-Grounded Dataset and Benchmark for Bengali Agricultural Advisory

arXiv:2606. 29243v1 Announce Type: new Abstract: We present KrishokChat, the first citation-grounded Bengali agricultural instruction-tuning dataset for crop advisory in low-resource settings.

By Khan Raiyan Ibne Reza, Omar Ibne Shahid

rag fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

Atompack: A Storage and Distribution Layer for Read-Heavy Atomistic ML Training Datasets

arXiv:2606. 29975v1 Announce Type: new Abstract: Atomistic machine learning datasets are increasingly used for training: large immutable snapshots are read repeatedly, shuffled across epochs, staged across clusters' storage systems, and republished as reusable scientific artifacts.

By Ali Ramlaoui, Daniel T. Speckhard, Sagar Pal, Fragkiskos D. Malliaros, Alexandre Duval, Victor Schmidt

efficiency benchmarks

arXiv AI

Jun 30

Early Cue Precision Shapes Visual Shortcut Learning in Controlled Cue-Manipulation Benchmarks

arXiv:2606. 30344v1 Announce Type: cross Abstract: Visual classifiers can achieve high matched-distribution accuracy while relying on low-level cues that fail under conflict or suppression.

By Chanho Park, Woochan Lee, Janyeong Oh, Geongho Gong, Minshu Kim, Yeachan Kwak, Seongim Choi

robotics fine-tuning benchmarks

arXiv AI

Jun 30

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation

arXiv:2606. 28410v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) enables text-guided segmentation of unseen objects, breaking fixed-class limitations to achieve open-world understanding.

By Shanwen Wang, Xin Sun, Sirui Wang, Xiao Xiang Zhu

llms computer-vision benchmarks safety

arXiv AI

Jun 30

An Agentic AI Pipeline for Appliance-Level Energy Anomaly Detection and LLM-Driven Recommendations

arXiv:2606. 28467v1 Announce Type: cross Abstract: Appliance-level energy monitoring in office buildings produces noisy alerts that non-expert facility managers struggle to use.

By Dihia Falouz, Aida Douaibia, Amine Bechar, Youssef Elmir, Abbes Amira, Adel Oulefki

llms rag agents benchmarks

arXiv AI

Jun 30

SVC-Probe: A Framework for Evaluating Perturbation Generalization in Spatial Foundation-Model Embeddings

arXiv:2606. 28465v1 Announce Type: cross Abstract: This work examines perturbation generalization in spatial foundation-model embeddings derived from fluorescence microscopy images.

By Jake Y. Chen, Huu Phong Nguyen, Fuad Al Abir, Ehsan Saghapour

rag benchmarks

arXiv AI

Jun 30

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

arXiv:2606. 29464v1 Announce Type: cross Abstract: Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-language models under strict data and compute budgets.

By Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon

llms rag efficiency multimodal benchmarks safety

arXiv AI

Jun 30

Reported Confidence in LLMs Tracks Commitment More Than Correctness

arXiv:2606. 29490v1 Announce Type: cross Abstract: Confidence is an estimate of the probability that a chosen answer is correct.

By Dharshan Kumaran

llms benchmarks

arXiv AI

Jun 30

One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models

arXiv:2606. 29600v1 Announce Type: cross Abstract: A faithful 3D world representation should account for layered geometry, where a single camera ray may contain multiple visible and geometrically valid surfaces.

By Xiaohao Xu, Feng Xue, Xiang Li, Haowei Li, Shusheng Yang, Tianyi Zhang, Matthew Johnson-Roberson, Xiaonan Huang

llms computer-vision benchmarks

arXiv AI

Jun 30

ProSpec RL: Plan Ahead, then Execute

arXiv:2407. 21359v2 Announce Type: replace-cross Abstract: Imagining potential outcomes of actions before execution helps agents make more informed decisions, a prospective thinking ability fundamental to human cognition.

By Liangliang Liu, Yi Guan, BoRan Wang, Rujia Shen, Yi Lin, Chaoran Kong, Lian Yan, Jingchi Jiang

agents reinforcement-learning benchmarks

arXiv AI

Jun 30

Can Fine-Tuning Erase Your Edits? On the Fragile Coexistence of Knowledge Editing and Adaptation

arXiv:2511. 05852v4 Announce Type: replace-cross Abstract: Knowledge editing (KE) offers a lightweight alternative to retraining for updating large language models (LLMs).

By Yinjie Cheng, Paul Youssef, Christin Seifert, J\"org Schl\"otterer, Zhixue Zhao

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing

arXiv:2603. 03143v2 Announce Type: replace-cross Abstract: Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm.

By Jiyuan Wang, Chunyu Lin, Lei Sun, Zhi Cao, Yuyang Yin, Lang Nie, Zhenlong Yuan, Xiangxiang Chu, Yunchao Wei, Kang Liao, Guosheng Lin

diffusion computer-vision reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?

arXiv:2511. 06090v3 Announce Type: replace-cross Abstract: Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness.

By Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh, Kevin Swersky, Ofir Press, Enhui Li, Vijay Janapa Reddi, Parthasarathy Ranganathan

llms agents benchmarks data-engineering

arXiv AI

Jun 30

When, How Long and How Much? Interpretable Neural Networks for Time Series Regression by Learning to Mask and Aggregate

arXiv:2512. 03578v3 Announce Type: replace-cross Abstract: Time series extrinsic regression (TSER) refers to the task of predicting a continuous target variable from an input time series.

By Florent Forest, Amaury Wei, Olga Fink

benchmarks safety

arXiv AI

Jun 30

ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

arXiv:2512. 09066v2 Announce Type: replace-cross Abstract: Reliable assessment of the abilities of large audio language models (LALMs) is essential to advancing the state of the art.

By \v{S}imon Sedl\'a\v{c}ek, Sara Barahona, Bolaji Yusuf, Laura Herrera-Alarc\'on, Santosh Kesiraju, Cecilia Bola\~nos, Alicia Lozano-Diez, Sathvik Udupa, Fernando L\'opez, Allison Ferner, Ramani Duraiswami, Jan \v{C}ernock\'y

llms nlp benchmarks

Get the daily briefing