Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

27,828 stories · RSS feed

arXiv AI
1d ago

Grounding in the Dark: Text-Only Training for Zero-Shot Video Temporal Localization

The paper explores training a large language model solely on textual video descriptions to perform zero‑shot video temporal localization, eliminating the need for paired video‑text data during training. Using a 134K‑example corpus and adapting Qwen3‑8B with LoRA on CLIP or LLM2CLIP embeddings, the authors evaluate on Charades‑STA and QV‑Highlights, achieving modest performance when directly replacing text with frame embeddings. They introduce Nearest Entity Replacement (NER), a training‑free mapping that represents each video frame by the mean of its nearest text‑entity embeddings, which significantly improves results and narrows the modality gap.

By Divyanshu Mishra, Sabine Sternig, Rakshith Shetty, Erhan Gundogdu
arXiv AI
1d ago

CoSafe: Cooperative Runtime Safety Assurance for Mixed-Traffic Autonomous Driving

CoSafe is a modular, ADS-agnostic framework that extends runtime safety assurance from ego-centric takeover to scene-level cooperative intervention in mixed-traffic autonomous driving. It uses vehicle-to-everything communication to allow roadside units to monitor hazards and jointly mitigate risks involving multiple vehicles. Evaluation on the InterDrive benchmark shows that CoSafe improves driving scores by 70.03%, reduces safety violations by 87.80%, and accurately identifies unsafe autonomous vehicles with an F3 score of 0.930 while meeting real-time constraints.

By Dingji Wang, You Lu, Haowen Jiang, Yanggang Chen, Yuyang Bi, Bihuan Chen, Shenglin Bao, Xin Peng
arXiv AI
1d ago

Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains

Rubric Curriculum RL (RcRL) is a reinforcement learning approach that leverages the generation‑verification gap in non‑verifiable domains by using pairwise preferences over rollouts and a curriculum over rubric criteria. Unlike traditional methods that rely on absolute scoring and stationary judges, RcRL provides a more discriminative, non‑stationary signal, allowing policies to maintain output entropy and continue improving beyond 1,000 training steps. In experiments on creative writing and HealthBench, RcRL outperforms baseline models, achieving a 70.5% human win rate and significant accuracy gains over the base model.

By Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
arXiv AI
1d ago

From Admission to Retention: Predicting Durable Offspring in Quality-Diversity Search

The paper investigates how to predict which offspring in a Quality‑Diversity (QD) archive will be retained over time, rather than just entering the archive. By analyzing signals available before reproduction, the authors find that lineage depth is the most reliable indicator of long‑term persistence, though its predictive power varies with population size. Despite this insight, biasing parent selection toward these signals does not improve overall search performance and can even lower the QD‑score.

By Corrado Calamita, Matteo Mendicini, Luca Ferrari, Riccardo Bertamini
arXiv AI
1d ago

Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images

Trick-or-Treat is an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs to test cross‑modal reasoning. Each condition has matched pairs where the radiograph and photograph suggest the same disease, and mismatched pairs where the photograph shows an alternative finding. In tests of three frontier models, hallucinations of clinically significant findings rose from 25‑37.5% for single photographs to 45‑57.5% when mismatched pairs were presented, with models often ignoring mismatches and generating findings that better aligned with the radiograph.

By Oishi Banerjee, Eujin Lee, Sung Eun Kim, Alexandra N. Willauer, Christine Y. Zhou, Zhizhou Yang, Pranav Rajpurkar
arXiv AI
1d ago

CounterFactual Retrieval for Evidence-Aware Individual Animal Identification

The paper introduces Counterfactual Foreground Retrieval (CFR), an evidence‑aware evaluation protocol for automated individual animal identification. CFR assesses retrieval performance under three conditions: full images, foreground‑only, and shortcut‑only (background or silhouette). By comparing these views, CFR produces an evidence profile that reveals whether models rely on animal features or on identity‑correlated shortcuts, and the authors provide masks and code for 34 datasets to support this analysis.

By Ankit K. Upadhyay, Aleksandr Algasov, Ekaterina Nepovinnykh, Fedor Zolotarev, Tuomas Eerola, Heikki K\"alvi\"ainen, Charles V. Stewart, Lukas Picek
arXiv AI
1d ago

Learning to Learn from Multimodal Experience

The paper introduces AutoMMemo, a framework that lets agents automatically design memory mechanisms for learning from multimodal experience. Each memory mechanism is represented as an executable memo program that specifies how past episodes are stored, organized, and retrieved. Through iterative evaluation, reflection-guided mutation, and budget-aware tree search, AutoMMemo learns effective memory designs that improve agent performance on GUI/Web navigation and multimodal visual reasoning tasks while keeping token and interaction costs reasonable.

By Xingyu Sui, Weixiang Zhao, Yongxin Tang, Yanyan Zhao, Yang Wu, Dandan Tu, Bing Qin
arXiv Machine Learning
1d ago

SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning

arXiv:2602.03201v4 Announce Type: replace Abstract: Model-based reinforcement learning (MBRL) is sample-efficient but struggles in sparse reward settings. A critical bottleneck arises from the lack o...

By Yao-Hui Li, Zeyu Wang, Xin Li, Wei Pang, Yingfang Yuan, Zhengkun Chen, Boya Zhang, Riashat Islam, Alex Lamb, Yonggang Zhang