Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,477 stories · RSS feed

arXiv AI

Jun 4

Metric-Aware Hybrid Forecasting for the CTF4Science Lorenz Challenge

arXiv:2606. 04191v1 Announce Type: cross Abstract: We describe our approach to the CTF4Science Lorenz challenge, a benchmark that mixes short-horizon forecasting, long-time distribution matching, and trajectory reconstruction across nine task pairs.

By Cen Lu

benchmarks

arXiv AI

Jun 4

Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models

arXiv:2602. 19101v2 Announce Type: replace-cross Abstract: Value alignment of Large Language Models (LLMs) requires us to empirically measure these models' actual, acquired representation of value.

By Seong Hah Cho, Junyi Li, Anna Leshinskaya

llms rag benchmarks safety

arXiv AI

Jun 4

EvoPrompt: Guided Prompt Evolution for Vision-Language Models Adaptation

arXiv:2603. 09493v2 Announce Type: replace-cross Abstract: The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge.

By Enming Zhang, Jiayang Li, Yanlong Wang, Yanru Wu, Zhenyu Liu, Yang Li

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jun 4

SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

arXiv:2606. 04202v1 Announce Type: new Abstract: As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation.

By Joel Sol, Homayoun Najjaran

llms agents benchmarks

arXiv AI

Jun 4

Vectorized Online POMDP Planning

arXiv:2510. 27191v5 Announce Type: replace-cross Abstract: Planning under partial observability is an essential capability of autonomous robots.

By Marcus Hoerger, Muhammad Sudrajat, Hanna Kurniawati

agents reinforcement-learning robotics benchmarks

arXiv AI

Jun 4

StepPRM-RTL: Stepwise Process-Reward Guided LLM Fine-Tuning for Enhanced RTL Synthesis

arXiv:2606. 04246v1 Announce Type: new Abstract: Automatic generation of RTL code for digital hardware designs remains challenging due to long-horizon reasoning, multi-step dependencies, and strict correctness constraints in Verilog and VHDL.

By Prashanth Vijayaraghavan, Apoorva Nitsure, Luyao Shi, Ehsan Degan, Vandana Mukherjee

llms rag reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 4

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

arXiv:2606. 04613v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited.

By Alessandro Gambetti, Qiwei Han, Cl\'audia Soares, Hong Shen

llms multimodal benchmarks safety

arXiv AI

Jun 4

MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&A

arXiv:2606. 04231v1 Announce Type: cross Abstract: Recent advances in multimodal retrieval-augmented generation (MM-RAG) have shifted toward minimal parsing, relying on page-level images for producing retriever embeddings and for answer generation.

By Hanoz Bhathena, Parin Rajesh Jhaveri, Rohan Mittal, Prateek Singh, Aymen Kallala, Rachneet Kaur, Yiqiao Jin, Zhen Zeng, Adwait Ratnaparkhi, Denis Kochedykov

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Jun 4

You Only Train Once: Differentiable Subset Selection for Omics Data

arXiv:2512. 17678v2 Announce Type: replace-cross Abstract: Selecting compact and informative gene subsets from single-cell transcriptomic data is essential for biomarker discovery, improving interpretability, and cost-effective profiling.

By Daphn\'e Chopard, Jorge da Silva Gon\c{c}alves, Irene Cannistraci, Thomas M. Sutter, Julia E. Vogt

efficiency benchmarks safety

arXiv AI

Jun 4

SAM 3D: 3Dfy Anything in Images

arXiv:2511. 16624v2 Announce Type: replace-cross Abstract: We present SAM 3D, a generative model for visually grounded 3D object reconstruction, predicting geometry, texture, and layout from a single image.

By SAM 3D Team, Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, Aohan Lin, Jiawei Liu, Ziqi Ma, Anushka Sagar, Bowen Song, Xiaodong Wang, Jianing Yang, Bowen Zhang, Piotr Doll\'ar, Georgia Gkioxari, Matt Feiszli, Jitendra Malik

benchmarks safety

arXiv AI

Jun 4

R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

arXiv:2602. 00104v3 Announce Type: replace-cross Abstract: Vision-centric retrieval for VQA requires retrieving images to supply missing visual cues and integrating them into the reasoning process.

By Zhuohong Chen, Zhengxian Wu, Zirui Liao, Shenao Jiang, Hangrui Xu, Yang Chen, Chaokui Su, Xiaoyu Liu, Haoqian Wang

rag benchmarks

arXiv AI

Jun 4

SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systems

arXiv:2605. 10246v2 Announce Type: replace Abstract: AI scientist systems are increasingly deployed for autonomous research, yet their academic integrity has never been systematically evaluated.

By Zonglin Yang, Xingtong Liu, Xinyan Xu

llms agents benchmarks safety

arXiv AI

Jun 4

SePO: Self-Evolving Prompt Agent for System Prompt Optimization

arXiv:2606. 04465v1 Announce Type: cross Abstract: System prompt optimization improves agent behavior without modifying the underlying model, yielding human-readable, model-agnostic instructions.

By Wangcheng Tao, Han Wu, Weng-Fai Wong

agents fine-tuning benchmarks

arXiv AI

Jun 4

What If Prompt Injection Never Left? Exploring Cross-Session Stored Prompt Injection in Agentic Systems

arXiv:2606. 04425v1 Announce Type: cross Abstract: Modern agentic systems transform LLMs from session-bounded assistants into stateful systems that persist and evolve shared world state across sessions through memories, filesystems, tools, and other long-lived contextual artifacts.

By Yuanbo Xie, Tianyun Liu, Yingjie Zhang, Suchen Liu, Yulin Li, Liya Su, Tingwen Liu

llms agents benchmarks safety

arXiv Machine Learning

Jun 4

Representation Matters in Randomized Smoothing for Audio Classification

arXiv:2606. 04210v1 Announce Type: cross Abstract: Randomized smoothing (RS) certifies robustness in the vector space where Gaussian noise is added.

By Jong-Ik Park, Shreyas Chaudhari, Jos\'e M. F. Moura, Carlee Joe-Wong

benchmarks

arXiv Machine Learning

Jun 4

\textsc{Lethe}: Principled Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning

arXiv:2601. 22601v3 Announce Type: replace Abstract: Federated unlearning (FU) aims to erase knowledge from a global model.

By Wentai Wu, Hanwei Tan, Yijun Quan, Haixia Peng, Ligang He, Bin Yang, C. L. Philip Chen

nlp benchmarks

arXiv AI

Jun 4

Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

arXiv:2603. 10044v2 Announce Type: replace-cross Abstract: A safety score earned on a benchmark need not predict how the same model behaves once it is wrapped in an agentic scaffold the benchmark never tested.

By David Gringras

llms agents benchmarks safety

arXiv AI

Jun 4

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

arXiv:2606. 04381v1 Announce Type: cross Abstract: Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely \emph{symbolic}, arising from pattern matching over spatial language rather than true \emph{geometric} reasoning over space.

By Chen Chu, Bita Azarijoo, Li Xiong, Khurram Shafique, Cyrus Shahabi

llms multimodal benchmarks

arXiv Machine Learning

Jun 4

PointAction: 3D Points as Universal Action Representations for Robot Control

arXiv:2606. 03943v1 Announce Type: cross Abstract: Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation.

By Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu

diffusion robotics benchmarks

arXiv Machine Learning

Jun 4

Graph Set Transformer

arXiv:2606. 05116v1 Announce Type: new Abstract: We introduce the Graph Set Transformer (GST), a neural network architecture for learning on sets of graphs, designed for tasks in which per-element predictions depend on set-wide context as well as local structure.

By Jose E. Escrig Molina, Baoquan Chen, Daniel Probst

llms rag computer-vision benchmarks

Get the daily briefing