Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv AI
Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney
arXiv AI
Jun 2

Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing

arXiv:2606. 02218v1 Announce Type: cross Abstract: Synchronous reinforcement learning methods such as Group Relative Policy Optimization (GRPO) provide stable and reproducible on-policy training, but they are highly vulnerable to stragglers, a single unusually long rollout can delay reward computation and parameter updates for the entire group.

By Azal Ahmad Khan, Ammar Ahmed, Zeshan Fayyaz, Sheng Di, Mingyi Hong, Ali Anwar
arXiv AI
Jun 2

SPARC: Spatial-Aware Path Planning via Attentive Agent Communication

arXiv:2603. 02845v5 Announce Type: replace-cross Abstract: Efficient communication is critical for decentralized Multi-Robot Path Planning (MRPP), yet existing learned communication methods treat all neighboring robots equally regardless of their spatial proximity, leading to diluted attention in congested regions where coordination matters most.

By Sayang Mu, Xiangyu Wu, Bo An
arXiv AI
Jun 2

Improving Diffusion Planners by Self-Supervised Action Gating with Energies

arXiv:2603. 02650v2 Announce Type: replace-cross Abstract: Diffusion planners are a strong approach for offline reinforcement learning, but they can fail when value-guided selection favours trajectories that score well yet are locally inconsistent with the environment dynamics, resulting in brittle execution.

By Yuan Lu, Dongqi Han, Yansen Wang, Dongsheng Li
arXiv AI
Jun 2

The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue

arXiv:2606. 01901v1 Announce Type: cross Abstract: We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image generator across multiple turns, making accumulated common ground directly observable as a rendered image.

By Sherzod Hakimov, Mattia D'Agostini, Ivan Samodelkin, David Schlangen
arXiv Machine Learning
Jun 2

PFT: Phonon Fine-tuning for Machine Learned Interatomic Potentials

arXiv:2601. 07742v4 Announce Type: replace-cross Abstract: Many materials properties depend on higher-order derivatives of the potential energy surface, yet machine learned interatomic potentials (MLIPs) trained with a standard loss on energy, force, and stress errors can exhibit error in curvature, degrading the prediction of vibrational properties.

By Teddy Koker, Abhijeet Gangan, Mit Kotak, Jaime Marian, Tess Smidt