Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv AI

Jun 30

Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL

arXiv:2604. 28123v3 Announce Type: replace-cross Abstract: The standard post-training recipe for large multimodal models (LMMs) applies supervised fine-tuning (SFT) on curated demonstrations followed by reinforcement learning with verifiable rewards (RLVR).

By Sudong Wang, Weiquan Huang, Xiaomin Yu, Zuhao Yang, Hehai Lin, Keming Wu, Chaojun Xiao, Chen Chen, Wenxuan Wang, Beier Zhu, Yunjian Zhang, Chengwei Qin

llms reinforcement-learning fine-tuning efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Experience Augmented Policy Optimization for LLM Reasoning

arXiv:2606. 30420v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for improving the reasoning capabilities of large language models (LLMs).

By Jinda Lu, Kexin Huang, Junkang Wu, Shuo Yang, Jinghan Li, Chiyu Ma, Shaohang Wei, Xiang Wang, Guoyin Wang, Jingren Zhou

llms reinforcement-learning benchmarks

arXiv AI

Jun 30

MirrorCode: AI can rebuild entire programs from behavior alone

arXiv:2606. 30182v1 Announce Type: new Abstract: AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler.

By Tom Adamczewski, David Owen, David Rein, Florian Brand, Giles Edkins, Allen Hart, Daniel O'Connell

agents benchmarks

arXiv AI

Jun 30

Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering

arXiv:2606. 30294v1 Announce Type: new Abstract: Live product demonstrations are a recurring, high-cost activity in software organizations: a human presenter must select features, dispatch the corresponding interactions on a running application, narrate them coherently, and answer questions in real time.

By Rahul Khedar, Mayank Malhotra, Avinash Karn, Mouli V, Prakhar Mehrotra

agents nlp multimodal benchmarks

arXiv AI

Jun 30

An AI agent for treatment reasoning over a biomedical tool universe

arXiv:2606. 28692v1 Announce Type: new Abstract: Treatment reasoning underpins every therapeutic decision, integrating disease context, comorbidities, medications, contraindications, and evolving biomedical knowledge to select an appropriate therapy.

By Shanghua Gao, Ayush Noori, Richard Zhu, Curtis Ginder, Zhenglun Kong, Xiaorui Su, Justin Kauffman, Benjamin S. Glicksberg, Joshua Lampert, Ankit Sakhuja, Ashwin Sawant, ATHENA-R1 Evaluation Consortium, David A. Clifton, Noa Dagan, Ran Balicer, Marinka Zitnik

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF

arXiv:2606. 29758v1 Announce Type: cross Abstract: Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training.

By Doo Hwan Hwang, Kee-Eung Kim

llms reinforcement-learning benchmarks

arXiv AI

Jun 30

LAMP: Lean-based Agentic framework with MCP and Proof Repair

arXiv:2606. 28841v1 Announce Type: cross Abstract: Large language models are increasingly capable of mathematical reasoning, but the proofs they generate are often unreliable and hard to verify.

By Santhana Srinivasan R, Maithilee Patawar

llms agents fine-tuning benchmarks

arXiv AI

Jun 30

HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning

arXiv:2606. 28349v1 Announce Type: cross Abstract: Long-context reasoning requires models to access, retrieve, and integrate evidence scattered across documents, dialogues, and accumulated interaction histories.

By Zeju Li, Ziyang Zheng, Yizhou Zhou, Qiang Xu

rag agents benchmarks

arXiv AI

Jun 30

MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes

arXiv:2606. 28900v1 Announce Type: new Abstract: Doctor agents are moving beyond single-turn answer generation toward evolving clinical decision systems.

By Hui Zhang

agents benchmarks

arXiv AI

Jun 30

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

arXiv:2606. 29957v1 Announce Type: cross Abstract: Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code.

By Yifan Wu, Zhuokai Zhao, Songlin Li, Ho Hin Lee, Jiacheng Zhu, Shirley Wu, Tianhe Yu, Serena Li, Lizhu Zhang, Xiangjun Fan, Shengzhi Li

llms agents benchmarks

arXiv AI

Jun 30

A Task-Driven and Quality-Assured Agent Framework for SAR Data Generation

arXiv:2606. 28896v1 Announce Type: cross Abstract: Synthetic aperture radar (SAR) data augmentation is important for improving the generalization of data-driven SAR interpretation models, yet practical augmentation workflows are often hindered by heterogeneous dataset formats, task-dependent metadata requirements, diverse generation methods, and weak validation of generated samples.

By Xuanting Wu, Fan Zhanga, Fei Ma, Ling Guan, Guochun Ma, Yongsheng Zhou

llms agents benchmarks

arXiv Machine Learning

Jun 30

Spatial Deconfounder: Interference-Aware Deconfounding for Spatial Causal Inference

arXiv:2510. 08762v2 Announce Type: replace Abstract: Causal inference in spatial domains faces two intertwined challenges: (1) unmeasured spatial factors, such as weather, air pollution, or mobility, that confound treatment and outcome, and (2) interference from nearby treatments that violate standard no-interference assumptions.

By Ayush Khot, Miruna Oprescu, Maresa Schr\"oder, Ai Kagawa, Xihaier Luo

benchmarks

arXiv AI

Jun 30

Domain-Informed Multi-View Self-Distillation for Astronomical Light-Curve Representation Learning with JEPA

arXiv:2606. 28446v1 Announce Type: cross Abstract: Light curves describe temporal variations in the brightness of celestial objects.

By Yicheng Rui

rag nlp fine-tuning efficiency benchmarks

arXiv AI

Jun 30

HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

arXiv:2606. 28831v1 Announce Type: cross Abstract: Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.

By Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, Huan Li

llms efficiency benchmarks

arXiv Machine Learning

Jun 30

Diffusion Fine-tuning with Rewarded Moment Matching Distillation

arXiv:2606. 30414v1 Announce Type: new Abstract: Distillation and Reinforcement Learning (RL) fine-tuning are the primary pillars of diffusion post-training.

By Alexis Jacq, Guillaume Couairon, Valentin De Bortoli, Quentin Berthet, Arnaud Doucet, Romuald Elie

diffusion reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 30

How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD

arXiv:2606. 29733v1 Announce Type: cross Abstract: Organizations that cannot send data to a cloud API increasingly ask: how good is Text-to-SQL if the model must run on-premises on open weights, and which popular accuracy "recipes" are worth their compute?

By Vladimir Beskorovainyi

llms rag benchmarks data-engineering

arXiv AI

Jun 30

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.

By Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang

benchmarks safety

arXiv AI

Jun 30

Selective Memory Retention for Long-Horizon LLM Agents

arXiv:2606. 29178v1 Announce Type: new Abstract: When does retention matter for memory-augmented LLM agents?

By Pranath Reddy

llms rag agents benchmarks

arXiv AI

Jun 30

Defeat Devices in AI Systems

arXiv:2606. 28863v1 Announce Type: cross Abstract: AI systems increasingly exhibit behavior that differs systematically between evaluation and deployment contexts.

By Emilio Ferrara

benchmarks safety

arXiv Machine Learning

Jun 30

Local-Minima-Preserving Continuous Relaxation of Ising Problems

arXiv:2606. 30333v1 Announce Type: cross Abstract: The generalized Ising problem captures a broad spectrum of hard combinatorial problems, including MAX-CUT, Number Partitioning (NPP), and Maximum Independent Set.

By Debraj Banerjee, Santanu Mahapatra, Kunal N. Chaudhury

benchmarks

Get the daily briefing