Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv AI

Jul 1

Beyond expert users: agents should help users construct preferences, not just elicit them

arXiv:2606. 30863v1 Announce Type: new Abstract: Agents typically assume an expert user -- one with well-formed preferences about what they want -- and default to clarifying questions whenever the task is underspecified.

By Irena Saracay, Ludwig Schmidt, Carlos Guestrin

agents benchmarks

arXiv Machine Learning

Jul 1

ShardNet: Training Neural Controllers with Hard, Non-Convex Constraints

arXiv:2606. 30935v1 Announce Type: cross Abstract: While neural network control policies are powerful, their deployment on safety critical systems depends on ensuring that they obey strict constraints.

By Long Kiu Chung, Shreyas Kousik

benchmarks safety

arXiv AI

Jul 1

Learning Structurally Consistent Representations for Multi-View Radar Semantic Segmentation

arXiv:2606. 31609v1 Announce Type: cross Abstract: Radar sensors provide reliable perception under adverse weather and lighting conditions, but their sparse, noisy, and weakly semantic measurements make dense semantic segmentation challenging.

By Ali Zia, Muhammad Umer Ramzan, Abdelwahed Khamis, Usman Ali, Abdul Rehman

computer-vision efficiency benchmarks safety

arXiv AI

Jul 1

From Idea to Prototype in an Afternoon: Scaffolded, AI-Assisted Rapid VA Prototyping

arXiv:2606. 31311v1 Announce Type: cross Abstract: Testing a new visual-analytics idea usually takes months: one needs to find a realistic data set, clean it, and implement an interactive prototype.

By Gennady Andrienko, Natalia Andrienko

benchmarks

arXiv AI

Jul 1

Seeing Through Multiple Views: Parameter-Efficient Fine-Tuning via Selective Neurons for Consistent Radiology Report Generation

arXiv:2606. 31099v1 Announce Type: cross Abstract: Recent years have seen substantial advances in radiology report generation (RRG), yet existing approaches predominantly adopt direct feature fusion when handling multi-view X-ray images.

By Yucheng Chen, Jinjing Zhu, Yang Yu, Yufei Shi, Hane Naghshbandi, Jinhua Liu, Angela S. Koh, Fang Fen, Kian Eng Ong, Si Yong Yeo

llms fine-tuning benchmarks

arXiv AI

Jul 1

ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

arXiv:2606. 31054v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image.

By Zhiyuan Yao, Zheren Fu, Zhixiao Zheng, Jiajun Li, Yi Tu, Zhendong Mao

llms diffusion reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 1

LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent

arXiv:2604. 17931v3 Announce Type: replace Abstract: Reinforcement Learning (RL) has emerged as a powerful training paradigm for LLM-based agents.

By Wanli Li, Bince Qu, Bo Pan, Jianyu Zhang, Zheng Liu, Pan Zhang, Wei Chen, Bo Zhang

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 1

Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection

arXiv:2606. 30675v1 Announce Type: cross Abstract: Early detection of dementia through speech analysis offers a non-invasive screening alternative, but capturing both acoustic and linguistic biomarkers remains challenging.

By Olivier Jiyoun Jung, Jonghyeon Park, Myungwoo Oh

llms rag multimodal benchmarks

arXiv AI

Jul 1

Budget-Adaptive Routing: Skipping the Weak When the Strong Answers Anyway

arXiv:2606. 30919v1 Announce Type: cross Abstract: Edge-cloud inference collaborations are often designed with a routing estimator that decides whether to offload each frame from weak models at the edge to stronger models in the cloud.

By Wei Geng, Nitinder Mohan, J\"org Ott

benchmarks

arXiv Machine Learning

Jul 1

Structure-Regularized Interpretable TCR-Epitope Prediction

arXiv:2606. 30902v1 Announce Type: cross Abstract: T cell receptor (TCR)-epitope binding prediction is essential for understanding adaptive immunity and developing immunotherapies.

By Jiarui Li, Zixiang Yin, Yunbei Zhang, Janet Wang, Samuel J. Landry, Zhengming Ding, Ramgopal R. Mettu

llms rag benchmarks safety

arXiv AI

Jul 1

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

arXiv:2606. 31329v1 Announce Type: cross Abstract: Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation.

By Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

llms robotics multimodal benchmarks

arXiv AI

Jul 1

A Reproducible Benchmark of Lightweight CNNs: Accuracy, Efficiency, and the Impact of Pretrained Initialization

arXiv:2505. 03303v3 Announce Type: replace-cross Abstract: Lightweight convolutional neural networks are often compared using results obtained with different training recipes, input settings, and pretrained checkpoints.

By Tasnim Shahriar

fine-tuning benchmarks

arXiv AI

Jul 1

Are Video Reasoning Models Ready to Go Outside?

arXiv:2603. 10652v3 Announce Type: replace-cross Abstract: In real-world deployment, vision-language models often encounter disturbances such as weather, occlusion, and camera motion.

By Yangfan He, Changgyu Boo, Jaehong Yoon

llms robotics multimodal benchmarks

arXiv AI

Jul 1

BP-TTA: Balanced and Prototype-Guided Test-Time Adaptation in Dynamic Scenarios

arXiv:2606. 31420v1 Announce Type: new Abstract: Test-Time Adaptation (TTA) enables models trained on a source domain to adapt online to unlabeled test data under distribution shifts.

By Shaoyang Huang, Yashi Zhu, Yichen Yu, Lei Zhang, Zhang Yi, Tao He

benchmarks safety

arXiv AI

Jul 1

IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO

arXiv:2606. 23032v3 Announce Type: replace Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language models on financial tasks.

By Mostapha Benhenda

llms agents benchmarks safety

arXiv AI

Jul 1

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

arXiv:2511. 00810v4 Announce Type: replace-cross Abstract: Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen.

By Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil, Wanrong Zhu, Changyou Chen, Ruiyi Zhang

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jul 1

Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

arXiv:2606. 31397v1 Announce Type: cross Abstract: State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency.

By Duc Anh Nguyen, Tien Ngoc Luu, Tung Pham, Toan Tran

llms fine-tuning benchmarks

arXiv AI

Jul 1

ECHO: Prune to act, trace to learn with selective turn memory in agentic RL

arXiv:2606. 31650v1 Announce Type: cross Abstract: Long-horizon language agents must repeatedly interact with tools, accumulate evidence, and make decisions under bounded context windows.

By Zijun Xie, Binbin Zheng, Enlei Gong, Jihua Liu, Yuyang You, Lingfeng Liu, Jiayao Tang, Guanqun Zhao, Aoqi Hu, Zeyu Chen

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 1

Predictable GRPO: A Closed-Form Model of Training Dynamics

arXiv:2606. 30789v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) has become a standard tool for improving the reasoning ability of large language models, yet its training dynamics are still described empirically: reward trajectories are fit with low-parameter functional forms whose constants carry no mechanistic meaning, and hyperparameter choices remain a matter of trial and error.

By Rajat Ghosh, Datta Nimmaturi, Aryan Singhal, Vaishnavi Bhargava, Henry Wong, Johnu George, Debojyoti Dutta

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 1

WIDER-FAIR: An Annotated Version of the WIDER-FACE Dataset for Fairness Evaluation

arXiv:2606. 31704v1 Announce Type: cross Abstract: The deployment of face detection models in real-world applications raises important fairness concerns, as these systems may showcase performance disparities across demographic groups.

By Maxime Moussi, Beno\^it Ronval, Siegfried Nijssen, F\'elicien Schiltz

rag benchmarks safety

Get the daily briefing