Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv AI

Jun 11

PRInTS: Reward Modeling for Long-Horizon Information Seeking

arXiv:2511. 19314v2 Announce Type: replace Abstract: Information-seeking is a core capability for AI agents, requiring them to gather and reason over tool-generated information across long trajectories.

By Jaewoo Lee, Archiki Prasad, Justin Chih-Yao Chen, Zaid Khan, Elias Stengel-Eskin, Mohit Bansal

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jun 11

MARIC: Multi-Agent Reasoning for Image Classification

arXiv:2509. 14860v2 Announce Type: replace-cross Abstract: Image classification has traditionally relied on parameter-intensive model training, requiring large-scale annotated datasets and extensive fine tuning to achieve competitive performance.

By Wonduk Seo, Minhyeong Yu, Hyunjin An, Seunghyun Lee

llms agents computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 11

Semantic search for 100M+ galaxy images using AI-generated captions

arXiv:2512. 11982v2 Announce Type: replace-cross Abstract: Finding scientifically interesting phenomena through slow manual labeling campaigns severely limits our ability to explore the billions of galaxy images produced by telescopes.

By Nolan Koblischke, Liam Parker, Francois Lanusse, Jo Bovy, Irina Espejo, Shirley Ho

llms rag multimodal benchmarks

arXiv AI

Jun 11

Knowing When to Ask: Self-Gated Clarification for Hierarchical Language Agents

arXiv:2606. 11349v1 Announce Type: new Abstract: In hierarchical reasoning, failures often originate at intermediate decision points where the agent commits to a wrong branch without recognizing that it lacks critical information.

By Aijing Gao, Yiming Kang, Mengdie Flora Wang, Jae Oh Woo

llms agents benchmarks

arXiv AI

Jun 11

Noise-Aware Framework for Correcting Corrupted Labels

arXiv:2606. 11695v1 Announce Type: cross Abstract: High-quality labeled data is essential for training reliable ML/DL models.

By Ha-Linh Nguyen, Hong-Anh Nguyen, Minh-Duc La, Phong Lam, Thu-Trang Nguyen, Son Nguyen, Hieu Dinh Vo

benchmarks

arXiv AI

Jun 11

DiffCold: A Diffusion-based Generative Model for Cold-Start Item Recommendation

arXiv:2606. 12245v1 Announce Type: cross Abstract: Cold-start item recommendation remains a persistent challenge in real-world systems due to the absence of interaction histories.

By Kangning Zhang, Yingjie Qin, Weinan Zhang, Yong Yu, Jianghao Lin

rag diffusion benchmarks safety

arXiv Machine Learning

Jun 11

FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

arXiv:2601. 04203v2 Announce Type: replace-cross Abstract: We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback.

By Xueqing Wu, Zihan Xue, Da Yin, Shuyan Zhou, Kai-Wei Chang, Nanyun Peng, Yeming Wen

llms agents multimodal benchmarks

arXiv AI

Jun 11

Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

arXiv:2606. 11634v1 Announce Type: new Abstract: The rapid progress of reasoning and agentic large language models (LLMs) has increased the demand for long-context inference, but self-attention (SA) scales quadratically with context length.

By Kai Liu, Peijie Dong, Xinchen Xie, Jianfei Gao, Qipeng Guo, Xiaowen Chu, Shaoting Zhang, Kai Chen

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 11

Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite

arXiv:2606. 11257v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) pipelines are compute-intensive, combining embedding, retrieval, reranking, and large language model (LLM) generation.

By Zhiyuan Cheng, Longying Lai

llms rag efficiency benchmarks safety

arXiv AI

Jun 11

ARGUS: Stacked Multi-View Identity Mosaic Injection for Subject-Preserving Video Generation

arXiv:2606. 11670v1 Announce Type: cross Abstract: Subject-preserving video generation is not solved by frontal-face similarity alone: a generated person must remain recognizable across motion, large viewpoint changes, expression shifts, occlusion, scale variation, and conflicts among text, first-frame, and identity references.

By Zijie Meng, Jiwen Liu, Yufei Liu, Chengzhuo Tong, Xiaoqiang Liu, Yuanxing Zhang, Yulong Xu, Pengfei Wan

diffusion benchmarks

arXiv AI

Jun 11

MedCTA: A Benchmark for Clinical Tool Agents

arXiv:2606. 11702v1 Announce Type: cross Abstract: To make clinically grounded decisions, medical AI agents are expected to go beyond simple recognition and be capable of tool retrieval, evidence acquisition, and integration.

By Tajamul Ashraf, Hyewon Jeong, Fida Mohammad Thoker, Bernard Ghanem

agents nlp multimodal benchmarks

arXiv AI

Jun 11

Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction

arXiv:2605. 04221v2 Announce Type: replace-cross Abstract: Clinical named entity recognition from dental progress notes is challenging because documentation is highly unstructured, domain-specific, and often privacy-sensitive.

By Yao-Shun Chuang, Tushti Mody, Uday Pratap Singh, Shirindokht Shiraz, Chun-Teh Lee, Ryan Brandon, Muhammad F Walji, Xiaoqian Jiang, Bunmi Tokede

llms nlp reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 11

Can AI Agents Synthesize Scientific Conclusions?

arXiv:2606. 11337v1 Announce Type: new Abstract: Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions.

By Hayoung Jung, Pedro Viana Diniz, Jos\'e Reinaldo Corr\^ea Roveda, Abner Fernandes da Silva, Haeun Jung, Enoch Tsai, Aleksandra Korolova, Manoel Horta Ribeiro

agents benchmarks

arXiv AI

Jun 11

When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

arXiv:2605. 26418v2 Announce Type: replace-cross Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help?

By Guilin Zhang, Chuanyi Sun, Kai Zhao, Shahryar Sarkani, John Fossaceca

agents reinforcement-learning benchmarks

arXiv AI

Jun 11

OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

arXiv:2601. 22725v4 Announce Type: replace-cross Abstract: Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck.

By Jin Li, Tao Chen, Kai Wen, Siqi Yin, Shuai Jiang, Weijie Wang, Jingwen Luo, Chenhui Wu

llms diffusion computer-vision multimodal benchmarks safety

arXiv Machine Learning

Jun 11

Mechanisms of Introspective Awareness

arXiv:2603. 21396v5 Announce Type: replace Abstract: Recent work has shown that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept -- a phenomenon termed "introspective awareness.

By Uzay Macar, Li Yang, Atticus Wang, Peter Wallich, Emmanuel Ameisen, Jack Lindsey

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 11

Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking

arXiv:2602. 10743v2 Announce Type: replace Abstract: State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but their linear state updates limit expressivity and robust state tracking.

By Vaisakh Shaj, Cameron Barker, Aidan Scannell, Andras Szecsenyi, Elliot J. Crowley, Amos Storkey

llms robotics benchmarks

arXiv AI

Jun 11

TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation

arXiv:2606. 11637v1 Announce Type: new Abstract: Touch is a key modality for embodied agents to understand the physical world.

By Kailin Lyu, Di Wu, Pengwei Zhang, Yuhang Zheng, Yingxin Lai, Long Xiao, Kangyi Wu, Pengna Li, Chen Gao, Lianyu Hu, Xiaobin Hu, Jie Hao, Ce Hao, Weihao Yuan, Shuicheng Yan

agents robotics benchmarks

arXiv AI

Jun 11

INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration

arXiv:2606. 11440v1 Announce Type: new Abstract: Existing multi-agent LLM orchestration methods, ranging from brute-force ensembles to learned routers, select models and topologies based on task and model features.

By Ahasan Kabir, Jiaqi Xue, Mengxin Zheng, Qian Lou

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 11

Privacy-Preserving Federated Autoencoder for ECG Anomaly Detection on Edge Devices

arXiv:2606. 11556v1 Announce Type: cross Abstract: Continuous electrocardiography (ECG) monitoring could surface rhythm abnormalities before they escalate into cardiovascular events.

By Kaan Arda Akyol, Jakub Kacper Szel\k{a}g, Aydin Abadi, Maha Alghamdi, Ghadah Albalawi, Ghouse Ibrahim Kaleelullah, Hilal Tutus, Sarah Al Subaiei, Shardul Kapse, Syed Mohammed Raheeb, Mujeeb Ahmed, Rehmat Ullah

efficiency benchmarks safety

Get the daily briefing