Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,880 stories · RSS feed

arXiv AI

Jun 11

Task-Aligned Stability Analysis of Vision-Language Models for Autonomous Driving Hazard Detection

arXiv:2606. 11889v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used for scene understanding in autonomous driving, but robustness analysis often relies on task-agnostic embedding stability alone.

By Everett Richards

llms rag agents robotics multimodal benchmarks

arXiv AI

Jun 11

Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark

arXiv:2602. 19502v2 Announce Type: replace Abstract: Agentic AI systems are increasingly capable of autonomous data science workflows, yet clinical prediction tasks demand domain expertise that purely automated approaches struggle to provide.

By Lalitha Pranathi Pulavarthy, Raajitha Muthyala, Aravind V Kuruvikkattil, Zhenan Yin, Rashmita Kudamala, Saptarshi Purkayastha

agents multimodal benchmarks safety

arXiv Machine Learning

Jun 11

From Persistence to Survival: Hypothesis Testing, Effect Sizes and Vectorisation for Topological Features

arXiv:2606. 11911v1 Announce Type: cross Abstract: Persistence diagrams are common representations in topological data analysis, but they do not naturally live in a vector space, and the statistical tools developed for comparing them have largely evolved separately from those used for downstream prediction.

By Juliette Murris, Bernadette Stolz, Karsten Borgwardt

benchmarks

arXiv AI

Jun 11

Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning

arXiv:2601. 00791v2 Announce Type: replace-cross Abstract: Verifying whether a language model is genuinely reasoning or pattern-matching remains an open problem: learned verifiers are expensive, and output-based heuristics are brittle.

By Valentin No\"el

llms rag benchmarks

arXiv AI

Jun 11

CoVar: Confidence-Variance-Guided Pseudo-Label Selection for Semi-Supervised Learning

arXiv:2601. 11670v3 Announce Type: replace-cross Abstract: Pseudo-label selection in semi-supervised learning is commonly driven by maximum-confidence thresholds, yet confidence alone can be unreliable under model overconfidence and class imbalance.

By Jinshi Liu, Lei He, Pan Liu

computer-vision benchmarks

arXiv AI

Jun 11

Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs

arXiv:2606. 11232v1 Announce Type: cross Abstract: Existing LLM moral benchmarks usually ask which isolated moral act, value, or foundation a model prefers.

By Weijia Zhang, Ruiqi Chen, Yunze Xiao, Weihao Xuan

llms benchmarks

arXiv Machine Learning

Jun 11

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

arXiv:2605. 04893v2 Announce Type: replace Abstract: When a language model processes a hallucinated response, its attention routing tends to fail in one of two shapes: over-concentrating on a narrow set of positions, or spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal.

By Dominik Dahlem, Diego Maniloff, Mac Misiura

llms benchmarks

arXiv AI

Jun 11

Grounding Computer Use Agents on Human Demonstrations

arXiv:2511. 07332v2 Announce Type: replace-cross Abstract: Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements.

By Aarash Feizi, Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Kaixin Li, Rabiul Awal, Xing Han L\`u, Johan Obando-Ceron, Juan A. Rodriguez, Nicolas Chapados, David Vazquez, Adriana Romero-Soriano, Reihaneh Rabbany, Perouz Taslakian, Christopher Pal, Spandana Gella, Sai Rajeswar

agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jun 11

CaReTS: A Multi-Task Framework Unifying Classification and Regression for Time Series Forecasting

arXiv:2511. 09789v2 Announce Type: replace Abstract: Recent advances in deep forecasting models have achieved remarkable performance, yet most approaches still struggle to provide both accurate predictions and interpretable insights into temporal dynamics.

By Fulong Yao, Wanqing Zhao, Chao Zheng, Xiaofei Han

llms benchmarks

arXiv AI

Jun 11

Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment

arXiv:2510. 03520v2 Announce Type: replace-cross Abstract: Ensuring safety is a foundational requirement for large language models (LLMs).

By Kartik Pandit, Sourav Ganguly, Arnesh Banerjee, Shaahin Angizi, Arnob Ghosh

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 11

OCSVM-Guided Representation Learning for Unsupervised Anomaly Detection

arXiv:2507. 21164v2 Announce Type: replace-cross Abstract: Unsupervised anomaly detection (UAD) aims to detect anomalies without labeled data, a necessity in many machine learning applications where anomalous samples are rare or not available.

By Nicolas Pinon (MYRIAD), Robin Trombetta (MYRIAD), Carole Lartizien (MYRIAD)

benchmarks

arXiv Machine Learning

Jun 11

Analytic Bijections for Smooth and Interpretable Normalizing Flows

arXiv:2601. 10774v2 Announce Type: replace Abstract: A key challenge in normalizing flows is finding expressive invertible scalar bijections.

By Mathis Gerdes, Miranda C. N. Cheng

benchmarks

arXiv AI

Jun 11

Privacy-Preserving Federated Autoencoder for ECG Anomaly Detection on Edge Devices

arXiv:2606. 11556v1 Announce Type: cross Abstract: Continuous electrocardiography (ECG) monitoring could surface rhythm abnormalities before they escalate into cardiovascular events.

By Kaan Arda Akyol, Jakub Kacper Szel\k{a}g, Aydin Abadi, Maha Alghamdi, Ghadah Albalawi, Ghouse Ibrahim Kaleelullah, Hilal Tutus, Sarah Al Subaiei, Shardul Kapse, Syed Mohammed Raheeb, Mujeeb Ahmed, Rehmat Ullah

efficiency benchmarks safety

arXiv Machine Learning

Jun 11

Family-Aware Residual Architecture for Predicting Quantum Circuit Simulation Performance

arXiv:2606. 11620v1 Announce Type: cross Abstract: Approximate tensor-network simulators enable classical simulation of quantum circuits beyond the reach of exact methods, but selecting optimal approximation parameters -- such as bond dimension thresholds -- remains a costly trial-and-error process.

By Honjar Xing, Yehong Jiang, Xianbang Wang, Zehua Wang, Zhicheng Jiang

benchmarks

arXiv AI

Jun 11

Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

arXiv:2606. 11324v1 Announce Type: cross Abstract: We introduce Embodied-R1.

By Yifu Yuan, Yaoting Huang, Xianze Yao, Yutong Li, Shuoheng Zhang, Linqi Han, Pengyi Li, Jiangeng Sun, Wenting Jia, Zhao Zhang, Yuhao Liu, Ruihao Liao, Yucheng Hu, Qiyu Wu, Yuxiao Li, Zibin Dong, Fei Ni, Yan Zheng, Shuyang Gu, Yi Ma, Hongyao Tang, Han Hu, Jianye Hao

llms reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 11

My Chemical Harness: Evolutionary Molecular Design over Synthetic Pathways with Large Language Model Agents

arXiv:2606. 11256v1 Announce Type: cross Abstract: Designing molecules with target properties is most useful when candidate structures are accompanied by feasible synthetic routes.

By C\'esar Ojeda, Darius A. Faroughy, Maryam Karimi, Payam Zarrintaj, Mir Mehdi Seyedebrahimi, Mart\'in Carballo-Pacheco

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jun 11

Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite

arXiv:2606. 11257v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) pipelines are compute-intensive, combining embedding, retrieval, reranking, and large language model (LLM) generation.

By Zhiyuan Cheng, Longying Lai

llms rag efficiency benchmarks safety

arXiv Machine Learning

Jun 11

MASK: Multi-Agent Semantic K-Scheduling for Risk-Sensitive 6G Robotics

arXiv:2606. 11249v1 Announce Type: cross Abstract: Realizing the vision of 6G connected robotics requires reconciling high-performance collaborative control with the rigid spectral limitations of physical wireless channels.

By Ahmet Gunhan Aydin, Elif Tugce Ceran

agents robotics efficiency benchmarks

arXiv Machine Learning

Jun 11

Context-Driven Incremental Compression for Multi-Turn Dialogue Generation

arXiv:2606. 12411v1 Announce Type: cross Abstract: Modern conversational agents condition on an ever-growing dialogue history at each turn, incurring redundant attention and encoding costs that grow with conversation length.

By Yeongseo Jung, Jaehyeok Kim, Eunseo Jung, Jiachuan Wang, Yongqi Zhang, Ka Chun Cheung, Simon See, Lei Chen

agents nlp efficiency benchmarks

arXiv AI

Jun 11

SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving

arXiv:2606. 11244v1 Announce Type: cross Abstract: Efficient large language model (LLM) serving is increasingly constrained by deployment cost.

By Hongyuan Liu, Yawei Li, Zhiqiang Que, Qinli Yang, Junming Shao, Guosheng Hu

llms efficiency benchmarks

Get the daily briefing