Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,273 stories · RSS feed

arXiv AI

Jun 8

MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

arXiv:2605. 05225v3 Announce Type: replace-cross Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.

By Bo Li, Chuan Wu, Shaolin Zhu

llms multimodal benchmarks

arXiv Machine Learning

Jun 8

Leveraging Error Diversity in Group Rollouts for Reinforcement Learning

arXiv:2605. 17333v2 Announce Type: replace Abstract: Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness, yet the collective structure of the group output, specifically the distribution of errors, is largely discarded.

By Wenpu Liu, Yuqi Xu, Weichu Xie, Yongfu Zhu, Shuai Dong, Ziyue Wang, Wenqi Shao, Xiaoying Zhang, Tong Yang, Nan Duan, Jiaqi Wang

reinforcement-learning benchmarks

arXiv AI

Jun 8

Auditing Training Data in Domain-adapted LLMs: LoRA-MINT

arXiv:2606. 06946v1 Announce Type: cross Abstract: We present LoRA-MINT, a new methodology for Membership Inference Test (MINT) applied to recent Large Language Models (LLMs) fine-tuned for specific Natural Language Processing (NLP) tasks through Low-Rank Adaptation (LoRA).

By Gonzalo Mancera, Daniel DeAlcala, Aythami Morales, Julian Fierrez, Ruben Tolosana, Francisco Jurado

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Jun 8

GraphWalker: Patient Analogy Meets Information Gain for Clinical Reasoning with Large Language Models

arXiv:2604. 06684v2 Announce Type: replace Abstract: Clinical reasoning over electronic health records (EHRs) is a fundamental yet challenging task in modern healthcare.

By Yue Fang, Weibin Liao, Yuxin Guo, Jiaran Gao, Hongxin Ding, Jinyang Zhang, Xinke Jiang, Zhibang Yang, Junfeng Zhao, Yasha Wang, Liantao Ma

llms agents benchmarks

arXiv AI

Jun 8

Proxy Reconstruction Pre-training for Ramp Flow Prediction at Highway Interchanges

arXiv:2510. 03381v3 Announce Type: replace-cross Abstract: Interchanges are crucial nodes for vehicle transfers between highways, yet the lack of real-time ramp detectors creates blind spots in traffic prediction.

By Yongchao Li, Jun Chen, Zhuoxuan Li, Chao Gao, Yang Li, Chu Zhang, Changyin Dong

multimodal benchmarks

arXiv AI

Jun 8

ShallowBench: Benchmarking Generative Drug Design Models on Shallow-Pocket Targets

arXiv:2606. 06717v1 Announce Type: cross Abstract: While generative AI models have demonstrated remarkable success in structure-based drug design, they predominantly rely on deep binding pockets and struggle to sample effective ligands for challenging low-pocketability targets, such as the historically "undruggable" oncology targets KRAS and MYC.

By Saket Reddy, Shiwei Liu

benchmarks

arXiv AI

Jun 8

TSAQA: Time Series Analysis Question And Answering Benchmark

arXiv:2601. 23204v2 Announce Type: replace Abstract: Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science.

By Baoyu Jing, Sanhorn Chen, Lecheng Zheng, Boyu Liu, Zihao Li, Jiaru Zou, Tianxin Wei, Zhining Liu, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Yuchen Yan, Dongqi Fu, Jingchao Ni, Jingrui He, Hanghang Tong

llms nlp fine-tuning benchmarks

arXiv AI

Jun 8

Neuro-Symbolic Learning for Long-Horizon Task Planning Under Complex Logical Constraints

arXiv:2606. 06877v1 Announce Type: cross Abstract: Task planning often suffers from severe efficiency bottlenecks when robots must reason over long-horizon action sequences under complex logical constraints, including object affordances, spatial relationships, and sequential action dependencies.

By Qiwei Du, Zitong Zhan, Shaoshu Su, Bowen Li, Yi Du, Zhipeng Zhao, Taimeng Fu, Sebastian Scherer, Jiaoyang Li, Chen Wang

robotics benchmarks safety

arXiv AI

Jun 8

SW-$A^2$-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web

arXiv:2604. 04226v2 Announce Type: replace-cross Abstract: The Agentic Web is emerging as a paradigm in which autonomous software agents interact with online resources and with each other to accomplish user goals.

By Linyao Chen, Bo Huang, Qinlao Zhao, Shuai Shao, Zhi Han, Zicai Cui, Ziheng Zhang, Guangtao Zeng, Wenzheng Tang, Yikun Wang, Yuanjian Zhou, Zimian Peng, Yong Yu, Weiwen Liu, Hiroki Kobayashi, Weinan Zhang

agents benchmarks

arXiv AI

Jun 8

Human Adults and LLMs as Scientists: Who Benefits from Active Exploration?

arXiv:2606. 06464v1 Announce Type: cross Abstract: A long-standing finding in the causal learning literature is that adults struggle to identify conjunctive causal rules, where an effect requires the simultaneous presence of multiple causes, while performing better in disjunctive settings.

By Mandana Samiei, Eunice Yiu, Anthony GX-Chen, Dongyan Lin, Jocelyn Shen, Blake A. Richards, Alison Gopnik, Doina Precup

llms benchmarks safety

arXiv AI

Jun 8

Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

arXiv:2606. 06976v1 Announce Type: new Abstract: Large language model (LLM)-based agents often make suboptimal tool-use decisions, including unsupported tool invocation and hallucinated direct responses, which may accumulate errors throughout multi-step interactions.

By Yijin Zhou, Linqian Zeng, Xiaoya Lu, Wenyuan Xie, Dongrui Liu, Junchi Yan, Jing Shao

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 8

Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.

By Tao Liu, Ye Lu, Ruohua Zhang, Siyu Song, Wentao Liu, Aimin Zhou, Hao Hao

llms agents benchmarks safety

arXiv Machine Learning

Jun 8

Controllable Molecular Generative Foundation Models

arXiv:2605. 15354v2 Announce Type: replace Abstract: Despite the success of foundation models in language and vision, molecular graph generation still lacks a unified framework for heterogeneous design tasks with reliable controllability.

By Yihan Zhu, Yuhan Liu, Weijiang Li, Tengfei Luo, Meng Jiang

rag diffusion reinforcement-learning benchmarks

arXiv AI

Jun 8

OpenSkill: Open-World Self-Evolution for LLM Agents

arXiv:2606. 06741v1 Announce Type: new Abstract: Self-evolving agents requires adaptation after deployment, but existing approaches assume a usable learning loop, such as curated skills, successful trajectories, or verifier signals.

By Zhiling Yan, Dingjie Song, Hanrong Zhang, Wei Liang, Yuxuan Zhang, Yutong Dai, Lifang He, Philip S. Yu, Ran Xu, Xiang Li, Lichao Sun

llms agents benchmarks

arXiv Machine Learning

Jun 8

Closed-Form Spectral Regularization for Multi-Task Model Merging

arXiv:2606. 07289v1 Announce Type: new Abstract: Model merging combines several independently fine-tuned experts into a single multi-task model without any training data, reducing the storage, serving, and decentralized-development costs of large foundation models.

By Yongxian Wei, Runxi Cheng, Xingxuan Zhang, Li Shen, Chun Yuan, Peng Cui, Dacheng Tao

computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 8

It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents

arXiv:2512. 23128v2 Announce Type: replace-cross Abstract: Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.

By Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki, Piotr B{\l}aszczyk, Will Howard, Lukas Aichberger, Chris Russell, Philip H. S. Torr, Adam Mahdi, Adel Bibi

llms agents benchmarks safety

arXiv AI

Jun 8

dots.tts Technical Report

arXiv:2606. 07080v1 Announce Type: cross Abstract: We present dots.

By Shi Lian, Changtao Li, Bohan Li, Hankun Wang, Da Zheng, Junfeng Tian, Yufeng Ma, Colin Zhang, Kai Yu

diffusion efficiency multimodal benchmarks

arXiv AI

Jun 8

What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media

arXiv:2606. 06784v1 Announce Type: cross Abstract: Public social media posts can reveal private information through weak cues scattered across text, images, or metadata.

By Zifan Peng, Yini Huang, Aiwen Lu, Qiming Ye, Peixian Zhang, Jingyi Zheng, Yule Liu, Xuechao Wang, Xinlei He, Jiaheng Wei

agents multimodal benchmarks safety

arXiv Machine Learning

Jun 8

CoMetaPNS: Continually Meta-learning Personalized Neural Surrogates for Cardiac Electrophysiology Simulations

arXiv:2606. 07488v1 Announce Type: new Abstract: Personalized virtual heart simulations face challenges in model personalization and computational cost.

By Ryan Missel, Xiajun Jiang, Linwei Wang

benchmarks

arXiv AI

Jun 8

Detecting and Mitigating Bias by Treating Fairness as a Symmetry Operation

arXiv:2606. 06514v1 Announce Type: new Abstract: Machine learning systems deployed in high stakes socioeconomic settings routinely display bias.

By Nishit Singh

benchmarks safety

Get the daily briefing