Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 1

VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation

arXiv:2609.38512v1 Announce Type: new Abstract: Voice agents in production must handle several requests, background speech, and customers who lose patience. We introduce VAmoS Energy, a benchmark tha...

By Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi

llms agents benchmarks

arXiv AI

Oct 1

Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimization

arXiv:2609.38757v1 Announce Type: new Abstract: Large language models are increasingly participating in complex real-world tasks in the form of algorithm-design agents, designing and refining algorit...

By Chen Lu, Ke Xue, Siyuan Xu, Mingxuan Yuan, Chao Qian

llms agents efficiency benchmarks

arXiv AI

Oct 1

Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents

arXiv:2609.39026v1 Announce Type: new Abstract: Deep Research agents synthesize evidence into cited reports, yet a well-cited report can still reach a misleading conclusion. Citation correctness chec...

By Shuyao Xiao, Shengling Wang, Xuan Chen, Ke Chao, Ming Cui, Feifei Qian, Chaoyang Mei, Fanlin Meng, Lulu Wang, Ziming Yu, Junxi Yin

agents benchmarks safety

arXiv AI

Oct 1

Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation

arXiv:2609.39168v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet existing...

By Zhihan Zhang, Lizi Liao

llms reinforcement-learning multimodal benchmarks

arXiv AI

Oct 1

EHR-RobustGym: Benchmarking and Training Agents for Robust Clinical Reasoning

arXiv:2609.39371v1 Announce Type: new Abstract: In hospital workflows, electronic health records (EHRs) are often noisy, and may not contain the evidence needed to confirm events or measurements refe...

By Yitong Qiao, Yancheng Jin, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan Chu

llms agents reinforcement-learning fine-tuning benchmarks data-engineering

arXiv AI

Oct 1

ArchitectureIQ: On the Measure of Training Intuition

arXiv:2609.39714v1 Announce Type: new Abstract: Top researchers have good intuition, but do language models have as good intuition about model training as top AI researchers? To measure model intuiti...

By Zirui Ren, Shaoyang Guo, Chencheng Tang, Jinxin Wang, Chengyu Xiong, Shanbin Yu, Peihang Li, Yidi Wu, Bangzhe Huang, Qingyu Qu, Leqian Yang, Ziming Liu

llms rag benchmarks

arXiv AI

Oct 1

What Was Said, Not What Was 'Thought': Type-6 Logic for CoT Verification

arXiv:2609.38420v1 Announce Type: cross Abstract: We introduce Type-6 logic, a variant of dynamic epistemic logic augmented with two operators (uncertainty and recurrence), designed to model the infe...

By Adrian de Wynter

llms benchmarks

arXiv AI

Oct 1

Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation

arXiv:2609.38660v1 Announce Type: cross Abstract: Long-form subtitle translation requires reasoning over discourse and cultural context spanning episodes or entire series, while maintaining consisten...

By Haibo Jin, Xinjie Li, Najmeh Sadoughi, Yang Liu, Yibo Wang, Zhu Liu, Yuzong Liu

llms agents benchmarks

arXiv AI

Oct 1

In a Streaming World, Should You Stand Still? A Comprehensive Benchmark of Anomaly Detection in Streams

arXiv:2609.39215v1 Announce Type: cross Abstract: Time series anomaly detection (TSAD) is increasingly deployed in streaming settings, where data arrive sequentially and may exhibit non-stationarity....

By Magali Parrino, Antoine Ajenjo, Emmanuel Remy, Pierre Stephan, Pierre Senellart, Paul Boniol

benchmarks

arXiv AI

Oct 1

Self-Spec Verifiable Code Generation

arXiv:2609.39568v1 Announce Type: cross Abstract: Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable...

By Jiaru Qian, Yihong Dong, Yongmin Li, Hao Zhu, Bin Gu, Ge Li

llms benchmarks

arXiv AI

Oct 1

When Masking Helps or Hurts Robustness in Compressed CLIP: A Pre-Deployment Diagnostic

arXiv:2609.39704v1 Announce Type: cross Abstract: This paper demonstrate that whether masking-based token pruning helps or hurts worst-group robustness can be predicted before deployment, without lab...

By Muhammad Zawish, Steven Davy

computer-vision fine-tuning efficiency benchmarks

arXiv AI

Oct 1

Pseudo-Label-Triggered Retraining from Forecast Errors for Online Time Series Forecasting

arXiv:2609.39789v1 Announce Type: cross Abstract: Real-world time series forecasting systems operate under non-stationary data streams, where forecasting performance may degrade over time. Although r...

By Yeryeong Kwak, Yoo-Min Jung, Jonghun Park

benchmarks

arXiv AI

Oct 1

CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion

arXiv:2609.39902v1 Announce Type: cross Abstract: Large language models have achieved remarkable capabilities across diverse domains, yet their safety alignment remains vulnerable to jailbreak attack...

By Zhen Liang, Hai Huang, Wentao Chen

llms diffusion benchmarks safety

arXiv AI

Oct 1

DoGBench: Can Agents Meet Expert Standards for User-Facing Documentation?

arXiv:2609.39909v1 Announce Type: cross Abstract: We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing sof...

By Frances Liu, Manny Silva, Paige Calvert, Ayu Adiati, Sarah Sanders

agents benchmarks

arXiv AI

Oct 1

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding

arXiv:2609.39924v1 Announce Type: cross Abstract: Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, m...

By Yulong Liu, Xiaotian Han, Junyuan Shang, Yuchen Ding, Zhenyu Zhang, Shuohuan Wang, Guibo Zhu, Sirui Han, Dianhai Yu

llms rag multimodal benchmarks

arXiv AI

Oct 1

LongEmo: Towards Emotion Understanding and Reasoning in Long Videos

arXiv:2609.40079v1 Announce Type: cross Abstract: While recent Multimodal Large Language Models (MLLMs) have shown promise in affective computing, their reasoning capabilities are largely confined to...

By Shuo Zhang, Yifan Zhou, Han Wang, Jinsong Zhang, Jingyu Li, Hongbing Li, Zhejun Zhang, Chengyi Zhao, Yuquan Hao, Yitong Liu, Jiyin Li, Ruiqi Tang, Zixuan Lin, Yi Luo, Xurui Zhang, Ronghao Chen, Huacan Wang, Lei Li

llms agents multimodal benchmarks

arXiv AI

Oct 1

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

arXiv:2609.40195v1 Announce Type: cross Abstract: Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spann...

By Guangzhi Xiong, Xinyuan Zhang, Xiao Yang, Hyokun Yun, Kai Zhang, Shiun-Zu Kuo, Hyeonjeong Ha, Xilun Chen, Kai Sun, Lucas Liang, Guangqiang Dong, Ejaz Ahmed, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Aidong Zhang, Xin Luna Dong

agents reinforcement-learning multimodal benchmarks

arXiv AI

Oct 1

cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

arXiv:2609.40284v1 Announce Type: cross Abstract: Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on...

By Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh

agents benchmarks

arXiv AI

Oct 1

Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models

arXiv:2609.40219v1 Announce Type: cross Abstract: World Action Models (WAMs) couple visual dynamics prediction with action generation, yet they do not explicitly support the reuse of action experienc...

By Qi Lyu, Jiahua Dong, Hao Shen, Xudong Wang, Hongyuan Yu, Baichen Liu, Henghui Ding, Zhi Han, Nicu Sebe, Ivan Laptev, Fahad Shahbaz Khan, Salman Khan

rag nlp robotics benchmarks

arXiv AI

Oct 1

SkillMaster: Toward Autonomous Skill Mastery in LLM Agents

arXiv:2605.08693v3 Announce Type: replace Abstract: Skills provide an effective mechanism for improving LLM agents on complex tasks, yet in existing agent frameworks, their creation, refinement, and...

By Min Yang, Jinghua Piao, Xu Xia, Xiaochong Lan, Jiaju Chen, Yongshun Gong, Yong Li

llms agents benchmarks

Get the daily briefing