Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 15

The Agentic Company OS: Substrate Inversion for Sustained Enterprise Agent Deployment

arXiv:2609.13334v1 Announce Type: cross Abstract: Enterprise AI agents often succeed in a demonstration and then stall once they must operate day after day. An industry report estimates that most pil...

By Oliver Aleksander Larsen, Mahyar T. Moghaddam

llms agents benchmarks

arXiv AI

Sep 15

Learning to Solve Hard Problems in RL for LLMs by Never Giving Up

arXiv:2609.13443v1 Announce Type: cross Abstract: We demonstrate that training LLMs with RL does not improve performance equally across a dataset. RL shows large improvements on easy problems that an...

By Michael Noukhovitch, Hamish Ivison, Nathan Lambert, Aaron Courville

llms reinforcement-learning benchmarks

arXiv AI

Sep 15

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

arXiv:2609.14118v1 Announce Type: cross Abstract: Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM)...

By Feiyu Du, Xi He, Jia Li, Yapeng Tian, Weili Wu

multimodal benchmarks

arXiv AI

Sep 15

ATTRICITE: Training an Open 4B Model for Citation Recovery toward Faithful Attribution

arXiv:2609.14248v1 Announce Type: cross Abstract: Faithful citation attribution begins with identifying the intended source for a scientific claim. We study this source-identification capability thro...

By Yee Man Choi, Xuehang Guo, Songcheng Cai, Yimu Wang, Yi R. Fung, Qingyun Wang

llms fine-tuning benchmarks

arXiv AI

Sep 15

Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization

arXiv:2609.14839v1 Announce Type: cross Abstract: The integration of Large Language Models (LLMs) into automated code optimization introduces a critical reliability risk we term the Efficiency Halluc...

By Sarah Wilson, Gail Kaiser, Patrick Musau

llms benchmarks safety

arXiv AI

Sep 15

VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding

arXiv:2609.15606v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress on short video understanding yet remain limited on long videos due to the...

By Weixin Xu, Zhenyu Yang, Bing Wang, Shengsheng Qian, Changsheng Xu

llms agents reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Sep 15

FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learning

arXiv:2609.15625v1 Announce Type: cross Abstract: Driven by the escalating demand for privacy-preserving computing, Federated Learning (FL) has witnessed remarkable progress, becoming a cornerstone t...

By Changkun Lin, Junxiao Wang

benchmarks safety

arXiv AI

Sep 15

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

arXiv:2609.15939v1 Announce Type: cross Abstract: Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can dete...

By Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, Amin Karbasi

llms agents benchmarks

arXiv AI

Sep 15

From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions

arXiv:2603.13359v2 Announce Type: replace Abstract: Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusa...

By Rishab Alagharu, Ishneet Sukhvinder Singh, Shaibi Shamsudeen, Zhen Wu, Ashwinee Panda

llms fine-tuning benchmarks safety

arXiv AI

Sep 15

CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflection

arXiv:2604.07487v2 Announce Type: replace Abstract: Large language model agents rely on effective model context to obtain task-relevant information for decision-making. Many existing context engineer...

By Linbo Liu, Guande Wu, Han Ding, Yawei Wang, Qiang Zhou, Yuzhe Lu, Zhichao Xu, Huan Song, Panpan Xu, Lin Lee Cheong

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Sep 15

Valley3: Scaling Omni Foundation Models for E-commerce

arXiv:2605.01278v3 Announce Type: replace Abstract: In this work, we present Valley3, an omni multimodal large language model (MLLM) developed for diverse global e-commerce tasks, with unified unders...

By Zeyu Chen, Guanghao Zhou, Min Yang, Qixiang Yin, Ziwang Zhao, Huanjin Yao, Pengjiu Xia, Cen Chen, Minghui Qiu

llms agents multimodal benchmarks

arXiv AI

Sep 15

Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost

arXiv:2605.06165v2 Announce Type: replace Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribut...

By Richmond Sin Jing Xuan, Rishabh Bhardwaj, Soujanya Poria

llms efficiency benchmarks

arXiv AI

Sep 15

TRACTA: Benchmarking Temporal Reasoning over Semantic Trajectories

arXiv:2607.22365v2 Announce Type: replace Abstract: High-complexity operational environments require methods that characterize temporally distributed patterns rather than classify isolated events. Th...

By Michael Romei De Socio, Gian Luca Pozzato, Alessio Merlo

benchmarks

arXiv AI

Sep 15

ChartAnno: Benchmarking Multimodal Large Language Models for Chart Annotation Generation

arXiv:2608.03464v2 Announce Type: replace Abstract: Annotations are essential to communicative visualization, helping explain data, emphasize key findings, and guide attention. While multimodal large...

By Zhenghan Chen, Zekai Shao, Lidan Tan, Xin Lin, Xingchen Zeng, Yi Shan, Ziyue Lin, Xiaoliang Fu, Xinyuan Liu, Yuetong Guo, Fen Wang, Bongshin Lee, Siming Chen

llms multimodal benchmarks

arXiv AI

Sep 15

Beyond Generation and Accuracy: Diagnosing and Enhancing Visual Chain-of-Thought for Geometry Problem Solving

arXiv:2609.12606v2 Announce Type: replace Abstract: While multimodal reasoning has advanced rapidly, solving complex geometry problems critically hinges on active visual assistance, such as construct...

By Zhitong Dong, Jicai Pan, Yingguo Gao, Jingting Ding, Hao Chen, Jinjie Gu

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Sep 15

Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation

arXiv:2503.06868v2 Announce Type: replace-cross Abstract: Existing long-text generation methods produce lengthy outputs from short inputs, leaving long-input-to-long-output generation underexplored....

By Junhao Zhang, Richong Zhang, Fanshuang Kong, Ziyang Miao, Yanhan Ye, Yaowei Zheng

llms rag benchmarks safety

arXiv AI

Sep 15

ModiGen: A Large Language Model-Based Workflow for Multi-Task Modelica Code Generation

arXiv:2503.18460v2 Announce Type: replace-cross Abstract: Modelica is a widely adopted language for simulating complex physical systems, yet effective model creation and optimization require substant...

By Jiahui Xiang, Tong Ye, Peiyu Liu, Yinan Zhang, Wenhai Wang

llms rag fine-tuning benchmarks

arXiv AI

Sep 15

Exploring the Potential of Diffusion Large Language Models in Code Generation

arXiv:2509.11252v3 Announce Type: replace-cross Abstract: LLMs have become the mainstream approaches to code generation. Existing LLMs mainly employ autoregressive generation, i.e. generating code to...

By Chengze Li, Yitong Zhang, Jia Li, Liyi Cai, Ge Li

llms diffusion benchmarks

arXiv AI

Sep 15

Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Models

arXiv:2509.19375v2 Announce Type: replace-cross Abstract: Large language models are increasingly used for clinical text classification, where overconfident misclassifications can directly affect pati...

By Mridul Sharma, Adeetya Patel, Zaneta D' Souza, Samira Abbasgholizadeh Rahimi, Siva Reddy, Sreenath Madathil

llms rag nlp benchmarks

arXiv AI

Sep 15

CapGeo-Bench: Decoupling Visual Perception from Reasoning and Evaluating Geometric Understanding

arXiv:2510.09302v2 Announce Type: replace-cross Abstract: While Multimodal Large Language Models (MLLMs) have achieved remarkable success in difficult purely textual mathematical reasoning tasks, eve...

By Yuying Li, Siyi Qian, Hao Liang, Leqi Zheng, Ruichuan An, Linzhuang Sun, Jiajun Zhang, Wentao Zhang

llms multimodal benchmarks

Get the daily briefing