Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 5

Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents

arXiv:2610.02627v1 Announce Type: new Abstract: An email assistant should not complete less work simply because a user phrases the same request differently. Yet most benchmarks test each task with on...

By Feng Chen, Ritam Dutt, Atnaz Taheri, Alex Williams

rag agents benchmarks

arXiv AI

Oct 5

Designing the Future of User Feedback for Generative AI

arXiv:2610.02631v1 Announce Type: new Abstract: Post-deployment feedback from users can be a cost-effective, scalable, and representative means to monitor and improve generative AI systems and featur...

By Alisa Frik, Julia Bernd, Amitis Karami, Mohammad Tahaei

benchmarks

arXiv AI

Oct 5

Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

arXiv:2610.02715v1 Announce Type: new Abstract: Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses its...

By Qinchuan Cheng, Zhantao Gong, Pengzhan Sun, Angela Yao, Shijie Li

agents benchmarks

arXiv AI

Oct 5

Learning to Revise Reasoning with Segment-wise On-Policy Distillation

arXiv:2610.02703v1 Announce Type: new Abstract: On-policy distillation (OPD) improves large language model reasoning by training students on their own rollouts with dense token-wise supervision from...

By Yuxiang Zhang, Ding Cao, Shuting Cui, Lei Wang, Weijieying Ren, Tianxiang Zhao

llms efficiency benchmarks

arXiv AI

Oct 5

Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

arXiv:2610.02438v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic kn...

By Nickil Maveli, Antonio Vergari, Shay B. Cohen

llms fine-tuning benchmarks

arXiv AI

Oct 5

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

arXiv:2610.02824v1 Announce Type: new Abstract: Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requireme...

By Yuxuan Fan, Jaehong Yoon

reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Oct 5

Evaluation of Sampling Strategies and Physics-Informed Kolmogorov--Arnold Networks in Unbounded Domains

arXiv:2512.12074v2 Announce Type: replace Abstract: Physics-informed neural networks (PINNs) have emerged as an effective approach for solving partial differential equations (PDEs) by incorporating p...

By Gregorio P\'erez-Bernal, Oscar Rinc\'on-Carde\~no, Silvana Montoya-Noguera, Nicol\'as Guar\'in-Zapata

rag benchmarks

arXiv Machine Learning

Oct 5

ChronoSpike: An Adaptive Spiking Graph Neural Network for Dynamic Graphs

arXiv:2602.01124v4 Announce Type: replace Abstract: Dynamic graph representation learning requires capturing both structural relations and temporal evolution, yet existing approaches face a core trad...

By Md Abrar Jahin, Taufikur Rahman Fuad, Jay Pujara, Craig Knoblock

llms efficiency benchmarks safety

arXiv Machine Learning

Oct 5

Characterizing the Performance Gap in Human Activity Recognition for Older Adults

arXiv:2610.02711v1 Announce Type: cross Abstract: Human activity recognition (HAR) from wrist-worn accelerometers is increasingly used for health and behavioral tracking. Yet, most wearable HAR model...

By Hossein Khayami, Sungjin Hwang, Eshed Ohn-Bar, David E. Conroy, Amanda Lazar, Eun Kyoung Choe, Hernisa Kacorri

benchmarks

arXiv AI

Oct 5

OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine

arXiv:2610.02563v1 Announce Type: cross Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in...

By Eray Turkel, Mengsha Sun, Kartik Ayyar, Sean Dunigan, Jack Lu, Vlad Shcherban, Hsiang-Shun Shih, Xin Wang, Tiantian Zhang

llms agents benchmarks

arXiv Machine Learning

Oct 5

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent...

By Siyu Wang, Yifan Wang, Yuecheng He

llms benchmarks

arXiv Machine Learning

Oct 5

ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometry

arXiv:2610.03125v1 Announce Type: cross Abstract: Speech delivery varies with both the requested paralinguistic attribute and the linguistic content. We introduce ParaGeo, a matched-content decomposi...

By Yuhan Liu, Yuxuan Ou, Ruoxi Su, Mohamed Ahmed Zaki, Yunbo Long

llms benchmarks

arXiv Machine Learning

Oct 5

Does Physics Live in the Activations? Localizing Physical Quantities in Video Diffusion Models

arXiv:2610.03154v1 Announce Type: cross Abstract: Video generation models produce strikingly realistic sequences and are increasingly proposed as world models, yet recent benchmarks reveal pronounced...

By Jonas Kneifl, Jakub Skalski, Bart{\l}omiej Twardowski, Kamil Deja

llms diffusion benchmarks

arXiv Machine Learning

Oct 5

Can AI Understand the Language of Origami?

arXiv:2603.13856v3 Announce Type: replace Abstract: Building AI systems that can plan, act, and create in the physical world requires more than pattern recognition. Such systems must reason about the...

By Naaisha Agarwal, Yihan Wu, Xin Guan, Ayaan Garg, Yikuan Hu, Mohan Li, Vincenzo Collura, Wang-Zhou Dai, Yao-Xiang Ding, Emanuele Sansone

llms multimodal benchmarks

arXiv Machine Learning

Oct 5

PerturbCellRL: Aligning Distributions and Grounding Biology via Post-Training Perturbation Generators

arXiv:2606.27752v2 Announce Type: replace Abstract: Single-cell perturbation models can reduce costly wet-lab screening by predicting how cells respond transcriptionally to interventions. Recent adva...

By Dongxia Wu, Mingyu Li, Yuhui Zhang, Anurendra Kumar, Emma Lundberg, Serena Yeung-Levy, Emily B. Fox

reinforcement-learning benchmarks safety

arXiv AI

Oct 5

Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agents

arXiv:2610.02330v1 Announce Type: new Abstract: Large language models (LLMs) rely on long-horizon tool invocation sequences for complex tasks, where each invocation can alter the task state and condi...

By Yu Li, Zheng Zhang, Xin Liu, Shengtian Yang, Guangfeng Cai, Lei Feng

llms agents benchmarks

arXiv AI

Oct 5

Tailoring the Quantization Space for 1-Bit KV Cache Compression

arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...

By Minsoo Cheong, Donghyun Son, Sungjoo Yoo

llms efficiency benchmarks

arXiv AI

Oct 5

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

arXiv:2610.03084v1 Announce Type: cross Abstract: Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complem...

By Omar Elfatairy, Maria A. Bravo, Jessica Bader, Zeynep Akata

diffusion multimodal benchmarks

arXiv AI

Oct 5

WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites

arXiv:2610.03036v1 Announce Type: cross Abstract: We present WebFovea, a vision-based web agent that placed 2nd in the WebRetriever Challenge 2026 with a final score of 57.0 out of 100. The challenge...

By Jiangang Han

llms agents multimodal benchmarks safety

arXiv Computation and Language

Oct 5

Last Layer Logits to Logic: Empowering LLMs with Logic-Consistent Structured Knowledge Reasoning

arXiv:2511.07910v3 Announce Type: replace Abstract: Large Language Models (LLMs) achieve excellent performance in natural language reasoning tasks through pre-training on vast unstructured text, enab...

By Songze Li, Zhiqiang Liu, Zhaoyan Gong, Xiaoke Guo, Zhongpu Bo, Zhengke Gui, Lei Liang, Huajun Chen, Wen Zhang

llms nlp benchmarks

Get the daily briefing