Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

Learnable Low-Rank Polynomial Sketch for Effective Linear Attention

arXiv:2610.14219v1 Announce Type: cross Abstract: Softmax attention in Transformers suffers from quadratic complexity in sequence length, making it impractical for long-context applications. Linear a...

By Zongyue Qin, Haoran Deng, Jason Cong, Yizhou Sun

llms benchmarks

arXiv AI

Oct 15

RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work

arXiv:2610.14397v2 Announce Type: cross Abstract: Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to...

By Andrew Perrault (Relantic, Inc., Columbus, OH, USA), Anmol Takiar (Relantic, Inc., Columbus, OH, USA), Dominik Winecki (Relantic, Inc., Columbus, OH, USA), Arnab Nandi (Relantic, Inc., Columbus, OH, USA)

llms agents multimodal benchmarks

arXiv AI

Oct 15

Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling

arXiv:2610.14440v1 Announce Type: cross Abstract: Asynchronous reinforcement-learning post-training of large language models is computationally efficient but inherently off-policy: the learner update...

By Shayan Mohajer Hamidi, Kayhan Behdin, Wenhui Zhu, Aida Rahmattalabi, Jincheng Cao, Jelena Markovic-Voronov, Alborz Geramifard

llms benchmarks

arXiv AI

Oct 15

Specification-Based Testing of Large Language Models

arXiv:2610.14443v1 Announce Type: cross Abstract: Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benc...

By Kausar Moshood, Jace Bolante, Roshan Aditya Murali, Manish Motwani

llms benchmarks safety

arXiv AI

Oct 15

SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning

arXiv:2610.14439v1 Announce Type: cross Abstract: Large multimodal models (LMMs) need to understand how entities move and interact as a scene develops. This requires following the same entities over...

By Chaoyue Li, Shoudong Han, Haoyang Wu, Boxue Yang, Sijia Chen, Yufan Chen, Ruping Zhang

nlp multimodal benchmarks

arXiv AI

Oct 15

RES-MR: Risk-Aware Reasoning for Explainable and Safe Medication Recommendation

arXiv:2610.14474v1 Announce Type: cross Abstract: Medication recommendations seek to deliver personalized drug combinations based on patients' clinical information, where ensuring transparent decisio...

By Cong Wang, Jin Li, Shoujin Wang, Yishuo Li, Huilin Gu, Wenpeng Lu

llms fine-tuning efficiency benchmarks safety

arXiv AI

Oct 15

How Do Uncertainty Estimators Align with Hallucination Occurrence in LLMs?

arXiv:2605.27016v2 Announce Type: replace-cross Abstract: Large language models (LLMs) can hallucinate, but do model uncertainty estimators reliably reveal when they do? Although uncertainty estimato...

By Yedidia Agnimo, Anna Korba, Annabelle Blangero, Nicolas Chesneau, Karteek Alahari

llms benchmarks safety

arXiv Computation and Language

Oct 15

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

arXiv:2601.08689v3 Announce Type: replace Abstract: Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitati...

By Zhaolu Kang, Junhao Gong, Wenqing Hu, Yingjie He, Kehan Jiang, Leqi Zheng, Shuo Yin, Zhicheng Fang, Chunlei Meng, Eric Hanchen Jiang, Yunfei Feng, Yitong Leng, Junfan Zhu, Xiaoyou Chen, Xi Yang, Richeng Xuan

llms nlp reinforcement-learning fine-tuning benchmarks

arXiv AI

Oct 15

MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision

arXiv:2610.17856v2 Announce Type: cross Abstract: Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations deriv...

By Ziyang Zhang, Shenyi Li, Yilin Wang, Ziyun Cui, Bowen Zhou, Wen Wu, Chao Zhang

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Oct 15

Bridging Academia and Industry: A Comprehensive Benchmark for Attributed Graph Clustering

arXiv:2602.08519v2 Announce Type: replace Abstract: Attributed Graph Clustering (AGC) is a fundamental unsupervised task that integrates structural topology and node attributes to uncover latent patt...

By Yunhui Liu, Pengyu Qiu, Yu Xing, Yongchao Liu, Peng Du, Chuntao Hong, Jiajun Zheng, Tao Zheng, Tieke He

computer-vision benchmarks

arXiv Machine Learning

Oct 15

Wrong Operator or Blind Design? A Post-Training Diagnostic for Physics-Informed Coefficient Learning

arXiv:2608.16925v3 Announce Type: replace Abstract: Physics-informed and hybrid models integrate data and physics, and read learned coefficients as physical. Yet a small prediction error guarantees n...

By Eric Fock

diffusion benchmarks

arXiv Computation and Language

Oct 15

Dual-Evolving Reinforcement Learning for Automated Reward Design

arXiv:2512.13399v3 Announce Type: replace-cross Abstract: Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning. Existing methods for auto...

By Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, Difan Zou

agents reinforcement-learning robotics benchmarks

arXiv AI

Oct 15

Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries

arXiv:2610.06269v2 Announce Type: replace Abstract: Open-ended scientific discovery often requires repeatedly proposing and evaluating candidate solutions. LLM-based systems can support this process...

By Chonghe Jiang, Ao Qu, Siyuan Liu, Ruoyun Ma, Zijian Zhou, Dingyi Zhuang, Bo Liu, Han Zheng, Hanfei Yu, Baichuan Mo, Jinhua Zhao, Paul Pu Liang

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 15

ASIWorld: Benchmarking Autonomous Agents in a Persistent Survival Sandbox

arXiv:2610.14807v1 Announce Type: new Abstract: Evaluating autonomous agents requires examining whether they can translate goals into coherent action across extended interactions. In persistent envir...

By Jianfeng Lin, Runze Wang, Haorong Hong, Yuxuan Xie, Zeno Zeng, Zhongyuan Liu

agents benchmarks

arXiv AI

Oct 15

Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency

The paper introduces Counterfactual Semantic Saliency (CSS), a black-box, model-agnostic method that measures how the removal of objects from a scene shifts the semantic description produced by vision‑language models (VLMs). Using CSS, the authors compare prominent VLMs to a large human psychophysics baseline (16,603 responses across 307 scenes) and find that VLMs over‑rely on large, centrally located, and highly salient objects, while under‑rely on people compared to humans. The study highlights a significant scene‑comprehension gap between VLMs and human perception, with size bias emerging as a key factor driving model‑human divergence.

By Ziqi Wen, Parsa Madinei, Miguel P. Eckstein

llms multimodal benchmarks safety

arXiv AI

Oct 15

Beyond Questions: Evaluating the Stated Knowledge of LLMs

arXiv:2605.26937v3 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge ben...

By Luca Giordano, Simon Razniewski

llms benchmarks safety

arXiv AI

Oct 15

ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts

arXiv:2610.16175v1 Announce Type: new Abstract: Real-world studies (RWS) analyze routinely collected healthcare data, such as electronic health records and insurance claims, to estimate treatment eff...

By Zhaoyu Fan, Bowen Han, Jincheng Ou, Yanwei Ren, Zehua He, Kaiyu Huang, Pengcheng He, Shanshan Ni, Chengchen Gong, Shuai Xue, Qingjiang Shi

llms agents benchmarks safety

arXiv AI

Oct 15

Interactive 4D Volumetric Liquid Forecasting under Moving-Solid Interaction

arXiv:2610.16167v1 Announce Type: new Abstract: We propose the task of Interactive 4D Volumetric Liquid Forecasting: predicting the spatiotemporal evolution of a future dense 3D liquid velocity field...

By Jongwon Ryu, Sunjae Yoon, Junyeong Kim

diffusion robotics benchmarks

arXiv AI

Oct 15

Benchmarking Quantum Kernel SVMs for Credit Card Fraud Detection: Performance Realities Under NISQ Constraints

arXiv:2610.17027v1 Announce Type: new Abstract: Detecting credit card fraud is notoriously difficult because fraudulent charges make up less than 0.2% of total transactions. Quantum Machine Learning...

By Hitanshi Arora, Mohit Swarnkar, Aswani Kumar Cherukuri

rag benchmarks

arXiv AI

Oct 15

Neos: Learning to Interleave Search and Visual Reasoning

arXiv:2610.17043v1 Announce Type: new Abstract: Recent multimodal deep search agents have advanced tool use and visual content retrieval, yet they still struggle to convert intermediate visual eviden...

By Ekko He, Jack Hong, Shilin Yan, Xintong Zhang, Yujie Zhang, Wenjie Li, Haoran Sun, Kaixun Jiang, Jiyuan Fu, Guannan Zhang, Hongwei Xue

agents reinforcement-learning fine-tuning multimodal benchmarks

Get the daily briefing