← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI

Oct 1

From Charts to Code: A Hierarchical Benchmark for Multimodal Models

arXiv:2510.17932v5 Announce Type: replace-cross Abstract: We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (...

By Jiahao Tang, Henry Hengyuan Zhao, Lijian Wu, Zijian Zhang, Yifei Tao, Dongxing Mao, Yang Wan, Jingru Tan, Min Zeng, Min Li, Alex Jinpeng Wang

llms multimodal benchmarks

arXiv AI

Oct 1

Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

arXiv:2512.12870v2 Announce Type: replace-cross Abstract: Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are of...

By Pouya Ahadi, Blair Winograd, Camille Zaug, Karunesh Arora, Lijun Wang, Kamran Paynabar

arXiv AI

Oct 1

UniST-Pred: A Robust Unified Framework for Spatio-Temporal Traffic Forecasting in Transportation Networks Under Disruptions

arXiv:2602.14049v2 Announce Type: replace-cross Abstract: Spatio-temporal traffic forecasting is a core component of intelligent transportation systems, supporting various downstream tasks such as si...

By Yue Wang, Areg Karapetyan, Djellel Difallah, Samer Madanat

agents benchmarks

arXiv AI

Oct 1

When Does Self-Supervised Learning Transfer to Time-Series Tasks?

arXiv:2605.19462v2 Announce Type: replace-cross Abstract: Self-supervised learning (SSL) assumes that solving pretext tasks on unlabeled data yields representations that transfer effectively across d...

By Noam Major, Kathy Razmadze, Yoli Shavit

fine-tuning benchmarks

arXiv AI

Oct 1

GrepSeek: Training Search Agents for Direct Corpus Interaction

arXiv:2605.29307v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) search agents have shown strong promise on knowledge-intensive tasks through iterative reasoning and retrieval. Mo...

By Alireza Salemi, Chang Zeng, Atharva Nijasure, Jui-Hui Chung, Razieh Rahimi, Fernando Diaz, Hamed Zamani

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 1

SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents

arXiv:2609.35596v2 Announce Type: replace-cross Abstract: Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their con...

By Saswat Das, Parvati Viswanathan, Daniel Donnelly, Chang Huang, Sahar Abdelnabi, Ferdinando Fioretto

llms agents benchmarks safety

arXiv Computation and Language

Oct 1

4MT-VLM: How Coarse Is a VLMs Cognitive Map?

arXiv:2609.39238v1 Announce Type: new Abstract: An agent that moves must recognise a place from a viewpoint it has never seen. We introduce 4MT-VLM, a dataset of procedurally generated landscapes, ea...

By Markus Frey

llms agents multimodal benchmarks

arXiv Computation and Language

Oct 1

Marginal Response Surface Elicitation for Zero-Label Tabular Learning

arXiv:2609.39639v1 Announce Type: new Abstract: Tabular learning uses structured data to predict target outcomes. Traditionally, this process has relied on labeled data. However, large language model...

By Liangyu Teng, Yicheng Ding, Jing Liu, Hengsong Liu, Juncen Guo, Hongru Li, Jingyu Zhang, Liang Song

llms benchmarks

arXiv Computation and Language

Oct 1

Jev in Medicine: A Benchmark Evaluation. Preliminary Results

arXiv:2609.34024v1 Announce Type: cross Abstract: Jev is a non-generative "System One" model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and c...

By Alfredo Madrid-Garc\'ia, Beatriz Merino-Barbancho

llms benchmarks

arXiv Computation and Language

Oct 1

HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Control

arXiv:2609.38291v1 Announce Type: cross Abstract: Large language model (LLM) agents are vulnerable to safety risks such as injected malicious instructions or misleading information, motivating runtim...

By Zhuo Liu, Moxin Li, Zhixin Ma, Wentao Shi, Wenjie Wang, Fuli Feng

llms agents benchmarks safety

arXiv Computation and Language

Oct 1

TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

arXiv:2609.38371v1 Announce Type: cross Abstract: Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focu...

By Guangxin Zhao, Yiran Hu, Yuan Cao, Chenxi Jiang, Jianfei Yang, Yegang Du, Yasuyuki Taki, Yoshifumi Kitamura, Lin Gu, Zhi Zheng

llms robotics benchmarks safety

arXiv Computation and Language

Oct 1

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

arXiv:2609.39929v1 Announce Type: cross Abstract: Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and disting...

By Yuyang Wu, Yufeng Du, Hao Peng

llms benchmarks

arXiv Computation and Language

Oct 1

Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models

arXiv:2410.02343v2 Announce Type: replace Abstract: Large language models (LLMs) routinely fail to output the correct option in multiple-choice question answering (MCQA) while encoding the answer int...

By Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov

llms rag nlp benchmarks safety

arXiv Computation and Language

Oct 1

Lowest Span Confidence: Zero-Shot Hallucination Detection from a Single LLM Response

arXiv:2601.19918v2 Announce Type: replace Abstract: Hallucinations in Large Language Models (LLMs), i.e., plausible but non-factual generations, pose a significant challenge to reliable deployment in...

By Yitong Qiao, Licheng Pan, Yu Mi, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Fei Sun, Zhixuan Chu

llms benchmarks safety

arXiv Computation and Language

Oct 1

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

arXiv:2603.26164v2 Announce Type: replace-cross Abstract: Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters...

By Hao Liang, Zhengyang Zhao, Mingrui Chen, Meiyi Qiang, Lu Ma, Rongyi Yu, Hengyi Feng, Shixuan Sun, Zimo Meng, Xiaochen Ma, Xuanlin Yang, Qifeng Cai, Ruichuan An, Bohan Zeng, Zhen Hao Wong, Chengyu Shen, Runming He, Zhaoyang Han, Yaowei Zheng, Fangcheng Fu, Conghui He, Bin Cui, Zhiyu Li, Weinan E, Wentao Zhang

llms rag benchmarks

arXiv Computer Vision

Oct 1

ExploreNet: Learning Where to Explore in Diffusion GRPO

arXiv:2609.38329v1 Announce Type: new Abstract: Group-relative RL methods such as Flow-GRPO post-train image generators by exploring with isotropic Gaussian noise added at every denoising step. This...

By Shuyue Stella Li, Xiaochuang Han, Yulia Tsvetkov, Luke Zettlemoyer

diffusion reinforcement-learning benchmarks

arXiv Computer Vision

Oct 1

TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos

arXiv:2609.38347v1 Announce Type: new Abstract: Quantifying collective fish behavior requires accurate trajectories, yet multi-view 3D tracking remains challenging due to frequent occlusions, visuall...

By Patt Phurtivilai, Zhiyang Dou, Yifan Wu, Kinfung Chu, Yuan Liu, Lei Yang, Wenping Wang, Taku Komura

llms rag benchmarks

arXiv Computer Vision

Oct 1

MOBA-VL: Event-Localized Multi-Turn Reinforcement Learning for Real-Time MOBA Commentary

arXiv:2609.38428v1 Announce Type: new Abstract: Real-time commentary for Multiplayer Online Battle Arena (MOBA) esports requires a vision-language model (VLM) to narrate a live match second by second...

By Shengyun Zhong, Xinkang Zhao, Ziyuan Chu, Linchao Zhu

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv Computer Vision

Oct 1

PhyProbe: Rethinking Physical Consistency Evaluation in Generated Videos

arXiv:2609.38377v1 Announce Type: new Abstract: Evaluating the physical consistency of generated videos remains a fundamental challenge. Existing approaches rely on off-the-shelf vision-language mode...

By Max Ku, Jiaojiao Fan, Zekun Hao, Francesco Ferroni, Heng Wang, Wenhu Chen, Ming-Yu Liu, Prithvijit Chattopadhyay

llms fine-tuning multimodal benchmarks

arXiv Computer Vision

Oct 1

StereoGaussians: Feed-Forward 3D Gaussian Splatting from Stereo Images

arXiv:2609.38592v1 Announce Type: new Abstract: Feed-forward 3D Gaussian Splatting (3DGS) enables reconstruction without per- scene optimisation, but practical stereo-camera applications require near...

By Boyuan Tian, Huangying Zhan, Zhan Li, Shin-Fang Chng, Hanwen Yang, Zirui Wang, Yi Xu