Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language

Oct 27

A Russian, a Brazilian, and a German Walk into a Bar, Ask the Same Question-- And Get Three Different Answers: Profiling LLM Perspective Diversity on Contested Topics

arXiv:2610.34278v1 Announce Type: new Abstract: Large Language Models(LLMs) have shown strong performance on tasks with verifiable answers, but many real-world queries involve subjectivity, disagreem...

By Wiem Ben Rim, Yao Lu, Pontus Stenetorp, Emine Yilmaz

llms benchmarks

arXiv Computation and Language

Oct 27

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

arXiv:2610.34653v1 Announce Type: new Abstract: Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete de...

By Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu, Eng Siong Chng

llms agents benchmarks safety

arXiv Computation and Language

Oct 27

From Belief Inference to Task Action: A Multi-Level Evaluation of Theory of Mind on Verifiable Tasks

arXiv:2610.34759v1 Announce Type: new Abstract: Effective collaboration requires language models to understand what others know and believe, and to act accordingly. Models that assume others share th...

By Zhenshuai Yin, Yongxin Guo, Mingda Zhang, Zhongze Luo, Haoyue Liu, Lianmin Zhang, Xiaoying Tang

llms agents benchmarks

arXiv Computation and Language

Oct 27

RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge

arXiv:2610.34875v1 Announce Type: new Abstract: Rubric-based evaluation asks which level of a rubric a response meets. It is usually done by prompting a large language model (LLM) as a judge, which n...

By Balamurugan Paramasivam (CobuildX), Sudalai Rajkumar (CobuildX)

llms rag fine-tuning benchmarks

arXiv Computation and Language

Oct 27

Rethinking Evaluation for Principled Generalization Using a Kripkean Challenge to Meaning as Use

arXiv:2610.35111v1 Announce Type: new Abstract: Large language models operationalize an intuitive, view of meaning as use. We argue that it faces a rule-following challenge. Using Kripke's interpreta...

By Niyar R Barman, Omar Naim, Nicholas Asher

llms nlp benchmarks

arXiv Computation and Language

Oct 27

PreView: Enhancing GUI Grounding with Preprocessed Views and Cues

arXiv:2610.33645v1 Announce Type: cross Abstract: GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropp...

By Wenjun Hou, Kaishuai Xu

benchmarks

arXiv Computation and Language

Oct 27

MIS-VQA: An Annotation-Driven Surgical Vision-Language Dataset for Complex Surgical Understanding

arXiv:2610.34210v1 Announce Type: cross Abstract: Modern surgical AI increasingly requires multimodal understanding rather than isolated perception, yet existing datasets remain largely limited to si...

By Mohamed Hamdy, Iheb Zouari, Fatmaelzahraa Ahmed, Muraam Abdel-Ghani, Muhammed Arsalan, Ponnuthurai Suganthan, Khalid Al-Jalham, Abdulaziz Al-Ali, Shidin Balakrishnan

llms computer-vision nlp efficiency multimodal benchmarks

arXiv Computation and Language

Oct 27

Reason or Recite: Complementary Specialists with an LLM Judge for Traffic Anomaly Reasoning

arXiv:2610.34993v1 Announce Type: cross Abstract: Road anomalies such as collisions, violations, and sudden hazards demand more from video understanding than detection: a practical system must verify...

By Mayur Kulkarni

llms fine-tuning efficiency multimodal benchmarks

arXiv Computation and Language

Oct 27

Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs

arXiv:2509.23166v4 Announce Type: replace Abstract: Large Language Models (LLMs) employ multi-turn interaction as a fundamental paradigm for completing complex tasks. However, their performance often...

By Chenxing Wei, Hong Wang, Ying He, Fei Yu, Yao Shu

llms benchmarks

arXiv Computation and Language

Oct 27

Advancing the Pareto Frontier of Test-Time Scaling with Multi-Path Plan Aggregation

arXiv:2510.11620v3 Announce Type: replace Abstract: Test-time scaling seeks to improve reasoning through additional inference-time computation, commonly using solution sampling with majority voting t...

By Siheng Xiong, Ali Payani, Faramarz Fekri

reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 27

ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction

arXiv:2605.11212v4 Announce Type: replace Abstract: Computer-use agents (CUAs) rely on visual observations of graphical user interfaces, where each screenshot is encoded into a large number of visual...

By Amirhossein Abaskohi, Yuhang He, Peter West, Giuseppe Carenini, Pranit Chawla, Vibhav Vineet

llms agents multimodal benchmarks

arXiv Computation and Language

Oct 27

Adapt or Calibrate? How Local Labels Buy Certified Coverage in Medical VQA

arXiv:2606.15910v3 Announce Type: replace Abstract: An answer from a vision-language model (VLM) to a medical image question can be accepted without review only under a threshold bounding the error o...

By Reza Khanmohammadi, Kundan Thind, Mohammad M. Ghassemi

llms multimodal benchmarks

arXiv Computation and Language

Oct 27

RIVO: Lightweight Adaptive Rerouting for Deep Research Agents

arXiv:2610.15910v2 Announce Type: replace Abstract: Deep Research agents rely on long-horizon reasoning and iterative tool use to solve complex information-seeking tasks. However, adaptive decision-m...

By Mingchen Li, Jiatan Huang, Hong Yu

agents benchmarks safety

arXiv Computer Vision

Oct 27

Intuition with Evidence: Adaptive Dual Memory for Multimodal Agents

arXiv:2610.33019v1 Announce Type: new Abstract: As multimodal agents engage in sustained interactions, long-term memory serves two distinct but complementary functions: fine-grained evidence recall a...

By Qiuhe Hong, Jingya Wang, Cuzhen Geng, Shenyao Peng, Munan Ning, Yuyang Liu, Yonghong Tian

agents multimodal benchmarks safety

arXiv Computer Vision

Oct 27

Scalable Unrolled Chains for MRI Reconstruction

arXiv:2610.33695v1 Announce Type: new Abstract: Unrolled variational networks achieve state-of-the-art performance for accelerated MRI reconstruction, but standard end-to-end training becomes prohibi...

By Emiljo M\"ehillaj, Ender Konukoglu

benchmarks

arXiv Computer Vision

Oct 27

Road Infrastructure Perception for Autonomous Driving: Taxonomy, Representations, Tasks, and Challenges

arXiv:2610.33758v1 Announce Type: new Abstract: Road infrastructure perception is a fundamental component of autonomous driving, as it recovers the structured road elements that define roadway geomet...

By Guoyang Zhao, Weiqing Qi, Kai Zhang, Zhihai Bi, Tianshuai Hu, Zeying Gong, Dezong Zhao, Jun Ma

agents robotics benchmarks

arXiv Computer Vision

Oct 27

TurboWAM: Learning What Matters for Fast World Action Models

arXiv:2610.33941v1 Announce Type: new Abstract: World action models have demonstrated promising performance in robot policy learning, yet their high inference latency remains a critical bottleneck fo...

By Guile Wu, Shuangzhi Li, Xingxin Chen, Chengjie Huang, Sam Hosseini, Tongtong Cao, Yingxue Zhang, Dongfeng Bai

robotics efficiency benchmarks

arXiv Computer Vision

Oct 27

What Sketches Tell Us about LVLMs: Conventions, Grounding, and Localisation

arXiv:2610.34037v1 Announce Type: new Abstract: LVLM benchmarks tell us whether a model answers correctly, but not which visual conventions it has retained, nor where those conventions live. Sketches...

By Chaitat Utintu, Ahmed Bourouis, Arkaprabha Basu, Yi-Zhe Song

llms fine-tuning benchmarks

arXiv Computer Vision

Oct 27

GeoPrompt: Geometry-Conditioned Prompt Learning for Underwater Open-Vocabulary Instance Segmentation

arXiv:2610.34190v1 Announce Type: new Abstract: Open-vocabulary instance segmentation (OVIS) provides a promising solution for marine perception by recognizing and segmenting categories beyond predef...

By Shanshan Han, Yuxi Wang, Can Qin, Yujie Chen, Cong Pan, Haitao Li, Guoqiang Zhong

rag computer-vision multimodal benchmarks safety

arXiv Computer Vision

Oct 27

VTC-AUDIT: An Interventional Reliability Audit for Visual-Token Compression under Paired Image Shifts

arXiv:2610.34545v1 Announce Type: new Abstract: Visual-token compression is usually evaluated by clean accuracy and efficiency, but these averages do not reveal failures that compression introduces u...

By Isaac (Rucheng), Li

llms efficiency multimodal benchmarks

Get the daily briefing