Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computation and Language

Sep 22

NLPCC 2026 Task 10: Citation-Level Faithfulness Verification with DeBERTa Ensembles and Class-Wise Calibration

arXiv:2609.22774v1 Announce Type: new Abstract: This paper presents our system for Track 2 of the NLPCC 2026 Shared Task 10 on citation-level faithfulness in AI-assisted scientific reporting. Given a...

By Yanling Li, Zirui Li, Mingyu Wan

llms benchmarks

arXiv Computation and Language

Sep 22

To Consolidate or not to Consolidate? Evaluating the Impact of Consolidation in Multi-Reference Training using Peer Reviews

arXiv:2609.22805v1 Announce Type: new Abstract: Natural language generation (NLG) tasks span the spectrum of conditional entropy, ranging from highly constrained machine translation to open-ended dia...

By Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Yash More, Harshit Gupta, Manav Chaudhary, Manish Shrivastava, Vasudeva Varma

nlp benchmarks

arXiv Computation and Language

Sep 22

Chronologic: Measuring Language Models' Ability to Represent the Past

arXiv:2609.23178v1 Announce Type: new Abstract: Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its response...

By Ted Underwood, Ziliang Qiu, Sarah Griebel, Laura K. Nelson, Edwin Roland, Wenyi Shang, Matthew Wilkens

llms benchmarks

arXiv Computation and Language

Sep 22

Enhancing speech representation learning with cross-modal knowledge transfer with HGNN under low resource settings: the case study of Yemba

arXiv:2609.23194v1 Announce Type: new Abstract: Acoustic representation learning is crucial for speech processing, yet low-resource languages (LRLs) face severe data scarcity, limiting the effectiven...

By Yannick Yomie Nzeuhang, Paulin Melatagia Yonta, Marie Tahon

multimodal benchmarks

arXiv Computation and Language

Sep 22

Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap

arXiv:2609.23742v1 Announce Type: new Abstract: Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, functio...

By Akash Chavan

llms agents benchmarks

arXiv Computation and Language

Sep 22

You Can Tell Who's Asking: What the Web's Questions Are Made Of, and Where They Come From

arXiv:2609.24106v1 Announce Type: new Abstract: Questions scraped from the web are used across academia and industry as a proxy for what people want to know. Across QA training data, retrieval benchm...

By Calvin Zhou, Vincent McCloskey, Krishna Srinivasan

benchmarks

arXiv Computation and Language

Sep 22

Mitigating Entity Type Confusion in Cross-Domain NER via Multidimensional Quantification and Reasoning Enhancement

arXiv:2609.24357v1 Announce Type: new Abstract: Cross-domain Named Entity Recognition (CD-NER) aims to transfer the rich knowledge in the source domain to the target domain. Recent studies adopting d...

By Jingyu Wang, Shijie Wu, Fusheng Jin

llms nlp benchmarks

arXiv Computation and Language

Sep 22

Assessing Readability with LLMs: The Role of Reasoning and Few-Shot Prompting

arXiv:2609.24650v1 Announce Type: new Abstract: Readability assessment is essential for tailoring texts to intended audiences across educational, healthcare, and information retrieval domains. Howeve...

By Rapha\"el Thieffry, Matej Martinc

llms benchmarks

arXiv Computation and Language

Sep 22

MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents

arXiv:2609.24812v1 Announce Type: new Abstract: Voice provides a natural and immediate interface for AI agents. Many settings in which voice agents could be useful, including meetings, households, an...

By Chenxu Xiong, Dongming Shen, Yuzhi Tang, Wentao Ma, Mu Li, Alex Smola

agents benchmarks

arXiv Computation and Language

Sep 22

LLM-Based FORM Code Generation with Verification-Driven Fine-Tuning

arXiv:2609.23367v1 Announce Type: cross Abstract: FORM is a domain-specific symbolic manipulation language widely used in particle physics for processing the very large algebraic expressions arising...

By Bakar Chargeishvili

llms robotics fine-tuning efficiency benchmarks

arXiv Computation and Language

Sep 22

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

arXiv:2609.23570v1 Announce Type: cross Abstract: Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluatio...

By Liyang Fan, Yingcheng Shi, Yongbin Li, Chenghao Sun, Xin Chen, Xander Xu, Hu Wei, Shiwen Ni, Min Yang, Jieping Ye

agents benchmarks

arXiv Computation and Language

Sep 22

VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencoder

arXiv:2307.00852v3 Announce Type: replace Abstract: The natural language generation domain has witnessed great success thanks to Transformer models. Although they have achieved state-of-the-art gener...

By Yueen Ma, Dafeng Chi, Jingjing Li, Kai Song, Yuzheng Zhuang, Irwin King

llms rag diffusion benchmarks

arXiv Computation and Language

Sep 22

SG-FSM: A Self-Guiding Zero-Shot Prompting Paradigm for Multi-Hop Question Answering Based on Finite State Machine

arXiv:2410.17021v2 Announce Type: replace Abstract: Large Language Models with chain-of-thought prompting, such as OpenAI-o1, have shown impressive capabilities in natural language inference tasks. H...

By Xiaochen Wang, Liang Chen, Reza Haf Zhe Yang, Yiru Wang, Xiangdi Meng, Kunhao Pan, Zhifang Sui, Junqing He

llms nlp benchmarks safety

arXiv Computation and Language

Sep 22

Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions

arXiv:2411.10163v3 Announce Type: replace Abstract: Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmark...

By Yutao Hou, Yajing Luo, Zhiwen Ruan, Hongru Wang, Weifeng Ge, Yun Chen, Guanhua Chen

llms benchmarks

arXiv Computation and Language

Sep 22

The Harder Text Embedding Benchmark (HTEB): Beyond One-dimensional Static Robustness

arXiv:2605.28190v2 Announce Type: replace Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...

By Manuel Frank, Haithem Afli

llms rag benchmarks

arXiv Computation and Language

Sep 22

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

arXiv:2606.22419v3 Announce Type: replace Abstract: A recent Nature Medicine study reports that general-purpose frontier LLMs outperform specialized retrieval-augmented clinical tools on medical benc...

By Madhulatha Mandarapu, Sandeep Kunkunuru

llms rag agents nlp benchmarks

arXiv Computation and Language

Sep 22

X-GS: An Extensible Framework for Perceiving and Thinking with 3D Gaussian Splatting

arXiv:2603.09632v5 Announce Type: replace-cross Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, subsequently extending into numerous spatial AI ap...

By Yueen Ma, Zenglin Xu, Irwin King

efficiency multimodal benchmarks

arXiv Computer Vision

Sep 22

PanoSeg3R: Feed-Forward 3D Semantic Segmentation for Panoramic Images with an Automatic Data Curation Pipeline

arXiv:2609.22687v1 Announce Type: new Abstract: We present PanoSeg3R, a feed-forward framework for 3D panoramic semantic segmentation. Unlike existing methods designed for perspective inputs, PanoSeg...

By Heechan Yoon, Dongki Jung, Phuc Nguyen, Ming Lin, Dinesh Manocha

computer-vision benchmarks

arXiv Computer Vision

Sep 22

ZIL: Zero-shot Image-to-LiDAR Registration

arXiv:2609.22716v1 Announce Type: new Abstract: Image-to-LiDAR registration estimates the camera pose of an image with respect to a LiDAR point cloud. It has diverse applications in autonomous drivin...

By Zijun Li, Xiaotian Sun, Xuelun Shen, Yao Dai, Sheng Ao, Yangyang Shi, Jakob Engel, Zhipeng Cai, Cheng Wang

llms agents robotics benchmarks

arXiv Computer Vision

Sep 22

C$^{2}$-INR: Customized Convolutional Implicit Neural Representation

arXiv:2609.22807v1 Announce Type: new Abstract: Implicit Neural Representation (INR) leverages neural networks to represent discrete signals such as images as continuous ones, where the network weigh...

By Jinglei Shi, Xinran Chang, Jiaqi Cui, Yingjie Xia, Zhaolin Xiao, Chongyi Li

benchmarks

Get the daily briefing