Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computation and Language
Sep 22

XYEval: Agents say yes to bad advice

arXiv:2609.23939v1 Announce Type: new Abstract: Effective communication between users and AI agents is essential for human-AI collaboration. The XY problem is a well-known communication pitfall where...

By Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord, Been Kim
arXiv Computation and Language
Sep 22

Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidance

arXiv:2609.24066v1 Announce Type: new Abstract: Best-of-$N$ is a widely used inference strategy for complex reasoning, whose effectiveness depends on whether sampled candidates can cover diverse and...

By Hengyuan Zhang, Chenming Shang, Zunhai Su, Xiao Liang, Hui Shen, Jing Xiong, Dawei Li, Shiping Yang, Kailai Yang, Wei Zhang, Ruobing Xie, Hayden Kwok-Hay So, Ngai Wong
arXiv Computation and Language
Sep 22

Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

arXiv:2609.24199v1 Announce Type: new Abstract: Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlle...

By Kaushal Santosh Bhogale, Srija Anand, Sadakopa Ramakrishnan Thothathiri, Tahir Javed, Sshubam Verma, Mitesh M. Khapra
arXiv Computation and Language
Sep 22

Can LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning

arXiv:2605.25920v2 Announce Type: replace Abstract: While large language models (LLMs) augmented with agentic search capabilities show promise for legal reasoning, they overlook a fundamental constra...

By Wei Fan, Yining Zhou, Mufan Zhang, Yanbing Weng, Yiran HU, Tianshi Zheng, Baixuan Xu, Chunyang Li, Jianhui Yang, Haoran Li, Yangqiu Song