Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computation and Language

Oct 15

Anuvani-LID: Robust and Calibrated Spoken Language Identification for Indian Languages

arXiv:2610.20263v1 Announce Type: new Abstract: Every multilingual speech pipeline begins with a language decision, an error there propagates silently through recognition, translation and routing. In...

By Stephen Fernandes, Narayan Choudhary

llms multimodal benchmarks

arXiv Computation and Language

Oct 15

Cascaded Retrieval with Cross-Modal Query Refinement for Efficient Visual Document Search

arXiv:2610.20367v1 Announce Type: new Abstract: Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layou...

By Diana Turmakhan, Aseem Srivastava, Vivek Gupta, Iryna Gurevych, Thamar Solorio

multimodal benchmarks

arXiv Computation and Language

Oct 15

Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims

arXiv:2610.20457v1 Announce Type: new Abstract: Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, wr...

By Delip Rao, Ahmed Muharram, Chris Callison-Burch

llms agents benchmarks

arXiv Computation and Language

Oct 15

ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models

arXiv:2610.20503v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden featu...

By Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong, Chuan-Wei Kuo

llms efficiency benchmarks

arXiv Computation and Language

Oct 15

PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

arXiv:2610.20548v1 Announce Type: new Abstract: Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and mul...

By Ali Zamani, Soroush Esfahanian, Erfan Shahabi, Mohammad Taher Pilehvar, Azadeh Shakery, Yadollah Yaghoobzadeh

llms agents benchmarks

arXiv Computation and Language

Oct 15

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

arXiv:2610.19100v1 Announce Type: cross Abstract: Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen c...

By Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das

agents reinforcement-learning fine-tuning efficiency multimodal benchmarks

arXiv Computation and Language

Oct 15

What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning Capabilities

arXiv:2610.19476v1 Announce Type: cross Abstract: Post-training often improves the performance of large audio-language models on audio understanding and reasoning benchmarks. However, these improveme...

By Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu, Chih-Kai Yang, Yu-Xiang Lin, Sung-Feng Huang, Haibin Wu, Hung-yi Lee

llms benchmarks

arXiv Computation and Language

Oct 15

$\tau$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

arXiv:2610.19560v1 Announce Type: cross Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users....

By Rishitosh Kumar Singh, Amir Saeidi, Shamanthak Hegde, Shrinidhi Kumbhar, Chitta Baral

llms agents benchmarks

arXiv Computation and Language

Oct 15

Mitigating Video Hallucinations via Attention-Guided Representation Perturbation

arXiv:2610.19728v1 Announce Type: cross Abstract: Video large language models (Video LLMs) often generate responses unsupported by the input video. Contrastive decoding (CD) can mitigate these halluc...

By Zhixin Cai, Ruoqi Yuan, Xinchen Yang, Zhuofan Chen, Jun Bai, Chuantao Yin, Wenge Rong

llms benchmarks safety

arXiv Computation and Language

Oct 15

Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoning

arXiv:2610.19796v1 Announce Type: cross Abstract: In Theory-of-Mind (ToM) reasoning, large language models may correctly infer the agent's mental state but fail to use that inference when predicting...

By Soyoung Oh, Nursulu Sagimbayeva, Nicholas Jennings, Paulina Hussein, Mehwish Alam, Vera Demberg

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 15

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

arXiv:2610.20027v1 Announce Type: cross Abstract: Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other pa...

By Tongwei Deng, Jinyu Zhu, Yue Cao, Chenxu Wang, Yanzhen Zou, Bing Xie

llms fine-tuning benchmarks

arXiv Computation and Language

Oct 15

SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets

arXiv:2610.19869v1 Announce Type: cross Abstract: We introduce SEA-HELM Audio, a Southeast Asian (SEA) multilingual evaluation framework that extends the text-based SEA-HELM benchmark to assess perfo...

By Yixian Tan, Aulia Adila, Jonathan Heng, Kunat Pipatanakul, Donghang Wu, Anocha Sutaveephamochanon

llms agents multimodal benchmarks

arXiv Computation and Language

Oct 15

$\pi$-CoT: Prolog-Initialized Chain-of-Thought Prompting for Multi-Hop Question-Answering

arXiv:2506.20642v3 Announce Type: replace Abstract: Chain-of-Thought (CoT) prompting significantly enhances large language models' (LLMs) problem-solving capabilities, but still struggles with comple...

By Chao Wan, Albert Gong, Mihir Mishra, Carl-Leander Henneking, Claas Beger, Kilian Q. Weinberger

llms rag benchmarks

arXiv Computation and Language

Oct 15

MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs

arXiv:2601.22050v2 Announce Type: replace Abstract: In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the...

By Ghazal Kalhor, Behnam Bahrak

llms benchmarks

arXiv Computation and Language

Oct 15

ANCHOR-RE: An Agentic Biomedical Relation Extraction Framework with Retrieval Augmentation and Error-Driven Verification

arXiv:2608.03154v2 Announce Type: replace Abstract: Biomedical relation extraction (BioRE) supports knowledge graph construction and hypothesis generation. In real-world settings, most candidate enti...

By Shufan Ming, Yikun Han, Gibong Hong, Rui Zhang, Halil Kilicoglu

llms agents fine-tuning benchmarks safety

arXiv Computation and Language

Oct 15

Hybrid Gated Attention

arXiv:2608.11805v2 Announce Type: replace Abstract: Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention. To further extend its...

By Zekun Zhou, Ruobing Xie, Lanrui Wang, Weixuan Sun

benchmarks

arXiv Computation and Language

Oct 15

LycheeMemory V2: Efficient Long-Term Memory for LLM Agents via Semantic Segment-Level Consolidation

arXiv:2608.12990v2 Announce Type: replace Abstract: Long-horizon LLM agents must preserve information from past interactions to support future tasks. Existing memory systems typically rely on eager c...

By Dongfang Li, Zixuan Liu, Junmai Wang, Jiahe Huang, Fuhao Li, Lin Zhang, Bonian Jia, Baotian Hu, Min Zhang

llms agents nlp benchmarks

arXiv Computation and Language

Oct 15

Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills

arXiv:2610.14346v2 Announce Type: replace Abstract: Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this proble...

By Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee, Elena Khasanova, Quinten McNamara, Shashi Bhushan TN

llms agents benchmarks

arXiv Computation and Language

Oct 15

Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

arXiv:2603.00578v2 Announce Type: replace-cross Abstract: Long chain-of-thought~(CoT) reasoning improves the problem-solving capability of large reasoning models~(LRMs), but often incurs substantial...

By Jie Cao, Tianwei Lin, Juekai Lin, Zhenxuan Fan, Yunqi Cao, Bo Yuan, Ziyuan Zhao, Rolan Yan, Wenqiao Zhang, Siliang Tang

llms reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 15

T2J-BENCH: Benchmarking Coding Agents on Execution-Grounded PyTorch-to-JAX Repository Migration

arXiv:2605.29054v3 Announce Type: replace-cross Abstract: Migrating a machine-learning codebase across frameworks--for example from PyTorch to JAX--is a routine but costly engineering task that is fu...

By Tao Feng, Jiefeng Chen, Linxin Song, Yufan Zhuang, Yue Huang, Bhavana Dalvi Mishra, Chi Wang, Xiangru Tang, Jieyu Zhao, Ge Liu, Tomas Pfister, Jinsung Yoon

agents benchmarks

Get the daily briefing