Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Computation and Language

Oct 8

Cache the Encoder Within:Compact, Reusable Memory across LLM Queries

arXiv:2610.10058v1 Announce Type: new Abstract: Repeated queries over shared documents incur redundant encoding, while caching model states introduces persistent storage costs. Building on CoMem's in...

By Hanzuo Liu, Chunyu Liu, Chaofan Lin, Alex Lamb, Mingyu Gao

llms efficiency benchmarks

arXiv Computation and Language

Oct 8

LLM4Impact: Integrating Heterogeneous Information for Scientific Impact Prediction

arXiv:2610.10138v1 Announce Type: new Abstract: Predicting the future impact of a newly published paper is challenging because it must be inferred from heterogeneous evidence available at publication...

By Yong Cao, Markus Flicke, Haoyu He, Katrin Renz, Andreas Geiger

llms benchmarks

arXiv Computation and Language

Oct 8

Document-Level Text Simplification in Estonian Using Large Language Models

arXiv:2610.10378v1 Announce Type: new Abstract: Document-level text simplification involves transformations that go beyond sentence-internal edits, addressing discourse coherence, anaphora resolution...

By Meeri-Ly Muru, Eduard Barbu

llms agents benchmarks

arXiv Computation and Language

Oct 8

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete r...

By Haibo Jin, Peng Kuang, Xucheng Yu, Jerry Wang, Dehao Wu, Haohan Wang

llms agents benchmarks

arXiv Computation and Language

Oct 8

Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution

arXiv:2610.09240v1 Announce Type: cross Abstract: Modern web agents built on large vision-language models process webpages, select relevant UI elements, and translate model outputs into browser actio...

By Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang, Guanhong Tao

llms agents multimodal benchmarks safety

arXiv Machine Learning

Oct 8

PatchBench: Measuring Collateral Damage in Activation Patching

PatchBench is a new benchmark designed to evaluate the collateral damage of activation patching in large language models, particularly for jailbreak repairs. It curates 400 high‑confidence jailbreak failures from 27,870 prompts across 37 datasets and introduces PatchBench‑Local, an evaluation protocol that tests whether a patch corrects harmful prompts while preserving benign ones. The study shows that aggregate metrics can miss severe local regressions, highlighting the need for more precise evaluation methods.

By Alexi Canesse, Mathis Le Bail, Ma\"el Jenny, Cl\'ement Elliker, Mahammed El Sharkawy, Sonia Vanier

llms benchmarks safety

arXiv Computation and Language

Oct 8

Boundary-Free Contextual Biasing: Depth-Adaptive Gating and Reading-Space Matching for Unsegmented Languages

arXiv:2610.09467v1 Announce Type: cross Abstract: Contextual biasing supplies an ASR system with a list of expected words at inference time, but existing methods rely on word boundaries that Japanese...

By Muhammad Huzaifah, Yu Pan, Zachary Yeo, Ningjie Bai, Guangzhao Yang

multimodal benchmarks

arXiv Computation and Language

Oct 8

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

arXiv:2610.09872v1 Announce Type: cross Abstract: Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, w...

By Jun Zhao, Leiming Fu, Yanbo Wen, Yiding Wang, Xuantong Liu, Yang Shu, Yuyang Lu, Xuanran Xing, Jingqi Tong, Hao Xu, Qi Zhang, Xuanjing Huang

llms agents benchmarks

arXiv Computation and Language

Oct 8

Inverting Multi-Vector Visual Document Indices

arXiv:2610.09920v1 Announce Type: cross Abstract: Prevailing multi-vector visual document retrievers store each page as about a thousand patch vectors, often in vector databases run by a third party....

By Zhuchenyang Liu, Yao Zhang, Yu Xiao

llms diffusion multimodal benchmarks

arXiv Computation and Language

Oct 8

Handle with CARE: Can LLMs Reproduce How Online Communities React?

arXiv:2605.27388v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as proxies for computational social analysis, yet faithfully representing the "thick description...

By Nuan Wen, Chanbin Lim, Xuezhe Ma

llms benchmarks

arXiv Computation and Language

Oct 8

Progressive Disclosure for LLM-Maintained Wiki Knowledge Bases: a Preregistered Ablation

arXiv:2607.04576v2 Announce Type: replace Abstract: LLM agents now often answer questions from knowledge bases they help maintain. A common intuition says progressive disclosure should make this chea...

By Theodore O. Cochran

llms rag agents benchmarks

arXiv Computation and Language

Oct 8

FTA-Mem: Fact-Time-Affect Anchored Memory for Low-Density Long-Term Dialogue

arXiv:2608.16303v2 Announce Type: replace Abstract: Long-term emotional-support agents require memory mechanisms for personalized understanding across sessions. However, emotional-support dialogue is...

By Chang Liu, Junyi Zhao, Shuyi Zhang, Changsheng Ma, Yongfeng Tao, Minqiang Yang, Bin Hu

agents computer-vision nlp benchmarks

arXiv Computation and Language

Oct 8

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

arXiv:2609.34798v2 Announce Type: replace Abstract: Recent medical multimodal models have benefited from larger corpora, broader modality coverage, and stronger reasoning-oriented training, yet effec...

By Guanghao Zhu, Zeyu Liu, Zhitian Hou, Pengkai Wang, Zhijie Sang, Shuo Cai, Yang Yu, Yuanyi Wang, Yanggan Gu, Congkai Xie, Jianmin Wu, Hongxia Yang

nlp reinforcement-learning fine-tuning multimodal benchmarks

arXiv Computation and Language

Oct 8

Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning

arXiv:2610.05685v2 Announce Type: replace Abstract: Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed m...

By Runguo Li

efficiency benchmarks

arXiv Computation and Language

Oct 8

SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models

arXiv:2509.14093v3 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) prompting can substantially improve the reasoning ability of large language models (LLMs), but it often comes with hig...

By Kerui Huang, Shuhan Liu, Xing Hu, Tongtong Xu, Lingfeng Bao, Xin Xia

llms benchmarks

arXiv Computation and Language

Oct 8

Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition

arXiv:2603.29217v3 Announce Type: replace-cross Abstract: Phoneme-based ASR factorizes recognition into speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G), enabling cross-lingual acoustic sharing...

By Lukuan Dong, Ziwei Li, Saierdaer Yusuyin, Xianyu Zhao, Zhijian Ou

llms multimodal benchmarks

arXiv Computation and Language

Oct 8

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

arXiv:2605.12894v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...

By Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques

llms agents fine-tuning benchmarks

arXiv Computer Vision

Oct 8

Pre-training, Reasoning, Benchmarking: X-ray Report Generation on CheXpert Plus Dataset

arXiv:2610.08813v1 Announce Type: new Abstract: X-ray image-based Radiology Report Generation (RRG) constitutes a critical research direction within medical artificial intelligence, with great potent...

By Xiao Wang, Yuxiang Zhang, Dan Xu, Yuehang Li, Shiao Wang, Bo Jiang, Yaowei Wang, Yonghong Tian, Jin Tang

llms fine-tuning multimodal benchmarks safety

arXiv Computer Vision

Oct 8

VCR-Bench: A Modular Open-Source Benchmark for Video Classification Robustness

arXiv:2610.08936v1 Announce Type: new Abstract: Robustness of image classification has several benchmarks, but their video counterparts are absent. In video classification temporal dimension introduc...

By Maksim Plinskiy, Aleksandr Gushchin, Sergey Lavrushkin, Dmitriy S. Vatolin, Anastasia Antsiferova

computer-vision benchmarks safety

arXiv Computer Vision

Oct 8

RACER: Reflective Agent Coupling Query Interpretation and Tool-Based Retrieval for Frame Selection in Long Video Understanding

arXiv:2610.08954v1 Announce Type: new Abstract: Video large language models (Vid-LLMs) excel at diverse video-language tasks by reasoning over selected frames. However, frame selection for long video...

By Yiyang Huang, Yitian Zhang, Yizhou Wang, Jianglin Lu, Qihua Dong, Hailing Wang, Huimin Zeng, Mingyuan Zhang, Yun Fu

llms rag agents benchmarks

Get the daily briefing