← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning

Oct 27

Joint user-item allocation for e-commerce marketing campaigns

arXiv:2606.09623v2 Announce Type: replace Abstract: Web platforms run many marketing campaigns at once, and for each one they must decide which products to promote and which users to target. The best...

By Maja Lindstr\"om, Natalija Glisovic, Jan von Pichowski, Tommy L\"ofstedt, Martin Rosvall

reinforcement-learning benchmarks

arXiv Machine Learning

Oct 27

Flow-ERD: Agent-type Aware Flow Matching With Entropy-Regularized Distillation for Diverse Traffic Simulation

arXiv:2607.06957v2 Announce Type: replace-cross Abstract: Realistic and diverse traffic simulation is essential to autonomous driving development. Yet prevailing benchmarks predominantly reward reali...

By Seulbin Hwang, Kiyoung Om, Dae Jung Kim, Jinhan Lee

agents diffusion robotics efficiency multimodal benchmarks

arXiv Computation and Language

Oct 27

Linear-Time Context Compression with State-Space Encoding

arXiv:2610.33006v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compr...

By Marc Serra Ortega, Andrey Barsky, Carlos Boned Riera, Tom\'as Ockier, Dimosthenis Karatzas

llms efficiency benchmarks

arXiv Computation and Language

Oct 27

Preference Optimization Improves Definition Generation: Semantic Change in EU Institutional Sustainable Finance Disclosures

arXiv:2610.33357v1 Announce Type: new Abstract: Dictionary-like definition generators are known to be useful for lexical semantic change detection (LSCD). However, they suffer from overly granular se...

By Mariia Fedorova, Andrey Kutuzov, Gustavo Cadenas Delascio

reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 27

Perturbation-Based Data Selection for Medical Question Answering

arXiv:2610.33440v1 Announce Type: new Abstract: Selecting a compact training subset is an important step in adapting medical language models from existing question-answering banks. We investigate per...

By Yuexin Wu, Dayou Yu, Vasile Rus

llms nlp benchmarks

arXiv Computation and Language

Oct 27

Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

arXiv:2610.33741v1 Announce Type: new Abstract: LLM-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain po...

By Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev, Stepan Vysotsky, Nikolay Karpachev

llms nlp benchmarks

arXiv Computation and Language

Oct 27

Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems

arXiv:2610.33749v1 Announce Type: new Abstract: As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We crit...

By Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin

nlp benchmarks safety

arXiv Computation and Language

Oct 27

MTA-Zero: Self-Improving Multimodal Deep Search via Online QA-Solver Co-training

arXiv:2610.33799v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have become strong visual reasoners, yet remain limited in open-world, multi-step search that requires groundi...

By Xiangyu Peng, Jiaxin Zhang, An Yan, Yu Li, Ran Xu, Zeyuan Chen, Chien-Sheng Wu

llms agents multimodal benchmarks

arXiv Computation and Language

Oct 27

When Personas Become Harmful: Guiding LLMs with Global Citizenship Values Alignment

arXiv:2610.33899v1 Announce Type: new Abstract: LLMs are increasingly being used by users from diverse demographics with pluralistic values and distinct backgrounds. Maintaining safety while respecti...

By Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Rada Mihalcea

llms agents benchmarks safety

arXiv Computation and Language

Oct 27

Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

arXiv:2610.34012v1 Announce Type: new Abstract: Professionals send working documents to external large language models. A common safeguard is reversible pseudonymisation: personal data are replaced w...

By Vladimir Beskorovainyi

llms benchmarks

arXiv Computation and Language

Oct 27

The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs

arXiv:2610.34080v1 Announce Type: new Abstract: We introduce the Literature and Linguistics Analysis Benchmark (LALA), comprising 236 open-ended questions written by experts in linguistics and litera...

By Michael Foster, Barbara McGillivray, Gard B. Jenset, Jakub M\'acha, Jason Harding, Marcus Perlman, Devyani Sharma, Patrick Skeels, Jana Willer-Gold, Jeannette Littlemore

llms benchmarks

arXiv Computation and Language

Oct 27

DecisionTraceBench: Tracking Decision-Critical Information in Multi-Party Spoken Conversations

arXiv:2610.34746v1 Announce Type: new Abstract: In high-stakes domains such as finance, critical information evolves through multi-party interaction. Recovering these changes and their sources is fun...

By Wenchao Du, Fangxin Shang, Long Wu, Runze Cui, Qing Yang, Yehui Yang

arXiv Computation and Language

Oct 27

Position Matters More Than Channel: Talking a Language Model Out of a Correct Answer

arXiv:2610.34809v1 Announce Type: new Abstract: Language models now answer questions alongside other claims: a note in the system prompt, a message from another agent, their own earlier answer. We as...

By Xinyi Ni

llms agents benchmarks safety

arXiv Computation and Language

Oct 27

From Questions to Queries to Insights: A Schema-Grounded Agentic Workflow for Natural-Language Crash Data Analysis

arXiv:2610.35054v1 Announce Type: new Abstract: Transportation agencies maintain extensive crash databases to identify safety priorities, evaluate countermeasures, and guide resource allocation. Maki...

By Jingwen Zhu, Keshu Wu, Steven T. Parker, Bin Ran, David A. Noyce

rag agents benchmarks safety data-engineering

arXiv Computation and Language

Oct 27

The Lay of the Land in Idiom-Translation Quality Estimation

arXiv:2610.35261v1 Announce Type: new Abstract: Idioms have long been a challenge for machine translation, and while large language models (LLMs) have proven increasingly adept at handling them, whet...

By Malik Marmonier, Beno\^it Sagot, Rachel Bawden

llms nlp benchmarks safety

arXiv Computation and Language

Oct 27

Hierarchical Skill Evolution for Coding Agent

arXiv:2610.35616v1 Announce Type: new Abstract: Skills provide agents with reusable experience to guide future decisions without modifying the underlying model. However, coding trajectories are often...

By Jun Gao, Yun Peng, Changhai Zhou, Yuhan Zhu, Xiangyu Zeng, Shiyang Zhang Yuhua Zhou, Luozheng Qin, Qian Qiao, Xiaoteng Ma, Baian Chen Weiyi Shang, Zhen Ming Jack Jiang, Lizhi Liao, Xiaoxue Ren

llms agents benchmarks

arXiv Computation and Language

Oct 27

Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

arXiv:2610.33798v1 Announce Type: cross Abstract: An image is rarely described only once. Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still ju...

By Khang Nhat Hoang Vo, Belal Shoer, Abhirama Subramanyam Penamakuri

diffusion multimodal benchmarks

arXiv Computation and Language

Oct 27

Ranking Is Not Deciding: Benchmark Shortcuts and Operating-Point Collapse in Query-Time RAG Poisoning Defenses

arXiv:2610.33849v1 Announce Type: cross Abstract: Query-time filters that screen each retrieved document before it reaches the generator are the dominant published defense against retrieval-augmented...

By Pushkal Kumar, Arvin Sahaym

arXiv Computation and Language

Oct 27

A Domain-Aware Pipeline for Structured Extraction and Retrieval-Augmented Question Answering over arXiv Superconductivity Literature

arXiv:2610.34391v1 Announce Type: cross Abstract: Scientific literature in materials physics is growing faster than researchers can manually inspect it, yet most retrieval tools rely on keyword match...

By Ivan Iorsh, Aleksandr Kapitonov, Ivan Filianin, Matvey Lesonen, Nikita Chizhov

llms rag nlp benchmarks

arXiv Computation and Language

Oct 27

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

arXiv:2610.34756v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA...

By Fei Zuo, Yizhou Huang, Kezhi Wang

llms fine-tuning efficiency benchmarks safety