Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language

Sep 30

LatCom: Cross-Agent Latent Compression for Efficient Multi-Agent Collaboration

arXiv:2609.37017v1 Announce Type: new Abstract: LLM-based multi-agent systems (MAS) increasingly use latent collaboration to avoid the information loss and repeated encoding-decoding overhead of natu...

By Shinan Zhang, Tao Zhang, Qihui Zhu, Mengjie Zhang, Dong Jin, Yunpeng Hou, Shuangwu Chen, Xiaobin Tan, Quan Zheng, Jian Yang

llms agents efficiency benchmarks

arXiv Computation and Language

Sep 30

CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence

arXiv:2609.36987v1 Announce Type: new Abstract: Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the...

By Yuzhe Zhang, Weijie Zhu, Haolin Yang, Ziyun Zhang, Xianwei Xue, Mengke Chen, Qiutong Pan, Huaqian Cai

agents fine-tuning benchmarks

arXiv Computation and Language

Sep 30

LLM unbranding: Erasing Commercial Identity while Preserving Generic Utility

arXiv:2609.37127v1 Announce Type: new Abstract: Establishing unbranding as a critical practice to prevent visual logos from acquiring negative connotations is standard in image generation. Large Lang...

By Kajetan O\.z\'og, Alicja Wojciechowska, Dawid Malarz, Pawe{\l} Batorski, Artur Kasymov, Przemys{\l}aw Spurek

llms diffusion benchmarks

arXiv Computation and Language

Sep 30

SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search

arXiv:2609.37361v1 Announce Type: new Abstract: Operations research supports decision-making in domains such as energy, economics, and healthcare. Solving operations research problems typically begin...

By Zetong Zhou, Wentao Zhang, Jingyuan Wang, Yifan Yang, Zizhuo Wang, Shixi Hu

llms reinforcement-learning benchmarks

arXiv Computation and Language

Sep 30

Who Warmed the Archives? LLMs Overestimate Historical Warmth

arXiv:2609.37499v1 Announce Type: new Abstract: Historical archives are an under-used source for extending the instrumental climate record backward in time, and LLMs offer a way to extract the indice...

By Claudiu Creanga, Liviu P. Dinu

llms fine-tuning benchmarks safety

arXiv Computation and Language

Sep 30

Devils in Question Relay: Source-Conditioned Relay Steering to Mitigate Hallucinations in Audio-visual Large Language Models

arXiv:2609.37568v1 Announce Type: new Abstract: Audio-visual large language models (AVLLMs) have made remarkable progress in multimodal understanding and reasoning through interactions among visual,...

By Yu Zhang, Pingrui Zhang, Xuefeng Bai, Pengfei Zhang, Yang Xiang, Kehai Chen

llms multimodal benchmarks safety

arXiv Computation and Language

Sep 30

Corpus-Guided Dual-Path Propagation for Graph Retrieval-Augmented Generation

arXiv:2609.37661v1 Announce Type: new Abstract: Graph-based retrieval-augmented generation supports multi-hop retrieval by organizing corpus information into graphs. However, existing relation-free g...

By Baoxian Liu, Tong Wei

rag benchmarks

arXiv Computation and Language

Sep 30

Which papyrus HTR is good enough? Character-error-rate tolerance of four papyrological tasks on Greek texts

arXiv:2609.37755v1 Announce Type: new Abstract: Purpose: Most Greek papyri remain unpublished and undigitised; a handwritten text recognition (HTR) pipeline that transcribes them automatically would...

By Anton Repushko, Elena Chepel

benchmarks

arXiv Computation and Language

Sep 30

VehicleArena: A Realistic Urban Environment for Multi-Agent Driving

arXiv:2609.35916v1 Announce Type: cross Abstract: Real-world embodied agents often pursue independent objectives within a shared physical environment, where their actions can alter the conditions fac...

By Jie Yang, Jiajun Chen, Jiazheng Zhou, Mianqiu Huang, Yining Zheng, Yuxin Wang, Xipeng Qiu

llms agents robotics benchmarks

arXiv Computation and Language

Sep 30

LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems

arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wi...

By Yun Peng, Zihan Wu, Zeyang Zhuang, Xin Zhou, Rui Shu, Xu Han, Chun Yong Chong, Yuan Wang, Jiakun Liu

agents benchmarks

arXiv Computation and Language

Sep 30

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

arXiv:2609.38177v1 Announce Type: cross Abstract: Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs...

By Jaewoo Jung, Hyeonseo Yu, Honggyu An, Jisang Han, Mungyeom Kim, Minkyeong Jeon, Heeseong Shin, Wonjun Moon, Federico Tombari, Daniel Barath, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

llms multimodal benchmarks

arXiv Computation and Language

Sep 30

POET: Preference Optimization for Enhanced Text-to-Image Generation

arXiv:2510.12041v3 Announce Type: replace Abstract: Recent advances in text-to-image (T2I) generation have achieved impressive results, yet existing models often struggle with simple or underspecifie...

By Ruibo Chen, Jiacheng Pan, Heng Huang, Zhenheng Yang

llms diffusion reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Computation and Language

Sep 30

IESR:Efficient MCTS-Based Modular Reasoning for Text-to-SQL with Large Language Models

arXiv:2602.05385v2 Announce Type: replace Abstract: Text-to-SQL is a key natural language processing task that maps natural language questions to SQL queries, enabling intuitive interaction with web-...

By Tao Liu, Jiafan Lu, Bohan Yu, Pengcheng Wu, Liu Haixin, Guoyu Xu, Li Xiangheng, Lixiao Li, Jiaming Hou, Zhao Shijun, Xinglin Lyu, Kunli Zhang, Yuxiang Jia, Hongyin Zan

llms nlp fine-tuning benchmarks data-engineering

arXiv Computation and Language

Sep 30

RAWR: Reward Assignment Without Rollouts in Verifiable Domains

arXiv:2603.17815v2 Announce Type: replace Abstract: Understanding and evaluating multi-step reasoning in LLMs at the level of individual steps remains a key challenge. Process reward models (PRMs) pr...

By Corentin Royer (International Business Machines), Anna Hedstr\"om (ETH AI Center), Debarun Bhattacharjya (Lirio), Gaetano Rossiello (International Business Machines), Andrea Giovannini (International Business Machines), Mennatallah El-Assady (Department of Computer Science, ETH Zurich)

llms benchmarks data-engineering

arXiv AI

Sep 30

MERID: Multimodal Exploration via Recursive Self-Improvement Agents for Major Depression Analysis

MERID is a framework that uses recursive self‑improvement agents to autonomously develop multimodal pipelines for detecting major depressive disorder. It aligns multimodal records with depression targets, jointly modifies representations, fusion, and predictors, and guides revisions through evidence‑guided evolution to validate improvements before inheritance. Experiments on depression benchmarks show MERID outperforms existing multimodal and agent‑based baselines, especially highlighting the importance of acoustic and linguistic cues.

By Lei Liu, Zhaokang Liang, Qingcheng Zeng, Chenda Duan, Lu Mi, Zhen Tan, Tianyu Liu

agents multimodal benchmarks

arXiv Computation and Language

Sep 30

Evaluating Cross-lingual Knowledge Consistency in Code-Mixed vis-a-vis Indian Languages using IndicKLAR

arXiv:2605.29637v2 Announce Type: replace Abstract: Large language models often exhibit a substantial gap between their performance in English and in lower-resourced languages on equivalent knowledge...

By Debajyoti Mazumder, Divyansh Pathak, Prashant Kodali, Aditya Joshi, Akshay Agarwal, Jasabanta Patro

llms benchmarks safety

arXiv Machine Learning

Sep 30

Learning Social Navigation from Internet Videos in the Policy State Space

arXiv:2609.37476v1 Announce Type: cross Abstract: Training robust social-navigation policies requires simulators with diverse scene layouts, terrain, and human motion, but constructing such environme...

By Jiaming Wang, Duc Thang Nguyen, Jizhuo Chen, Volodymyr Shcherbyna, Diwen Liu, Zhengcheng Shen, Harold Soh

robotics fine-tuning benchmarks

arXiv Computation and Language

Sep 30

OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution

arXiv:2608.00677v2 Announce Type: replace Abstract: AI agents operate in persistent environments where early state changes can influence decisions far into the future. Unlike conventional language-mo...

By Yunhao Chen, Xin Wang, Yixu Wang, Yi Liu, Jie Li, Yan Teng, Xingjun Ma, Xia Hu, Yu-Gang Jiang

agents multimodal benchmarks safety

arXiv Computation and Language

Sep 30

MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation

arXiv:2608.03275v2 Announce Type: replace Abstract: Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of large language models, but existing MoE-based PEFT methods typically improve...

By Yiming Zeng, Lei Lu, Zexin Li, Zhuochun Li, Dehai Min, Shuoqiu Li, Shuyi Liao, Xidong Wu, Zeyu Zhang, Minmei Wang, Yu Zhao, Tingting Yu, Shangqian Gao

llms fine-tuning benchmarks

arXiv Computation and Language

Sep 30

VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model, and What Limits Its Visual Grounding

arXiv:2608.08477v5 Announce Type: replace Abstract: We build VectraYX-Vision-1B, a sub-2B Spanish/LATAM cybersecurity vision-language model for offline use, and measure what limits its visual groundi...

By Juan S. Santillana

llms fine-tuning multimodal benchmarks

Get the daily briefing