Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computation and Language

Oct 15

ConRAG: Consensus-Driven Evidence Flow for Multi-Hop Question Answering

arXiv:2605.28093v3 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) has emerged as a promising paradigm for enhancing large language models (LLMs) on multi-hop question answering...

By Yikai Zhu, Kunfeng Chen, Qihuang Zhong, Juhua Liu, Bo Du

llms rag nlp benchmarks

arXiv AI

Oct 15

Sequent-Prover: Training Agents for Formal, Checkable SMT-based Reasoning

arXiv:2610.15258v1 Announce Type: cross Abstract: We introduce Sequent-Prover, a family of verifier-using language-model agents, and the post-training framework that produces them. Agents reason in a...

By Debargha Ganguly, Weicong Chen, Mohsen Hariri, Vikash Singh, Sreehari Sankar, Shouren Wang, Wang Yang, Biyao Zhang, Chaoda Song, Alexander Nemecek, Kai Ye, Osama Zafar, Erman Ayday, Joe Stubbs, Srinivasan Iyengar, Vipin Chaudhary

agents fine-tuning efficiency benchmarks

arXiv AI

Oct 15

HASTE: Hierarchical and Attribute-aware Supervision for Generative Recommendation

arXiv:2610.17368v1 Announce Type: cross Abstract: Generative recommendation models have shown promising results by representing items as semantic IDs and generating these IDs autoregressively. We int...

By Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian Anderson

llms benchmarks

arXiv AI

Oct 15

REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming

arXiv:2610.15513v1 Announce Type: cross Abstract: Safety-aligned large language models (LLMs) remain vulnerable to adversarial prompts that bypass their safety training. Existing automated red-teamin...

By Abrar Alotaibi, Moataz Ahmed

llms multimodal benchmarks safety

arXiv AI

Oct 15

REVEAL: Reverse Engineering Agentic Latents from Frontier-Model Traces

arXiv:2610.16288v2 Announce Type: new Abstract: Frontier LLM agents perform complex interactive tasks, but their inference costs motivate transferring their experience to smaller open-source agents....

By Yuetian Du, Meng Wang, Yucheng Wang, Zhangzhe Chen, Wei Wang, Boyu Yang, Weiqi Zhai, Bing Zhao, Ming Kong, Zhan Zhou, Jie Liu, Qiang Zhu

llms agents efficiency benchmarks

arXiv AI

Oct 15

RetroPlanner: Multi-Route Expansion and Multi-Objective Reasoning for LLM-Based Retrosynthesis Planning

arXiv:2610.16391v1 Announce Type: new Abstract: Multi-step retrosynthesis planning is often considered solved once a route to purchasable starting materials is found, but such a route should also be...

By Ye Rin Kim, Yongjun Jeong, Seongjun Yun, Hanbum Ko, Soohyun Oh, Sungbin Lim, Sungwoong Kim

llms benchmarks

arXiv Computer Vision

Oct 15

Improving Category-Aware Active Domain Adaptation via Uncertainty-Guided Region Coverage

arXiv:2610.19803v1 Announce Type: new Abstract: Active domain adaptation researches focus on alleviating the annotation cost, which comprises source data and target data labelling, as well as enhanci...

By My Tra Nguyen Huynh, Viet Cuong Ta

fine-tuning benchmarks

arXiv Computer Vision

Oct 15

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

arXiv:2610.18836v2 Announce Type: replace Abstract: Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using the...

By Juntong Feng, Qingan Liu, Jingru Dong, Yue Zhu, Weiming Zheng, Yujie Cheng, Xinzhe Han, Shuai Hao, Yunzhi Zhuge, Haiwen Diao, Huchuan Lu

llms robotics multimodal benchmarks

arXiv AI

Oct 15

Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference

arXiv:2610.17531v1 Announce Type: cross Abstract: While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting imp...

By Petar Basta

llms efficiency benchmarks

arXiv AI

Oct 15

CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning

arXiv:2610.17733v1 Announce Type: cross Abstract: Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large lan...

By Guangcheng Zhu, Yingfan Ma, Xing Zheng, Haobo Wang, Shenzhi Yang, Kai Tang, Zhengqing Zang, Bowen Song, Weiqiang Wang, Gang Chen

llms reinforcement-learning efficiency benchmarks

arXiv AI

Oct 15

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

arXiv:2610.16402v1 Announce Type: new Abstract: Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily...

By David Nukrai, Bar Alon, Nativ Levy, Ido Andrew Atad, Ido Cohen, Itamar Zimerman, Lior Wolf, Eli David, Ilan Naiman

agents multimodal benchmarks

arXiv Machine Learning

Oct 15

Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization

arXiv:2610.19198v1 Announce Type: cross Abstract: With the growing number of LLMs available, selecting which one to use for a prompt has become a key question. A large part of the routing literature...

By Leonardo Torti, Emanuele Rimoldi, Tomer Galanti, Tomaso Poggio, Pierfrancesco Beneventano

llms rag benchmarks

arXiv AI

Oct 15

EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

arXiv:2610.16523v1 Announce Type: new Abstract: Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on th...

By Min Cai, Dan Zhang, Gao Xing, Libin Ma, Zhaohui Xu, Xi Ye

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 15

G$^2$WAM: World Action Models for Efficient and Geometry Grounded Image-Goal Navigation

arXiv:2610.16558v1 Announce Type: new Abstract: World models capture environment dynamics and support counterfactual reasoning for planning. In image-goal navigation, where the target often lies beyo...

By Ke Gui, Chenghao Shi, Ziyue Wang, Longwen Chen, Xieyuanli Chen, Huimin Lu

agents diffusion fine-tuning benchmarks

arXiv AI

Oct 15

The Agent Knew: Internal States Forecast Unauthorized Actions and Stop Them, and the Lies That Follow

arXiv:2610.17301v1 Announce Type: new Abstract: Language-model agents sometimes act beyond their authorization and then misreport what they did. On CAPE-Bench, matched agent tasks in which one fact d...

By Arya Fayyazi, Mehdi Kamal, Massoud Pedram

llms agents benchmarks

arXiv AI

Oct 15

Unsupervised Data Drift Monitoring for Deployed Deepfake Speech Detectors

arXiv:2610.16826v1 Announce Type: cross Abstract: Deepfake speech detectors are typically evaluated as binary classifiers on fixed benchmarks, but deployed systems face evolving attacks, codecs, spea...

By Luk\'a\v{s} Proke\v{s}, Vojt\v{e}ch Stan\v{e}k, Anton Firc, Kamil Malinka

rag benchmarks

arXiv AI

Oct 15

WildTrace: Evaluating Training Data Detection in Realistic Large-Scale LLM Post-Training

arXiv:2610.16152v1 Announce Type: cross Abstract: Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination an...

By Hengxiang Zhang, Junxian Liu, Hongxin Wei

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Oct 15

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

arXiv:2610.16441v1 Announce Type: cross Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Exp...

By Subhadeep Roy, Christoph Leiter, Steffen Eger

llms fine-tuning multimodal benchmarks

arXiv AI

Oct 15

Neural Bayesian Networks for Modular and Scalable Decision-Making Under Uncertainty

arXiv:2610.16902v1 Announce Type: new Abstract: Bayesian Networks (BNs) are established tools for letting agents reason under uncertainty in sequential decision-making. They split a joint distributio...

By Giovanni Briglia, Marco Lippi, Stefano Mariani, Franco Zambonelli

agents benchmarks

arXiv Computation and Language

Oct 15

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

arXiv:2610.20172v1 Announce Type: cross Abstract: Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel...

By Puneet Mathur, Dinesh Manocha

llms agents benchmarks

Get the daily briefing