Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 21

HealBench: A Fine-Grained Benchmark for Evaluating Psychological Comfort in Text-to-Image Generation

arXiv:2610.27516v1 Announce Type: cross Abstract: Despite advances in photorealism, semantic fidelity, and aesthetics, text-to-image generation and image editing remain understudied in a human-center...

By Junjie Xu, Yuzhe Yang, Rui Yang, Luwei Xiao, Wenxin Wu, Xingjiao Wu, Liang He

llms diffusion reinforcement-learning multimodal benchmarks safety

arXiv AI

Oct 21

ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values

arXiv:2610.27611v1 Announce Type: cross Abstract: Ensuring the alignment of Large Language Models (LLMs) with diverse cultural values is critical for their safe global deployment. However, existing a...

By Liping Yang, Yongqi Leng, Jingting Zheng, Yuqi Ren, Shaolin Zhu, Deyi Xiong

llms benchmarks safety

arXiv AI

Oct 21

Beyond Calibration: Temporal LLM Confidence Assessment Across Languages and Local Settings

arXiv:2610.27702v1 Announce Type: cross Abstract: Large language models are expected to estimate their own uncertainty, with confidence signals used to guide retrieval, adaptive compute allocation, r...

By Sahil Kale

llms benchmarks safety

arXiv AI

Oct 21

RELAY: Scaling Historical Build Restoration through Rule Reuse and Agentic Repair

arXiv:2610.27784v1 Announce Type: cross Abstract: Vulnerability benchmarks and software engineering research depend on rebuilding historical software. Yet builds rely on external resources, such as b...

By Olivier Nourry, Yoshiki Higo

llms agents benchmarks

arXiv AI

Oct 21

APRIL: Ambiguity-Aware Expert Routing for Respiratory Sound Domain-Incremental Learning

arXiv:2610.27942v1 Announce Type: cross Abstract: Automated respiratory sound classification enables continuous, non-invasive respiratory assessment, yet long-term deployment remains challenging due...

By Xinhong Li, Weiping Tu

fine-tuning benchmarks safety

arXiv AI

Oct 21

Auditing Where Concept Bottleneck Accuracy Comes From on CUB

arXiv:2610.28059v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) promise predictions that humans can inspect and correct through concepts such as "curved beak". Yet on CUB-200-2011,...

By Sascha Saralajew, Eleonora Poeta, Eliana Pastor, Ammar Shaker

rag benchmarks

arXiv AI

Oct 21

Africa-Papers: A 10-Billion-Token Multi-Domain Academic Corpus for African Foundation Language Models

arXiv:2610.28173v1 Announce Type: cross Abstract: Large Language Models (LLMs) depend fundamentally on the diversity, depth, and factual rigor of their pre-training corpora. However, contemporary fou...

By Mich-Seth Owusu

llms benchmarks

arXiv AI

Oct 21

JAXbounce: A Continuous-Control Pong-Like Benchmark for Active Inference Beyond Low Dimensional Environments

arXiv:2610.28203v1 Announce Type: cross Abstract: We introduce JAXbounce, a continuous control pong-like benchmark designed to evaluate active inference beyond discrete, low dimensional environments....

By Chaitanya Kaul, Markus Klar, Sebastian Stein, Fraser C. Paterson, John H. Williamson, Roderick Murray-Smith

agents benchmarks

arXiv AI

Oct 21

GALA: Geometry-Aware Language Model for Controllable Object Arrangement

arXiv:2610.28280v1 Announce Type: cross Abstract: Reasoning about fine-grained placement constraints, such as support and containment, is essential for realistic 3D scene creation. It remains a signi...

By Chucheng Xiang, Runze Wang, Zhi Deng, Ruchao Bao, Yuanwei Zhang, Yuxuan Xie, Hanliu Wang, Liangzhen Fei, Wenzheng Wu, Cheng Wan, Peifeng Li, Zhongyuan Liu, Ligang Liu

llms agents benchmarks

arXiv AI

Oct 21

ABACUS: A Benchmark for Image Editing Models to Assess Object Category and Counting Under Scene Conservation

arXiv:2610.28316v1 Announce Type: cross Abstract: Instruction-based image editing models can now add, remove a specified number of objects or change their category while preserving the scene. Yet the...

By Aymen Lassoued, Angelique Loesch, Romaric Audigier

llms multimodal benchmarks

arXiv AI

Oct 21

Context-Aware Prompt Injection Detection via Small Language Models in RAG Pipelines: nokast-secureRAG

arXiv:2610.28364v1 Announce Type: cross Abstract: Local Retrieval-Augmented Generation (RAG) systems are vulnerable to indirect prompt injection and retrieval poisoning, where malicious instructions...

By Abhishek Rai

llms rag fine-tuning efficiency benchmarks

arXiv AI

Oct 21

Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs

arXiv:2610.28369v1 Announce Type: cross Abstract: A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per t...

By Jakob Steimle, Thomas Elsken, Daniel Ehrhardt, Lukas Rinder, Daniel Cremers, Xi Wang

llms multimodal benchmarks

arXiv AI

Oct 21

VisualEvolve: A Skill Self-Evolution Framework for Visual Reasoning

arXiv:2610.28386v1 Announce Type: cross Abstract: Visual tool use enables vision-language models (VLMs) to acquire task-relevant evidence, but its effectiveness can be limited by predefined visual op...

By Jiawei Wang, Yucun Li, Xueqi Li, Shenghao Wang, Xinquan Gong, Dan Zhang, Zheqi Lv, Yuquan Le, Shaozu Yuan, An Zhang, Da Cao, Tat-Seng Chua

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Oct 21

Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation

arXiv:2610.28586v1 Announce Type: cross Abstract: Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of wher...

By Shu Zhao, Tan Yu

llms agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Oct 21

VRPAgent: LLM-Driven Discovery of Heuristic Operators for Vehicle Routing Problems

arXiv:2510.07073v2 Announce Type: replace Abstract: Designing high-performing heuristics for vehicle routing problems (VRPs) is a complex task that requires both intuition and deep domain knowledge....

By Andr\'e Hottung, Federico Berto, Chuanbo Hua, Nayeli Gast Zepeda, Daniel Wetzel, Michael R\"omer, Haoran Ye, Davide Zago, Michael Poli, Stefano Massaroli, Jinkyoo Park, Kevin Tierney

llms benchmarks

arXiv AI

Oct 21

Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

arXiv:2602.02589v2 Announce Type: replace Abstract: Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can ag...

By Yanki Margalit, Nurit Cohen-Inger, Erni Avram, Ran Taig, Seffi Cohen, Oded Margalit

llms agents benchmarks safety

arXiv AI

Oct 21

Repair the Core, Not the Symptom: Planning-Graph Repair for Long-Horizon Agents

arXiv:2607.01767v3 Announce Type: replace Abstract: Long-horizon language agents make interdependent decisions across multiple rounds, forming growing planning graphs in which later decisions depend...

By Xinyuan Song, Zekun Cai

llms agents benchmarks

arXiv AI

Oct 21

VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement

arXiv:2610.26342v2 Announce Type: replace Abstract: Recursive self-improvement (RSI) requires agents to diagnose system bottlenecks and coordinate improvements across training and inference. We refer...

By Ruotian Peng, Yuanyi Ren, Shuaiqi Liu, Xin Guo, Fang Zhu, Yangyi Huang, Yue Wu, Zheng Wang, Yandong Wen, Zhihang Fu

llms agents benchmarks

arXiv AI

Oct 21

PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models

PALATE is a new framework that improves the evaluation of deep generative models by applying the law of total expectation to accessible data. It builds on the Feature Likelihood Divergence (FLD) method but overcomes FLD’s computational limits, integrating with the MMD baseline metric and the DINOv3 feature extractor. Experiments show PALATE matches or surpasses state‑of‑the‑art methods while being more efficient and scalable, especially for detecting memorization and assessing generalization.

By Tadeusz Dziarmaga, Marcin K\k{a}dzio{\l}ka, Laura Musio{\l}, Patryk Marsza{\l}ek, Artur Kasymov, Piotr Ko\'scielniak, Marcin Mazur

nlp benchmarks

arXiv AI

Oct 21

WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

arXiv:2508.20976v2 Announce Type: replace-cross Abstract: Large audio-language models (LALMs) extend language understanding into the auditory domain, yet their ability to perform low-level listening,...

By Jaeyeon Kim, Heeseung Yun, Sang Hoon Woo, Chao-Han Huck Yang, Gunhee Kim

llms benchmarks

Get the daily briefing