Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computer Vision
Sep 30

Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Models

arXiv:2609.37537v1 Announce Type: new Abstract: Machine unlearning has emerged as a critical post-hoc safety measure to erase sensitive concepts from Text-to-Image (T2I) models without prohibitive re...

By Arian Komaei Koma, Seyed Amir Kasaei, Aida Aryafar, Matin Ghiasi, Ali Aghayari, Amirhossein Souri, Mohammad Mosayyebi, AmirMahdi Sadeghzadeh, Mohammad Hossein Rohban
arXiv AI
Sep 30

Multi-Site Real-World Performance of Commercial AI for Pulmonary and Incidental Pulmonary Embolism Detection

arXiv:2609.37750v1 Announce Type: cross Abstract: Pulmonary embolism (PE) is a leading cause of cardiovascular mortality, yet the real-world performance of FDA-cleared AI detection models remains inc...

By Aawez Mansuri, Mohammadreza Chavoshi, Theodorus Dapamede, Wasif Bala, Beatrice Brown-Mulry, Rohan Isaac, Bardia Khosravi, Hanzhou Li, Frank Li, John T. Moon, Chad Robichaux, Dan I. G. Cohen-Addad, Ninad V. Salastekar, Janice Newsome, Judy W. Gichoya, Hari Trivedi
arXiv Computer Vision
Sep 30

Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference

The paper introduces Q-TOFC, a query‑guided task‑oriented visual feature compression method that uses residual vector quantization to encode merged features as compact codebook index sequences. By incorporating query relevance into feature aggregation and adding a quantization error compensation adapter, Q‑TOFC reduces visual payload by 53.6% compared to previous TOFC while preserving task performance. Experiments across seven multimodal benchmarks and latency tests confirm its effectiveness under bandwidth‑constrained uplinks.

By Luning Pang, Cheng Yuan, Jiawei Shao, Mingtao Huang, Yuan Shen
arXiv AI
Sep 30

Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics

The paper benchmarks vision‑language models (VLMs) on two key tasks in connectomics: synapse detection (presence and polarity) and proofreading (split and merge errors). It evaluates 21 models (19 open, 2 closed) under zero‑shot, few‑shot, and LoRA fine‑tuning, comparing them to specialist models on datasets built from public resources. While most VLMs perform at chance in zero‑shot settings, LoRA fine‑tuning with a few thousand labels brings open models to specialist performance, and adapted VLMs outperform specialists on unseen species for merge‑error detection.

By Yicong Li, Junjie Wang, Leander Lauenburg, Ella Hugie, Alexandra Irger, Wanhua Li, Donglai Wei, Hanspeter Pfister
arXiv AI
Sep 30

Grab a Coffee: Future-Aware Guidance for Discrete Diffusion with Compiled Objectives

COFFEE is a plug‑and‑play framework that enables future‑aware guidance for discrete diffusion models by separating sequence dependence from the objective. It uses a target‑free carrier to absorb marginal token distributions and a compiled finite‑state model to capture how token combinations affect sequence‑level preferences, allowing global preferences to be transferred to unresolved positions without retraining the diffusion model. The framework supports both hard constraints and learned soft objectives and demonstrates strong control results across symbolic, language, and biological benchmarks.

By Hua (Edward), Xu, Dongxin Li, Gwen Yidou-Weng, Guy Van den Broeck, Wei Wang, Anji Liu
arXiv AI
Sep 30

Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models

The study evaluates multi‑agent debate (MAD) in small language models, testing whether cognitive diversity—via personas, sampling temperature, or model identity—drives performance gains. Across 23 models, five tasks, and over 5,500 runs, MAD consistently outperforms single‑model inference but, when matched for generation budget, it ties or falls behind self‑consistency sampling, with persona prompting actually reducing accuracy. The authors find that MAD’s benefits largely stem from the first answer exchange and that many reported gains are due to ensemble‑sampling effects rather than true diversity, highlighting the need for budget‑matched, contamination‑checked baselines. whyItMatters":"The findings clarify that MAD’s perceived advantages may be overestimated and that future debate mechanisms must be evaluated against rigorous, budget‑matched baselines to ensure genuine performance improvements."

By Leonardo Ferreira, Gardenia Liu, Kaden Zheng
arXiv AI
Sep 30

PowerZooJax: A JAX-based Power System Benchmark for Reinforcement Learning

PowerZooJax is a JAX-based benchmark suite designed for reinforcement learning in power system operation. It offers five constrained Markov decision process tasks covering generation, transmission, distribution, distributed energy resources, and data center microgrids. By implementing power flow, economic dispatch, market clearing, and device dynamics as JAX computation graphs, the entire training and evaluation loop runs on the GPU, yielding significant speedups over CPU-based simulations and enabling standardized evaluation of policy returns, safety violations, and out-of-distribution stress conditions.

By Zhanhua Pan, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu
arXiv AI
Sep 30

An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

The paper investigates how scaling a team of small language‑model agents affects performance across different orchestration architectures. By testing eight architectures on five short‑answer benchmarks and an executable‑code benchmark, it finds that team scaling yields large gains on arithmetic word‑problem tasks but only modest improvements on multiple‑choice and code generation tasks, with no single architecture dominating all tasks. The authors explain these patterns using a generate‑transform decomposition that separates coverage and transformation effects, showing that arithmetic tasks benefit from both coverage and critic‑guided transformation, while other tasks are limited by saturation or poor conversion.

By Blaz Bertalanic, Carolina Fortuna
arXiv AI
Sep 30

Principled Thoughts for Latent Recursive LLM Systems

The paper introduces REST (REpresentation‑Supervised Thoughts), a new training objective for latent recursive language‑model systems that supplements cross‑entropy loss with differentiable penalties enforcing causality, minimality, separability, and stability of internal thought representations. By applying REST to both single‑agent and multi‑agent setups without changing architectures or adding inference parameters, the authors achieve up to 7.5 percentage‑point gains in accuracy across seven diverse benchmarks and a 30 % improvement in convergence to the final answer. The method also yields more informative latent thoughts, improving interpretability of agent communication.

By Fahd Seddik, Fatemeh Fard
arXiv AI
Sep 30

HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworks

HyperZip introduces an efficient data compression framework that uses diffusion-based large language models (dLLMs) with Multi-Token Prediction to speed up compression. It addresses the trade‑off between throughput and compression rate by employing a hypernetwork that generates data‑specific updates from a context representation, allowing the dLLM to adapt to target data without costly fine‑tuning. Experiments show HyperZip outperforms state‑of‑the‑art baselines in both compression rate and speed.

By Thai Nguyen, Khang Tran, NhatHai Phan
arXiv AI
Sep 30

From Retrieval to Reasoning: Agentic Mechanism Prediction from Cell Painting Profiles

The paper introduces PhenoAIR, a reliability‑aware multi‑agent framework for predicting mechanisms of action (MOA) from Cell Painting morphological profiles. It treats retrieved neighbors as uncertain evidence, calibrating their reliability based on source, phenotype stability, and mechanism confusion, and refines predictions through controller‑guided evidence evaluation. PhenoAIR is benchmarked on JUMP Cell Painting data and outperforms both representation‑matching and LLM‑based baselines across controlled, realistic, and open‑world settings.

By Jiayuan Chen, Botao Yu, Tianyu Liu, Thai-Hoang Pham, Meng Wu, Ping Zhang
arXiv AI
Sep 30

Rethinking Reasoning Paths as Phase-Structured Trajectories

The paper proposes PAIR, a method that treats reasoning paths of large language models as phase‑structured trajectories within each question. By sampling multiple trajectories per question, aligning them to shared relative phases, and comparing successful versus unsuccessful paths only within the same phase, PAIR isolates path‑quality signals from question‑level variation. Experiments show that standard correctness probes lose predictive power under this within‑question evaluation, while PAIR improves trajectory ranking, Best‑of‑N selection, and enables phase‑wise steering of generation outcomes.

By Zhenghao He, Guangzhi Xiong, Sanchit Sinha, Bohan Liu, Wenqian Ye, Aidong Zhang
arXiv AI
Sep 30

MAADBench: The Refreshable Paradigm for Anomaly Detection in Multi-Agent Systems

MAADBench is a refreshable benchmark for anomaly detection in multi‑agent systems powered by large language models. It addresses the challenge of keeping benchmarks current by sampling and coupling generative tasks, generating trace data under configurable LLM backbones, and automatically providing deterministic step‑level labels. The authors evaluated 25 anomaly‑detection methods on 5,200 labeled traces, finding that existing approaches depend heavily on supervision, struggle with subtle MAS‑specific anomalies, and lack robustness across different LLM backbones.

By Lei Ma, Dennis Hofmann, Haowen Xu, Joshua DeOliveira, Peter VanNostrand, Lei Cao, Elke Rundensteiner
arXiv AI
Sep 30

BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning

The paper introduces BRIDGE, a bilevel optimization framework that jointly trains a large language model (LLM) and a retriever for agentic reinforcement learning (ARL). It demonstrates that adapting the retriever before the policy yields better rewards, and that BRIDGE outperforms existing methods on seven open‑domain QA benchmarks and medical QA tasks, achieving significant gains in accuracy and reasoning quality.

By Quan Xiao, Mingda Liu, Gaowen Liu, Katsuki Fujisawa, Tianyi Chen
arXiv AI
Sep 30

Neural Structural Reasoner: A Brain-inspired Architecture for Reasoning over Structured Knowledge

Neural Structural Reasoner (NSR) is a brain-inspired neural network that preserves relational structure directly in the connectivity and dynamics of coupled neuronal populations. It incorporates multi-layered architecture, stable entity representations, and path integration to perform link prediction by parallelizing over candidate relational structures and using confidence-weighted scores. NSR achieves competitive accuracy on knowledge-graph benchmarks, trains faster than several neural baselines, and offers native interpretability by tracking human-readable neuron activations and extracting latent relational hierarchies and compositional rules.

By Zixing Jia, Yuhang Pan, Ni Ji
arXiv AI
Sep 30

FairDiff: Mitigating the Self-Reinforcing Matthew Effect in Diffusion Recommender Models

FairDiff is a new fairness‑aware diffusion framework designed to mitigate the self‑reinforcing Matthew Effect in Diffusion Recommender Models (DRMs). It introduces Popularity Condition Guidance (PCG) to reweight inference‑time gradients and penalize high‑popularity items, and a Semantic Calibration (SC) module that aligns forward and reverse distributions via optimal transport. Experiments show FairDiff achieves state‑of‑the‑art performance while reducing popularity bias in DRMs.

By Song-Li Wu, Xianquan Wang, Zhaocheng Du, Weinan Gan, Jingyi Wang