Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 26

Guided Zero-Shot Anomaly Explanation for Anonymized Crowd Video Monitoring: Context, Categorization, and Spatial Localization

The paper introduces XAVAD, a unified approach for explainable anomaly detection in anonymized crowd video monitoring. It combines zero‑shot anonymization, anomaly type categorization, spatial localization, and scene context description using multimodal CLIP embeddings, LLM‑assisted prompting, and a prompt‑guided patching autofocus mechanism. Experiments on public CCTV datasets show XAVAD outperforms state‑of‑the‑art methods and offers insights into privacy‑explainability trade‑offs.

By Mulugeta Weldezgina Asres, Lei Jiao, Christian Walter Omlin
arXiv AI
Oct 26

ImageRSI: Verification-Driven Recursive Self-Improvement for Agentic Image Generation

ImageRSI is a framework that improves agentic image generation by keeping the generator frozen and integrating three components: a five‑dimensional verification and repair process, an experience library shared across tasks, and offline updating of the judge. Candidate experiences are added to the library only after paired A/B validation, and the judge is refined through supervised fine‑tuning and GRPO on accumulated image‑judgment records. The authors also introduce ImageReq‑Bench, a benchmark of 961 hard real‑world image‑generation requests, and report that ImageRSI achieves 52.6% Checklist Strict Accuracy on MindBench and 65.66% on ImageReq‑Bench.

By Shuo Han, Zhilin Wang, Peng Ling, Wenjia Gong, Yang Liu, Guanjun Jiang, Yukun Cao, Xike Xie
arXiv AI
Oct 26

Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting

The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.

By Jiazheng Li, Huaqing Zhang, Jing Xu, Tianle Cai, Jingzhao Zhang
arXiv AI
Oct 26

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection introduces a multi-turn reinforcement learning framework that uses a teacher model to provide Socratic feedback, guiding reasoning trajectories during rollout. This approach addresses localization ambiguity, reward sparsity, and verbosity bias in detecting hallucinations in large language models. Empirical results on the RAGTruth dataset show that SOPO (4B) outperforms leading proprietary models and standard RL baselines, achieving state‑of‑the‑art performance on multiple benchmarks.

By Jiawei Dong, Zilong Bai, Pengtian zhu, Peng Zhou
arXiv AI
Oct 26

Principia: Test-Time Hypotheses to Principle Construction for Scientific Discovery

Principia is an LLM-based framework that constructs and refines scientific principles during test-time discovery. It generates comparative causal hypotheses from related inputs and outcomes, synthesizes recurring relationships into inductive principles, and iteratively updates these principles as new observations are obtained. Across benchmarks in molecular optimization, protein engineering, gene perturbation, and molecular property prediction, Principia outperforms baselines that rely on internal knowledge, external sources, or experience summaries, producing plausible principles consistent with scientific literature.

By Dongki Kim, Sangwoo Park, Jinheon Baek, Soyeong Jeong, Sung Ju Hwang
arXiv AI
Oct 26

4DMemory: Benchmarking Long-Horizon Spatiotemporal Memory with Harness-Augmented Agents

The paper introduces 4DMemory, a benchmark for long‑horizon spatiotemporal memory that includes the LongVideo dataset—about 900 hours of video with over 5,000 manually reviewed questions—and the 4DMemoryBench. It also presents the 4DMemory Harness, a system that combines four complementary skills, more than 22 specialized tool models, and a retrieval‑augmented generation component. Evaluation of 13 leading models shows that standalone foundation models perform poorly on long‑horizon tasks, while the 4DMemory Harness boosts performance by over 50% and reveals a performance drop of more than 10% compared to previous benchmarks.

By Yanhao Jia, Yi Zhu, Yiran Zhong, Weigao Sun, Feida Zhu, Pengkun Jiao, Sihan Yang, Ming Ma, Steven Hoi, Erik Cambria
arXiv AI
Oct 26

DIGOM: Disentangled Interaction Graphs for Outcome prediction in Multimodal healthcare

DIGOM (Disentangled Interaction Graphs for Outcome prediction in Multimodal healthcare) is a graph-based framework that jointly models intra‑patient cross‑modal, cross‑patient same‑modal, and cross‑patient cross‑modal interactions. It learns a sparse multiplex graph with patient‑adaptive connectivity through a differentiable sampling mechanism, enabling end‑to‑end learning without a fixed neighbor count. Evaluated on MIMIC‑IV and NIH‑TB datasets across five clinical prediction tasks, DIGOM achieves the highest AUROC while using fewer connections than existing multi‑interaction approaches, and shows robustness to missing modalities and task‑specific connectivity patterns.

By Hakima Laribi, Olivier Lefebvre, Dan Poenaru, Martin Valli\`eres
arXiv AI
Oct 26

SP-Guide: Enhancing Aerial Vision-Language Navigation with Spatial Priors

SP-Guide is a transformer-based framework designed to improve aerial vision-language navigation by incorporating spatial priors. It introduces a Spatial Prior Cross-Modal Encoder, an Instruction Enhanced Decoder, and a Goal Convergence Filter to better align visual and linguistic information. Experiments on the CityNav benchmark show that SP-Guide increases success rates, especially in unseen environments, demonstrating the value of physically grounded spatial priors for robust navigation.

By Haoxiang Tang, Dong Zhou, Shen Tan, Minxuan Jiang, Xitong Li
arXiv AI
Oct 26

On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models

The paper introduces On-Policy Self-Training, a method that allows large language models to improve themselves without external teachers or curated datasets. By generating its own question-answer pairs and privileged reasoning demonstrations guided only by coarse topic descriptions, the model creates a synthetic curriculum that enhances performance across multiple domains. Experiments show consistent self-improvement on both text and multimodal benchmarks, including Visual Question Answering, while preserving abilities on non-self-trained domains.

By Leon Sick, Lukas Hoyer, Yannick Str\"umpler, Christian Reisswig, Timo Ropinski, Federico Tombari
arXiv AI
Oct 26

Green Code or Costly Code? Evaluating the Energy Costs of Iterative LLM-Based Code Optimizations

Large Language Models (LLMs) are increasingly used to optimize code for performance, but this study shows that such optimizations often increase energy consumption rather than reduce it. Across 11 open‑source models and 1,000 benchmark problems, inference energy ranged from 0.79 to 8.68 MJ, and 42 of 44 configurations produced code that used 4% to 886% more energy than human‑written reference solutions. Mixture‑of‑Experts architectures and token‑limit constraints can reduce inference energy, yet performance‑optimized code remains energy‑heavier, indicating that speed and sustainability are not always aligned goals.

By Sadra Emami, Benjamin Weigell, Mohammad Reza Mousavi, Gunel Jahangirova, Hugo Araujo, Bernhard Bauer
arXiv AI
Oct 26

RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

RobustRLlib is a unified library and benchmark that consolidates 17 robust reinforcement learning algorithms, providing a single interface to improve reusability. It includes an evaluation platform covering diverse real‑world shifts and advanced tasks such as humanoid and vision‑language‑action scenarios. Systematic tests across 273 perturbation configurations map the current capabilities of robust RL methods, highlighting both effective areas and remaining robustness gaps.

By Xi Chen, Fengqi Hu, Zeyu Shen, Jiaying Xu, Shangding Gu, Pan Xu, Anqi Liu, Laixi Shi
arXiv AI
Oct 26

MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph

MEDMKG is a new Medical Multimodal Knowledge Graph that combines visual data from MIMIC‑CXR with structured clinical knowledge from UMLS. The construction pipeline uses rule‑based tools and large language models for concept extraction and relationship modeling, and introduces a Neighbor‑aware Filtering algorithm to improve graph quality and compactness. Experiments on six datasets across three tasks show that MEDMKG enhances performance of twenty‑four baseline methods and four state‑of‑the‑art vision‑language backbones, providing a solid foundation for adaptive multimodal knowledge integration in medical AI.

By Xiaochen Wang, Yuan Zhong, Lingwei Zhang, Lisong Dai, Ting Wang, Fenglong Ma
arXiv AI
Oct 26

Uncertainty-Guided Latent Diagnostic Trajectory Learning for Sequential Clinical Diagnosis

The paper introduces Uncertainty-Guided Latent Diagnostic Trajectory Learning (LDTL), a framework that uses a planning LLM agent and a diagnostic LLM agent to model sequential clinical diagnosis under uncertainty. By treating diagnostic action sequences as latent paths and applying a posterior distribution that favors trajectories with higher diagnostic value, the approach encourages coherent evidence acquisition that progressively reduces uncertainty. Experiments on the MIMIC-CDM benchmark show that LDTL improves diagnostic accuracy while requiring fewer tests, and ablation studies confirm the importance of trajectory-level posterior alignment.

By Xuyang Shen, Haoran Liu, Dongjin Song, Martin Renqiang Min
arXiv AI
Oct 26

Noise-Robust Financial Numerical Entity Attribute Tagging

The paper introduces NORA, a noise‑robust tagging system for Financial Numerical Entity (FNE) attributes, addressing errors in XBRL‑derived labels and under‑emphasized attributes such as reporting‑time relation, measurement scale, and accounting sign. NORA employs task‑aware instance‑specific weighting to reduce the impact of noisy labels during training and proposes an evaluation protocol that filters noisy labels in real‑world test sets. A large benchmark of 11.4 million FNEs across 6.3 million sentences is constructed, and experiments show NORA outperforms prior models in mean Accuracy and Macro F1, especially for time‑relation prediction.

By Sheng-Hua Tsai, Chen-Yang Lai, Yi-Jhen Li, Ju-Chun Yen, Hsin-Min Lu
arXiv AI
Oct 26

Where Cross-Question Structure in LLM Errors Comes From: Shared Seeds, Shared Prompts, and Answer-Letter Preference

The study investigates why language models sometimes give consistent errors across related questions. By analyzing the correlation matrix of model responses, it finds that within a single model, independently drawn answers show no structure, but structure emerges when all questions share a random seed or few‑shot prompt. Across 24 models, the strongest additional structure is a preference for specific answer letters, especially in Llama‑3.2‑1B, and removing this effect leaves only a small excess of errors linked to a few model pairs.

By Izhar Ali
arXiv AI
Oct 26

EvoSkill v2: Fully Agentic Skill Evolution from Benchmark Adaptation to a Stateful Evolver

EvoSkill v2 is a three‑agent system that edits the instructions (skills) given to an agent while keeping its model and harness unchanged. The system comprises a benchmark adaptor that turns user‑supplied benchmarks and graders into plugins, a solver that attempts tasks, and a stateful evolver that rewrites skills each epoch based on the solver’s training rollouts. Compared to existing skill optimizers, EvoSkill v2’s evolver uses a single coding agent with a bash tool, accepts every epoch’s skills, and allows multiple attempts per training task, achieving higher held‑out pass rates and lower cost per performance gain on SpreadsheetBench v2 and $ au^3$‑bench banking. whyItMatters":"EvoSkill v2 demonstrates that a single, stateful evolver can more efficiently improve agent skills, enabling even less capable models to achieve significant performance gains in benchmark tasks."

By Dastin Huang, Baran Nama, Salaheddin Alzubi
arXiv AI
Oct 26

Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner

The paper introduces a benchmark that evaluates Vision Large Language Models (VLLMs) on table recognition tasks without training, highlighting low‑quality image input as a major challenge. To address this, the authors propose the Neighbor‑Guided Toolchain Reasoner (NGTR), which integrates lightweight visual tools and a reflection module to improve VLLM performance on unstructured tables. Experiments on public datasets show that NGTR significantly boosts the recognition capabilities of standard VLLMs.

By Yitong Zhou, Mingyue Cheng, Qingyang Mao, Jiahao Wang, Feiyang Xu, Xin Li