Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Machine Learning
Sep 10

CoRL: Co-Evolutionary Reinforcement Learning for Adaptive Indirect Prompt-Injection Attacks and Defenses

The paper introduces CoRL, a co-evolutionary reinforcement learning framework designed to defend against adaptive indirect prompt-injection attacks on tool-augmented language agents. CoRL operates in three stages—attacker fine‑tuning, bilateral Co‑PPO training, and defender fine‑tuning—using verifier‑grounded repairs to adapt to changing attack strategies. Experiments on 1,514 executions show that CoRL reduces attack success rates to 0% while improving task utility, demonstrating its effectiveness against adaptive adversaries.

By Boyang Zhang, Qingxin Xiao, Lingwei Dang, Qingyao Wu
arXiv Machine Learning
Sep 10

Conformalized Quantum DeepONet Ensembles: Towards Scalable Operator Learning with Distribution-Free Guarantees

The paper introduces Conformalized Quantum DeepONet Ensembles, a framework that combines Quantum Orthogonal Neural Networks (QOrthoNNs) with split conformal calibration to address the quadratic cost of dense neural layers and unreliable uncertainty quantification in operator learning. It demonstrates that QOrthoNNs achieve ×O(n) hidden‑layer running‑time scaling, improving on the ×O(n^2) cost of classical dense layers, while the ensemble approach provides a finite‑sample, distribution‑free lower bound on coverage for new input‑output function pairs. Experiments on synthetic benchmarks and real‑world power‑system dynamics confirm accurate predictions and empirical coverage close to the target under both ideal and noisy quantum simulations.

By Purav Matlia, Christian Moya, Guang Lin
arXiv Machine Learning
Sep 10

Hyperspectral Trajectory Image for Multi-Month Trajectory Anomaly Detection

The paper introduces TITAnD, a Trajectory Image Transformer that converts dense and sparse GPS trajectories into a Hyperspectral Trajectory Image (HTI) and applies vision-based classification and segmentation for anomaly detection. It employs a Cyclic Factorized Transformer (CFT) that splits attention along within-day and across-day axes, drastically reducing computational cost and enabling multi-month analysis. Empirical results show TITAnD outperforms existing sparse and dense benchmarks, achieving higher AUC-PR and faster inference than comparable Transformers.

By Md Awsafur Rahman, Chandrakanth Gudavalli, Hardik Prajapati, B. S. Manjunath
arXiv Computation and Language
Sep 10

Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements

Edu-QuRating is a pipeline that scores and curates educational data across multiple dimensions—accuracy, engagement, structure, and audience appropriateness—using an LLM judge to label document pairs and distill these preferences into reusable Edu-QuRaters. The best Edu-QuRater achieves 91.7% accuracy against held‑out GPT‑4.1‑mini judgments and is applied to filter 322.25 M FineWeb‑Edu‑Fortified documents, improving small‑model pre‑training performance on nine benchmarks. Additionally, Edu-QuRater scores serve as reward signals in GRPO post‑training, yielding responses that are preferred for pedagogical quality and instruction following over the Qwen3‑4B base model.

By Oliver G. B. Garrod, Robin A. A. Ince, Meng Liu, Mohamed Huti, Moritz Boos, Amy Waldock, Dominic Andrews, Paul Atherton
arXiv Computation and Language
Sep 10

Towards Automatic Evolution Tree Generation from Citation Graphs

The paper introduces EvoTree, a staged framework for automatically generating evolution trees from citation graphs. It separates backbone learning from temporal refinement, using a graph-aware encoder and hierarchical clustering to build a stable taxonomy, then fine-tunes temporally to attach marginal papers under monotonic-path constraints, and finally labels concepts with an LLM without changing the topology. The authors release an annotated benchmark across 11 AI subfields and report that EvoTree outperforms baselines in NMI, citation-direction accuracy, concept purity, and marginal-paper detection.

By Zexing Zhao, Yuntong Hu, Liang Zhao
arXiv Computation and Language
Sep 10

From Symbolic Perception to Logical Deduction: A Framework for Guiding Language Models in Geometric Reasoning

The paper introduces a framework that combines a Geometric Vision Parser and a Symbolic Solver to enable a Large Language Model to solve complex plane geometry problems. By translating diagrams into symbolic representations and performing formal deductions, the approach reduces hallucinations and produces interpretable, human-like solutions. Experiments on a new benchmark from 2025 Chinese Zhongkao exams show performance comparable to Gemini 2.5 Pro.

By Weichen Dai, Rafael Medeiros Cabral, Ziyi Shou, Yan Cao, Xin Shen, Dongcai Lu, Yi Zhou
arXiv Computer Vision
Sep 10

The Living Library: Transforming Archival Collections into Conversational Knowledge Systems -- Lessons from the Theodore Roosevelt Presidential Library

The Living Library is an end‑to‑end framework that converts fragmented digital archives into governed, conversational exhibit experiences. Developed at the Theodore Roosevelt Presidential Library, it digitizes a 300,000‑record collection, enriches it with OCR and metadata, and publishes it to a hybrid dense/semantic index. The system supports curator review via the Archivist App, powers a researcher interface, and runs Talk to TR—a museum exhibit where a digital human embodiment of Theodore Roosevelt answers visitors’ questions using Cross‑Era Analogical Grounding and dual‑path retrieval to keep responses grounded and responsive.

By Pengce Wang, Lucia Ronchi Darre, Matt Briney, Michaell Bakalars, Dan Rutkowski, Ursula Hardy, David Wolf, Laura Hoffman, Allen Kim, Shawn Wright, Juan Lavista Ferres
arXiv Computer Vision
Sep 10

VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning

VideoTIR introduces a reinforcement‑learning approach to improve long‑video understanding by encouraging multimodal large language models to use comprehensive multi‑level toolkits efficiently. It combines Zero‑RL and SFT cold‑starting strategies to help models retrieve and focus on meaningful video segments, images, and regions, thereby reducing hallucinations. The method includes Toolkit Action Grouped Policy Optimization (TAGPO) to streamline tool‑calling and a sandbox‑based trajectory synthesis framework for high‑quality data, achieving strong results on three long‑video QA benchmarks.

By Zhe Gao, Shiyu Shen, Taifeng Chai, Weinong Wang, Haotian Xu, Xing Wu, Wenbin Li, Qi Fan, Yang Gao, Dacheng Tao
arXiv AI
Sep 10

Towards a Resilience-Theoretic Foundation for Adversarial Robustness in Industrial Control System Anomaly Detection

The paper argues that adversarial robustness of anomaly detectors in industrial control systems (ICS) is a specific form of system resilience. It maps resilience concepts—disturbance class, absorption capacity, recovery trajectory, and degradation function—to adversarial machine learning, deriving a compositional resilience bound that identifies the coupling‑adjusted absorption capacity of nodes along an attack path as the key constraint. Empirical tests on the BATADAL water distribution benchmark reveal operationally significant effects, such as absorption‑degradation divergence under adversarial training and a paradox where hardening the most vulnerable node alone can reduce overall resilience.

By Branka Stojanovi\'c, Andreas Flatscher, Michael Somma
arXiv AI
Sep 10

LoGAN: Multilingual Font Localization with Generative Agents

LoGAN is a VLM-based agentic framework designed for few-shot multilingual font localization. It takes a handful of glyphs or logo letters and generates complete character sets across many languages, including CJK, by combining a glyph-level diffusion model, style finetuning, spacing/kerning transfer, and texture expansion. The method outperforms specialized font generators and state‑of‑the‑art image editors in glyph fidelity, style, texture, and kerning consistency on datasets covering more than 27 languages.

By Zhuoning Yuan, Ta-Ying Cheng, Benjamin Klein
arXiv AI
Sep 10

CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations

CALIPER is a new benchmark that tests whether pretrained visual encoders can infer physical properties such as mass and friction from images. The test involves striking an object twice at known speeds, showing a third strike only up to contact, and asking a linear readout on frozen features to predict how far the object slides. Results show that in clean, fixed‑camera scenes all representations perform similarly, but when camera, lighting, and clutter are varied, only encoders that truly infer physics—like V‑JEPA 2—maintain performance, while random or raw pixel representations fail.

By Aman Mehta, Riya Baviskar
arXiv AI
Sep 10

The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

The study examines how the design of audit questions influences perceived bias in large language models (LLMs). Using 40,726 requests across five models and three domains—hiring, lending, and medical triage—the authors find that demographic bias effects are not replicated when the audit is standardized. Instead, the audit’s construction, such as question phrasing and ordering, has a stronger impact on model responses than applicant demographics.

By Siddharth Vohra, Manikandan Ravikiran
arXiv AI
Sep 10

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

The paper introduces Fact-Ablated Evaluation (FAE), a framework that iteratively removes cited evidence to test whether large language models (LLMs) adjust their fact‑checking predictions accordingly. Experiments reveal that many off‑the‑shelf LLMs rely more on internal knowledge than on the provided evidence. To address this, the authors propose REAL, a training method that uses counterfactual evidence supervision to encourage LLMs to base veracity judgments on evidence, achieving better evidence‑dependent performance across four datasets.

By Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson
arXiv AI
Sep 10

Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking

The paper introduces RGBTR‑Motion, a new benchmark that synchronizes RGB, thermal, and radar data with dense moving‑instance masks and consistent identities for surveillance scenes. It also presents SAM‑Radar, a segmentation and tracking framework that fuses calibrated RGBT features with radar returns, using radar‑aware detection and motion supervision to reject clutter and maintain identity continuity during low visibility or occlusion. SAM‑Radar achieves state‑of‑the‑art performance, improving IoU, F1‑50, MOTA, HOTA, and IDF1 metrics over existing methods.

By Jue Wang, Xuan Wang, Hao Zhou, Ruixiang Zhou, Yixuan Zhou, Tianshuo Yuan, Jieming Ma, Jie Zhang, Fei Luo
arXiv AI
Sep 10

SUN: Reaching for Novelty in Reinforcement Learning

The paper introduces SUN, a reachability-aware goal-selection framework for reinforcement learning that integrates novelty and reachability using successor value functions. SUN provides theoretical guarantees, including recovery of count-based bonuses, bounds on short-horizon hitting probabilities, and rejection of unreachable goals. Empirical results show SUN consistently outperforms state-of-the-art methods across diverse environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.

By Wenyan Yang, Arsenii Mustafin, Dominik Baumann, Joni Pajarinen, Simone Parisi
arXiv Computation and Language
Sep 10

When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurors

The paper investigates how a defendant’s courtroom statement influences decisions made by large language model (LLM)–simulated jurors. Using the newly introduced JuryBench benchmark, the authors analyze 432,000 verdicts from 20 frontier LLMs, varying defendant background, emotional appeal, and rebuttal content. Results show that emotional persuasion can backfire, that jurors are harsher toward defendants from different backgrounds and more lenient toward same‑background defendants, and that juror ideology strongly shapes verdict severity.

By Cho-Ying Wu
arXiv AI
Sep 10

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.

By Shweta Mishra, Shashank Mishra
arXiv AI
Sep 10

Reasoning-Aware Compression: Identifying and Protecting Vulnerable Reasoning Circuits for Energy-Efficient LLM Deployment

The paper introduces a reasoning‑aware compression framework for Large Reasoning Models (LRMs) that identifies vulnerable reasoning circuits and selectively restores them to FP16 to avoid performance loss. By benchmarking INT4 quantization across five reasoning tasks (GSM8K, FOLIO, MATH‑500, ProofWriter, MuSiQue) and measuring GPU energy, the authors find that uniform quantization can actually increase energy consumption and that vulnerability varies by task and architecture. Selective compression yields Pareto‑optimal trade‑offs, achieving significant energy savings while improving or matching accuracy on held‑out data.

By Leonard Twagirayezu, Prasenjit Mitra
arXiv Machine Learning
Sep 10

Electricity Price Forecasting: Bridging Linear Models, Neural Networks and Online Learning

The paper presents a hybrid neural architecture that blends linear and nonlinear feed‑forward networks for day‑ahead electricity price forecasting. It introduces a partial online learning strategy with warm‑starting and stage‑specific hyperparameters to cut computational time, and employs Bernstein Online Aggregation to combine forecasts. Experiments on six years of major European markets show the method reduces RMSE by 11‑12% and MAE by 14‑17% compared to state‑of‑the‑art benchmarks while lowering computational cost.

By Btissame El Mahtout, Florian Ziel