Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 2

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

The paper introduces TALON, a Task‑Adaptive LoRA‑Teacher framework for Few‑Shot Class‑Incremental Learning. TALON assigns a dedicated LoRA‑Teacher to each incremental task, then distills the frozen teachers into a single LoRA‑Student via Ensemble Knowledge Transfer, using a semantic‑guided weighting scheme to reduce forgetting and overfitting. Experiments on four FSCIL benchmarks show that TALON matches or surpasses state‑of‑the‑art accuracy while using up to 33× fewer deployment parameters and cutting inference time by 41.7%.

By Hongwei Zhao (School of Computer Science,Engineering, Beihang University), Rui Liu (School of Computer Science,Engineering, Beihang University), Yansong Liu (School of Computer Science,Engineering, Beihang University), Zhiyuan Zou (School of Computer Science,Engineering, Beihang University), Yong Chen (School of Computer Science, Beijing University of Posts,Telecommunications)
arXiv Machine Learning
Oct 2

Beyond Model Ranking: Regime Diagnosis for Distributional-Statistical Misspecification in Industrial Time-Series Forecasting

The paper introduces a regime‑diagnosis framework for industrial time‑series forecasting, highlighting that canonical loss functions embed fixed statistical priors that are violated in real‑world demand regimes such as zero‑inflation, skewness, and high variability. It proposes the Regime‑wise Relative Bias Vector (RBV) as a metric‑agnostic diagnostic that decomposes bias into an intrinsic floor and an excess attributable to training. A large‑scale study across 13 loss objectives and 60,000+ series demonstrates that regime‑aware diagnosis distinguishes optimization‑from‑bias failures and that regime‑aware training can eliminate pooling‑induced bias that mere capacity scaling cannot.

By Pengyu Nie, Chenglang Xu, Yaoshi Chen, Chaogan Ren, Wei Hu, Chao Yang, Jiangong Zhang
arXiv Computer Vision
Oct 2

Discrete Annotation, Continuous Preference: Rethinking Supervision for Accurate and Generalizable Aesthetic Image Cropping

The paper introduces a new approach to aesthetic image cropping by modeling human preference as a continuous, multi-peaked field rather than relying on discrete, grid‑based annotations. It presents the Continuous Preference Field (CPF) that reconstructs a dense preference landscape from sparse labels, and uses this to train a VLM‑based cropping model (CPIC) that achieves state‑of‑the‑art accuracy and strong out‑of‑domain generalization. Additionally, the authors propose CPICD, a recalibrated benchmark that corrects grid‑bound artifacts in existing datasets, providing a more reliable evaluation framework.

By Ziqing Zhang, Xiao Liu, Kai Liu, Jianze Li, Weihang Zhang, Linghe Kong, Yulun Zhang
arXiv Computer Vision
Oct 2

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

VTR-Bench is a new benchmark designed to evaluate how well video generation models render text within scenes. It includes 300 prompts across five real-world scenarios such as advertisements and scientific videos, and uses an automated pipeline with human alignment to assess text fidelity and scene/motion requirements. Experiments on 11 state‑of‑the‑art models show that even the best performer has a word error rate of 0.250, underscoring widespread challenges in visual text rendering.

By Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei, Song Dai, Jiayu Yang, Deyuan Liu, Xiang Zheng, Xiaoshuang Shi, Hao Cheng, Kaidi Xu
arXiv Computer Vision
Oct 2

VoxelSynth3D: Interpretable Volumetric Image-Domain Metal Artifact Reduction with a Paired Synthetic CLINIC-Metal Benchmark

VoxelSynth3D is a training‑free 3D image‑domain framework that reduces metal artifacts in postoperative musculoskeletal CT by combining support masking, normalized tissue synthesis, deviation gating, and restricted edge refinement. The method preserves implant voxels while correcting surrounding tissue artifacts and was evaluated on a newly constructed Synthetic CLINIC‑Metal benchmark, showing a reduction in RMSE from 801.48 to 786.18 HU on 40 held‑out cases. Compared to a 3D Gaussian smoother, VoxelSynth3D achieved a 13.58 HU improvement and maintained clean-edge agreement beyond 5 mm from metal.

By Amritesh Banerjee, Abdul Basit, Renil Renji Joseph, Nouhaila Innan, Muhammad Shafique
arXiv Computer Vision
Oct 2

4DMulti: automated multicomponent identification at complex material interfaces

4DMulti is a physics‑guided learning framework that automates multicomponent identification from large‑scale four‑dimensional scanning transmission electron microscopy data. It leverages a 6‑million‑pattern diffraction database, a retrieval‑conditioned latent diffusion transformer (Sim2real) for realistic pattern generation, and a rotation‑invariant convolutional network for phase classification, achieving 98.82% accuracy on a five‑phase benchmark. The method introduces a diffraction‑inferred structural complexity metric and produces single‑nanometer‑resolution structural maps of complex material interfaces such as superconducting heterostructures, corroded alloys, and degraded solid‑state battery interfaces.

By Haoran Zhang, Zian Mao, Shufen Chu, Xiaoya He, Yuyan Guan, Antong Yang, Mingze Li, Xiaoqin Zeng, Yujun Xie
arXiv AI
Oct 2

DynGhost: Temporally-Modelled Transformer for Dynamic Ghost Imagings

DynGhost is a transformer-based architecture designed for dynamic ghost imaging, addressing two key limitations of prior methods: the lack of temporal coherence exploitation and the assumption of Gaussian noise. It alternates spatial and temporal attention blocks and incorporates a quantum-aware training framework that uses realistic detector simulations (SNSPDs, SPADs, SiPMs) and Anscombe variance-stabilizing normalization to handle Poissonian noise. Experiments on multiple benchmarks show that DynGhost surpasses traditional reconstruction methods and existing deep learning models, especially in dynamic and photon-starved scenarios.

By Vittorio Palladino, Ahmet Enis Cetin
arXiv AI
Oct 2

K-Dense BYOK: An Open-Source AI Research Assistant That Runs Locally and Keeps a Hash-Chained Lab Notebook

K‑Dense BYOK is a free, open‑source AI research assistant that runs locally on a researcher’s own computer. It provides a structured environment with scientific procedures, workflow templates, and a living lab notebook that logs all actions without allowing the agent to alter the record. The system emphasizes reproducibility by recording the software environment and offering commands to regenerate results, outperforming managed platforms on interdisciplinary research prompts.

By Aubrey M. Brueckner, Darshil Patel, Yuhuan He, Timothy Kassis
arXiv AI
Oct 2

Conflicting Supervision Moves Commitment, Not Capability: A 12.29{\sigma} arrangement effect that is exactly zero under a convention-agnostic score

The paper investigates how the ordering of training data written under two incompatible but correct conventions influences a model’s learned parameters. It shows that the learning‑rate schedule acts as an averaging operator that determines the ordering effect, with constant schedules producing larger effects than decaying ones. Experiments on a single corpus with fixed budget demonstrate a statistically significant 12.29‑sigma shift in parameter commitment, yet this shift is zero when measured with a convention‑agnostic metric.

By Wenhui Chen
arXiv AI
Oct 2

Characterizing a Configuration Where Inference-Time PRM-Pruned Fragment Grafting Is Inert: Evidence from Three Reasoning LMs

The paper investigates PRM‑Pruned Fragment Grafting (PPFG), an inference‑time technique that extracts high‑reward prefixes from a pruned chain‑of‑thought and grafts them into a sibling decoding process. Experiments on Qwen2.5‑7B‑Instruct with Math‑Shepherd across 500 MATH problems and multiple seeds show that PPFG performs statistically indistinguishable from a parallel‑CoT baseline, with only 14% of grafts targeting genuinely struggling chains. The study extends across three language models, six benchmarks, and multiple PRM configurations, concluding that PPFG’s inertness is not due to heuristic specifics and providing an equivalence‑testing framework for mechanism nulls.

By Khawaja Murad ul Hassan, Mehran Ebrahimi
arXiv AI
Oct 2

Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

The study evaluates whether detailed, profession‑specific system prompts improve performance on scientific tasks. Using an open‑source corpus of 503 agent profiles and Gemini 3.8 Flash, the authors compared matched profiles to four control prompts across nine text‑based science benchmarks and a tool‑using bioinformatics benchmark. Results show no consistent accuracy gains; matched profiles actually increased token usage and cost, and in some cases reduced success rates, with only a minor advantage in one benchmark likely due to prompt length rather than domain expertise.

By Timothy Kassis
arXiv AI
Oct 2

Graph Hierarchical Recurrence for Long-Range Generalization

Graph Hierarchical Recurrence (GHR) is a new framework that enhances Graph Neural Networks and Graph Transformers by jointly processing the input graph and a pooled hierarchical abstraction. It addresses the limitation of existing models in handling predictions that depend on correlations between distant graph regions, especially under out-of-range generalization where test instances require interactions beyond training distances. Across many long-range benchmarks, GHR consistently improves performance, achieving state‑of‑the‑art or competitive results on multiple tasks.

By Stefano Carotti, Marco Pacini, Alessio Gravina, Davide Bacciu, Bruno Lepri, Sebastiano Bontorin
arXiv AI
Oct 2

What Should an Agent Remember? Disentangling Retention from Retrieval in Bounded-Memory Evaluation

The paper introduces a streaming-recall benchmark that separates retention and selection decisions for persistent agents. It shows that query‑aware selection boosts recall by 15.5 points when access is fixed, while mixed comparisons inflate gains due to changes in history access. The study finds that under bounded retention, failures stem from eviction rather than ranking errors, and that dense retrieval can outperform lexical retrieval on natural text.

By Juli Huang
arXiv AI
Oct 2

Benchmarking Prompt Optimization of Large Language Models With Chess

arXiv:2610. 00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure.

By Timoth\'ee Lesort, Alejandra L\'opez de Aberasturi G\'omez, Tristan Karch, Tom Veniat, Philippe Modard, Karl Tuyls, Ludovic Denoyer
arXiv AI
Oct 2

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

The paper "Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers" introduces SciSlopBench, a dataset of 390 AI‑generated papers paired with human‑written counterparts, and defines six measures across Structure, Argument, and Artifacts to detect scientific slop. The authors show that these measures can identify AI papers with 85.9% accuracy and that higher slop correlates with lower ICLR ratings and distinguishes rejected from accepted papers. They also propose SciSlopHarness, a framework that guides a fixed LLM to revise only evidence‑supported sections, reducing the AI‑human gap by 63% without human reference targets.

By Yerim Oh, Young-Jun Lee, Jaewoo Ahn, Gunhee Kim, Dongyeop Kang
arXiv AI
Oct 2

Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents

Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.

By Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan
arXiv AI
Oct 2

Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI

The paper introduces a pipeline that automatically creates ontology‑grounded multiple‑choice question benchmarks for evaluating large language models (LLMs) on logical reasoning tasks in scientific AI. By using OWL 2 ontologies, correct answers are guaranteed by design and distractors are generated and formally verified as incorrect through an OWL reasoner. Experiments on three ontologies—Pizza, PMDco, and DOID—yielded 112, 2,491, and 15,216 MCQs, respectively, with high natural‑language quality and challenging zero‑shot performance for six LLMs.

By Nishtha N. Vaidya, Stephan Grimm, Thomas Hubauer, Thomas A. Runkler
arXiv AI
Oct 2

Finding the Right Fit: Model-Harness Interactions across Agent Tasks

The study evaluates 66 combinations of four configurable harnesses and five language models across three benchmark suites, revealing that model rankings and best harnesses vary significantly with the task. Notably, the openJiuwen harness consistently yields the highest scores for the Kimi model, while GPT performs best with the lean PI scaffold on Terminal‑Bench 4. The findings demonstrate that a model’s performance is highly dependent on the harness it is paired with, and that higher cost or a vendor’s own harness does not guarantee superior results.

By Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng, Zhiyi Lyu, Xuyu Dong, Feng Chen, Bo An
arXiv AI
Oct 2

Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimization

The paper introduces InFlowOp, a label‑free optimization framework that assigns costs to each decision in a multi‑agent workflow, balancing agent competence against execution time. It determines task granularity and agent assignment before execution and corrects faults during execution using the same cost metric. The authors also present Braid, a benchmark for multi‑agent coordination, and show that InFlowOp outperforms single‑agent baselines by up to 11.97% across various domains.

By Xuehang Guo, Haoyu Wang, Shengyu Chen, Zach Chen, Wei Cheng, Qingyun Wang, Haifeng Chen
arXiv AI
Oct 2

Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems

The paper introduces Independent–Communicate–Revise (ICR), a framework that isolates communication effects in large language model multi‑agent systems by fixing initial reasoning and measuring how messages influence answer revision. ICR evaluates correction, preservation, and selectivity across four reasoning benchmarks, revealing that similar overall accuracy can mask divergent revision behaviors. The study shows that richer messages can both improve and harm outcomes, and that receiver policies can shift preservation and correction dynamics differently across tasks.

By Shixuan Li, Wei Yang, Peiyu Zhang, Anzhe Cheng, Heng Ping, Paul Bogdan