Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 24

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

The paper introduces a generation benchmark for culturally specific kinship terms, evaluating five open‑weight large language models (LLMs) on Hindi, Tamil, and Korean. Unlike prior multiple‑choice tests that treat kinship understanding as a recognition task, the study prompts LLMs to generate terms across two communicative tasks and compares results to a matched option‑supported baseline. Findings show that while models like GPT‑OSS120B and Llama‑3.370B can select correct terms in over 90% and 78% of cases respectively, they produce the correct term only 36% and 24% of the time, indicating a significant evaluation‑format gap and highlighting the difficulty of culturally specific kinship generation even when relationships are explicitly stated.

By Sahil Pardasani, Madhusudan Singh
arXiv AI
Sep 24

An open benchmark for machine learning-based polymer property prediction

The article introduces PolyBench26, an open benchmark dataset for polymer property prediction that contains nearly 250,000 datapoints covering eight physical properties from experimental, DFT, and MD sources. It supports four evaluation tasks—property prediction, dataset-size scaling, repeat‑unit complexity, and transfer learning—across homopolymers and various copolymer architectures. The study compares language, graph, and descriptor models, finding graph-based approaches achieve the lowest errors and maintain robustness across training sizes and repeat‑unit complexity.

By Robert W. Learsch, Nicholas Liesen, Daniel S. Levine, Anna M. Hiszpanski, Evan R. Antoniuk
arXiv AI
Sep 24

When Clients Are Orchestrated: Strategic Gradient Manipulation to Defeat Federated Learning Servers with Efficient Defense

The paper introduces Fed-ADR, a coordinated attack framework where a malicious orchestrator server directs heterogeneous adversarial clients to adapt their gradient updates in real time, thereby evading existing federated learning defenses and drastically reducing global model accuracy. It also presents a lightweight detection mechanism that estimates true client gradients from historical data to spot coordinated attacks, and an in-situ recovery method that restores model performance without restarting training. Experiments on MNIST, Fashion‑MNIST, and CIFAR‑10 show the attack can drop accuracy from over 90% to below 10%, while the defense can recover accuracy to above 90% within a few rounds at a computational cost at least 20× lower than retraining from scratch.

By Mohamed Shaaban, Ahmed Abdelnaby, Mohamed Elmahallawy
arXiv AI
Sep 24

Self-Evolving Multimedia Verification through Memory Consolidation of Contestation Experiences

Self-Evolving Multimedia Verification through Memory Consolidation of Contestation Experiences (SEMV) is a multi-agent framework that uses provenance-bearing arguments to link evidence, reasoning, human contestation, and memory. It integrates arena-based quantitative bipolar argumentation, causal and scoped revision, and verification-gated memory consolidation with explicit conflict retention. On the COSMOS benchmark, SEMV achieves 91.88% accuracy, reducing negative transfer from 5.7% to 0.2%, and on the CTR benchmark it corrects 96.7% of initial errors while saving 52.8% of compute.

By Truong Thanh Hung Nguyen, Vo Thanh Khang Nguyen, Hoang-Loc Cao, Phuc Ho, Truong Thinh Nguyen, Van Pham, Hung Cao
arXiv AI
Sep 24

KATOsuper: Surrogate-accelerated neural topology optimization with sensitivity-consistent Fourier neural operators

KATOsuper is an objective‑agnostic framework that accelerates neural topology optimization by coupling neural‑reparameterized TO with a Sensitivity‑Consistent Fourier Neural Operator (SC‑FNO). It uses a forward_split architecture to ensure that sensitivities derived via automatic differentiation remain consistent with predicted objectives, enabling stable optimization. The method demonstrates significant deployment‑time speedups (15–110×) over MATLAB baselines while preserving optimality across 2D and 3D benchmark problems, including compliance and stress minimization, and supports zero‑shot extrapolation to higher resolutions.

By Shengyu Yan, Jasmin Jelovica
arXiv AI
Sep 24

Learning Spectral Allocation: A Fractional Diffusion Framework for Adaptive Volumetric Segmentation

The paper introduces FHEAT, a fractional diffusion operator derived from the discrete cosine transform, to learn how much spectral mixing each stage of a 3D medical segmentation network should perform. By reparameterizing the operator with a semigroup time, the authors enable the optimizer to decide whether global mixing is needed, resulting in a lightweight U‑shaped architecture (Light‑UNETR) paired with a Kolmogorov‑Arnold mixer (KAN3D). In semi‑supervised experiments, FHEAT‑Seg achieves state‑of‑the‑art Dice scores while dramatically reducing FLOPs through learned spectral sparsification.

By Yi-Hui Shen, Tie-Qiang Li
arXiv AI
Sep 24

Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition

Ruby-ASR introduces an evidence-preserving supervision method for Japanese automatic speech recognition that refines the conventional orthographic target into a span-bound orthographic–lexical-reading sequence. This ruby representation locally binds each written span to its spoken reading, enabling deterministic recovery of both orthographic and lexical views. Experiments on five Japanese benchmarks show that this refined target improves lexical-reading recovery while maintaining readable orthographic transcription.

By Hao Shi, Yun Liu, Xuehao Yang, Jun Liu, Chuanbo Hua, Xuanjun Chen, Lianbo Liu, Shiao Zhu, Zixiong Su
arXiv AI
Sep 24

Geometry-Conditioned Visual Place Recognition in Natural Environments

The paper introduces Depth‑Aware Distillation (DAD), a method that conditions a pretrained Vision Foundation Model’s token representations on geometry inferred by a Geometric Foundation Model, without using a depth sensor. DAD projects image‑aligned depth into the VFM token space and selectively modulates visual representations through channel‑wise geometric conditioning. On the WildCross benchmark, DAD raises average inter‑sequence Recall@1 from 61.41% to 66.37% and Recall@5 from 65.86% to 72.49%, especially improving performance under reverse traversal and long‑term appearance variation.

By Walter Nedov, Saimunur Rahman, Kavindie Katuwandeniya, David Hall, Kaushik Roy, Peyman Moghadam
arXiv AI
Sep 24

Automotive mmWave Spinning Radar Place Recognition with Spatially Gated Feature-Correlation Representation

The paper introduces SGCA‑Net, a framework for place recognition using automotive spinning FMCW radar. It combines rotation‑robust feature extraction with Spatially Gated Correlation Aggregation, which learns spatial weights to mitigate unstable radar regions while preserving informative pairwise correlations. Experiments on the MulRan dataset show SGCA‑Net outperforms state‑of‑the‑art methods in various environments, and tests on the HeRCULES dataset demonstrate its ability to generalize to unseen settings and sensors without fine‑tuning.

By Saimunur Rahman, Sagun Singh Shrestha, Abdelwahed Khamis, Peyman Moghadam
arXiv AI
Sep 24

What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit

The paper argues that apparent capability limits in vision‑language benchmarks often stem from the way answers are presented rather than from the models themselves. By comparing performance on COCO images with answer choices given as English names versus pixel coordinates, the authors show that models like Qwen3‑VL‑4B perform far better when answers are in natural language, and that the choice of answer format can swing model rankings by dozens of points. The study also demonstrates that different conventions (e.g., hue angles vs. pixel coordinates) reveal which formats a model can actually interpret, highlighting that a fixed answer vocabulary is not neutral across models.

By Alfredo F. Frontera Del Valle
arXiv AI
Sep 24

RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction

RelCheck is a training‑free post‑hoc correction pipeline that addresses relational hallucinations in multimodal large language models. It augments object‑level visual grounding with two forms of relational evidence—learned scene‑graph triples from RelTR and deterministic spatial predicates derived from bounding‑box geometry—forming a three‑layer visual knowledge base. When applied to LLaVA v1 13B, RelCheck improves the overall MME hallucination score from 585.0 to 630.0, with the most significant gain on spatial position accuracy.

By Siddhi Patil, Navrati Saxena, William B. Andreopoulos
arXiv AI
Sep 24

Schr\"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?

Schr"odinger's Repository (Schr"odingerRepo) is an evaluation framework that tests coding agents on dynamically instantiated repository representations to mitigate data leakage from static repository benchmarks. It transforms test repositories through four levels—problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting—to obscure familiar cues while preserving executable behavior. Experiments on popular LLMs using SWE-bench Verified and SWE-QA show that removing these cues consistently degrades performance and increases interaction costs, mainly due to harder repository exploration and localization.

By Silin Chen, Yufei Yang, Xiaodong Gu, Yuling Shi, Chengcheng Wan, Haibing Guan
arXiv AI
Sep 24

Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing

The paper examines whether state‑of‑the‑art large language models (LLMs) produce feedback that aligns with expert teachers’ pedagogical practices, focusing on feedback type and adaptivity. Using a refined taxonomy of seven feedback focus types, the authors annotate and compare teacher and LLM‑generated feedback from three university writing courses, creating the FeedType benchmark. Their analysis shows that while most LLMs cover many feedback types, they do not match teachers’ distribution patterns or adaptive behavior across draft stages and student performance levels.

By Norah Almousa, Shayan Peyghambari Oskoui, Raquel Coelho, Gayle Rogers, Xiang Lorraine Li, Diane Litman
arXiv AI
Sep 24

Fed-ReMasker: Federated Tabular Imputation under Feature-Level Missingness

Fed-ReMasker is a federated learning approach that adapts the ReMasker masked autoencoder for tabular data imputation, specifically addressing feature-level missingness where entire features are absent at some centers. The method enables centers to impute unobserved features by leveraging knowledge from collaborating institutions. In benchmark tests on synthetic and real-world datasets, Fed-ReMasker achieves the lowest imputation error in the majority of scenarios and remains robust to client heterogeneity, closely matching the performance of a centralized model.

By Ioannis Papathanail, Rooholla Poursoleymani, Lubnaa Abdur Rahman, Stavroula Georgia Mougiakakou
arXiv AI
Sep 24

When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

The paper introduces UECR-GRPO, a method that unifies on‑policy distillation and verifier‑based reinforcement learning for mathematical reasoning. It combines verifier rewards and teacher‑derived log‑ratios into a single KL‑regularized objective (Path‑Utility Unification) and then redistributes credit at the token level using entropy‑calibrated redistribution, preserving total task credit. Experiments on five benchmarks show that UECR‑GRPO improves average accuracy by up to 0.89 percentage points over the best baseline for both Qwen3‑1.7B and Qwen3‑4B students.

By Jie Zhang, Jingxiao Yang, Zhehao Huang, Yuhang Liu, Xiaolin Huang
arXiv AI
Sep 24

Agent-Editing World Model: Rethinking World Modeling for LLM Agents

The paper introduces the Agent-Editing World Model (AEWM), a new approach that models how reasoning and actions influence future task progress instead of simulating tool responses. AEWM includes an Action Judge that classifies decisions as Critical, Exploratory, or Noisy, and a State Revision mechanism that edits noisy reasoning–action continuations from the same observed history. The integrated system, EditAct, directly updates the underlying state during real execution, leading to significant performance gains across multiple benchmarks and agent backbones.

By Shuang Sun, Guoxin Chen, Fanzhe Meng, Jia Deng, Huatong Song, Jinhao Jiang, Wayne Xin Zhao, Hongteng Xu, Ji-Rong Wen
arXiv AI
Sep 24

Anon: Extrapolating Adaptivity Beyond SGD and Adam

The paper introduces Anon, an optimizer that extends adaptivity beyond the traditional bounds of SGD and Adam by allowing extrapolation across the entire real-number spectrum. It addresses the limitations of prior tunable optimizers that only interpolate between 0 and 1 adaptivity, showing that optimal adaptivity can require negative values for CNNs or values greater than one for Transformers. Anon incorporates Incremental Delay Update (IDU) to maintain provable stability and demonstrates competitive performance on image classification, diffusion, and large language modeling tasks.

By Yiheng Zhang, Kaiyan Zhao, Shaowu Wu, Yiming Wang, Jiajun Wu, Leong Hou U, Steve Drew, Xiaoguang Niu
arXiv Machine Learning
Sep 24

SR-Fraud: An Outcome-Supervised Reflective LLM Agent Framework for Non-Stationary Payment Fraud Detection

SR‑Fraud is a framework that uses a frozen, stateless LLM agent to score transactions in real time while an offline reflection agent proposes boundary hypotheses based on matured errors. The system then verifies these hypotheses deterministically before updating its knowledge state. On a production payment‑fraud benchmark, SR‑Fraud outperforms both static and periodically retrained CatBoost models and successfully detects an emerging fraud burst.

By Xuwei Tan, Yao Ma, Xueru Zhang
arXiv Computation and Language
Sep 24

Improving LLM-based Autonomous Web Agents with Filtering

The paper investigates how to improve large language model (LLM) based autonomous web agents by filtering irrelevant webpage content. The authors reproduce baseline models on the WebArena benchmark and identify failure modes caused by raw HTML input. They propose DeBERTa‑ and T5‑based retrieval models that rank HTML elements by relevance, fine‑tuned on Mind2Web data, and demonstrate that the DeBERTa model raises the LLaMA‑2‑70B agent’s success rate from 1.97% to 2.96%. Additionally, a zero‑shot ColBERT retriever achieves recall of 0.52 on Mind2Web and 0.47 on WebArena.

By Zhitong Guo, Jing Yu Koh, Ruiyu Li
arXiv Computer Vision
Sep 24

Beyond Balanced Accuracy: A Resolution and Parity-Controlled Benchmark for Vision-Language and Vision-Only Defect Assessment in UAV Power-Line Inspection

The paper evaluates the claim that vision‑language models (VLMs) outperform task‑specific vision backbones for UAV power‑line defect assessment using the ElecVQA‑Bench benchmark. Across various evaluation settings—partitioning, item sets, label spaces, replication, resolution, and side information—the performance gap between VLMs and traditional backbones is minimal or even reversed when controlling for resolution and token budget. The study concludes that VLM superiority is not universally supported and emphasizes the importance of rigorous benchmark audits.

By Linghao Zhang, Siyu Xiang, Junwei Kuang, Peiyu Yi