Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 16

Hierarchical Graph Convolutional Network Approach for Detecting Low-Quality Documents

The paper introduces a hierarchical graph convolutional network (HGCN) designed to detect low‑quality documents by identifying internal inconsistencies and mismatches between titles and bodies. The authors created the Inconsistency Dataset from news data and metadata to train the model, achieving 91.20% accuracy on this dataset and 92.00% on a public incongruent dataset. An ablation study shows that incorporating metadata significantly improves performance, suggesting the model’s broad applicability for filtering low‑quality content.

By Jaeyoung Lee, Joonwon Jang, Misuk Kim
arXiv AI
Oct 16

Learning to Reason with Selective Internal Feedback

The paper introduces SIF, a reinforcement learning technique that uses selective internal feedback to improve language models’ reasoning without external verifiers. SIF constructs comparative rewards by comparing token‑level self‑certainty across multiple windows and peers, rewarding only when both voting levels agree, and applies sample and group‑level filtering to decide which responses influence policy updates. Experiments on three backbone models across 12 mathematical and scientific benchmarks show that SIF’s final checkpoints outperform the best‑validation checkpoints of Intuitor and Vigor, achieving a 4.71‑point accuracy gain on Qwen2.5‑Math‑1.5B and maintaining accuracy during continued optimization with minimal test‑time rollouts.

By Xun Lu, Shumeng Yang, Xinrui Wang, Songjun Tu, Linjing Li, Daniel Dajun Zeng
arXiv AI
Oct 16

Watts If? A Benchmark for Counterfactual Reasoning About the AI Data Center Buildout

The paper introduces DCBench-440, a benchmark for evaluating large language models on counterfactual reasoning about AI data center buildouts. It comprises two parts: 200 generative STRIPS‑world questions covering five U.S. deal archetypes, and 240 multiple‑choice questions grounded in real site records, emission equations, and scenarios. Model performance varies widely, with generative accuracy ranging from 12.5 % to 71.5 % and multiple‑choice accuracy from 58.3 % to 92.5 %, while models struggle to list all possible actions at decision gates.

By Pankaj Chouhan, Rajarshi Das, Aditya Dubey, Manminder Tomar, Sreekar Reddy, Samta Sharma, Satyapragnya Kar, PR Panigrahi
arXiv AI
Oct 16

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

JevAgentHarness is a new framework that benchmarks, deploys, and trains decision models to share control with large language models (LLMs) in agentic tasks. It introduces AgentDecisionBench to evaluate decision models from isolated questions to full agent control, revealing that accurate isolated decisions still miss many tasks and that naïve delegation can reduce success. By fitting confidence thresholds per choice and training a 4B decision model on states after agent mistakes, JevAgentHarness achieves high task success while cutting cost and speeding up execution, with results and resources made publicly available.

By Kunlun Zhu, Tao Feng, Xu Han, Heng Wang, Ling Yang, Xiangru Tang, Jiaxuan You, Heng Ji
arXiv AI
Oct 16

ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing

ClaimScope is a 72‑task benchmark that audits mathematical claims by requiring a verdict, scope, counterexample, and repair, covering areas such as optimizer‑state transport and exact $2 imes2$ linear algebra. The study evaluates Qwen3‑8B using a witness checker that only examines the model’s submitted state, showing that verifier feedback improves counterexample acceptance and safe completions compared to generic feedback. The results demonstrate that separating claim classification from executable evidence can yield gains in witness‑structure correction without necessarily improving broader reasoning.

By Mushir Akhtar
arXiv AI
Oct 16

Data World: Scaling Data Environments from Scratch

Data World proposes a multi‑agent engine that generates synthetic data environments, tasks, and evaluation rubrics by first specifying the underlying world rather than curating real files. The system produces thousands of training tasks across diverse domains and enables independent audits of target recoverability, while human curation ensures realistic question selection. Benchmark results show that models trained on these synthesized tasks still face significant challenges, yet RL‑zero training on the synthetic data yields measurable performance gains over baseline models.

By Zhanli Li, Zeyuan Xu, Chaoxu Pang, Zeyuan Liu, Anyang Ji, Te Sun, Kongcheng Zhang, Wenbo Xu, Xubin Yue, Shipeng Luo, Junjie Zhao, Yingfan Hua, Jingqing Ruan, Zhongda Su, Feng Hong, Zhizhao Zeng, Ke Zeng, Xunliang Cai, Hanjia Ye
arXiv AI
Oct 16

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

CoWorld Reasoner is an agentic framework that leverages a Video World Model (VWM) as a unified visual reasoning engine for Vision‑Language Models (VLMs). It exposes multiple visual operations through tools and skills, allowing the VLM to decide when to imagine, select an operation, generate visual content, and interpret the resulting evidence. The framework includes a VLM‑VWM Harness Co‑Evolution mechanism that jointly refines tools, skills, and execution code based on feedback, while keeping model weights fixed. Across seven benchmarks covering spatial, temporal, causal, and perceptual reasoning, CoWorld Reasoner outperforms direct VLM reasoning and improves accuracy on all benchmarks, achieving notable gains on MMSI and ERQA.

By Dongjie Cheng, Ruifeng Yuan, Ran Wei, Ning Song, Yongqi Li, Wenjie Li
arXiv AI
Oct 16

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

PROBE is an algorithm that trains language‑model auditors to probe deployed tool‑using agents for policy violations through realistic, multi‑turn, tool‑mediated dialogue. By framing auditing as a two‑player Markov game and using a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity, PROBE outperforms zero‑shot adversaries in uncovering broader and higher‑yield policy violations. The study compares GRPO, DPO, and GFlowNet optimizations, demonstrating that multi‑turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing existing capability and safety benchmarks.

By Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou
arXiv AI
Oct 16

X-RIC: An Agentic RAN-Intelligent Controller Benchmark for Enhancing Reinforcement Learning

X‑RIC is a new framework that embeds agentic AI into O‑RAN architectures to enhance the RAN Intelligent Controller (RIC). It combines reactive reinforcement learning with strategic reasoning via automated behavioral cloning, a telecom language model, and closed‑loop co‑adaptation, and introduces BARX, a benchmark for evaluating agentic control. In simulations and testbed deployments, X‑RIC cuts operational costs by 89 %, meets SLA compliance, and converges 75 % faster with fewer training steps and lower variance.

By Adhwaa Al-Chaab, Ayman Younis, Dario Pompili
arXiv AI
Oct 16

Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

The paper introduces DIB-Bench, a benchmark that exposes deployment‑integrity blindness in large language model (LLM) evaluation by testing 7,560 scored examples across six tamper types and a range of evaluators from regex scorers to GPT‑5‑nano. It shows that stronger evaluators are more likely to accept tampered outputs—mean acceptance rises from 88.6% for rule‑based scorers to 94.1% for GPT‑5‑nano—and that certain trigger‑conditional tampering can produce harmful outputs in 48–100% of cases, increasing with model capability. The study also demonstrates that output‑only evaluation cannot detect tampering with zero output shift, while a manifest‑hash identity channel can detect it with zero samples, and proposes a graded deployment‑identity descriptor for evaluation reports.

By Jianchang Su, Dawei Xiang, Yifan Zhang, Wei Zhang
arXiv AI
Oct 16

Fractional Power-of-Two Quantization for Efficient and Effective Multiplier-Free LLM Inference

The paper introduces Fractional Power-of-Two (FPoT) quantization, a base‑√2 weight grid that enables 4‑bit weight quantization for large language models while keeping a fully integer shift‑add datapath. FPoT combines a fractional PoT grid with uniform integer activations, a calibration‑time grid‑alignment refinement, and a lightweight 1‑adder √2 approximation to reduce gate count compared to a 4‑bit multiplier. On Llama‑3‑8B, FPoT boosts zero‑shot commonsense reasoning by 8.72% over the standard PoT baseline and stays within 0.4% of uniform integer performance, with benefits extending across seven models and three bit‑widths.

By Sunghyun Wee, Geunjae Choi, Hyeonjin Kim, Suyoung Kim, Nojun Kwak
arXiv AI
Oct 16

When do quantum views help? Complementary classical-quantum representations in multi-view fusion

The paper investigates when quantum machine learning can enhance predictive performance by fusing classical and quantum representations in multi‑view learning. It introduces a framework that preserves full classical inputs while adding simulated quantum features, and uses concatenation and attention to combine the views. Experiments on synthetic angular‑polynomial data and real datasets (Wine, Wisconsin Diagnostic Breast Cancer, Fashion‑MNIST, Steel Plates Faults) show that fusion improves predictions for targets that depend on complementary classical and quantum relationships, while classical‑only baselines perform best on other tasks.

By Azadeh Alavi, Fatemeh Kouchmeshki, Hossein Akhoundi, Yongli Re, Abdolrahman Alavi, Muhammad Usman
arXiv AI
Oct 16

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

Qwen-Audio-Jev extends Qwen-Audio-3.1-ASR-Flash-Next with a dedicated decision head that unifies selection, binary judgment, and scoring into three primitives: CHOICE, NOUL, and SCORE. The model delivers competitive accuracies and error rates across 18 labeled configurations, achieving 78.70% on MMAU-mini, 84.06% on MMSU, 6.50% equal error rate on VoxCeleb1-O, and a 0.1062 normalized MAE on NISQA. Compared to its predecessor, Qwen-Audio-Jev offers roughly a 20‑fold inference speedup, enabling efficient high‑throughput audio decision‑making for tasks such as large‑scale screening, speaker verification, and automated quality assessment.

By Ying Fang, Zhifu Gao, Qing Gu, Jiajun He, Kangqi Jing, Xiangang Li, Yu Pu, Wupeng Wang
arXiv AI
Oct 16

Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

The paper critiques current agent memory benchmarks that rely on downstream QA accuracy, arguing they conflate memory correctness with retrieval and generation quality, leading to misleading results. It introduces FEB-30, a new benchmark of 30 fact‑evolution trajectories across six diagnostic categories and three difficulty levels, generated automatically to provide deterministic ground truth. Using FEB-30, the authors show a significant divergence between intrinsic and extrinsic evaluations, revealing that a representative memory system correctly classifies memory operations only 20% of the time—an error invisible to traditional QA metrics.

By Sebastian-Antonio Toma, Jean de Kernier, Mike Ply, Oliviu Matei
arXiv AI
Oct 16

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

AVC-Reward introduces an event‑centric framework for evaluating audio‑video consistency, breaking down scenes into individual audio and visual events and assessing them at three levels: global narrative harmony, event‑level temporal synchronization, and fine‑grained attribute alignment. The metric, instantiated as a trainable reward model, is validated against human judgments using the AVC‑Benchmark, which contains diverse positive and negative samples from various generation models. AVC‑Reward outperforms existing automatic methods, offers interpretable failure localization, and enhances consistency when used as a reinforcement‑learning signal.

By Yuruo Wu, Kaihang Pan, Kai Shen, Jiacheng Li, Shuyuan Tu, Jisheng Dang, Li Tuoyu, Juncheng Li, Siliang Tang
arXiv AI
Oct 16

Design the Feedback, Not Just the Reward: Environments for Open-Ended Reinforcement Learning

The paper proposes a reinforcement learning framework that emphasizes designing feedback environments rather than solely relying on rewards. It integrates human domain knowledge via rubrics and model reasoning with hierarchical rewards to evaluate complex, partially successful outputs in open‑ended tasks. Applied to interactive web application generation, the model achieves 41.25% pass rate on MiniAppBench and 76.19% on ArtifactsBench, with ablation studies confirming the value of each component.

By Rui Jin, Jikai Chen, Chengyue Yu, Yihan Chen, Hao Zhou, Demin Zhu, Kaichen Yang, Dong Wang, Linjian Mo, Chenyi Zhuang
arXiv AI
Oct 16

STAMP: Symmetric Temporal Averaging Message Passing for Domain Adaptation in Chronologically Split Graphs

The paper introduces STAMP, a timestamp‑balanced message‑passing operator designed for domain adaptation in chronologically split graph data. STAMP replaces standard neighborhood aggregation with temporally aligned propagation, enforcing first‑moment invariance across time while remaining scalable and easy to implement. Experiments on several temporal graph benchmarks, including ogbn‑arxiv, ogbn‑mag, and a temporal‑split Pokec dataset, show that STAMP and its variant STAMP‑mag consistently improve node classification accuracy and achieve state‑of‑the‑art performance.

By Joo Young Park, Sejun Park, Hyunwoo Park
arXiv AI
Oct 16

Right Answer, Wrong Tokens: Tail-Aware Credit Calibration for RLVR

The paper introduces Tail‑Aware Credit Calibration (TACO), a method that adjusts the positive credit given to tokens in reinforcement learning for large language models (LLMs) by evaluating each token’s rarity relative to the policy’s expectations. TACO reduces credit for high tail‑risk tokens while still allowing them to influence learning, thereby mitigating the positive‑credit contamination problem seen in GRPO‑style methods. Experiments on four LLMs across eight reasoning benchmarks demonstrate that TACO consistently outperforms GRPO baselines and improves training stability for long‑horizon RL tasks.

By Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang, Hoang Anh Duy Le, Zhaozhuo Xu, Guanchu Wang, Vladimir Braverman
arXiv AI
Oct 16

DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation

DexMimic is a generalist dexterous hand‑object tracking policy trained across diverse human hand‑object interaction trajectories. It uses Value‑Guided Reset Allocation to adapt the initialization distribution, prioritizing difficult phases and selecting reset states that maximize expected tracking returns. The authors present a benchmark for training, fine‑tuning, and generalization, demonstrating improved tracking success, training efficiency, faster fine‑tuning, and potential zero‑shot generalization to unseen trajectories and objects, and validate the approach on a real‑world dexterous manipulation platform.

By Zhexi Luo, Zhixuan Xu, Rui Yang, Zeyu Xiao, Yiwen Hou, Lin Shao
arXiv Machine Learning
Oct 16

Flexible Doubly Robust Proximal Causal Inference for Censored Outcomes

The paper introduces Proximal Survival Bridge learning (PSB), a flexible doubly robust framework for estimating causal effects on right‑censored time‑to‑event outcomes under unmeasured confounding. PSB builds on proximal causal inference to produce outcome‑bridge, treatment‑bridge, and doubly robust estimators, and offers two flexible bridge learners: PSB‑K, a kernel estimator with low‑rank time decomposition, and PSB‑N, a neural estimator trained with a projected maximum‑moment‑restriction loss. Experiments on a standard SEM benchmark, a proximal semi‑synthetic ACTG study, and the real‑world Framingham study show PSB‑K excels on SEM and ACTG data, while PSB‑N provides greater stability on Framingham, with additional analyses demonstrating robustness to proxy quality and censoring levels.

By Dhanajit Brahma, Ruolin Meng, Lawrence Carin, Ricardo Henao