Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 18

VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion proposes a new inference-time framework that improves steering of frozen masked discrete diffusion models. By reducing the variance of guidance estimates, applying reward tilting to clean-token logits, and adapting the selection temperature at each step, VGAS addresses three default choices in existing pipelines. Experiments on regulatory DNA, protein, and small-molecule benchmarks show that VGAS achieves the best training-free reward performance and matches or surpasses reward-fine-tuned generators.

By Kwanyoung Kim
arXiv Machine Learning
Sep 18

Evaluating Out-of-Distribution Robustness in Graph-Based Android Malware Classification: A New Principled Benchmark

The paper introduces a new benchmark for assessing out-of-distribution robustness in graph-based Android malware classifiers, highlighting that current models drop up to 45% accuracy on unseen malware variants. It presents two scenarios—MalNet-Tiny-Common for covariate shift and MalNet-Tiny-Distinct for domain shift—and identifies a limitation in existing benchmarks that rely solely on structure-only function call graphs. To address this, the authors propose a semantic enrichment framework that augments graph topology with function-level attributes and LLM-based code embeddings, demonstrating that this data-centric approach improves robustness under distribution shift and complements model-based methods.

By Ngoc N. Tran, Anwar Said, Waseem Abbas, Tyler Derr, Xenofon D. Koutsoukos
arXiv Computation and Language
Sep 18

Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition

The paper introduces the Modality Discrepancy Transformer (MDT), a model designed to detect ambivalence and hesitancy in clinical videos by capturing cross‑modal disagreement across facial, vocal, and linguistic signals. MDT expands a 6‑token representation to 9 tokens that include modality embeddings, absolute‑difference features, and Hadamard‑product discrepancy features, which are processed through Transformer self‑attention with FiLM‑based text conditioning and LoRA fine‑tuning. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves a Macro F1 score of 0.7408 on the labelled test split and 0.7368 on the private leaderboard, surpassing the strongest baseline by over 10 points while training in under 20 minutes on a single GPU.

By Shiyu Luo, Yu Wang, Jiawen Huang, Zhaoxiang Xiao, Chenxi Huang, Qi Zhang, Bin Liu
arXiv Computation and Language
Sep 18

Neo-Classic: A Benchmark for Evaluating Linguistic-Aesthetic Reasoning in Classical Chinese Poetry

Neo-Classic is a new benchmark designed to evaluate linguistic‑aesthetic reasoning in Classical Chinese poetry. It uses an out‑of‑sample dataset of strictly metrical poems written by contemporary experts and a set of reverse‑understanding probes, avoiding reliance on historical corpora. Experiments with leading LLMs show a 20–50% performance drop on contemporary texts and low accuracy (0–13%) on discourse‑level ordering, indicating that current models excel at local patterns but struggle with global hierarchical planning.

By Han Zhang, Zihan Gu, Zhiyuan Wang, Tianyi Ma, Jiacheng Lu, Xinyan Zhang, Yuhao Wei, Cheng Hua
arXiv Computation and Language
Sep 18

Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes

Reflective Recovery is a self‑supervised method that turns failed reasoning attempts into training data, enabling large language models to learn how to correct mistakes during inference. By extracting initial segments of erroneous trajectories and using them as prompts, the approach teaches models to recognize and recover from errors without external critics. Experiments show significant accuracy gains on benchmarks such as AIME 2025 and Minerva, and the method overcomes the scaling collapse problem, fostering emergent self‑correction behaviors.

By Qirui Chen, Renjie Pi, Jiahui Gao, Lingpeng Kong
arXiv Computation and Language
Sep 18

Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asia

Compass‑v3 is a 245B‑parameter Mixture‑of‑Experts language model tailored for Southeast Asian e‑commerce, featuring 71B active parameters per token and hardware‑efficient expert parallelism. It is trained on 12 trillion multilingual tokens and synthetic e‑commerce instructions, and incorporates Optimal‑Transport Direct Preference Optimization to improve instruction adherence. Benchmarks show it outperforms GPT‑4, DeepSeek‑V3.1, and Qwen3‑235B, and it is already deployed at scale on Shopee, handling over 70% of the platform’s LLM traffic.

By Sophia Maria
arXiv Computation and Language
Sep 18

CORTEX: High-Quality Cross-Domain Organization of Web-Scale Corpora through Ontological Corpus Graph

CORTEX is a novel framework that transforms web‑scale corpus construction from flat document filtering into structured knowledge organization using an Ontological Corpus Graph (OCG). The OCG comprises a quality‑refined content layer, a lightweight ontology layer that evolves via LLMs, and a cross‑domain alignment layer that supports arbitrary taxonomic resolution. Experiments demonstrate CORTEX’s effectiveness, and the authors release a 24.14 B‑token refined corpus, its OCG, and a cross‑domain benchmark called CortexBench for evaluating large language models.

By Chengtao Gan, Xiaoke Guo, Yushan Zhu, Zhaoyan Gong, Zhiqiang Liu, Songze Li, Huajun Chen, Wen Zhang
arXiv Computer Vision
Sep 18

Instance Segmentation and Fine-grained Classification for Urban Buildings with Adaptive Region Dividing and Spatially-Supervised Contrastive Learning

The paper introduces an adaptive region‑dividing strategy that projects a 3D point cloud onto a bird’s‑eye‑view plane to detect building regions, then back‑projects bounding boxes to create structure‑aligned training blocks for unified scene‑level evaluation. It also proposes a fine‑grained classification model using a point transformer classifier and a spatially‑supervised contrastive loss to improve inter‑class discriminability, addressing class imbalance with a weighted cross‑entropy. Experiments on UrbanBIS and STPLS3D datasets show the method outperforms state‑of‑the‑art approaches in both building instance segmentation and fine‑grained classification.

By Weiyuan Zhang, Qi Zhang, Hui Huang
arXiv Computer Vision
Sep 18

WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing

WeVisDoc is a two‑stage data‑centric framework designed to improve end‑to‑end document parsing. Stage I expands coverage by adding heterogeneous data and applying structure‑preserving degradation synthesis, while Stage II evaluates residual errors with a held‑out probe and uses those diagnostics to target data construction and token budget reallocation. The resulting WeVisDoc‑4B model achieves an overall score of 95.38 on OmniDocBench v1.6 and outperforms competing parsers across all evaluated settings, with Stage II delivering notable gains on degraded tracks.

By Hao Yu, Kang Liu, Linnan Zhao, Jiabo Zhan, Chong Sun, Chen Li, Jing Lyu
arXiv Computer Vision
Sep 18

Latent-Centroid Steering: Single-Pass Classifier-Free Guidance for Command-Aligned Autonomous Driving

The paper introduces Latent-Centroid Steering (LCS), a single-pass classifier-free guidance method for vision‑language autonomous driving models. LCS replaces instance‑level residuals with class‑level latent shifts, projecting conditional representations toward precomputed command‑specific centroids to enhance command adherence. Experiments on Bench2Drive and nuScenes show that LCS cuts inference latency by about 50% while improving driving performance.

By Meibo Hu, Jiamian Wang, Pichao Wang, Zhiqiang Tao
arXiv AI
Sep 18

FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA

FINSKILLOPS is a multi‑agent system designed to improve SEC filing question‑answering after deployment by creating reusable skills from evidence‑grounded failure diagnoses. It manages these skills through targeted validation, regression checks, negative controls, and versioned replacement or retirement, ensuring that new patches do not introduce regressions. Across six benchmarks, a frozen skill registry outperforms other systems, and in a 12‑round operational study the system reduced the non‑correct rate from 20.0% to 12.5% while promoting only six of 33 proposed skills.

By Yanzhang Ma, Zhenghan Tai, Hanwei Wu, Sizhe Guan, Jianliang Lei, Hailin He, Chaolong Jiang, Jijun Chi, Tung Sum Thomas Kwok, Bohuai Xiao, Jingrui Tian, Xinlu Wu, Xingao Zhan, Peng Lu, Muzhi Li, Yihong Wu, Liheng Ma, Sicheng Lyu, Tianshuo Yan, Junhao Zhu, Yaqian Xu, Lei Ding, Yufei Cui, Ziquan Liu, Boyu Han, Hengli Liu, Ling Zhou, Xinyu Wang
arXiv AI
Sep 18

Contagion on the Trading Floor: How Adversarial Signals Spread in Multi-Agent Trading Systems

The paper introduces the Generic Multi-Agent Trading System (GMATS), a framework for studying how large language model (LLM) based trading stacks react to black-box, input-only attacks that inject plausible social‑media content. It defines contagion metrics—belief‑shift scores at analyst and coordinator layers and attack‑clean deltas on backtest metrics—to trace the spread of adversarial signals. Experiments on a safe offline benchmark show that even simple attackers can significantly degrade risk‑return profiles, while certain multi‑agent topologies and coordinator prompts can mitigate these effects.

By Qi Rong Sua, Junhao Dong, Nguyen Duc Thai, Yuqing Wen, Cheston Tan, Yew-Soon Ong
arXiv AI
Sep 18

Form Over Content In Gradient-Based Data Attribution Methods

The paper investigates what gradient similarity measures in data attribution for large language models. By independently varying task and answer format in supervised fine‑tuning benchmarks, the authors show that gradient alignment is driven by answer format rather than task semantics, with strong alignment for shared formats and none for differing formats. This pattern persists across training stages, model sizes, and families, and is evident in the selections of the LESS data‑selection method, which over‑represents its own answer format.

By Sunwoo Kim, Seokwon Jung, Sohyung Kim, Seong Joon Oh, Alice Oh
arXiv Machine Learning
Sep 18

Learning-Based Reconstruction of Optical Properties in Bilayered Media from Single-distance Time-Resolved Reflectance Measurements

The paper presents a machine‑learning framework for reconstructing absorption and scattering coefficients in bilayered biological media from single‑distance, time‑resolved reflectance data. By training on a synthetic dataset generated with exact Monte Carlo simulations, the method outperforms traditional diffusion‑equation‑based inverse solvers in both speed and accuracy. It also estimates the dimensionality of the parameter space without prior knowledge of the number of layers, and suggests that future work could further improve accuracy using multi‑distance data.

By Caterina Amendola, Giulia Maffeis, Lorenzo Buffoni, Lorenzo Chicchi, Francesco Coghi, Duccio Fanelli, Raffaele Marino, Fabrizio Martelli, Riccardo Paoli, Lorenzo Pattelli, Lorenzo Spinelli
arXiv Machine Learning
Sep 18

SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting

SETTer is a transformer-based model designed for long‑term multivariate time‑series forecasting. It introduces decoupled self‑attention and hybrid masking to better handle high dimensionality and complex relationships, while adding explainable structures to highlight discriminative patterns. Experiments on real‑world benchmarks show that SETTer outperforms state‑of‑the‑art models in 88% of scenarios.

By Abraham Ezema, Chijioke Eze, Ferdinanda Ponci, Antonello Monti
arXiv Machine Learning
Sep 18

ZeroHAT: Behavior-Conditioned Zero-Shot Human Activity Trace Generation

ZeroHAT is a new framework for generating synthetic human activity traces (HATs) in a target region without any real data from that region. It transfers behavioral patterns learned from real HATs in source regions and adapts them using publicly available contextual information about the target region. The system includes a consistency-aware intent extractor, a cross-region behavioral cloning module, and a behavior-conditioned activity realization module, and it outperforms the strongest baseline by 4.5–6.4× in downstream utility and improves fidelity by 15.6–40.8% across ten cities.

By Rongchao Xu, Dahai Yu, Lin Jiang, Guang Wang
arXiv Machine Learning
Sep 18

Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction

This study independently reproduces the dissociation reported by Zhao (2026) regarding chain-of-thought entropy in large language models. It confirms that the shape of the entropy trajectory predicts answer correctness, while the total entropy drop magnitude does not, across four open-weight models and two benchmarks (GSM8K and MATH‑500). The reproduction also maps settings where the magnitude signal holds or fails and documents protocol differences not reported in the original work.

By Theodore O. Cochran
arXiv Machine Learning
Sep 18

Traffic Engineering in Large-scale Networks with Generalizable Graph Neural Networks

The paper introduces TELGEN, a traffic engineering algorithm that uses graph neural networks to predict an optimal TE algorithm rather than a direct solution. TELGEN generalizes across diverse network topologies and traffic patterns, achieving less than a 3% optimality gap on networks up to 5,000 nodes and 3.6 million links, while reducing solving time by up to 84% and training time by up to 79.6% compared to existing methods.

By Fangtong Zhou, Xiaorui Liu, Ruozhou Yu, Guoliang Xue
arXiv Machine Learning
Sep 18

Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents

The paper introduces AutoMIA, a framework that uses large language model agents to automatically design and implement new membership inference attack (MIA) signal computations. By systematically exploring a wide range of attack strategies, AutoMIA discovers novel MIAs tailored to specific target models and datasets, achieving up to a 0.18 absolute improvement in AUC over existing methods. This demonstrates that LLM agents can serve as an effective and scalable approach for creating state‑of‑the‑art MIAs.

By Toan Tran, Olivera Kotevska, Li Xiong
arXiv Computation and Language
Sep 18

CovR: Coverage-Aware Hardware Verification via Reasoning-Guided Reinforcement Learning

CovR is an agentic framework that automates testbench generation for hardware verification by combining self-reflection loops with simulation-based feedback to maximize coverage. It builds a large dataset of 16,514 specification–RTL reasoning tuples and uses reinforcement learning with tool-derived rewards to train a student model, achieving high coverage scores on VerilogEval, RTLLM V2.0, and CVDP. When deployed as a plug-in stimulus engine, CovR boosts coverage by nearly 19% and improves mutation detection while uncovering previously undetected failures.

By Manar Abdelatty, Maryam Nouh, Sherief Reda