Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 9

HistCAD: Constraint-Aware Parametric CAD Histories for Evaluating Editability

HistCAD is an executable representation and dataset comprising 180,495 parametric construction histories with sketch constraints and feature operations, split into Academic and Industrial collections. The benchmark evaluates whether predicted constraints can enable a history to reproduce an initial model and perform a specified dimensional edit, measuring success when the model matches the required geometry, reaches the target value, preserves relations, and rebuilds through the entire history. Predictors trained on both collections achieve 52.4% edit success on Academic data and 29.0% on Industrial data, with models retaining only endpoint-connectivity constraints sometimes reaching the target but failing to preserve all relations.

By Xintong Dong, Chuanyang Li, Peng Zheng, Chuqi Han, Jiaxin Jing, Hailong Shen, Yanzhi Song, Zhouwang Yang
arXiv AI
Oct 9

READ-Bench: Benchmarking Historical Instance Retrieval for Time-Series Diagnosis

READ‑Bench is a new benchmark for historical‑case retrieval in time‑series diagnosis, covering 12 multivariate datasets and defining relevance by shared fault or event type rather than signal shape. The study evaluates a wide range of retrieval methods—including classical distances, symbolic retrievers, self‑supervised and foundation‑model embeddings, and label‑aware or language‑model rerankers—under a unified protocol that varies supervision, pollution, and corpus size. Results show that pretrained embeddings do not outperform strong classical baselines for search alone, but a small amount of resolved‑case supervision via a Gaussian‑process reranker significantly boosts performance, especially when combined with a reciprocal‑rank fusion of a normal‑residual embedder and dynamic time warping. whyItMatters":"The benchmark demonstrates that modest supervised reranking can outperform sophisticated representations, guiding future research toward efficient, data‑light retrieval strategies for time‑series diagnosis."

By Gerardo Pastrana, Haojun Li, Dhruv Mehta, Anoushka Vyas, Sina Khoshfetrat Pakazad, Henrik Ohlsson, John Paparrizos
arXiv AI
Oct 9

Humanize: Judgement Engineering for Agentic Coding

Humanize is a multi‑agent workflow for agentic coding that uses explicit, mechanically enforced decisions—called judgement engineering—to separate planning, implementation, review, and learning. A human approves a plan contract, a builder agent implements it in rounds, and a reviewer agent from another vendor decides completion, with deterministic hooks and 72 mechanical gates ensuring that a defect survives only if both models miss it. The system has been deployed in over 68 versions in 108 days, earned 1,468 GitHub stars, and powered applications such as a 567‑file gem5 build‑system migration, Kernel Design Agents that excelled in MLSys 2026 contests, and Humanize Olympiad Agents that achieved top scores in multiple international competitions and high rankings on PutnamBench and Lean‑Eval. whyItMatters":"Humanize demonstrates that a structured, multi‑agent approach with explicit mechanical gates can improve reliability and performance in automated code generation, as evidenced by its adoption, star count, and success in competitive benchmarks."

By Sihao Liu, Ligeng Zhu, Zijian Zhang, Dongyun Zou, Zhengyang Zhang, Changye Li, Song Bian, Song Han, Tony Nowatzki
arXiv AI
Oct 9

How Should a Prompt Optimizer Spend a Tight Budget? BudgetAPO with Noise-Adaptive Evaluation

The paper introduces BudgetAPO, a single‑stage prompt optimizer designed for tight‑budget scenarios where only a few hundred model calls are feasible. BudgetAPO adapts the evaluation slice size to each task’s noise level, uses a fixed slice to turn accept/reject decisions into paired comparisons, and applies a reflective operator to jointly rewrite reasoning strategy and output format. Across seven benchmarks and five subject models, BudgetAPO outperforms existing methods, ranking first on every subject and requiring far fewer calls to achieve comparable or superior performance.

By Haoyue Liu, Zhichao Wang, Huanyu Yan, Xiaoying Tang
arXiv AI
Oct 9

BEAT: Balanced Frequency Adaptive Tuning for Long-Term Time-Series Forecasting

BEAT (Balanced Frequency Adaptive Tuning) is a framework for long‑term time‑series forecasting that uses frequency‑specific error monitoring and adaptive gradient modulation. It introduces a Frequency‑Specific Monitor to compare predicted and target wavelet coefficients in a normalized space, and a Dynamical Gradient Balancer that assigns larger gradient weights to components with higher relative errors. Experiments on seven real‑world datasets demonstrate that BEAT achieves competitive performance against state‑of‑the‑art forecasting methods.

By Zhixuan Li, Naipeng Chen, Seonghwa Choi, Sanghoon Lee, Weisi Lin
arXiv AI
Oct 9

Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover

The paper investigates how adversarial prompt‑injection attacks can shift the scaling of attack success from a slow polynomial growth to an exponential growth as the number of inference‑time samples increases. By proposing a minimal statistical framework and a spin‑glass generative model, the authors explain that short injected prompts act as a weak magnetic field leading to power‑law scaling, while long prompts act as a strong field producing exponential scaling. The phenomenon is observed consistently across large language models ranging from 3B to 70B parameters and across various attack methods and benchmark datasets.

By Indranil Halder, Annesya Banerjee, Cengiz Pehlevan
arXiv AI
Oct 9

DCVD: Dual-Channel Cross-Modal Fusion for Joint Vulnerability Detection and Localization

DCVD (Dual-Channel Cross-Modal Vulnerability Detection) is a unified framework that jointly performs function-level vulnerability detection and statement-level localization. It extracts control-dependency and semantic features via two parallel branches and fuses them using contrastive alignment and bidirectional cross-attention. The model incorporates explicit supervision at both granularities, achieving superior performance on a large real-world vulnerability benchmark compared to state-of-the-art methods.

By Wenxin Tang, Junliang Liu, Wenbin Li, Jingyu Xiao, Xi Xiao, Mingzhe Liu, Jinlong Yang, Xuan Liu, Yuehe Ma, Wang Luo, Qing Li, Lei Wang, Peng Xiangli
arXiv Machine Learning
Oct 9

Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning

The paper investigates whether large language models (LLMs) can learn from rewards presented in context, a process termed in‑context reinforcement learning (ICRL). Through experiments on four benchmarks and six models, the authors find that while LLMs read reward signals, altering or removing these signals has little effect on performance, and that trajectory structure—rather than reward content—drives improvement. These results suggest that direct ICRL behaves more like standard in‑context learning (ICL) than true inference‑time reinforcement learning.

By Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim
arXiv Machine Learning
Oct 9

Learning infinite context windows in recurrent architectures via spatial neural computing

The paper proposes a second‑order recurrent neural network that replaces traditional neuron‑to‑neuron communication with a spatially evolving field governed by discretized partial differential equations. This design creates a structured infinite‑order RNN where each state depends on its entire history, providing an unbounded receptive field while keeping the number of parameters fixed. The authors derive conditions for marginal stability that prevent vanishing or exploding gradients, and demonstrate that the architecture outperforms other recurrent models on long‑horizon benchmarks with fewer parameters.

By Aleix Salvador-Pomarol, Arthur N. Montanari, Earl K. Miller, Adilson E. Motter, Jorge Cort\'es
arXiv Machine Learning
Oct 9

SPD-MetaFormer is what you need for small-data brain decoding

The paper introduces SPD-MetaFormer, an attention‑free architecture for decoding brain signals from small datasets. It shows that existing attention‑based models on the SPD manifold (MAtt and GBWAtt) learn nearly uniform weights, and that replacing learned weights with uniform ones does not degrade performance. SPD-MetaFormer uses uniformly weighted Fréchet aggregation with a geodesic residual and a shared spectral feedforward map, achieving competitive results on three EEG benchmarks while maintaining SPD‑valued token states until classification.

By Zhida Wang, Wei Lyu, Guo Yu, Sui Tang
arXiv Machine Learning
Oct 9

SR-TTA: Spatial-Redundancy Test-Time Adaptation for Interference-Robust Respiration Sensing

The paper introduces SR‑TTA, a spatial‑redundancy test‑time adaptation method for respiration sensing using a 64‑antenna cell‑free massive MIMO base station. It replaces hand‑crafted fusion with a learned complex‑weight beamformer that spatially nulls in‑band motion interference, and further closes the performance gap to an oracle through label‑free adaptation. Experiments show the respiration‑rate error drops from 5.8 to 0.8 breaths per minute under simulated interference, and real‑world recordings see failure rates fall from 47% to 7% with a single cross‑subject calibration.

By Jingyuan Liu, Zheng Chang, Haoqiu Xiong, Zhuangzhuang Cui, Sofie Pollin
arXiv Machine Learning
Oct 9

Camera-Noise Residuals for Face-Swap Detection: Redundant, Not Complementary, and Why

The study evaluates whether adding a camera‑noise fingerprint (Noiseprint++) to an RGB Xception backbone improves face‑swap detection on FaceForensics++. A three‑model ablation shows that fusion does not outperform RGB alone and that the residual branch alone performs near chance. Diagnostic analysis reveals that the noise maps contain a statistical signal in their per‑sample moments, but an InstanceNorm layer standardizes these away, rendering the residual redundant rather than complementary to RGB features.

By Danil Davydov, Bader Rasheed, Dmitriy Vatolin
arXiv Machine Learning
Oct 9

GaussianBench: Physics-Fidelity Evaluation for Gaussian Scene Representations

GaussianBench is a physics‑fidelity evaluation suite designed for physics‑integrated Gaussian scene representations. It tests key physical properties—such as conservation, continuum response, material coupling, covariance transport, rendering, thermal phase change, and counterfactual response—using frozen scenes, simulator‑independent scorers, and analytical or measured references. The benchmark distinguishes physical failures from unsupported capabilities and invalid comparisons, and the authors also present GaussianFlesh, a thermomechanical reference system that couples rendering primitives with continuum material points via an MPM solver.

By Chukwudalu Dumebi-Kachikwu
arXiv Machine Learning
Oct 9

RIFT: Relative Isolation From Trees For Anomaly Detection

RIFT (Relative Isolation From Trees) is a deterministic anomaly detection method that builds a minimum spanning tree and scores points by the sum of apparent edge sizes from each point. For one-dimensional data, RIFT reproduces the closed-form limit of Isolation Forest exactly, while in higher dimensions it offers a parameter‑free, robust alternative that avoids axis‑parallel artifacts and handles varying density and clustered anomalies. An ensemble variant is proposed for large datasets, and experiments on synthetic data and the ADBench benchmark show comparable accuracy to Isolation Forest with reduced variance across random seeds.

By Mark Daniel Szalai, Gabor Horvath
arXiv Machine Learning
Oct 9

FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs

FloorSAV is a framework that enhances audio‑visual large language models (AV‑LLMs) by embedding a dynamic 2D floormap derived from 3D point clouds, camera trajectories, spatial audio cues, and object landmarks. The floormap is fed into the AV‑LLM as a synchronized stream with egocentric video, enabling the model to jointly reason over visual, auditory, and geometric information in a single inference. The authors also introduce SAVED‑Bench, a benchmark for spatial reasoning tasks, and show that FloorSAV improves AV‑LLM performance on both SAVED‑Bench and SAVVY‑Bench, with ground‑truth floormaps highlighting its potential.

By Kyeong-Rae Kim, Sungnyun Kim, Tae-Hyun Oh
arXiv Computation and Language
Oct 9

Event-Centric Memory with Query-Aware Graph Augmentation for Long-Term Conversational Agents

The paper introduces QGMem, a memory framework for conversational agents that organizes dialogue history into event-indexed atomic units and consolidates related units into dynamic memory traces. When a query is posed, QGMem retrieves complementary candidate memories, reranks them to form a compact working memory, and represents this working memory as a local graph to expose relational dependencies. Experiments on six benchmarks demonstrate consistent improvements in retrieval, multi‑hop evidence composition, conflict resolution, and ultra‑long dialogue reasoning while keeping context size and inference cost moderate.

By Yichen Liu, Chunfeng Yuan, Haowei Liu, Wenjuan Li, Zefeng Lin, Bing Li, Xu Chen, Weiming Hu
arXiv Computation and Language
Oct 9

UXBench Pro: Benchmarking Personalized User Experience in Multi-Turn Dialogue Interactions

UXBench Pro is a new benchmark that evaluates personalized user experience in multi-turn dialogue interactions. It contains 1,000 test instances from real user interactions across 12 task scenarios and 82 domains, each paired with a FACTORS user profile describing seven behavioral facets. The benchmark introduces a dual-perspective evaluation: a personalized User Reward Model for third‑person judgment and Sim4Eval, a user simulator for first‑person evaluation across four cognitive state dimensions, along with meta‑benchmarks URMBench and USimBench to assess evaluator reliability.

By Mengze Hong, Zeyang Lei, Wenbo Shang, Xia Zeng, Xiying Zhao, Qi Zhu, Chen Jason Zhang, Di Jiang, Taiming Fu, Qiongyi Zhou, Qinghe Chang, Fubao Zhang, Chenxuan Ma, Minlong Peng, Jinfeng Huang, Zineng Zhou, Jindou Wu, Muge Qi, Sijun He, Xin Cui, Di Liang, Yuan Hua, Davey Chen
arXiv Computation and Language
Oct 9

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

SpaceCast-Bench is a new benchmark that evaluates predictive spatial reasoning in vision‑language models, moving beyond simple spatial perception to tasks that require constructing scenes, anticipating interventions, and reasoning about unseen outcomes. It contains 3,862 questions from 182 real‑world scenes across 16 task types and three difficulty levels—static perception, local prediction, and global prediction—testing scene understanding, spatial state updating, and relational inference. Evaluation of 21 models shows a large performance gap, with the best model achieving only 58.0% versus 87.2% human accuracy, and fine‑tuning on the benchmark’s programmatically generated data can substantially improve model performance.

By Hongxing Li, Jinyue Su, Dingming Li, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
arXiv AI
Oct 9

How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense

The paper investigates how narrative wrapping can trick safety‑aligned language models into refusing a harmful request only when it is presented directly, but answering it when embedded in a role‑play or narrative context. It introduces GUISE, a multilingual benchmark that tests this vulnerability across English, modern Chinese, and Classical Chinese, and shows high attack success rates (up to 95.7%). The authors propose AXIS, a defense that aligns harmful‑request representations with the refusal direction and trains the model to refuse outright, achieving superior safety and usability scores on several large language models.

By Zhankai Ye, Yanning Wang, Yukai Jin, Bo Mei, Fangyi Li, Wei Wang, Shangqian Gao, Xin Liu