Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 12

Constrained Probabilistic Numerics: Gaussian Process Solvers for Linear PDEs under Linear Inequality Constraints

The paper introduces a method to incorporate linear inequality constraints—such as nonnegativity, monotonicity, and no‑arbitrage—into Gaussian process solvers for linear partial differential equations. By proving that conditioning a Gaussian process jointly on equality observations and a finite‑dimensional inequality event is equivalent to sequential conditioning, the authors obtain a non‑Gaussian but computable posterior that respects both PDE physics and inequality constraints. Experiments on Poisson problems, option‑price surfaces, and air‑quality data show substantial error reductions and improved adherence to constraints when the new method is applied.

By Boyuan Deng, Kshitiz Upadhyay, Michael Shields
arXiv Machine Learning
Oct 12

Bridge Matching Sampler: Scalable Sampling via Generalized Fixed-Point Diffusion Matching

Bridge Matching Sampler (BMS) is a new diffusion-based sampling method that generalizes least-squares matching objectives into fixed-point iterations using Nelson's relation. It learns a stochastic transport map between any prior and target distribution with a single, scalable, and stable objective, and a damped variant adds regularization to reduce mode collapse. Experiments show BMS can sample at unprecedented scales while preserving mode diversity, achieving state‑of‑the‑art results on complex synthetic densities and high‑dimensional molecular benchmarks.

By Denis Blessing, Lorenz Richter, Julius Berner, Egor Malitskiy, Gerhard Neumann
arXiv Computer Vision
Oct 12

What Visual Representations Do World Action Models Need?

The paper investigates how different visual representations and integration mechanisms affect the performance of World Action Models (WAMs). It finds that strong general‑purpose visual representations yield the largest robustness gains, that channel‑wise joint prediction is the most effective integration strategy, and that additional sources should be chosen based on the unique task‑relevant information they provide. Using these insights, the authors propose R$^2$‑WAM, which achieves state‑of‑the‑art success rates on RoboCasa GR1 Tabletop and RoboTwin 2.0.

By Kai Liu, Jiangran Lyu, Tingyu Deng, Yixin Zheng, Jingtian Wan, Wenbo Cui, Mi Yan, Yizhou Zhou, Haoran Li, Dongbin Zhao, Zhizheng Zhang, He Wang
arXiv Computer Vision
Oct 12

ThinkDrag: Semantic Drag-Based Image Editing with Visual Reasoning

ThinkDrag is a multimodal framework for drag-based image editing that incorporates visual reasoning to interpret sparse point constraints as meaningful object- and scene-level transformations. It can generate an explicit reasoning trace to clarify the intended edit, improving interpretability for ambiguous drags. The system introduces Drag Endpoint Markers for grounding drags in image space and is evaluated on DragBench++, a benchmark of challenging drag-based editing scenarios, achieving state‑of‑the‑art performance in generation quality and plausibility while maintaining competitive point-following precision.

By Armando Fortes, Tianyi Wei, Chindanai Trakantannarong, Shangchen Zhou, Xingang Pan
arXiv Computer Vision
Oct 12

FedHUG: Federated Heterogeneous Unsupervised Generalization for Remote Physiological Measurements

FedHUG introduces a Federated Unsupervised Domain Generalization protocol to address privacy and labeling challenges in remote physiological measurement. The framework includes a Minimal Bias Aggregation module that adapts weights to heterogeneous non‑IID features and a Global Distribution‑aware Learning Controller that adjusts client training to mitigate label skew and long‑tail issues. Experiments demonstrate superior performance over state‑of‑the‑art methods using RGB video or mmWave radar data.

By Xiao Yang, Dengbo He, Duoxun Tang, Jiyao Wang, Kaishun Wu
arXiv Computer Vision
Oct 12

FlowSGS: Improving Flow Priors for Inverse Imaging with Stochastic Interpolants

FlowSGS introduces a flow-based posterior sampling method that combines Split Gibbs Sampling with annealed coupling and Stochastic Interpolants to improve flow priors for inverse imaging. The approach includes a novel timestep correction and accelerated warm-up using one-step flow models, resulting in faster and more accurate performance across various imaging tasks. It also demonstrates the first application of flow-based samplers to a nonlinear inverse problem, namely Fourier phase retrieval, and extends one-step flow models to principled Bayesian inverse imaging.

By Tianao Li, Xinhui Qian, Emma Alexander
arXiv Computer Vision
Oct 12

Riemannian Shape Analysis of the Corpus Callosum in Kendall Space: Aging and Alzheimer's Disease

The study introduces a Riemannian shape‑space framework for analyzing age‑related morphological changes in the midsagittal corpus callosum (CC) using 128 landmarks embedded in Kendall shape space. Applied to the OASIS‑1 cohort, multivariate geodesic regression identifies anterior sub‑regions (Genu and Anterior Body) as the primary sites of age‑related deformation, outperforming Euclidean benchmarks. In Alzheimer’s patients, age explains less than 0.5% of shape variance, and a tangent‑space classifier achieves an age‑group AUC of 0.791 from 2D contour data alone.

By Olakunle S. Abawonse, Fatou Fall
arXiv Machine Learning
Oct 12

TernaryLLM: Ternarized Large Language Model

TernaryLLM introduces Dual Learnable Ternarization (DLT) to allow both scales and shifts to be learned, addressing asymmetric outliers and non‑zero means in weights. It also proposes Outlier‑Friendly Feature Knowledge Distillation (OFF), which maximizes mutual information between ternarized and floating‑point features using cosine similarity to recover lost information. Experiments show TernaryLLM outperforms prior low‑bit quantization methods on text generation and zero‑shot benchmarks, notably improving LLaMA‑3’s perplexity and accuracy.

By Tianqi Chen, Zhe Li, Weixiang Xu, Zeyu Zhu, Dong Li, Lu Tian, Emad Barsoum, Peisong Wang, Jian Cheng
arXiv Computer Vision
Oct 12

High-Throughput Generative Video Super-Resolution on a Single Consumer GPU

The paper presents a zero‑shot framework for high‑throughput generative video super‑resolution (VSR) that operates on a single high‑end consumer GPU without retraining. It decouples spatial upscaling, handled by a time‑distilled one‑step diffusion backbone, from temporal stabilization, achieved with a lightweight pretrained flow‑based interpolator. A resolution‑aware hybrid attention scheme replaces high‑resolution global attention with neighborhood attention to improve scalability, achieving up to ~32 FPS at 720p and a 44% speedup at 4K on an NVIDIA RTX 4090, while maintaining competitive perceptual quality and temporal consistency on standard VSR benchmarks.

By Georgios Panaretou, Sotirios Chatzis
arXiv Machine Learning
Oct 12

SynAct: Synergistic Active Feedback Learning for LLM Agents

SynAct is an active learning framework that allocates scarce expert feedback to individual decisions in large language model agents, ranking decisions by the mutual information between the chosen action and the final outcome across repeated rollouts. By querying experts at the highest‑value decisions and distilling corrections into reusable rules, SynAct improves held‑out accuracy by an average of 11.7 percentage points across 16 settings and outperforms budget‑matched routers in inference‑time routing. The method is theoretically grounded, showing that outcome‑separating decisions have a minimum mutual information of H2(m/k) bits, enabling exact separation for any threshold within that range.

By Ning Liu, Ian Li, Kristina Klinkner
arXiv Machine Learning
Oct 12

NV-Tesseract-AD: Curriculum-Enhanced Conditional Diffusion for Multivariate Time-Series Anomaly Detection

NV-Tesseract-AD is a curriculum learning‑enhanced conditional diffusion framework designed for multivariate time‑series anomaly detection. It structures reconstruction difficulty along masking severity and diffusion noise level, using a Transformer‑based diffusion backbone and a three‑stage dual curriculum that keeps anomalous samples throughout training. The method achieves significant performance gains, including a 3.7‑fold F1 improvement and 56% MSE reduction over a non‑curriculum baseline, and demonstrates strong zero‑shot results on several benchmark families.

By Aditi Gautam, Manikandan Ravikiran, Saira Qureshi, Sena Fatma Ekiz, Charan Jagwani, Mahmoud Elhadidy, Sanjeev Roka
arXiv Machine Learning
Oct 12

Controlled-Minimax: Stabilizing Adversarial Optimization via Control Theory

Controlled-Minimax is a new framework that stabilizes adversarial minimax optimization by modeling training dynamics as a second-order ODE and applying a feedback controller derived from control theory. The method guarantees local exponential stability regardless of the Hessian spectrum, addressing persistent limit cycles and divergence that plague traditional gradient descent-ascent approaches. Experiments on benchmarks such as Dirac-GAN, 2D Gaussian mixtures, Swiss Roll, and StyleGAN2-ada show that Controlled-Minimax suppresses oscillations and achieves fast, robust convergence.

By Tianjiao Luo, Jianfei Chen, Jun Zhu
arXiv Computer Vision
Oct 12

GuidedNav: Shaping Vision-Language Navigation Representations through Pre-action Attention and Spatial Guidance

GuidedNav is a new framework for vision‑and‑language navigation that explicitly shapes the representations linking instructions to actions. It uses subtask and landmark guidance to focus pre‑action attention on the relevant instruction segment and visual regions, and spatial guidance to align internal representations with geometric features of the environment. The method achieves state‑of‑the‑art success rates on R2R‑CE and RxR‑CE val‑unseen, improving by 8.1% and 13.1% respectively, while also providing faster inference and introducing two new annotation‑enriched datasets, Guided‑R2R and Guided‑RxR.

By Jingyu Guo, Jiaxin Huang, Chengxing Lin, Xiaoyu Luo, Cheng Wen, Shanshan Ye, Tingjin Chu, Shaoli Huang, Mingming Gong
arXiv Computation and Language
Oct 12

Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

The paper introduces a memory-augmented retrieval method that leverages token-level evidence instead of relying solely on single-vector embeddings. By performing sparse token matching and aggregating scores with whole-memory similarity, the approach improves retrieval performance across eight benchmarks and various model families without extra training. An optional low-rank query adapter can further boost results when supervised.

By Han Luo, Zhan Zhuang, Xu He, Jujia Zhao, Yu Li, Suqin Yuan, Dong Li, Lei Feng
arXiv AI
Oct 12

A Baseline Assessment for the Agentic Web Technology Stack

The paper introduces MASTIF, a Multi‑Agent System TestIng Framework, to evaluate how different protocols, frameworks, and libraries combine into an Agentic Web solution. It presents a baseline assessment using 3 large language models, 3 protocols, and 4 agent implementation frameworks across 300 Mind2Web benchmark tasks. The results show that task‑specific technology stacks can improve completion rates, efficiency, and social value stability while reducing verbosity and task time.

By Vagner Figueredo de Santana, Diogo Cortiz
arXiv Machine Learning
Oct 12

Algebraic Integer Quantization for Machine Learning Applications

The paper introduces algebraic integer number formats (AINF) as a framework for highly configurable and accurate compressed computation, emphasizing its reduction to purely integer arithmetic for error‑free multiply‑accumulate operations. A matrix multiplication engine leveraging AINF achieves performance close to the most efficient numerical counterparts on modern hardware accelerators. The authors further propose algebraic integer quantization (AIQ) as a lossy compression format for large language model inference, demonstrating competitive accuracy against the industry‑standard MX format at equivalent bit‑per‑parameter budgets, and suggest that higher‑order algebraic integers could yield even greater computational benefits.

By Anthony Kostal-Vazquez, Vassil Dimitrov, Branden Butler
arXiv Computer Vision
Oct 12

NovaSplat: Model-Free 6D Object Pose Estimation via 3D Gaussian Splatting Onboarding and Refinement

NovaSplat is a model‑free pipeline for 6D object pose estimation that builds a 3D Gaussian Splatting (3DGS) model from a short smartphone video during onboarding. The 3DGS model serves both as a template for detection and coarse pose estimation and as a differentiable backbone for pose refinement, which optimizes pose using photometric and geometric losses under a continuous rotation parametrization. The method outperforms all model‑based methods on a household‑object benchmark and all model‑free competitors on a challenging industrial benchmark, while a faster variant, NovaSplat‑Lite, achieves comparable accuracy with a 3.1× speedup.

By Georgios Floros, Evangelos Sartinas, Orestis Vangelis, Maria Pateraki
arXiv Computer Vision
Oct 12

Video Codec-Guided Training-Free Token Compression for Video Large Language Models

Video large language models (VLLMs) need long visual token sequences, making token compression essential for efficient inference. This paper introduces a training‑free token compression method that leverages existing video codec information—such as reference‑frame dependencies, motion vectors, and residuals—to guide low‑cost search for token protection and temporal merging. Experiments on three VLLMs and four benchmarks show that the method matches or surpasses the FlashVID baseline while dramatically reducing feature comparisons.

By Xihua Sheng, Ruijie Liu, Chang Wen Chen
arXiv Computer Vision
Oct 12

Stride: Scale-Aware Multi-Agent Evolutionary Refinement for Infographic Generation

Stride is a multi‑agent framework for infographic generation that separates constraint evaluation from visual grounding, routing global revisions to a composer subagent and local corrections to a retoucher subagent. Each subagent plans internally and produces multiple candidates, and Stride accepts only those that improve a verifier score, thereby preventing regression across iterations. This scale‑aware, redundancy‑rich approach yields stable improvement and outperforms proprietary systems on several benchmarks, achieving state‑of‑the‑art results on WISE and strong performance on BizGenEval and IGenBench.

By Yan Zhou, Yang Yang
arXiv AI
Oct 12

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

The paper introduces MagicBeanStalk, a recursive data‑synthesis method that transforms seed math problems into new, policy‑ and budget‑calibrated training problems. By avoiding the zero‑advantage trap found in large open‑source RLVR corpora, the approach yields significant performance gains on multiple math benchmarks and financial datasets. Experiments show that continual RLVR with MagicBeanStalk problems boosts GPT‑OSS‑120B from 61.2 to 69.8 and DeepSeek v4Flash from 72.4 to 77.9 on a 16K‑token budget, outperforming training on the full source corpus.

By Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi, David Acuna