Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
5d ago

Geometry of Relaxed Fair Regression: A Unified Framework for Aware and Unaware Settings

The paper introduces a unified framework for relaxed fair regression that applies to both aware and unaware settings. By modeling regression with a demographic parity penalty as an optimal transport problem, it characterizes optimal prediction functions under squared Wasserstein‑2 and Total Variation penalties, revealing distinct fairness philosophies. The authors also present FAUST, an efficient algorithm that consistently matches or surpasses state‑of‑the‑art baselines on real‑world benchmarks.

By Marie Generali Lince, Vincent Divol, R\'emi Flamary, Solenne Gaucher, Patrick Loiseau
arXiv Machine Learning
5d ago

Cost-Aware Stopping for Freeze-Thaw Bayesian Optimization: When Does One-Step Lookahead Suffice?

The paper introduces a cost‑aware stopping rule for Freeze‑Thaw Bayesian Optimization called EI stopping, which uses a one‑step lookahead to decide whether to continue evaluating a candidate. EI stopping compares the expected improvement of the next increment to its cost and stops when no improvement exceeds the cost, guaranteeing that it never performs worse than stopping immediately. The authors provide a theoretical certificate that identifies when EI stopping is premature and show through experiments that EI stopping performs comparably to a two‑step lookahead correction in high‑cost scenarios, while the correction outperforms EI stopping at low costs.

By Long Vu, Jongseong Brad Choi, Binoy Ravindran, Zhoulai Fu
arXiv Computation and Language
5d ago

GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs

GraphNIAH is a new benchmark for long‑context reasoning that uses real citation relationships among scientific papers published after a model’s knowledge cutoff. It automatically samples subgraphs of the citation graph and turns each into a long‑context QA example with a verifiable answer, avoiding synthetic data, dataset reuse, manual annotation, and trivial reasoning steps. Evaluation of eight large language models shows that accuracy drops sharply as reasoning depth increases, indicating that current models often fail to fully explore citation relationships even when all necessary facts are present.

By Minh-Nam Tran, Abulhair Saparov
arXiv Computation and Language
5d ago

Keepace.ai: A Technical Report on a Large Language Model for Sports and Health

Keepace.ai is a domain‑specialized large language model for sports and health, built on Qwen3.5‑27B through post‑training. It is optimized for Chinese‑language exercise guidance across three tasks: workout plan generation, sports‑and‑health knowledge QA, and personal sports data analysis. Internal rubric‑based evaluations with domain experts show that Keepace outperforms its base model and larger general‑purpose baselines, especially in exercise safety, boundary compliance, structured output completeness, and scientific accuracy.

By ai Team, Bowen Wu, Heng Wang, Jufeng Mi, Kang Liu, Kun Li, Ling Man, Long He, Suli Fan, Xiaosen Jiang, Yi Wang, Ying He, Yu Tang, Yunhan Li, Zifang Liu
arXiv Machine Learning
5d ago

Intervention-Conditioned Counterfactual Human Digital Twin Generator

The paper introduces CHDT, an intervention‑conditioned Human Digital Twin generator designed to model multimodal longitudinal patient trajectories. It explicitly accounts for irregular visit intervals and cross‑mediator dependencies, which are often neglected in existing models. Experiments on ADNI and ALPACA datasets demonstrate that CHDT achieves higher long‑horizon patient‑state fidelity compared to baseline methods, with irregular‑time conditioning providing the main benefit and cross‑mediator refinement adding further improvement.

By Saeed Farzi, Sofia Michopoulou, Christopher Kipps, Rahman Attar
arXiv Machine Learning
5d ago

Dual-Corrected Stochastic Block Model ($\text{DC}^2\text{SBM}$): A Generalized Structure Detection Method for Attributed Networks

The paper introduces the Dual-Corrected Stochastic Block Model (DC²SBM), a generative framework for community detection in attributed networks that decouples node activity into independent topological and attribute correction factors. By incorporating these factors into link‑generation and attribute‑generation channels and adding a global scaling parameter, the model preserves computational efficiency while improving detection accuracy, especially when topological signals are weak. Experiments on synthetic and real datasets demonstrate that DC²SBM outperforms existing methods and that a single shared activity factor is insufficient when topology and attributes are decoupled.

By Zhenhai Chang, Shuo Wang, Yumeng Liu, Wei Liu
arXiv Machine Learning
5d ago

Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

Mage‑VL‑Cowork is a 4‑billion‑parameter hybrid computer‑use agent that integrates visual GUI interaction with code and terminal execution in a single policy. It is trained via a progressive five‑stage curriculum—mid‑training, supervised fine‑tuning, and reinforcement learning—starting from a multimodally aligned checkpoint. The training pipeline, Mage‑CUA‑Forge, automates task generation and skill conditioning, producing nearly 300,000 desktop and web tasks while cutting per‑task costs by 82.6%, and the resulting model achieves 66.0% on OSWorld‑Verified, setting a new state‑of‑the‑art for its size.

By Xiaoyi Zhang, Haoqing Wang, Xinjie Zhang, Xiang Ming, Xing Wang, Senqiao Yang, Kaichen Zhang, Zhaoyang Jia, Andong Tan, Xinjiang Wang, Jinghao Guo, Yifei Shen, Xiao Li, Peng Zhang, Xun Guo, Jiahao Li, Shicheng Zheng, Jinglu Wang, Zongyu Guo, Wenxuan Xie, Bin Li, Yan Lu
arXiv Machine Learning
5d ago

GraphTFactor: A graph-based approach for proteins transcription factor prediction

GraphTFactor is a new end-to-end framework that represents protein sequences as graphs to directly classify transcription factors. By encoding sequences into graph structures, it handles variable-length inputs and preserves relational information without length constraints. Experiments show that GraphTFactor outperforms state‑of‑the‑art methods on benchmark datasets and offers interpretable insights into sequence regions driving predictions, with an open-source package and web app for easy use.

By Domenico Amato, Salvatore Calderaro, Giosu\`e Lo Bosco, Riccardo Rizzo, Filippo Vella
arXiv Machine Learning
5d ago

Steering Vectors as a Training Signal in LLM Post-Training

Steering vectors (SVs) are residual-stream directions that can shift LLM behavior when added to hidden states during inference. This study investigates whether the same SVs can guide parameter updates during post‑training, exploring supervised learning (SFT and distillation) and reinforcement learning with verifiable rewards (RLVR). The authors find that SV injection improves performance over plain SFT and distillation, achieving up to a 12% gain in certain distillation settings, and that applying a correctness‑contrast direction as advantage repair in RL recovers learning signals for held‑out math benchmarks. They also identify failure boundaries related to code generation, cross‑family layer transfer, and inference‑time application on SV‑trained checkpoints.

By Tiejin Chen, Maunil Vyas, Huaiyuan Yao, Hua Wei
arXiv Machine Learning
5d ago

Differential rank preservation manifold learning

The paper introduces Smooth Rank t‑SNE (SR‑tSNE), a variant of t‑SNE that incorporates a differentiable rank term using a sigmoid approximation to preserve the ordering of neighbors in the embedding. It also proposes Ordinal DLP (O‑DLP), a metric that rewards neighbors only when they appear at their original rank. Across eight diverse benchmarks, SR‑tSNE outperforms 12 baselines in rank preservation, neighborhood overlap, and downstream tasks such as label transfer, cell‑type discovery, and intraday trading.

By Henry Han, Luyu Niu, Diane Li, Jiacun Wang
arXiv Machine Learning
5d ago

TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents

The paper introduces TSBPO, a tree‑structured sampling‑to‑update framework for training interactive language agents. It tackles three intertwined challenges—sampling task and state continuations, assigning delayed feedback, and aggregating token losses—by using a likelihood‑fitted Root/Branch allocator, trajectory and observation‑group advantages, and a clipped token surrogate aggregation. Experiments with a Qwen2.5‑1.5B actor show high success rates on ALFWorld and WebShop benchmarks, demonstrating the effectiveness of the proposed training configuration.

By Rui Yu, Chong Mu, Jinchuan Zhang
arXiv Computer Vision
5d ago

Hawkeye: Reading Illegible Text with Vision Language Models

The paper introduces BSTRB, a benchmark of 8,398 noisy image samples with high‑fidelity annotations from 1,699 real‑world video sequences, designed to evaluate Vision‑Language Models (VLMs) on degraded text. It presents Hawkeye, a VLM fine‑tuned on BSTRB that reads and reasons about illegible text by using contextual priors and pixel‑level alignment. Hawkeye performs comparably to larger VLMs overall and surpasses them on text at or below 20 pixels, making it suitable for real‑time deployment in noisy egocentric settings.

By Brian Tang, Kang G. Shin
arXiv Machine Learning
5d ago

WASABI: Wasserstein Sampling via Bridge Interpolants

WASABI (Wasserstein Sampling via Bridge Interpolants) is a diffusion‑model based method for sampling from Boltzmann distributions using only the energy function. It trains the model by minimizing the KL divergence to the target distribution through Wasserstein gradient flows, avoiding the need to compute the model’s log‑density gradient. After training, WASABI can generate new samples without further energy or gradient evaluations, achieving superior performance on alanine dipeptide and competitive results on interacting particle benchmarks.

By Sebastian Salazar, Dilum Fernando, Michal Kucer, Emily Casleton, Luhuan Wu, David M. Blei
arXiv Computer Vision
5d ago

TwinPast: Benchmarking Historical Evidence Use in Streaming Video Understanding with Evidence Contrastive Video Pairs

TwinPast is a new benchmark that tests whether streaming video models use historical evidence by presenting pairs of videos that differ only in critical past events while keeping recent context, questions, and options the same. The study evaluates 34 model configurations on 350 human‑reviewed pairs across five historical tasks, finding that most models rely on recent frames or the question alone, with only a few showing improved performance when historical inputs are provided. Even the best streaming system scores only 49.14% PairAcc, far below general‑purpose models, and case studies reveal that models often select states from the wrong time or contradict the recorded event order.

By Haotian Yang, Yong Chen, Cheng Peng, Haobo Wang, Zhongle Xie, Xuanru Li, Gang Chen
arXiv Computer Vision
5d ago

ReLift: Dense 3D Tracking as Lifted Trajectory Repair

ReLift presents a feed‑forward framework that treats dense 3D tracking as a problem of repairing and completing lifted trajectories. By alternating attention along and across trajectories, it corrects depth and correspondence errors, and fills in missing motion, all while training solely on simulated 3D trajectory data. The method improves lifted trajectories from any 2D tracker or geometry model, achieving state‑of‑the‑art accuracy on WorldTrack and running faster than leading 4D trackers.

By Assaf Singer, Maroun Khriesh, Noam Rotstein, Yoni Kasten, Or Litany
arXiv Machine Learning
5d ago

Latent Reasoning in Continuous Space for Unified Multimodal Models

The paper introduces Latent Reasoning in Continuous space (LARC), a framework that replaces discrete token-based reasoning with reasoning over continuous hidden states to better align with image generation’s continuous nature. LARC is trained in two stages: curriculum supervised fine‑tuning that gradually shifts from token to latent reasoning, followed by an information‑gain reinforcement learning stage that refines latent reasoning without ground‑truth supervision. Experiments on multiple text‑to‑image benchmarks show that LARC improves generation quality, prompt adherence, and compositional fidelity, achieving state‑of‑the‑art results on GenEval and demonstrating robust test‑time scaling.

By Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, Jinwoo Shin
arXiv Machine Learning
5d ago

MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering

MLLMEraser is a training‑free, input‑aware framework that achieves test‑time unlearning for multimodal large language models by steering activations. It constructs an erasure direction through adversarial contrast between knowledge‑recall and knowledge‑erasure image‑text pairs, and applies this direction selectively to preserve retained knowledge while forgetting specified content. Experiments on LLaVA‑1.5 and Qwen‑2.5‑VL show that MLLMEraser outperforms existing unlearning baselines with lower computational cost and minimal utility loss.

By Chenlu Ding, Jiancan Wu, Leheng Sheng, Fan Zhang, Yancheng Yuan, Xiang Wang, Xiangnan He
arXiv Computer Vision
5d ago

MIC-BEV: Multi-Infrastructure Camera Bird's-Eye-View Transformer with Relation-Aware Fusion for 3D Object Detection

MIC-BEV is a Transformer-based bird's-eye-view perception framework designed for infrastructure-based multi-camera 3D object detection. It supports a variable number of heterogeneous cameras and employs a graph-enhanced fusion module to adaptively integrate multi-view image features by modeling geometric relationships between cameras and the BEV grid. The authors introduce the M2I synthetic dataset and evaluate MIC-BEV on both M2I and the real-world RoScenes dataset, demonstrating strong detection performance, improved robustness under challenging conditions, and practical feasibility for deployment.

By Yun Zhang, Zhaoliang Zheng, Johnson Liu, Zhiyu Huang, Zewei Zhou, Zonglin Meng, Tianhui Cai, Jiaqi Ma