Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Machine Learning
Sep 30

Relative Kinetic Utility: Calibrating Cross-Layer Credit for Global Structured LLM Pruning

The paper introduces Global Relative Kinetic Utility (Global RKU), a label‑free method for calibrating cross‑layer credit in global structured pruning of large language models. Global RKU estimates channel importance via a final‑hidden‑state activation‑gradient signal and applies block‑relative normalization to remove block‑common scale while preserving within‑block ordering, enabling a single‑stage static pruning topology. Experiments on Qwen‑2.5‑7B show significant performance gains at various sparsity levels, and ablation studies confirm the effectiveness of the relative‑normalization step.

By Tianhao Qian, Guilin Qi, Jiayu Chen
arXiv Machine Learning
Sep 30

Alignment-Guided Flow Transformer for Efficient Vision-Language-Action Policy Learning

The paper introduces Alignment‑Guided Flow Transformer (AGFT), a framework for Vision‑Language‑Action (VLA) models that explicitly enforces tri‑modal alignment among vision, language, and action through a dedicated alignment loss. AGFT bridges representational gaps across modalities, improving task adaptation and robustness, and employs a flow‑matching objective to reduce inference steps compared to diffusion‑based policies. Experiments on a large benchmark demonstrate that AGFT achieves higher success rates and lower inference latency than state‑of‑the‑art baselines, highlighting tri‑modal alignment as crucial for scalable VLA manipulation.

By Shengchao Hu, Peng Wang, Qiyang Zhou, Guodong Zheng, Yuqi Huang, Li Shen, Ya Zhang, Dacheng Tao
arXiv Machine Learning
Sep 30

ALICE: In-context, Zero-shot, Mutual Information Estimation

ALICE is a foundation model that estimates mutual information (MI) without per‑distribution training. Trained only on synthetic distributions, it acts as an in‑context estimator of rectified‑flow velocity fields, producing MI via a fixed identity that integrates squared differences between joint and conditional fields. The authors validate ALICE on a challenging benchmark and demonstrate its applicability to unseen data in biology, genetics, and neuroscience, achieving performance comparable to neural estimators trained separately for each distribution.

By Giulio Franzese, Simone Rossi, Pietro Michiardi
arXiv Machine Learning
Sep 30

Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping

The paper introduces AttWarp, a lightweight technique that uses a multimodal large language model’s cross‑modal attention to perform rectilinear warping of input images at test time. By reallocating spatial resolution toward query‑relevant regions without altering model weights or architecture, AttWarp preserves global context while making small objects and subtle relationships easier for the model to read. Experiments on five benchmarks and four MLLMs show consistent accuracy gains, improved compositional reasoning, and reduced hallucinations compared to baseline image‑manipulation methods.

By Dwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim, Madhav Kanda, Hyeonjeong Ha, Svetlana Lazebnik, Heng Ji, Unnat Jain
arXiv Machine Learning
Sep 30

Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference

The paper introduces Minimum Specification Perturbation (MSP), a metric that counts the smallest number of analyst decisions that must be altered to make a causal study’s confidence interval include zero. MSP is small under the null hypothesis, grows with effect size, and provides a distance‑to‑falsification measure that traditional dispersion‑based robustness tools cannot capture. The authors demonstrate that MSP and the Fragility Index assess different vulnerabilities, and show that on the LaLonde benchmark MSP equals one, meaning a single decision change would render the estimate statistically insignificant.

By Hoang Dang, Luan Pham, Minh Nguyen
arXiv Computer Vision
Sep 30

EGSD: Event-Grounded Self-Distillation for Streaming Video Understanding

The paper introduces Event‑Grounded Self‑Distillation (EGSD), a method for real‑time video understanding that treats streaming memory as incremental updates over verifiable events such as key visual entities, actions, and details. EGSD adapts on‑policy self‑distillation by combining a multiplicative weight with outcome reward to align teacher and student preferences, and re‑weights the teacher with event information plus an entity‑coverage reward to counter question‑relevance bias. Experiments on StreamingBench and OVO‑Bench Real‑Time track show EGSD achieves 79.8 % and 73.4 % accuracy respectively, while improving effective‑entity recall by 17.4 % with only a 6.8 % increase in memory length.

By Yuwei Miao, Xuesheng Zhang, Wenhao Zou, Jixia Zhang, Jianwei Lv, Bo Yuan, Junfeng Wang, Shiao Xie