Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 8

Multi-Agent Coordination via Support-Preserving Distillation

The paper introduces Mode‑Support Semi‑Discrete Optimal Transport (MoSDOT) to address a failure mode in offline multi‑agent reinforcement learning where flow‑based teachers produce conflicting coordination modes. MoSDOT aggregates multimodal replay into a finite mode support with specified capacities and assigns each noise sample to a single mode before teacher training, thereby preventing the teacher from generating samples between valid modes. Experiments on controlled diagnostics and offline MARL benchmarks show that MoSDOT improves endpoint quality and routing consistency, especially on datasets with multimodal joint behavior.

By Sangmin Lee, Youngju Na, Chanmi Lee, Sung-eui Yoon
arXiv Computer Vision
Oct 8

Video Prediction Policy 2: Predict Better, Act Better

Video Prediction Policy 2 (VPP2) is a new world action model that improves zero‑shot generalization for both video prediction and action generation. It achieves this by pretraining a large, diverse manipulation video dataset with event‑level supervision, then distilling the model into a single‑step visual planner and adding a mixture‑of‑transformers action module. Experiments show VPP2 outperforms leading baselines on open‑ended video prediction, real‑world zero‑shot manipulation, and several challenging benchmarks.

By Yanjiang Guo, Haodong Yan, Zhide Zhong, Zhongru Zhang, Qingyuan Yang, Qingzhou Lu, Xiaoyu Chen, Yen-Jen Wang, Shuying Deng, Chenghan Yang, Puzhen Yuan, Chenxin Liu, Tun Ban, Xiang Zhu, Yichen Liu, Kun Feng, Haoang Li, Jianyu Chen
arXiv Machine Learning
Oct 8

Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance

The paper investigates how preprocessing choices influence time‑series forecasting performance. By evaluating 11 models across 16 reversible preprocessing pipelines on 29,000 M4 series, it shows that optimizing preprocessing can improve accuracy by 27% to 87%, especially for models without built‑in preprocessing. This demonstrates that simpler architectures can rival complex state‑of‑the‑art models when preprocessing is properly handled.

By Guilherme Afonso Galindo Padilha, Paulo Salgado Gomes de Mattos Neto, Rafael Menelau Oliveira e Cruz
Simon Willison
Oct 7

Claude Haiku 5.5

Claude Haiku 5.5 is Anthropic’s new fast, low‑cost model that matches GPT‑6 Luna’s pricing of $0.10 per 1,000 input tokens and $0.50 per 1,000 output tokens up to 100,000 tokens, with a 5× increase beyond that threshold. It uses a less generous tokenizer, meaning the same long prompt consumes about 1.25× more tokens than Haiku 4.5, which raises hidden costs. Benchmarks show Haiku 5.5 outperforms Luna within the 100,000‑token limit, but Luna becomes more economical for larger workloads.