Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 2

Representation Transitions Reveal Emerging Safety Risks in Multi-Turn LLM Agents

arXiv:2610.00400v1 Announce Type: cross Abstract: Multi-turn attacks on agentic systems can compose individually permissible actions into harmful outcomes, challenging defenses that assess actions or...

By Haoyu Wang, Wei Zhao, Yedi Zhang, Christopher M. Poskitt, Jun Sun

llms agents benchmarks safety

arXiv AI

Oct 2

Gumbel Straight Flow: Distilling Autoregressive Models into One-step Flow Maps

arXiv:2610.00497v1 Announce Type: cross Abstract: We present Gumbel Straight Flow (GSF), a continuous flow map language model that leverages the noise-data coupling of a pretrained autoregressive lan...

By Yeongmin Kim, Arnaud Doucet, Andrew Campbell, Valentin De Bortoli, Thomas Mensink, David Ruhe

llms benchmarks

arXiv AI

Oct 2

ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning

arXiv:2610.00592v1 Announce Type: cross Abstract: In partially observable reinforcement learning (RL), a later observation can make stored information obsolete or change what it implies for the next...

By Oleg Shchendrigin, Egor Cherepanov, Aleksandr I. Panov, Alexey K. Kovalev

agents reinforcement-learning benchmarks

arXiv AI

Oct 2

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation

arXiv:2610.00604v1 Announce Type: cross Abstract: Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappe...

By Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov, Alexey K. Kovalev

rag robotics fine-tuning multimodal benchmarks

arXiv AI

Oct 2

SHARPO: Segment-Level Credit Assignment for Agentic Reinforcement Learning

arXiv:2610.00838v1 Announce Type: cross Abstract: Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error...

By Xinchen Du, Zhengze Zhou, Wenhui Zhu, Han Yu, Sen Na, Rohit Jain, Alborz Geramifard

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Oct 2

Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads

arXiv:2610.00888v1 Announce Type: cross Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per fo...

By Prachi Badarayani, Aidan Jay, Chenghui Zhou, Dayquan Julienne, Yuan Gao, Tianwei Chen, George Zerveas, Ishmam Zabir, Xiren Zhou, Chris Quirk, Xia Song

llms efficiency benchmarks

arXiv AI

Oct 2

Structure-agnostic Causal Representation Learning

arXiv:2610.00968v1 Announce Type: cross Abstract: Causal representation learning aims to discover robust features by exploiting the causal structure underlying data generation. Existing methods requi...

By Arman Behnam, Binghui Wang

benchmarks

arXiv AI

Oct 2

Generalist Representation, Specialist Detection: TS-Router for Time-Series Anomaly Detection

arXiv:2610.00978v1 Announce Type: cross Abstract: Time-series anomaly detection (TSAD) is difficult to generalize across datasets because heterogeneous temporal dynamics imply different notions of no...

By Tian Lan, Yifei Gao, Yimeng Lu, Xuming An, Meng Wang, Yue Pan, Wenjun He, Chenghao Liu, Chen Zhang

benchmarks

arXiv AI

Oct 2

Optimal Transport Reweighting for Robust Learning under Spurious Correlations and Label Noise

arXiv:2610.01028v1 Announce Type: cross Abstract: Machine learning models often suffer performance degradation under subpopulation shift, particularly when spurious correlations cause models to rely...

By Sung Ho Jo, Seonghwi Kim, Wonsang Yun, Minwoo Chae

benchmarks safety

arXiv AI

Oct 2

Parameter-Efficient Distributionally Robust Adaptation of Tabular Foundation Models under Subpopulation Shift

arXiv:2610.01143v1 Announce Type: cross Abstract: Despite strong mean accuracy, tabular foundation models (TFMs) can perform poorly on underrepresented groups under subpopulation shift, where group p...

By Seonghwi Kim, Sung Ho Jo, Minwoo Chae

fine-tuning benchmarks

arXiv AI

Oct 2

Does Scaling Reinforcement Learning Really Require More Training?

arXiv:2610.01133v1 Announce Type: cross Abstract: Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yiel...

By Bangji Yang, Jiajun Fan, Hongba Ma, Ruihan Guo, Ge Liu

reinforcement-learning benchmarks

arXiv AI

Oct 2

Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability

arXiv:2610.01168v1 Announce Type: cross Abstract: Time Series Anomaly Detection has received increasing attention, driven by the growing availability of complex time series data. This surge has led t...

By Roberto Stanzione, Jules Barbe, Magali Parrino, J\'er\'emie Fourmann, Paul Boniol

llms benchmarks safety

arXiv AI

Oct 2

Counterfactual Generation via Flow Matching: Coupling-Sensitive End-to-End Rates

arXiv:2610.01193v1 Announce Type: cross Abstract: Counterfactual generation seeks to sample outcomes under a hypothetical intervention or decision using observational data collected under the factual...

By Yunrui Guan, Krishnakumar Balasubramanian, Shiva Prasad Kasiviswanathan

diffusion benchmarks

arXiv AI

Oct 2

Have an LLM Write Your Anomaly Detector: Autonomous Discovery of Compact, Interpretable Detectors for Time Series

arXiv:2610.01223v1 Announce Type: cross Abstract: Time-series anomaly detection trades off predictive accuracy, computational efficiency, and interpretability. We use a large language model not as th...

By David Berghaus

llms agents benchmarks safety

arXiv AI

Oct 2

Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training

arXiv:2610.01318v1 Announce Type: cross Abstract: Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable atte...

By Hanna Malet, Gabriel Turinici

diffusion benchmarks

arXiv AI

Oct 2

Auto-Formalizing Neuro-Symbolic Predictors

arXiv:2610.01519v1 Announce Type: cross Abstract: Neuro-Symbolic (NeSy) predictors incorporate prior knowledge into the prediction process of neural networks, ensuring that outputs satisfy specified...

By Samuele Bortolotti, Weixin Chen, Han Zhao, Andrea Passerini, Stefano Teso, Antonio Vergari

llms benchmarks

arXiv AI

Oct 2

CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning

arXiv:2610.02039v1 Announce Type: cross Abstract: Recent years have witnessed the rapid adoption of reinforcement learning (RL) in large language model (LLM) post-training, with substantial gains in...

By Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang

llms reinforcement-learning benchmarks

arXiv AI

Oct 2

Higher-Order Molecular Grammars for Generative and Foundation Models in Chemistry

arXiv:2610.02186v1 Announce Type: cross Abstract: Molecular learning models are strongly shaped by their underlying representations. Yet standard sequential and graph formalisms struggle to explicitl...

By Yiming Huang, Yujie Zeng, Vijay Prakash Dwivedi, Simone Foti, Jianmin Wang, Jure Leskovec, Tolga Birdal

fine-tuning benchmarks safety

arXiv AI

Oct 2

FERPO: Forward Entropy-Regularized Policy Optimization

arXiv:2610.02198v1 Announce Type: cross Abstract: Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic....

By Sebastian Sanokowski, Alireza Sarmadi, Majid Khadiv

reinforcement-learning benchmarks

arXiv Computer Vision

Oct 2

Encoded but Disconnected: Decomposing Vision-Language Model Failures under a Patching Null

arXiv:2610.00024v1 Announce Type: new Abstract: Across three vision-language model architectures (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL3-8B), we report a universal negative finding for mid-layer inte...

By Genpei Zhang

llms multimodal benchmarks safety

Get the daily briefing