Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv Machine Learning

Sep 14

On Identifying Adversarial Intent Injection in AI-Native 6G Networks

The paper introduces a threat model for malicious intent injection in AI‑native 6G networks and examines four injection strategies: stealth‑mode, random distribution, increasing frequency, and decreasing frequency. It proposes a dual‑path detection framework combining a CNN with TF‑IDF features for supervised detection and an AutoEncoder trained on benign data for one‑class detection. Evaluation shows the framework achieves higher accuracy (0.97) and F1‑score (0.98) than the state‑of‑the‑art baseline.

By Nilesh Chakraborty, Petar Djukic, Burak Kantarci

benchmarks safety

arXiv Machine Learning

Sep 14

Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage

The paper examines binary-choice truthfulness benchmarks, showing that systematic differences in surface-level features between correct and incorrect answers allow models to perform well without genuine reasoning. Using a six-feature logistic classifier, the authors demonstrate that such leakage is detectable and exploitable, and they find similar artifacts in multiple benchmarks. To mitigate this, they propose a cleaning method called Audit‑Prune that removes the most leakage‑reinforcing answer pairs, releasing a revised TruthfulQA dataset with reduced surface‑feature leakage.

By Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Cullen Anderson, Narmeen Fatimah Oozeer, Jeff M. Phillips

llms benchmarks

arXiv Machine Learning

Sep 14

DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models

DiffusionOPD introduces a multi-task training framework for diffusion models that leverages Online Policy Distillation (OPD). The method trains task-specific teachers separately and then distills their knowledge into a single student model using the student's own rollout trajectories, thereby separating exploration from integration. The authors extend OPD from discrete tokens to continuous-state Markov processes, deriving a closed-form per-step KL objective that unifies stochastic SDE and deterministic ODE refinement, and show that this analytic gradient yields lower variance and better generality than PPO-style gradients. Experiments demonstrate that DiffusionOPD outperforms both multi-reward RL and cascade RL baselines in training efficiency and final performance, achieving state-of-the-art results across all evaluated benchmarks.

By Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, Zuxuan Wu

diffusion reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Sep 14

AnyView: Synthesizing Any Novel View in Dynamic Scenes

AnyView is a diffusion-based video generation framework designed for dynamic view synthesis, requiring minimal inductive biases or geometric assumptions. It trains a generalist spatiotemporal implicit representation using diverse data sources—monocular, multi-view static, and multi-view dynamic—to produce zero-shot novel videos from arbitrary camera locations and trajectories. The authors evaluate AnyView on standard benchmarks, introduce a new challenging benchmark called AnyViewBench for extreme dynamic view synthesis, and demonstrate that AnyView outperforms baselines in maintaining realistic, plausible, and spatiotemporally consistent videos across diverse real-world scenarios.

By Basile Van Hoorick, Dian Chen, Shun Iwase, Pavel Tokmakov, Muhammad Zubair Irshad, Igor Vasiljevic, Swati Gupta, Fangzhou Cheng, Sergey Zakharov, Vitor Campagnolo Guizilini

diffusion benchmarks