Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computer Vision

Sep 14

EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use

EventMemAgent is an active online video agent that uses a hierarchical memory module to handle continuous perception and long‑range reasoning in streaming video. The framework employs a short‑term memory layer to detect event boundaries and sample frames within a fixed buffer, while a long‑term memory layer archives observations event‑by‑event. It also incorporates a multi‑granular perception toolkit and Agentic Reinforcement Learning to internalize reasoning and tool‑use strategies, achieving competitive results on online video benchmarks.

By Siwei Wen, Zhangcheng Wang, Xingjian Zhang, Lei Huang, Wenjun Wu

llms agents reinforcement-learning multimodal benchmarks

arXiv Computer Vision

Sep 14

Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap

ReVisIT is a train‑free framework that turns retrieved image‑label pairs into units of visual thought, combining structured class definitions, multimodal retrieval, and alternating user/assistant injection before joint decoding. On several benchmarks—including Fast Open MiniImageNet, Bongard‑OpenWorld, and the newly released MAAC‑Bench—ReVisIT achieves performance comparable to or surpassing large, trained models while using far fewer parameters. The approach demonstrates that high‑quality retrieval and a simple turns layer can provide a universal performance boost across diverse multimodal tasks.

By Bingchen Huang, Zhiling Wang, Yifu Chen, Yuanchao Du

llms multimodal benchmarks

arXiv Machine Learning

Sep 14

On Identifying Adversarial Intent Injection in AI-Native 6G Networks

The paper introduces a threat model for malicious intent injection in AI‑native 6G networks and examines four injection strategies: stealth‑mode, random distribution, increasing frequency, and decreasing frequency. It proposes a dual‑path detection framework combining a CNN with TF‑IDF features for supervised detection and an AutoEncoder trained on benign data for one‑class detection. Evaluation shows the framework achieves higher accuracy (0.97) and F1‑score (0.98) than the state‑of‑the‑art baseline.

By Nilesh Chakraborty, Petar Djukic, Burak Kantarci

benchmarks safety

arXiv Machine Learning

Sep 14

Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage

The paper examines binary-choice truthfulness benchmarks, showing that systematic differences in surface-level features between correct and incorrect answers allow models to perform well without genuine reasoning. Using a six-feature logistic classifier, the authors demonstrate that such leakage is detectable and exploitable, and they find similar artifacts in multiple benchmarks. To mitigate this, they propose a cleaning method called Audit‑Prune that removes the most leakage‑reinforcing answer pairs, releasing a revised TruthfulQA dataset with reduced surface‑feature leakage.

By Foad Namjoo, Remy Ogasawara, Amirali Abdullah, Cullen Anderson, Narmeen Fatimah Oozeer, Jeff M. Phillips

llms benchmarks

arXiv Machine Learning

Sep 14

DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models

DiffusionOPD introduces a multi-task training framework for diffusion models that leverages Online Policy Distillation (OPD). The method trains task-specific teachers separately and then distills their knowledge into a single student model using the student's own rollout trajectories, thereby separating exploration from integration. The authors extend OPD from discrete tokens to continuous-state Markov processes, deriving a closed-form per-step KL objective that unifies stochastic SDE and deterministic ODE refinement, and show that this analytic gradient yields lower variance and better generality than PPO-style gradients. Experiments demonstrate that DiffusionOPD outperforms both multi-reward RL and cascade RL baselines in training efficiency and final performance, achieving state-of-the-art results across all evaluated benchmarks.

By Quanhao Li, Junqiu Yu, Kaixun Jiang, Yujie Wei, Zhen Xing, Pandeng Li, Ruihang Chu, Shiwei Zhang, Yu Liu, Zuxuan Wu

diffusion reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Sep 14

AnyView: Synthesizing Any Novel View in Dynamic Scenes

AnyView is a diffusion-based video generation framework designed for dynamic view synthesis, requiring minimal inductive biases or geometric assumptions. It trains a generalist spatiotemporal implicit representation using diverse data sources—monocular, multi-view static, and multi-view dynamic—to produce zero-shot novel videos from arbitrary camera locations and trajectories. The authors evaluate AnyView on standard benchmarks, introduce a new challenging benchmark called AnyViewBench for extreme dynamic view synthesis, and demonstrate that AnyView outperforms baselines in maintaining realistic, plausible, and spatiotemporally consistent videos across diverse real-world scenarios.

By Basile Van Hoorick, Dian Chen, Shun Iwase, Pavel Tokmakov, Muhammad Zubair Irshad, Igor Vasiljevic, Swati Gupta, Fangzhou Cheng, Sergey Zakharov, Vitor Campagnolo Guizilini

diffusion benchmarks