Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computer Vision
Oct 21

IndusAgent: Agentic Reinforcement Learning for Cross-Category Industrial Anomaly Detection

arXiv:2605.20682v3 Announce Type: replace Abstract: Small defects in unfamiliar product categories can be difficult to distinguish from benign appearance variations in a global view. Additional reaso...

By Rongbin Tan, Fangfang Lin, Jinguo Luo, Min Qiu, Kejin Cui, Mengmeng Wang, Yi Wang, Jiyuan Wang, Yue Wang, Zhenlong Yuan, Shuhan Song, Huawei Cao
arXiv AI
Oct 21

MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems

MAS-Guard is a unified framework designed to safeguard both single- and multi-agent large language model (LLM) systems. It introduces a local Agent Guardrail (A-Guard) layer that uses task-specific experts to evaluate agentic events, and a system-level Control Plane that applies deterministic structural checks and semantic assessments to routes, capabilities, and message content. The framework enforces interventions—allow, reflection, confirmation, and block—across benchmarks, reducing attack success rates by 71.5‑86.3% while maintaining utility on benign tasks.

By Mohammad Kachuee, Phu Mon Htut, Sheng Li, Nikolaos Pappas, Miguel Ballesteros Martinez, Yassine Benajiba, Sujith Ravi, Dan Roth
arXiv AI
Oct 21

UniWALM: Unified World Action Language Model for Embodied Robot Control

UniWALM is a unified world-action-language model that jointly learns video, action, and language in a single shared representation, using a mixture-of-transformers backbone with diffusion heads that co-denoise across modalities. By generating language as part of the same pass as action, the model closes a language‑as‑memory loop without an external planner, and introduces Flow Matching with Correction to maintain reliable action streams. Experiments show UniWALM outperforms baselines on compositional pick‑and‑place tasks and memory‑dependent benchmarks, and dramatically improves success on a real‑world Find‑the‑Item task when using generated textual memory.

By Rahul Sajnani, Diego Romeres, Mingyuan Zhang, Kai Gao, Carl Qi, Baichuan Huang, Srinath Sridhar, Dylan Randle, Lillian J. Ratliff, Jose Barreiros, Manikantan Nambi, Jeroen Vanbaar
arXiv Computation and Language
Oct 21

TeCoP: Accelerating Diffusion Large Language Model Decoding via Tentative Context Pipelining

TeCoP is a training‑free decoder that accelerates diffusion large language model (dLLM) decoding by decoupling prediction readiness from context utility. It uses tentative context pipelining and draft‑and‑verify to reuse stable but uncommitted predictions as provisional cross‑block context, with commitment based on confidence and stability checks. Across four dLLMs on GSM8K, Minerva MATH, and MBPP, TeCoP cuts forward evaluations by up to 87% and boosts throughput up to 7.5‑fold without KV caching and 11.77‑fold with caching, outperforming Fast‑dLLM and DSB while maintaining accuracy within one percentage point.

By Jui-Chien Tsou, Hsuan-Chi Liu, Wei-Chin Wang, Yu-Syuan Xu, Chun-Yi Lee
arXiv Computation and Language
Oct 21

A Two-Way Street: Surveying the Interplay between Code-Mixing and Native Languages in Text and Speech

The paper surveys how code‑mixed (CM) text and speech systems transfer knowledge between CM data and the native languages involved, a process that can go in both directions. It introduces a taxonomy of transfer direction, route, and modality, and finds that most work transfers from native languages to CM, while the reverse is newer and mainly aligns the two languages. The study notes that text and speech methods overlap but rarely cite each other, and that native data often outperforms new architectures on shared speech benchmarks, highlighting open questions about data mixing and evaluation across scripts.

By Debajyoti Mazumder, Akash Anil
arXiv Computation and Language
Oct 21

SciDiagBench: A Type-Specific Benchmark for Human-Aligned Evaluation of Scientific Diagrams

SciDiagBench is a new benchmark designed to evaluate scientific diagram generation in a way that mirrors human peer‑review practices. It provides a dataset of 165 diagrams from 10 domains, systematically classified into 11 distinct types such as graphical abstracts and technical roadmaps. The benchmark also introduces a preference‑guided calibration method that incorporates human preferences into type‑specific evaluation criteria, aiming to improve the reliability of automated assessments.

By Hao Wang, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Tat-Seng Chua, Tian Tian
arXiv AI
Oct 21

Navalia: Advancing Cowork Agents through Verifiable Task Synthesis and Long-Horizon Post-Training

Navalia is a framework that advances cowork agents by combining verifiable task synthesis with long‑horizon post‑training. It selects supporting evidence from corporate financial reports, builds executable computation chains, and uses independent LLM review and programmatic checks to ensure evidence consistency and answer correctness. The verified tasks are then used for trajectory collection, supervised fine‑tuning, and reinforcement learning, yielding consistent performance gains over the base model across five financial and office‑task benchmarks.

By Hengqi Liu, Ziyuan Liu, Yang Qin, Wanting Zhou, Zichuan Wang, Jiachen Liang, Xu Chu, Shaowei Chen, Zhaokai Luo, Mu Chuan