Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Oct 1

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

OmniReasoning introduces a new benchmark, OmniReasoningBench, that requires both audio and visual evidence for answering 1,150 multiple-choice and open-ended questions across two tasks. The authors also develop OmniQA, a data engine that automatically generates evidence‑grounded QA pairs with time‑stamped clue chains, producing training datasets OmniReasoning‑SFT‑112K and OmniReasoning‑RL‑19K. Finally, they propose Modality‑Factored Self‑Distillation (MFSD), an on‑policy self‑distillation method that assigns token‑level credit by evaluating responses under modality‑specific clue contexts, enabling the OmniReasoning‑30B‑A3B model to achieve significant performance gains on both the new benchmark and existing video benchmarks.

By Junming Lin, Yuxuan Wang, Zhenxin Lei, Yuxin Liu, Ruixun Liu, Yinsong Yan, Ling Wang, Minghao Han, Yunfei Chu, Shun Lei, Xueyao Zhang, Qize Yang, Jin Xu, Yiwu Zhong
arXiv AI
Oct 1

Policy-Conditioned AI-Use Detection: An Evidentiary Framework for Academic Publishing

The paper introduces a policy‑conditioned AI‑use detection framework for academic publishing, arguing that traditional AI detection tools misalign with venue rules by merely identifying AI‑generated text. Instead, the proposed system treats the governing policy as an explicit input, generating hypotheses, evidence, calibration, and uncertainty rather than binary verdicts. It outlines how to benchmark compliance, evaluate true positive rates at venue‑specified false positive thresholds, and emphasizes the need for structured disclosure, tool routing, and contestable findings.

By Jairo Diaz-Rodriguez, Mumin Jia
arXiv AI
Oct 1

Diversity Combining for Multi-Path LLM Reasoning

The paper treats multi‑path large language model reasoning as a diversity‑combining problem, analogous to noisy channel observations in wireless communications. It shows that the optimal symmetric linear combiner of latent embeddings is uniform, justifying majority vote in standard self‑consistency while allowing for weighting or pruning when prompt‑template branches are heterogeneous. By reducing path correlation through prompt‑template diversity, the authors propose an Adaptive‑K rule that selects an optimal number of reasoning paths, preserving most of the accuracy achieved with a fixed large number of paths across multiple models and benchmarks.

By Guangsheng Yu, Litianyi Zhang, Qin Wang, Xu Wang, Mingyuan Li, Shaoxiong Ji, Ren Ping Liu, Massimo Piccardi