Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computer Vision
Oct 21

Learning the 4D World through Point Trajectory Forecasting

arXiv:2610.28594v1 Announce Type: new Abstract: Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Mo...

By Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin
arXiv Computer Vision
Oct 21

MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction

arXiv:2606.18558v2 Announce Type: replace Abstract: Motion forecasting is central to visual intelligence: agents must anticipate how objects will move in order to plan actions, reason about physical...

By Jianing Zhang, Chenhao Zheng, Yajun Yang, Max Argus, Rustin Soraki, Winson Han, Taira Anderson, Chun-Liang Li, Shuo Liu, Jiafei Duan, Zhongzheng Ren, Jieyu Zhang, Ranjay Krishna
arXiv Computer Vision
Oct 21

PCXR-Copilot: Device-aware pediatric chest radiographic analysis with segmentation, suppression, and edema-related cues

arXiv:2610.25486v2 Announce Type: replace Abstract: Patients admitted to pediatric ICU and cardiac ICU often require intensive perioperative support and frequent chest radiography to monitor support...

By Xueyang Li, Alec Girten, Peng Zeng, Mingze Jiang, Shaoyang Huang, Gelei Xu, Danny Z. Chen, Yiyu Shi, John N. Kheir
arXiv Computation and Language
Oct 21

RealControlBench: A Benchmark for Interpretability-Informed Fine-Grained Control of LLMs and Agents

RealControlBench is a new benchmark that evaluates fine‑grained steering of large language models and agents across diverse behavioral targets and interaction settings. It extends beyond isolated generation tasks to include complex, execution‑grounded, multi‑turn agent interactions, covering instruction‑following control, tool‑use decision control, reasoning performance, reasoning length, and safety awareness. The benchmark jointly measures target controllability and capability preservation, testing robustness across prompts, domains, and model architectures, and finds that current steering methods are often unreliable in realistic scenarios.

By Xinying Cai, Bin Hu, Mingyu Jin, Qiming Zhang, Yang Ouyang, Bangwei Guo, Minghao Guo, Yutong Yin, Lang Gao, Feijiang Han, Sheldon Yu, Kaiqu Liang, Qingcheng Zeng, Jingcheng Niu, Fan Yang, Wei Cheng, Zhen Xiang, Julian McAuley, Zhaoran Wang
arXiv Computation and Language
Oct 21

Judging the Justification: Decision Certificate Supervision for Generative Reward Models

The paper introduces DeCert, a framework that evaluates the validity and verdict consistency of rationales generated by generative reward models (GenRMs) without needing reference explanations. DeCert translates these evaluations into preference supervision that considers both verdict correctness and justification quality, enabling the construction of rationale preference pairs that differentiate stronger from weaker justifications. Experiments across multiple model families and benchmarks show that DeCert outperforms strong generative judge baselines, improves rationale quality, and mitigates positional bias, raising position-consistent accuracy by up to 14.8 points.

By Guiquan Sun, Bing Bai, Martin Renqiang Min, Dongjin Song
arXiv Computation and Language
Oct 21

EviDial: When Correct Depression Predictions Lack Evidence

EviDial is a new benchmark for depression dialogue that separates label agreement from evidence grounding. It contains 382 dialogues and 2,744 participant-symptom cells, each annotated as absent, partial, conflicting, or sufficient evidence. The study finds that most predictions that match benchmark labels lack sufficient textual evidence, highlighting a gap between label accuracy and evidence-based assessment.

By Junjie Xu, Jin Zhu, Yuzhe Yang, Luhan Wang, Xingjiao Wu, Daoguo Dong, Jing Yang, Wenxin Wu, Liang He
arXiv Computation and Language
Oct 21

From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution

The paper introduces domain-specific languages (DSLs) as an explicit representation layer to bridge the gap between high-level natural-language abstractions and concrete actions. Experiments on the visually-grounded “Hexagons” task and the “Text2CAD” benchmark show that code generation using a human-curated DSL yields dramatic performance gains, especially at higher abstraction levels, while automatically induced DSLs only partially recover these gains. The study highlights that the right representation can close much of the performance gap caused by abstraction, but that automatically learning such representations remains a challenge.

By Amit Lisha, Reut Tsarfaty
arXiv AI
Oct 21

MAMath-Bench: Process-Aware and Robust Evaluation of Multimodal Multi-Agent Mathematical Reasoning

MAMath-Bench is a new benchmark that evaluates multimodal mathematical reasoning in multi‑agent systems (MASs). It contains 1,003 problems across 11 categories, multiple difficulty levels, and modalities such as text, images, and audio, with fine‑grained milestone annotations for correctness, reasoning‑process reliability, and robustness to adversarial perturbations. The study finds that while collaboration can improve intermediate reasoning coverage, it does not consistently boost final‑answer accuracy and can increase redundancy, and current systems remain vulnerable to subtle adversarial errors.

By Xinping Wang, Jie Ma, Rui Xing, Wensong Guo, Xiayi Zhang, Jianhao Deng, Zihao Zhou, Boqian Yang, Feijuan He