Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 30

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

The paper introduces a decision‑theoretic framework that splits a large language model’s decision loss into belief formation and action selection components. Using a synthetic benchmark, it evaluates how reinforcement‑learning interventions on beliefs, decisions, or both affect these components across three domains. The study finds that targeting a single component improves that part but may not transfer to others, while jointly targeting both improves both only when training and evaluation formats match.

By Huaman Sun, Dingcheng Wang, Jason Hartline, Jessica Hullman
arXiv Computer Vision
Sep 30

MeshSplatBench: A Unified Benchmark for Triangle- and Mesh-Based Neural Rendering

MeshSplatBench is the first benchmark designed to evaluate triangle- and mesh-based neural rendering methods from native rendering to deployment in graphics engines such as Unity and Blender. It introduces a hierarchical deployment protocol with standard and dedicated options, and a structural audit for mesh splatting to assess topological and geometric integrity. The benchmark’s evaluations show that graphics engine deployment degrades quality, dedicated deployment preserves fidelity at a significant speed cost, and current mesh splatting methods lack sufficient connectivity for manifoldness.

By Kaixuan Zhang, Minxian Li, Mingwu Ren, Xiatian Zhu
arXiv AI
Sep 30

Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO

The paper introduces FastRL, a reinforcement learning framework designed to enhance the efficiency of Group Relative Policy Optimization (GRPO) and its variants. FastRL employs an advantage-aware pruning strategy that retains high-advantage trajectories while preserving gradient diversity, and an adaptive rollout sampling mechanism that adjusts sampling scale during training based on historical pruning data. Experiments show that FastRL can be integrated into GRPO, DAPO, and GSPO, yielding a 2.07× speedup on Geometry3K and GeoQA8K-R1V and a 1.64% accuracy improvement on visual reasoning benchmarks.

By Jiahua Yang, Zhiwei Yang, Xianpeng Zhang, Dongyu Chen, Xing Chen, Tianhuang Su, Haonan Lu, Quanlong Guan, Kai Tang, Chuangchuang Wang