Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computer Vision
Sep 17

PolyLayout: Multi-room Manhattan Layout Estimation

arXiv:2608.03323v2 Announce Type: replace Abstract: Estimating room layouts from multi-view imagery is a core task for indoor scene understanding. Existing methods are typically limited either by poo...

By Gustav Hanning, Shaohui Liu, R\'emi Pautrat, Marc Pollefeys, Kalle {\AA}str\"om, Viktor Larsson
arXiv Computer Vision
Sep 17

ChatGPT Images 2.5 on Forgery Tasks: Testing Advertised Improvements Against Known Answers

arXiv:2609.13617v2 Announce Type: replace Abstract: OpenAI released ChatGPT Images 2.5 on 8 September 2026, advertising more precise local edits, better consistency across edits, more faithful refere...

By Ankit Raj, Yuxin Zhang, Kidus Zewde, Tommy Duong, Jiaqi Gan, Xingyu Shen, Yuchen Zhou, Huaiyu Guo, Siyu Zhang, Simiao Ren
arXiv Machine Learning
Sep 17

FedPGT: Progressive Gradient Transmission for Vehicular Federated Learning over Time-Varying Channels

FedPGT introduces a progressive gradient transmission scheme for vehicular federated learning over time‑varying channels, where vehicles send high‑magnitude gradient entries according to instantaneous channel conditions. The authors derive a convergence bound showing diminishing returns governed by a power‑law decay, and formulate a stochastic optimization problem that is solved via a Lyapunov drift‑plus‑penalty approach with per‑slot surrogate variables. A low‑complexity resource allocation algorithm is proposed, and experiments on CIFAR‑10 and Argoverse demonstrate a 3.65% accuracy gain and a 12.66% reduction in displacement error compared to state‑of‑the‑art baselines.

By Jintao Yan, Tan Chen, Yuxuan Sun, Sheng Zhou, Zhisheng Niu
arXiv AI
Sep 17

OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning

OBC‑Prune introduces an outcome‑based calibration approach for pruning large reasoning models, focusing on the causal importance of each reasoning sentence rather than uniform activation salience. By pairing correct and incorrect rollouts and using intervention‑based analysis, it assigns per‑token weights that guide one‑shot pruning methods such as SparseGPT, Wanda, and ALPS. Experiments on DeepSeek‑R1‑Distill‑Qwen models show consistent accuracy gains and shorter reasoning traces across multiple benchmarks at 40‑50% sparsity.

By Ha Lan Nguyen, Huy Hoang Tran, Trac-Duy Tran, Dung D. Le