Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 23

WeightBridge: An Efficient Weight Transfer Library for Reinforcement Learning

WeightBridge is a lightweight library that streamlines weight transfer between trainers and rollout generators in reinforcement learning systems, particularly for large language models. It automatically maps trainer and rollout weight layouts, then performs redundancy‑free, load‑balanced transfers while supporting various synchronization modes. Experiments show that WeightBridge can cut GPU stall time by up to 42× compared to leading open‑source RL frameworks, and it was easily integrated into two different frameworks by a coding agent.

By Xuanlin Jiang, Samuel Hsia, Michael Kuchnik, Zachary DeVito, Minlan Yu, Carole-Jean Wu
arXiv Computation and Language
Sep 23

LLM-Driven Training-free Location-Attribute Synergic Fusion: A Closed-Loop Paradigm for Dual-source Encrypted POIs and LULC Mapping

The paper introduces a novel, training‑free, LLM‑driven closed‑loop framework for fusing dual‑source encrypted points of interest (DSEP) to improve land‑use/land‑cover (LULC) mapping. By iteratively refining location transformations and attribute correspondences—using LLM‑based matching, particle swarm optimization, and fuzzy attribute reassessment—the method reduces matching complexity from O(N²) to O(N) and converges in about two iterations. Experiments across 31 Chinese provincial capitals demonstrate a 4.58 m average location residual and 95.12 % attribute accuracy, outperforming existing baselines by 1.77 m and 14.87 % respectively, and enabling georeferencing of encrypted vector data to WGS‑84 without ground control points.

By Chang Li, Xingtao Peng, Yongjun Zhang, Yinfei He, Cairun Huang
arXiv Computation and Language
Sep 23

SpecialEduBench: Benchmarking Vision-Language Models on Knowledge, Skill, and Attitude in Language Intervention for Autistic Children

SpecialEduBench is a new benchmark for vision‑language models that evaluates their pedagogical competence in language intervention for autistic children across knowledge, skill, and attitude dimensions. It contains 4,537 knowledge items, 200 skill items, and 68 attitude items derived from recorded interventions, with 192 response cells for attitude items that combine pressure and monitoring. Eight leading vision‑language models were tested, none achieving full performance, especially in honesty cells, indicating that current models still struggle with situated teaching tasks.

By Jihoi Na, Taeyeong Kim, Sungjune Kong, Jaemin Jung, Min Joung Park, Kyungtae Joo, Ahhyun Kim, Shim Jaechang, Sooyoung Joo, Dongjin Ka, SeJoong Kim, Jimin Kim, HyunJin Jung, Unggi Lee
arXiv Computer Vision
Sep 23

Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Models

Video-HopChain introduces a new dataset of 22,550 multi‑hop video questions over 13,378 videos, each question consisting of three to six yes/no sub‑questions whose integer answers sum to a verifiable reward. Training a Qwen3‑VL‑8B model with GRPO on this dataset improves performance across eight video‑understanding benchmarks from 55.4 to 57.9, and the addition of Confidence‑Gated Exploration (CGE) raises the mean to 59.3. The authors release the dataset, checkpoint, and training code for further research.

By Trung Nguyen Quang, Yuhao Dong, Shuo Sun, Shuai Liu, Shulin Tian, Kim-Hui Yap, Ziwei Liu