Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language
Sep 25

Artificial Societies Benchmark: A Validation Framework for Synthetic Research

The article introduces the Artificial Societies Benchmark, a validation framework designed to evaluate synthetic populations used in research. It comprises eleven tests covering internal, construct, and external validity, drawing on twenty human data sources and comparing nine language models. The benchmark links specific research uses to the evidence required and assesses how results vary with different respondent information, revealing that strong performance in one domain does not guarantee fidelity in others.

By Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis, James K. He
arXiv Computation and Language
Sep 25

Reward-Tilted On-Policy Distillation for Acoustic Grounding in Audio-Language Models

The paper introduces Reward‑Tilted On‑Policy Distillation (RT‑OPD), a method that enhances acoustic grounding in audio‑language models by using a frozen teacher to generate a reward based on the contrast between token predictions with and without audio. This reward reshapes the teacher distribution for reverse‑KL distillation, encouraging students to rely more on acoustic evidence. Experiments on two compact students across three benchmarks show that RT‑OPD consistently outperforms vanilla OPD, and a 3B model trained with RT‑OPD achieves 72.72% accuracy on the MMAU benchmark, surpassing other 3B models and rivaling larger 7B and 8B models.

By Kaiyang Li, Shaobo Han, Yue Tian, Shihao Ji
arXiv Computation and Language
Sep 25

Free the Language Model From the Vision Encoder: Semantic Serialization as a Perception Interface for Small Language Models

The paper introduces a perception interface that separates vision from language in vision‑language models. A frozen perception stack detects objects, a deterministic semantic serializer converts the perceived state into text, and a standard text‑only large language model (LLM) answers questions. Experiments on a campus‑robot benchmark show that this serialized interface outperforms a zero‑shot VLM of the same language‑model size, especially as the language model shrinks, and that the advantage persists under paraphrase and different supervision regimes.

By Cong Xu, Ravi Sankar
arXiv Computation and Language
Sep 25

ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

The paper introduces ModularSQL, a lightweight runtime guardrail designed to detect and correct multiplicity errors—such as missing DISTINCT clauses, inflated aggregates, and Cartesian join explosions—in Text-to-SQL systems. It highlights the Multiplicity Blind Spot (MBS), where standard set-based accuracy metrics fail to capture these errors, and proposes Multiset-EX as a more comprehensive evaluation criterion. Experiments on several models show that ModularSQL can improve multiplicity-aware accuracy while adding minimal computational overhead.

By Tianxin Zhou, Ruixi Lin
arXiv Computer Vision
Sep 25

Interpretable Similarity of Synthetic Image Utility

The paper introduces Interpretable Utility Similarity (IUS), a novel metric for quantifying how closely a synthetic image set matches a real image set in terms of usefulness for deep‑learning clinical decision support systems. IUS is interpretable, leveraging generalized neural additive models to explain why one synthetic dataset may outperform another based on clinically relevant image features. Experiments on color medical imaging modalities—endoscopic, dermoscopic, and fundus—show that selecting synthetic images with high IUS can boost classification performance by up to 54.6%, and the method also generalizes to grayscale X‑ray and ultrasound data.

By Panagiota Gatoula, George Dimas, Dimitris K. Iakovidis
arXiv Computer Vision
Sep 25

OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding

OptiSAR-Net++ introduces a new cross‑domain remote sensing visual grounding task (CD‑RSVG) and the first large‑scale benchmark dataset, OptSAR‑RSVG. The framework replaces Transformer decoding with a CLIP‑based contrastive approach, employing a patch‑level Low‑Rank Adaptation Mixture of Experts for efficient cross‑domain feature decoupling and a text‑guided dual‑gate fusion module for improved semantic‑visual alignment. Experiments show state‑of‑the‑art performance on OptSAR‑RSVG and DIOR‑RSVG, with notable gains in localization accuracy and computational efficiency.

By Xiaoyu Tang, Jun Dong, Jintao Cheng, Rui Fan
arXiv Computer Vision
Sep 25

Bridging the Inter-Domain Gap through Low-Level Features for Cross-Modal Medical Image Segmentation

The paper introduces LowBridge, a method for cross‑modal medical image segmentation that leverages shared low‑level features such as edges between MRI and CT scans. It trains a generative model to reconstruct source‑modality images from edge maps and then trains a segmentation network on these generated images. At test time, edge features from target‑modality images are fed into the generative model to produce source‑style images, which are segmented by the pretrained network, achieving state‑of‑the‑art results on multiple public datasets.

By Pengfei Lyu, Pak-Hei Yeung, Jing Xia, De Hu, Xiaosheng Yu, Jianning Chi, Chengdong Wu, Jagath C. Rajapakse
arXiv Statistics ML
Sep 25

DeepGOF-1: A Pretrained Convolutional Goodness-of-Fit Test for Logistic Regression with a Computable Consistency Certificate

DeepGOF-1 introduces a pretrained convolutional network as a goodness‑of‑fit test for logistic regression, where the network reads a grid of standardized residuals as an image and outputs a test statistic. The test is fully calibrated via the analyst’s own bootstrap, ensuring the nominal level is maintained regardless of the network’s training. The authors prove exactness under pivotality, asymptotic exactness without it, and provide a computable consistency certificate from the frozen weights, demonstrating superior stability and power across multiple benchmarks and sample sizes.

By Ebrahim Khaled Ebrahim
arXiv Machine Learning
Sep 25

Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models

The paper investigates ‘thinking leakage’ in post‑training hybrid reasoning models that operate in NoThink mode. Using a causal mediation framework and bidirectional interventions, the authors show that the performance gains attributed to NoThink training largely stem from the model drifting toward its Think mode. Across three models and three training methods on competition math benchmarks, leakage ratios between 42% and 79% were observed, indicating that much of the accuracy improvement is due to re‑invoking existing Think behavior rather than genuine NoThink capability.

By Zehao Liu, Vasant G. Honavar
arXiv Computation and Language
Sep 25

JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

The study investigates whether Jev, a typed classifier that outputs probabilities over allowed answers without generating text, can replace large language model (LLM) rubric judges. Across nine panels from seven benchmarks, Jev’s accuracy differed significantly from LLM judges in only 8 of 27 paired comparisons, performing best on binary criteria and worse only on graded ones, while most other comparisons were inconclusive. In terms of cost and speed, Jev was 29 to 325 times cheaper and 30 to 220 times faster than the flash‑tier LLM judges, and a cascade approach that defers uncertain Jev verdicts to an LLM yielded only modest gains. whyItMatters":"The findings suggest that a lightweight classifier like Jev can serve as an efficient first‑stage evaluator, potentially reducing the reliance on expensive and slow LLM judges in automated grading pipelines."

By Delip Rao, Chris Callison-Burch
arXiv Computation and Language
Sep 25

What, When, and How: Audio Description as Constrained Global Optimization

The paper presents a new approach to automatic audio description (AD) that treats the task as a constrained global optimization problem. It jointly decides what visual content is narratively important, when it can be spoken without overlapping dialogue, and how to phrase it within time limits. Using large language models for salience estimation and a mixed‑integer linear program for scheduling, the system outperforms prior methods on the REFRAMED benchmark, especially in temporal placement and narrative relevance.

By Igor Sterner, Mirella Lapata, Alex Lascarides, Frank Keller
arXiv Computation and Language
Sep 25

Small yet Assistive: Spatially-Aware Post-Training for Low Vision

The paper introduces Smol‑VL‑BLV, a compact vision‑language model designed for blind and low‑vision users. It employs a 500M decoder transformer with teacher‑student distillation and Group Relative Policy Optimization to add spatial detail, directional cues, and hazard detection to post‑training. After a lightweight finetuning step, the model achieves significant gains on spatial, social, OCR, and VQA benchmarks while remaining under 450 MB and running entirely offline on a mid‑range Android phone.

By Rishabh Choudhary, Shreyansh Raj, Umesh Goyal, Shubh Kashyap, Shrestha Kumar, Sushovan Jena, Komal Kumar, Hisham Cholakkal, Aditya Nigam
arXiv Computer Vision
Sep 25

SplatLabel: Pseudo-Labelling through 4D Gaussian Splatting

SplatLabel is an automated pipeline that uses a 4D Gaussian representation to generate LiDAR segmentation and semantic occupancy grids with predictive confidence. It models dynamic scenes through an explicit temporal manifold, tracking moving actors without requiring pre‑annotated 3D bounding boxes. By integrating 360‑degree LiDAR depth maps and distilling soft probabilities from 2D models, it resolves semantic ambiguities over time and space, and evaluates pseudo‑labels via a selective classification framework that balances precision and recall.

By Nitya Nanvani, Andras Palffy, Holger Caesar
arXiv Computer Vision
Sep 25

Training-Free Hold-Usage Detection in Sport Climbing with Foundation Pose Models

The paper presents a training‑free method for detecting which holds a climber uses in sport climbing videos by leveraging a frozen foundation pose model (Sapiens) that provides fingertip and toe keypoints. Using a simple proximity test, mutual exclusion, and a temporal‑persistence rule, the approach achieves high F_1 scores (up to 90.2%) on the Way Up dataset without any climbing‑specific training, outperforming repurposed pose pipelines. The resulting automatic predictions enable accurate coaching statistics, such as climb time and pace, with Pearson correlations of 1.00 and 0.94 respectively.

By Abu Bakar, Abdullah Aftab, Amir Hamza
arXiv AI
Sep 25

CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

CounterRoute is an online reinforcement‑learning framework that jointly learns how to route a language model’s reasoning between a ‘think’ and a ‘direct answer’ mode, using a single shared policy derived from a dual‑mode checkpoint. It employs counterfactual rollouts to credit routing decisions and a curriculum that starts with forced dual‑mode rollouts before shifting to self‑routed updates, achieving better accuracy‑efficiency trade‑offs across nine benchmarks. The method reduces generated tokens by up to 51% on Qwen3‑8B while improving macro‑average accuracy, and its routing strategy generalizes to unseen coding, science, and commonsense tasks.

By Ruochen Jiao, Besnik Fetahu, Zhenyu Shi, Priyanka Nigam
arXiv AI
Sep 25

Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models

The study investigates whether large language models (LLMs) can identify code they have generated, potentially leading to self‑favoring or collusive behavior. Experiments across 15 model‑benchmark pairs show that models can attribute authorship with balanced accuracy between 49% and 58%, but this ability largely stems from superficial cues such as solution length. Removing surface features like docstrings, comments, and type hints reduces attribution accuracy to chance, indicating that surface cues drive the effect.

By Ehsan Barkhordar, Surendrabikram Thapa
arXiv Computation and Language
Sep 25

Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

The paper investigates whether large language models (LLMs) assess politeness in ways that match human judgments. Using two English datasets—one with continuous ratings and another with three‑way categorical labels—the authors compare seven LLMs to human annotations. They find that models agree more with each other than with humans, show systematic neutral bias in categorical predictions, and that alignment varies with explicit linguistic cues and rapport‑building strategies.

By Rong Wang, Kun Sun, Yadong Guo
arXiv AI
Sep 25

Can Classical Semantic-Extractive Summarization Be Evaluated in Hindi? A Replication Study

The study replicates a distributional‑semantics extractive summarisation method for Hindi, adapting all language‑specific components to Devanagari. Evaluated on the Hindi portions of XL‑Sum and FIRE ILSUM 2.0 with a Devanagari‑aware ROUGE scorer, the replicated system performs significantly worse than a simple three‑sentence lead baseline. Feature ablation shows that sentence position alone reproduces the lead baseline, while other features only steer extraction toward long, entity‑dense body sentences, and TextRank performs identically. "whyItMatters":"The results indicate that current Hindi summarisation benchmarks cannot reward non‑lead content selection, highlighting the need for purpose‑built evaluation resources."

By Showket Ahmad Khan, Mudasir Mohd, Nasrullah Sheikh, Mohsin Altaf Wani, Abid Hussain Wani, Hilal Ahmad Khanday, Niyaz Ahmad Wani
arXiv AI
Sep 25

IterSynth: Rethinking Deep Search Agents via Role-Decoupled Iterative Synthesis

IterSynth introduces a role-decoupled, iterative synthesis framework for deep search agents, separating planning and synthesis into distinct Planner and Synthesizer modules that maintain a persistent summary state. This design mitigates role coupling and context noise, while the new Role-Decoupled Policy Optimization (RDPO) enhances training by combining outcome rewards with turn-level rubric evaluations. Experiments on five long-horizon benchmarks show IterSynth-8B outperforming prior ≤8B agents by 4.2% and delivering significant zero-shot gains over ReAct on proprietary models.

By Xingyu Wu, Yuchen Yan, Zhengxi Lu, Siqi Chen, Xin ZHANG, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Jian Shao, Jun Xiao, Yongliang Shen
arXiv AI
Sep 25

CoSWA-YOLOv12: Scale-Invariant Tiny Object Detection and Segmentation of Malaria Parasites

CoSWA-YOLOv12 is a compact YOLOv12 instance‑segmentation detector designed to improve detection of tiny malaria parasites in microscopy images. It introduces a Cooperative Scale‑adaptive Wasserstein Assignment that applies a Wasserstein distance‑based label assignment inversely proportional to object size, a wavelet detail residual, and a min‑max Gaussian regression loss, all of which preserve pretrained weights. On a five‑class Rwandan thick‑smear dataset, the method raises P. falciparum recall from 0.63 to 0.74, increases mAP@50 from 0.73 to 0.81, and reduces missed detections from 38% to 15%.

By Ahmed Tahiru Issah, Carine Mukamakuza