Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language
5d ago

Saidzoda-Tajik-LFM-2.6B: Adapting a 2.6B Hybrid Model to Tajik Tool Calling, and What Continued Pretraining Takes Away

Saidzoda‑Tajik‑LFM‑2.6B is a 2.6‑billion‑parameter hybrid model adapted for Tajik tool calling. The adaptation uses two stages: LoRA continued pretraining on 1.49 million cleaned Tajik web documents followed by one epoch of supervised fine‑tuning on machine‑translated Tajik tool calls, English tool calls, and Tajik instructions. On a 174‑item Tajik tool‑calling test set, strict full‑call accuracy rises from 20.69 % for the base model to 64.14 %, while English performance remains unchanged; however, continued pretraining removes the native tool‑call format and part of the skill, and the model shows limited general assistant capabilities.

By Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon Shodyeva
arXiv Computation and Language
5d ago

Precision--Recall Trade-off on the Inference Level for LLMs in Grammatical Error Correction

The paper examines how large language models (LLMs) can be tuned for grammatical error correction (GEC) by adjusting inference-time parameters. It demonstrates that the repetition penalty can be used to control the precision–recall trade-off during decoding, without changing the model architecture, training, or prompt. Experiments on standard GEC benchmarks confirm that this simple, broadly applicable method allows inference-time adjustment of the precision–recall balance, similar to prior sequence‑to‑sequence techniques such as Align‑and‑Predict Decoding.

By Ryszard Staruch, Filip Grali\'nski
arXiv Computation and Language
5d ago

Asynchronous Flow Map Language Models

The paper introduces asynchronous flow map language models (aFMLMs), a new one‑step parallel language modeling approach that incorporates token prediction order via learned or prescribed asynchronous flows. These flows define a token‑wise time schedule controlling denoising progress, and the authors prove that such flows yield valid flow maps and develop distillation methods. Experiments on Sudoku solving and code generation for GSM8K math word problems show that aFMLMs achieve state‑of‑the‑art performance in one‑step generation.

By Chanhyuk Lee, Wonjung Kim, Patrick Pynadath, Jaehoon Yoo, Satoshi Hayakawa, Ruqi Zhang, Jinwoo Kim, Seunghoon Hong
arXiv Computation and Language
5d ago

DecisionRank: On-Policy Distillation for Decision-Model Rerankers

DecisionRank introduces an on-policy distillation (OPD) method that trains decision models to replicate a generative LLM’s ranking knowledge by converting the student’s document scores into a ranked sequence and using a teacher to provide conditional feedback. The approach combines item‑space OPD with offline ranking supervision, achieving state‑of‑the‑art results on retrieval benchmarks such as TREC DL, BEIR, and BRIGHT. Even with a 1.7B parameter model, DecisionRank outperforms its 27B generative teacher on TREC DL19–23, and at 8B it surpasses the generative listwise counterpart while running 24.5× faster on TREC DL and 16.7× faster on BRIGHT.

By Yanran Wu, Dian Yu, Chongyang Bai, Jieyi Zhang, Rui Liu, Tianyi Chen, Shike Mei, Hang Cui, Frank Shyu, Sandeep Pandey, Luke Simon, Chunwei Liu, Xi Liu
arXiv Computation and Language
5d ago

Lineage Is Memory: Parent Inheritance and Feedback Disclosure in Short-Budget LLM-Guided Optimization

The paper investigates how iterative language‑model optimizers improve prompts and programs by evaluating candidates, exposing failing cases to a proposer model, and retaining revisions that enhance performance. By conducting 1,750 pre‑registered experiments across nine benchmarks, the authors disentangle the effects of parent inheritance (keeping a surviving parent) from feedback disclosure (providing failure summaries) under equal computational budgets. They find that greedy parent inheritance alone drives most optimization gains, outperforming seed‑reset strategies even when the latter receive comprehensive failure summaries or larger proposer models, and that the cost of proposer reflection is minimal compared to validation and minibatch gating.

By Shanglin Yang, Zhan Shi
arXiv Computation and Language
5d ago

ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science

ProSearch is a new benchmark for evaluating multi‑constraint protocol retrieval in experimental science. It features realistic queries drawn from real experimental records, a complex set of 60 interdependent experimental fields, and 3–5 explicit constraints per query, including numerical and negation constraints. The benchmark contains 1,515 queries across six domains and is built using LLM‑based extraction, data‑driven constraint selection, controlled query generation, and evidence‑based relevance validation.

By Wenliang Liang, Zhiyuan Ning, Haolin Chen, Yuanchun Zhou, Wenjuan Cui, Yi Du
arXiv Computation and Language
5d ago

AgentCollab: A Self-Evaluation-Driven Collaboration Paradigm for Efficient LLM Agents

AgentCollab introduces a self-evaluation-driven collaboration paradigm for large language model agents, dynamically coordinating models of varying reasoning capacities during task execution. By using the agent’s own self-reflection to decide when to elevate to a stronger model, it avoids reliance on external routing modules and allocates additional reasoning budget only when recent failures indicate difficulty. Experiments on multi-step agent benchmarks demonstrate that this two-level small-large model setup consistently improves the accuracy-efficiency Pareto frontier of LLM agents.

By Wenbo Gao, Renxi Liu, Xian Wang, Fang Guo, Shuai Yang, Xi Chen, Hui-Ling Zhen, Hanting Chen, Weizhe Lin, Xiaosong Li, Yaoyuan Wang
arXiv Computation and Language
5d ago

When Support Crosses the Line: Evaluating Ethical Boundaries in LLM-Based Mental Health Support in the Chinese Context

The paper introduces MindEth-Bench, a benchmark designed to assess whether large language models (LLMs) maintain professional ethical boundaries during multi‑turn mental health interactions in China. It uses Monte Carlo Tree Search to generate challenging dialogue trajectories and evaluates nine LLMs, finding that models can shift from appropriate to unethical responses over time. The study demonstrates that single‑turn tests may overestimate ethical reliability, underscoring the need for comprehensive multi‑turn evaluation.

By Bichen Wang, Yufeng Jin, Yixin Sun, Yanyan Zhao, Bing Qin
arXiv Computation and Language
5d ago

Unlimited OCR Works 2

Unlimited OCR 2 is a document parsing model that processes only the text it transcribes, avoiding re‑scanning earlier pages. It replaces all decoder attention with Sparse Reference Sliding Window Attention (SR‑SWA), keeping computational costs constant regardless of document length. The model achieves significant per‑step speedups over Full Attention and R‑SWA, and sets new state‑of‑the‑art results on the MPDocBench and MarathonDocBench benchmarks for hundred‑page documents.

By Shu Wei, Jingjing Wu, Qunyi Xie, Lingshu Zhang, Xiangcheng Du, Yannian Fu, Mengjie Wang, Zuyi Wang, Zhichao Liu, Yangliu Xu, Youyang Yin, Huanhuan Liu, YY, Yi Yang
arXiv Computer Vision
5d ago

Drone-PUR: From Visual Clue Discovery to Understanding and Reasoning in UAV Scenes

Drone-PUR is a new dataset and benchmark for vision‑language models that focuses on visual clue discovery, interpretation, and integration in UAV scenes. It contains 11 tasks with 175k training and 20k test samples across single images, image sequences, cross‑modal images, and multiple viewpoints, all of which are human‑reviewed. Evaluation of 22 general‑purpose VLMs shows persistent gaps in spatial judgment and cross‑modal/cross‑view reasoning, while fine‑tuning on Drone‑PUR improves accuracy by up to 18 percentage points.

By Jiahe Wu, Bing Cao, Xingyue Guo, Ruipu Zhao, Beiming Yu, Jiekang Feng, Qinghua Hu, Pengfei Zhu
arXiv Computer Vision
5d ago

RADI3N: GeometRic ImAge EDiting via ParametrIc 3D CoNtrol

RADI3N is a new framework for part‑level image editing that provides parametric 3D control. It introduces a CS‑ARAP deformation module that smooths rotation discontinuities at part boundaries, and an inversion‑free selective texture transfer mechanism that reduces positional bias and enhances semantic matching. The method achieves state‑of‑the‑art results in geometric fidelity, viewpoint consistency, and controllability.

By Abdul Wasi, Akhil Gorugantu, Mahesh Bhosale, Anjan Dutta, Junsong Yuan, Sauradip Nag, David Doermann
arXiv Computer Vision
5d ago

CompoSET: A Controlled Single-Edit Benchmark for Vision-Language Compositionality

CompoSET is a new vision‑language benchmark that tests compositionality by presenting each image‑caption pair with a foil that differs by only a single controlled edit—such as color, count, spatial relation, or object state—while keeping the overall scene largely unchanged. The dataset contains 1,776 instances across 80 scenes, 16 edit types in four semantic categories, and captions at three verbosity levels, with an average of only 8% pixel change between base and foil images. Evaluation of 22 dual‑encoder VLMs shows that all models perform at or below chance on CompoSET, contrasting with their strong performance on prior benchmarks that conflate global image differences with compositional reasoning.

By Laure Delisle, Pietro Perona
arXiv Computer Vision
5d ago

From Evaluation to Evolution: Benchmarking Image Editing and Evolving Image Models

The paper introduces WeGenEditBench, a benchmark with 2,740 evaluation instances and 148 task types for image editing, focusing on local failure detection and prioritization for model improvement. It also presents WeGenEvoStudio, a multi‑agent harness that supports recursive self‑improvement by diagnosing failures, constructing retrieval‑grounded data, and adapting models. Experiments show that models such as GPT‑Image‑2 and Seedream‑5.0‑Pro excel in single‑image and text‑intensive editing, and evolution experiments demonstrate significant targeted gains on selected benchmarks.

By Jun He, Canmiao Fu, Ying Zhang, Zhipeng Huang, Chenyu Mu, Yazhen Xie, Chen Li, Jing Lyu, Weijia Li, Xiang Zhang
arXiv Machine Learning
5d ago

A Theory-guided Weighted $L^2$ Loss for solving the BGK model via Physics-informed neural networks

arXiv:2604. 04971v2 Announce Type: replace Abstract: While Physics-Informed Neural Networks offer a promising framework for solving partial differential equations, the standard $L^2$ loss formulation is fundamentally insufficient when applied to the Bhatnagar-Gross-Krook (BGK) model.

By Gyounghun Ko, Sung-Jun Son, Seung Yeon Cho, Myeong-Su Lee
arXiv Computer Vision
5d ago

Parse Before You Execute: Efficient Spatial Reasoning with Executable Primitives

The paper introduces PEPS, a training‑free framework that improves spatial reasoning in Vision‑Language Models by parsing queries into a compact set of executable requirements before execution. PEPS uses a typed FESM schema (Frame, Entity, State, Metric) to commit queries, then selectively grounds primitives via fixed geometric APIs and verifies sufficiency, allowing early termination or targeted repair. Experiments on spatial reasoning benchmarks show that PEPS boosts accuracy while reducing refinement rounds, API cost, and latency compared to existing tool‑integrated baselines.

By Xiwei Liu, Yulong Li, Xinling Zhuang, Xuhui Li, Yichen Li, Haolin Yang, Zhixiang Lu, Yutong Xie
arXiv Computer Vision
5d ago

VoxLution: Evolving Voxels for High-Fidelity 3D Scene Generation

VoxLution is a coarse‑to‑fine transformer‑based method for generating high‑fidelity 3D indoor scenes. It first creates a low‑resolution spatial layout and then progressively refines the geometry in local chunks using a sequence‑modeling approach over multi‑resolution sparse voxel grids. The method introduces a resolution‑invariant tokenization scheme that encodes octants into a single decimal number and incorporates adjacent chunk boundaries to maintain coherence, achieving state‑of‑the‑art performance on 3D‑FRONT and ScanNet++ benchmarks.

By Daoyi Gao, Xinyi Zhang, Alexey Bokhovkin, Lei Li, Angela Dai
arXiv Computer Vision
5d ago

Attention Maps Know When to Think: Training-Free Difficulty-Aware Chain-of-Thought for Reasoning Segmentation

The paper introduces Difficulty-Aware Chain-of-Thought (DA‑CoT), a training‑free framework that selectively applies chain‑of‑thought prompting in reasoning segmentation tasks based on the entropy of attention maps. Low entropy signals confident, localized grounding, while high entropy indicates uncertainty and the potential benefit of additional reasoning. DA‑CoT also incorporates a Bidirectional Rethinking mechanism to automatically reverse reasoning decisions when initial outputs fail, thereby improving robustness and inference efficiency while achieving state‑of‑the‑art results on Reasoning Segmentation benchmarks.

By Chanseul Cho, Jaesung Jun, Dongheon Lee, Youngmin Ro
arXiv Computer Vision
5d ago

OT-CAL: Optimal-Transport-Guided Adaptive Sample Selection for Efficient Visual Learning

OT-CAL is an adaptive sample selection method for visual learning that uses optimal transport to weight training samples based on five informativeness features such as difficulty, learnability, and uncertainty. It dynamically updates a curriculum vector during training, allowing it to focus on the most informative samples without a fixed schedule. Experiments on three visual benchmarks show that selecting only 50% of each mini‑batch with OT‑CAL yields 1.3–2.1 point performance gains over AdaSelection and speeds up training by 1.18–1.24× compared to full‑data training.

By Suryansh Talukdar, Jin Soung Yoo
arXiv Computer Vision
5d ago

SAR-TS-BERT: Physics-Guided Self-Supervised Pre-Training With Polarimetric Features for SAR Time-Series Crop Classification

SAR-TS-BERT is a physics‑guided, BERT‑style self‑supervised pre‑training framework designed for multitemporal synthetic aperture radar (SAR) crop classification. It fuses backscatter intensities, polarimetric ratios, and Cloude‑Pottier decomposition parameters into a six‑band input, employing a span‑masking strategy that targets key phenological transitions and a dual‑branch Transformer encoder with cross‑attention fusion. The method achieves high accuracies—96.78 % on Estonia’s EuroCropsML and 87.29 % on the USA’s California CDL—while significantly improving performance with limited labeled data.

By Jieqiu Wang, Yuhan Zhang, Bo Zheng, Shaoming Zhang
arXiv Computer Vision
5d ago

WorkIng: A Real-World and Synthetic Highway Work-Zone Benchmark Dataset for Autonomous Navigation

WorkIng is a dual-domain benchmark for autonomous navigation in highway work zones, offering a real-world dataset (WorkIng-Highway) with 360° multi‑modal sensor data and over 12,000 annotated frames, and a synthetic counterpart (WorkIng-CARLA) that simulates 24 regulation‑aware scenarios across six topologies and traffic standards. The benchmark includes 227,399 tracked 3D bounding boxes and 100+ custom 3D assets. Experiments show that real‑world data drives performance, while scenario‑stratified synthetic augmentation improves 2D recognition, 3D detection, and trajectory planning.

By Xujun Xie, Dominik R\"o{\ss}le, Adithya Mohan, Daniel Cremers, Torsten Sch\"on