Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language
5d ago

GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning

The paper introduces GRIT, a training framework that adds visual grounding to reinforcement learning with verifiable rewards (RLVR) for vision-language models, addressing the issue of Correct-but-Visually-Misgrounded Reasoning (CVMR). CVMR occurs when models produce correct final answers while misperceiving visual information during intermediate reasoning steps, a problem that standard RLVR rewards cannot detect. By analyzing token-level attention patterns, the authors show that CVMR tokens differ from well-grounded tokens and use this insight to guide GRIT, which improves both reasoning accuracy and grounding fidelity. Additionally, the authors present GRIT-Bench, a new benchmark for evaluating text-rich reasoning across varying text lengths and reasoning depths.

By Chanseul Cho, Noori Bae, Dongheon Lee, Youngmin Ro
arXiv Computation and Language
5d ago

Holistic Diversity Management in LLM-Guided Evolutionary Search

The paper introduces HoliDivEvolve, a method for managing diversity in large language model (LLM)-guided evolutionary search. It enhances the traditional island model by adding diverse initialization, trajectory‑adaptive island selection, and growth‑adaptive migration to maintain distinct search directions and reduce premature convergence. Experiments on various benchmarks demonstrate that HoliDivEvolve achieves competitive performance and the authors analyze how each design choice influences search behavior and outcomes.

By Seockbean Song, Youngsik Yoon, Sungjae Lee, Siwei Wang, Wei Chen, Jungseul Ok
arXiv Computer Vision
5d ago

PathoJev: Efficient Pathology Decisions from Shared Visual Context

PathoJev is a pathology model that predicts answer probabilities for whole-slide and region-level assessments by encoding each slide once and evaluating multiple question suffixes in batches over a shared visual context. It reuses hybrid backbone recurrent states to limit computation growth with slide-context length, achieving mean accuracies of 77.2% and 78.1% on three whole-slide benchmarks and outperforming SlideChat by 5.4–6.3 percentage points. The 9B model runs 9.1× faster than SlideChat and accelerates multi-question workloads up to 11.2× for 64 questions, providing a reusable probabilistic interface for combining multiple visual judgments.

By Chengyang Zhang, Wenchuan Zhang, Bo Li, Xinyu Liu, Jiaming Yang, Mengran Li, Jie Chen, Wei Ju, Yuhao Yi, Hong Bu, Jiancheng Lv
arXiv Machine Learning
5d ago

What Does a Harness Repair? Three Preregistered Studies Around a Longevity-Tuned Language Model

The paper reports three preregistered studies evaluating a longevity‑tuned 1.7B language model’s performance when a harness is applied. Across multiple tasks, the harness sometimes improves test accuracy over a capped thinking setting, but its gains are often comparable to or only slightly better than a constant label baseline. Replication experiments confirm that the observed improvements can be attributed to specific unparsed rows and pipeline artifacts rather than a general model enhancement.

By Bowen Xu, Boyu Chen
arXiv AI
5d ago

STAR: Steering Action Representations during Inference

The paper introduces STAR (Steering Action Representations), a training‑free inference technique for vision‑language‑action models that addresses a late‑layer grounding gap. STAR applies an exponential moving average over late‑layer residual updates, enabling later action representations to inherit object‑aware update directions from earlier layers without changing the model. By incorporating recent proprioceptive states to adjust mixing strength, STAR improves object awareness in late layers and boosts average task success rates by 4.0 percentage points on robot manipulation benchmarks.

By Guangyu Zhao, Yongchang Zhang, Haoxuan Ru, Kaichen He, Haoran Wen, Kun Zhan, Zeyu Zhang, Yin Li, Yiwu Zhong
arXiv Machine Learning
5d ago

CodeLift: Your Test Suite Is Secretly a Dense SWE Curriculum

CodeLift is a reinforcement learning approach that transforms a software engineering test suite into an adaptive curriculum by dynamically calibrating pass‑rate rewards. It groups tests that kill the same patches and those that pass the same patches, then prioritizes test groups that fewer patch groups pass, thereby reducing the influence of trivial tests. Experiments on SWE benchmarks demonstrate that CodeLift consistently improves pass@1/2/4/8 metrics compared to binary and pass‑rate baselines, providing denser and more stable learning signals.

By Junjielong Xu, Huachao Zhu, Alyssa Wu, Zhichao Hu, Pinjia He
arXiv Computation and Language
5d ago

BlindToChange: AI Agents Can Find Changes When Asked But Do They Look When Needed?

BlindToChange is a new benchmark that tests whether AI agents can proactively discover recent real‑world events that affect their responses, even when those events are not explicitly mentioned in user requests. The benchmark uses automatically extracted events from Wikidata to create recommendation tasks where an obvious candidate is no longer applicable due to a recent change. Experiments show a large performance gap—up to 27.5 points—when agents fail to verify recent changes, revealing that many failures stem from not directly searching for updated facts.

By Anirudh Phukan, Tanya Goyal, Danish Pruthi
arXiv Computer Vision
5d ago

MATS: A Multi-Agent Framework for Controllable and Reflective Tumor Synthesis

MATS is a multi-agent framework that enables text‑controlled synthesis of realistic tumors across different organs. It profiles the morphological distribution of a target disease, constructs a generation plan that emphasizes underrepresented subtypes, and uses a two‑stage diffusion process to create anatomically valid tumor masks and textures. A closed‑loop reflective mechanism checks attribute alignment at multiple stages and guides re‑generation when necessary, leading to improved downstream segmentation performance on several medical imaging benchmarks.

By Zhongnan Shi, Keyang Zhou, Siyi Li, Qichao Ying, Xinchen Ma, Shuyi Gong, Wei Tang, Sheng Li, Zhenxing Qian
arXiv Computer Vision
5d ago

Beyond Trial-and-Error: A Geospatial-Guided Active Reasoning Framework for Robust Aerial Vision-and-Language Navigation

The paper introduces GeoReasoner, a framework for Aerial Vision-and-Language Navigation that replaces reactive trial-and-error with geospatial-guided active reasoning. It first uses pre-search geospatial reasoning to generate region-level priors and limit the goal hypothesis space, then performs hypothesis-driven exploration to gather discriminative evidence, and finally applies consistency-aware goal verification to avoid premature confirmations. Experiments on the CityNav benchmark show GeoReasoner achieves state‑of‑the‑art zero‑shot performance, rivaling supervised methods.

By Jiawei Ge, Wenkang Gong, Jiuxin Cao, Bo Liu, Lei Cui, Haoran Wang, Xuelin Zhu, Biwei Cao, Chang Liu, Chen Feng, Ioannis Patras
arXiv Computer Vision
5d ago

Scout-Refine-Correct: Teaching Flow Models to Refine Their Own Samples

Scout-Refine-Correct (SRC) is a training recipe that equips flow-matching image generation models with an inference-time refinement capability. During sampling, the model performs a scout rollout to predict a full sample, uses this prediction as a self-conditioning signal, and then takes the actual step; this process can be repeated for multiple refinement rounds. On ImageNet-256, SRC improves LightningDiT-B’s FID from ~14.5 to ~7.26 with 550 normalised flow evaluations, matching a best-of‑N baseline that relies on an oracle verifier, and further reduces FID to 3.81 when combined with best-of‑N.

By Sergey Karpukhin, Aibek Alanov
arXiv Machine Learning
5d ago

CAP: Learning to Optimize from Sparse Observations of Related Tasks

The paper introduces CAP (Candidate Acquisition Policy), a method that learns a query policy from sparse historical records of related optimization tasks. CAP models shared structure and task-specific variation to generate interactive practice functions, then trains a feedback-conditioned policy that operates under a fixed evaluation budget. Experiments on synthetic task families and engineering simulation benchmarks (BOPTEST and SU2) show CAP achieving the lowest median final-budget values compared to other methods.

By Lin Fu, Siwei Fu, Dong Ni
arXiv Computer Vision
5d ago

Measuring What Matters: A Causal Reliance Score for Histopathology Image Segmentation

The paper introduces a causality-inspired evaluation framework for histopathology image segmentation that uses controlled interventions to probe model sensitivity to foreground tissue structure versus background/domain variation. It proposes the Causal Reliance Score (CRS), a metric that quantifies how much model predictions depend on foreground perturbations compared to background changes. The authors also design augmentation strategies that act as implicit counterfactual interventions during training, encouraging invariance to nuisance factors while preserving morphology, and demonstrate that these interventions reveal behavioural differences between models not captured by standard metrics such as Dice or IoU.

By Samuel Thomas, Daniel Tweneboah Anyimadu, Ahmed Karam Eldaly, Somaia Elsheikh, Mohammed M. Abdelsamea
arXiv AI
5d ago

Structure-Aware Skill Retrieval and Arbitration for Optimization Modeling

The paper introduces “ArchOR”, a skill agent that enhances “OptSkills” with two retrieval modules: Centroid-Aware Representation Learning (CARL) and Signature-Guided Retrieval (S3R). CARL refines cached skill representations using a low-rank prototype residual, while S3R attaches structural signatures to skills, prioritizes candidates via weighted Jaccard overlap, and limits semantic re-ranking to a top‑K set. Using the same DeepSeek-V4.1-Flash backbone and released skill library, “ArchOR” achieves a 65.00% Micro‑Avg. Pass@1 across five benchmarks, outperforming “OptSkills” by 1.64 pp Micro‑Avg. and 2.68 pp Macro‑Avg.

By Anran Liu, Benzhao Hou, Zongsheng Cao, Daichao Wang, Lang Chen, Shitong Chen
arXiv Machine Learning
5d ago

Anomalies as Residuals: Cauchy Displacement Networks

The paper introduces the Cauchy Displacement Network (CDN), a single‑hidden‑layer neural network that approximates neighbourhood‑based anomaly detectors by regressing each point onto the mean of its k nearest training neighbours. CDN uses Cauchy activations to compute a displacement vector whose length serves as an anomaly score, achieving state‑of‑the‑art performance on 46 benchmark datasets while being up to three orders of magnitude faster than traditional methods and independent of training‑set size. The authors provide theoretical guarantees, showing that Cauchy units represent Student‑t expert densities and that the network’s output grows linearly with residuals in any direction.

By Yang Cao, Zihao Guo
arXiv Computer Vision
5d ago

AVID: A Weakly Aligned RGB-Infrared Dataset and Perturbation Benchmark for Tiny Aerial Targets

AVID is a new benchmark dataset for detecting tiny aerial targets, comprising 32,040 RGB‑infrared frame pairs from 115 videos with 35,952 bird and drone annotations at 320×256 resolution. The dataset focuses on very small objects, with 68.1% of targets having a geometric‑mean bounding‑box side of 8–16 pixels and 99.0% meeting the COCO small‑object criterion. It introduces metrics for cross‑modal alignment and an AVID‑C extension that simulates seven families of image degradation at five severity levels to study the impact of scale, degradation, and alignment on detection performance.

By Dharini Raghavan, Vishnu Rao, Sethu Selvi S
arXiv Computer Vision
5d ago

FlowMoDE: Coarse-to-fine Flow Matching for Detail-aware Sim-to-Real Monocular Depth Estimation

FlowMoDE introduces a coarse‑to‑fine flow‑matching framework for monocular depth estimation that balances efficiency and detail reconstruction. The method first decodes coarse depth conditioned on diffused features from a pre‑trained semantic encoder, then refines fine structural patterns with expanded decoding layers. Trained only on a synthetic indoor dataset, FlowMoDE achieves a 14.1% lower relative error and a 9.9% lower Chamfer distance compared to state‑of‑the‑art methods across five real‑world indoor and outdoor datasets, and demonstrates strong generalization to realistic robotic scenes.

By Liangjing Shao, Wanhao Liu, Jinsong Lin, Zhiwei Fang, Hongliang Ren
arXiv Computer Vision
5d ago

SketchDirector: Complex Scene Sketching via Agentic Layout Reasoning and Detail Refinement

SketchDirector is a new framework for generating complex scene sketches guided by text prompts. It uses a large language model to break a scene into draft assets described by Bézier curves, then an agentic reasoning loop plans their spatial arrangement with evolving memory, and finally a diffusion-based refinement module improves sketch quality while keeping geometry intact. The authors also release the MCSketch8K dataset, providing editable vector assets, compositional layouts, and object-level manipulation annotations to support future research.

By Yiqi Liang, Xiaodong Cun
arXiv Computer Vision
5d ago

When Depth Lies: Benchmarking Vision-Language Models on Mirror-Induced RGB-D Ambiguity

The paper introduces MIRA-Bench, a mirror‑centric RGB‑D dataset and benchmark designed to test vision‑language models (VLMs) in reflective scenes where depth information is misleading. It contains 1,011 RGB‑D images from 525 real‑world scenes, annotated with reflective surface masks and object labels for both direct and reflected views, and 14,140 QA pairs across eight sub‑tasks. Evaluation shows that state‑of‑the‑art VLMs consistently fail to distinguish real from reflected objects, misalign reflected instances with their real counterparts, and produce incorrect spatial inferences, with depth data even degrading performance. whyItMatters":"The study reveals that current VLMs cannot handle mirror‑induced depth ambiguity, highlighting a critical limitation in spatial reasoning for real‑world applications involving reflective surfaces."

By Hao Yin, Tianchen Guo, Heming Du, Yan Ke, Yanbin Liu, Xin Yu
arXiv Computer Vision
5d ago

PGGT-Loc: Primitive-Grounded Geometry Transformer for Feed-Forward Camera Localization

PGGT-Loc introduces a Primitive‑Grounded Geometry Transformer (PGGT) that performs feed‑forward 6‑DoF camera localization by grounding a single query image to planar map primitives. The method trains a network to predict multiple localization targets, enabling generalization to unseen environments. It achieves state‑of‑the‑art performance on both dense surface maps and minimal layout models using a straightforward, scalable architecture.

By Hanqiao Ye, Yuzhou Liu, Mengqi Rong, Jian Liu, Yangdong Liu, Shuhan Shen
arXiv Machine Learning
5d ago

When the Past Shapes Action: Understanding Embodied Memory

The paper "When the Past Shapes Action: Understanding Embodied Memory" surveys how embodied AI agents can preserve and use past observations, actions, and experiences to improve decision-making in dynamic environments. It categorizes existing approaches along six memory requirements—world-state persistence, task progress, belief maintenance, experiential learning, etc.—and eight architectural styles such as recurrent states, fast weights, and spatial memory. The authors evaluate how current benchmarks test memory-dependent behavior and highlight open challenges in selecting relevant experiences, maintaining consistent beliefs, and consolidating knowledge across episodes.

By Siqi Li, Ziyi Bai, Xinrun Xu, Tinglei Huang, B\"orje F. Karlsson