Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision
Oct 23

No Pose, No Problem in 4D: Feed-Forward Dynamic Gaussians from Unposed Multi-View Videos

NoPo4D is a feed‑forward system that reconstructs dynamic 3D scenes from unposed multi‑view videos by using 4D Gaussian splatting. It introduces a velocity decomposition that separates per‑pixel image‑plane shifts from depth changes, enabling supervision with pseudo optical flow and avoiding the need for accurate camera poses or 3D motion ground truth. The method also employs a bidirectional motion encoder and view‑dependent opacity to improve cross‑view and cross‑timestep alignment, achieving state‑of‑the‑art results on multiple benchmarks while being much faster than per‑scene optimization approaches.

By Matteo Balice, Yanik Kunzi, Chenyangguang Zhang, Matteo Matteucci, Marc Pollefeys, Sunghwan Hong
arXiv Computer Vision
Oct 23

PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment

PortraitAes: Intent-Conditioned Structured Portrait Aesthetic Assessment introduces a new benchmark, PortraitAes-Bench, with 11,000 images and a detailed rubric covering nine photographic intents, six first-level dimensions, and 22 secondary criteria. The model predicts aesthetic scores conditioned on photographic intent, using multi-task supervision and Gaussian score calibration to improve comparability. On the benchmark, PortraitAes achieves high Pearson (0.924) and Spearman (0.934) correlations, outperforming general-purpose MLLMs and specialized baselines.

By Junzhou Xie, Haozhong Xiong, Xunyun Tian, Kaile Du, Tianchen Yu, Qiang Li, Wei Liu, Jiaming Liu, Ruihua Huang, Yang Shi, Guangcan Liu
arXiv Computer Vision
Oct 23

Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

The paper investigates how to best integrate large language model (LLM) guidance into visual question answering (VQA) systems. By prompting the 70‑billion‑parameter Llama 3.3 model to identify visual concepts, distractors, and external facts for each question, the authors inject this guidance into a frozen Vision Transformer (ViT) and a T5 text‑to‑text model via two methods: decision‑level late fusion and attention‑matrix injection. On the OK‑VQA benchmark, the shallow late‑fusion approach outperforms the deeper attention injection, achieving 39.4% exact match versus 31.8%, and a paraphrasing pipeline further expands the training set tenfold, indicating that the depth of integration should match the data regime.

By Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, Dhruv Bansal
arXiv AI
Oct 23

Speedrunning Relational Foundation Model Pretraining

The paper introduces Speedrunning Relational Foundation Model Pretraining, a new leaderboard called modded‑nanoRT for Relational Transformers that decentralizes the search for efficient pretraining recipes. By calibrating the benchmark and ranking candidates based on a 45‑minute learning curve, the authors demonstrate a recipe that achieves the target in 305.63 seconds on a single NVIDIA H200—8.11× faster than the reference implementation—and wins 13 of 15 comparisons outside the development database. The authors plan to release the benchmark, frozen artifacts, and record runs, with code available on GitHub.

By Gabriel Wedson Mendon\c{c}a de Souza Veloso, Luan Borges Teodoro Reis Sena, Denis Oliveira Correa, Francisco Galuppo Azevedo
arXiv Computation and Language
Oct 23

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

The paper investigates how mathematical reasoning improvements from Reinforcement Post-Training (RPT) in English-centric large language models (LLMs) transfer to other languages. Experiments show that transfer varies greatly depending on the initial model, target language, and training approach, and that stronger English initialization does not guarantee better transfer efficiency. Adding a single parallel language to the training data yields the largest boost, with additional languages providing smaller but consistent gains.

By Wen Yang, Junhong Wu, Chong Li, Chengqing Zong, Jiajun Zhang
arXiv Computation and Language
Oct 23

Language Models as Higher-Order Planning Formalizers

The paper examines the limitations of using large language models (LLMs) to translate natural language planning problems into PDDL representations, noting that standard benchmarks often map directly to PDDL and may overstate success. It introduces the concept of "unraveling problems," where concise natural language descriptions correspond to large PDDL structures, and shows that traditional LLM formalizers struggle to scale on such variants. To address this, the authors propose a new paradigm, LLM-as-Higher-Order-Formalizer, where the LLM first generates a high‑level program that encapsulates recurrent logic, which then produces the full PDDL representation, thereby decoupling token output from combinatorial explosion and improving performance on complex tasks.

By Owen Jiang, Cassie Huang, Ashish Sabharwal, Li Zhang
arXiv Computation and Language
Oct 23

Efficient Best-Of-N with Radial Consensus Score

The paper introduces Radial Consensus Score (RCS), a training‑free method for selecting the best answer from a set of candidate responses generated by large language models. RCS computes a weighted Fréchet mean of answer embeddings to form a semantic center and ranks candidates by their radial distance to this center, thereby capturing semantic consistency rather than surface agreement. Experiments on five open‑weight models and multiple benchmarks show that RCS matches or surpasses strong baselines across all sampling budgets, especially when correct answers are lexically fragmented, and can replace majority voting in multi‑agent debate and black‑box settings.

By Manh Nguyen, Sunil Gupta, Hung Le
arXiv Computation and Language
Oct 23

Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation

The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.

By Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu
arXiv Computation and Language
Oct 23

Beyond Recall Saturation: Scaling Hard-Negative Reranker Training for Industrial Legal Retrieval

The paper investigates how scaling hard‑negative training for rerankers can improve industrial legal retrieval systems once embedding recall saturates. On Chinese legal benchmarks, a pipeline that first uses embedding‑mined negatives and then self‑mined negatives yields the best trade‑off, and a 4B‑parameter Stage2 reranker outperforms the native 4B model on STARD, CSAID, and internal datasets. The study demonstrates that evaluating retrieval and reranking stages separately is beneficial for optimizing industrial retrieval performance.

By Yunhan Li, Mingjie Xie, Zeyang Shi, Gengshen Wu, Min Yang
arXiv Computer Vision
Oct 23

FilmFlow: Agentic System for 3D Virtual Film Cinematography

FilmFlow is an agentic framework that transforms natural‑language instructions or reference videos into explicit camera trajectories for dynamic 3D environments. It uses a Director agent that composes high‑level motions from concurrent atomic operations, supported by Interpreter, Scout, and Critic agents for intent extraction, viewpoint selection, and iterative refinement. The authors also introduce CinemaBench, a benchmark of 150 Blender scenes with 750 text‑directed tasks and 60 video‑reference tasks, where FilmFlow outperforms baselines and receives the highest user preference.

By Yiming Zhang, Jiarui Guo, Haojia Wei, Yifei Liu, Yifan Yang, Hongjie Zhang, Xue Yang, Zhihang Zhong
arXiv Computer Vision
Oct 23

ICGF-CXRMate: Image-Conditioned Clinical Context Compression and Gated Fusion for Chest X-ray Report Generation

ICGF-CXRMate is a framework that compresses heterogeneous patient data into a fixed set of clinical context tokens conditioned on chest X‑ray images. These tokens are selectively fused with visual features via gated cross‑attention and also retained in the decoder prompt, yielding shorter prompts and higher report quality. Experiments on MIMIC‑CXR and MIMIC‑IV‑ED show significant gains in RadGraph‑F1, CXR‑BERT similarity, and BLEU‑4, while reducing prompt length by 22.9%.

By Vu Minh Tran, Thuc Huynh, Son Thanh Luu, Doanh C. Bui
arXiv Computer Vision
Oct 23

GigaBench: Benchmarking Learnable Representations for Gigapixel Image Compression and Super-Resolution

GigaBench is the first benchmark designed to evaluate learnable representations (LRs) for gigapixel image compression and super‑resolution. It compiles a diverse set of gigapixel images—collected, acquired, and synthetically rendered—from various domains and tests representative LR methods (Implicit, Hybrid, and Explicit) on consumer hardware. The study systematically measures reconstruction quality, model size, compression, and inference times, revealing consistent trends and offering actionable guidance for future research.

By Matteo Fusconi, Alex Costanzino, Pierluigi Zama Ramirez, Luigi Di Stefano
arXiv Computer Vision
Oct 23

AA-ViT: Anatomically Aware Vision Transformer with Structural and Frequency Guidance for Contrast Enhanced Brain MRI Synthesis

The paper introduces AA‑ViT, an anatomically aware vision transformer that synthesizes contrast‑enhanced brain MRI from pre‑contrast modalities (T1, T2, FLAIR) by incorporating structural and frequency guidance. Experiments on the BraTS 2021 dataset show that AA‑ViT preserves anatomical and lesion boundaries, achieving higher PSNR and SSIM than existing methods. A clinical evaluation by four experts on 19 glioma cases yielded a mean score of 3.94/5, indicating promising clinical relevance.

By Talha Meraj, Tom Flannery, Charlie Cummins, Matt Townend, Thomas C Booth, Peter Crossley, Michael McCann, Ian Overton, Saritha Unnikrishnan
arXiv AI
Oct 23

Automated Materials Data and Knowledge Extraction using Machine Learning and Large Language Models

The review traces the development of machine‑learning techniques for extracting materials data and knowledge from scientific literature, progressing from rule‑based pipelines to NLP models, transformer encoders, and finally large language models and agentic systems that handle text, tables, and figures. It highlights how these methods enable the creation of queryable materials databases, knowledge graphs, generic question answering, AI co‑scientists, chatbots, and closed‑loop discovery with autonomous labs. The paper critically evaluates datasets, extraction accuracy, computational costs, and offers practical guidelines, while outlining open challenges such as full tetrahedron extraction, benchmark scarcity, and terminology resolution.

By Aravindan Kamatchi Sundaram, B. Pabitramohan Prusty, Rohit Batra
arXiv AI
Oct 23

DualView: Adaptive Local-Global Fusion for Multi-Hop Document Reranking

DualView is an efficient dual‑view cascaded reranking framework designed for multi‑hop document reranking in question answering. It combines a Local Scorer that uses stacked cross‑attention for fine‑grained query‑document relevance with a Global Scorer that models inter‑document dependencies via Transformer‑based context aggregation. An Adaptive Gate dynamically fuses these views based on query semantics, achieving high recall and low latency on benchmarks such as MuSiQue.

By Litong Zhang, Jiaxin Li, Kuo Zhao
arXiv Machine Learning
Oct 23

MoG-VLM: Efficient Video Anomaly Detection through Motion-Guided Vision-Language Models

MoG-VLM is a motion‑guided vision‑language model designed for efficient video anomaly detection. It uses a lightweight motion signal, Temporal Gradient Magnitude, to perform content‑aware sampling and activates dense multi‑frame VLM reasoning only around frames with high anomaly likelihood via a hierarchical dual‑stream design. The method achieves state‑of‑the‑art AUC‑ROC scores on four benchmarks and delivers a 1.31× inference speedup on XD‑Violence.

By Khaled Waleed Dawoud, Muhammad Zaigham Zaheer, Khan Muhammad, Muhammad Haris Khan
arXiv Machine Learning
Oct 23

LACToS: Local Anomaly Detection via Conditional Typicality of Superpixels

LACToS is a local anomaly detection method that identifies spatially confined out‑of‑distribution regions within images. It uses statistical typicality, grouping pixels into superpixels and employing a computationally efficient proxy to estimate typicality. The approach is evaluated on medical imaging, human faces, and X‑ray security screening, outperforming prior methods across these diverse benchmarks.

By Jihoon Hong, Alireza Kheirandish, Rohan Sanda, Sara Fridovich-Keil
arXiv Machine Learning
Oct 23

Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch

The paper investigates how distribution mismatch—when the sampling distribution used to estimate policy gradient updates differs from the discounted state distribution required by theory—affects on‑policy policy gradient methods. For tabular direct and softmax parameterizations, it proves global convergence of objective values under biased updates and, for tabular softmax policies, derives exact leading‑order coefficients of asymptotic objective errors, providing a criterion to determine whether mismatch speeds up or slows convergence. The analysis is extended to general policy parameterizations, yielding explicit bounds on state distribution mismatch and normalized policy‑gradient mismatch in both episodic and continuing MDPs, showing these mismatches vanish at least linearly as the discount factor approaches one and bounding the squared norm of the normalized exact policy‑gradient direction under smoothness assumptions.

By Weizhen Wang, Xinzhe Zhou, Jianping He, Xiaoming Duan
arXiv Machine Learning
Oct 23

Projecting Latent RL Actions: Towards Generalizable and Scalable Graph Combinatorial Optimization

The paper introduces projection agents for graph combinatorial optimization, leveraging the Wolpertinger framework and a shared graph neural network latent space to predict latent actions in a single forward pass. This approach achieves up to 6.41× faster inference and 75% better generalization across diverse benchmarks compared to existing methods, while enabling strong reinforcement learning performance in large decision spaces. The authors also release the LaGCO‑RL Python library to facilitate reproducibility and adaptation to new GCO benchmarks.

By Franco Terranova (UL, LORIA, RESIST), Guillermo Bernardez (UC Santa Barbara - UCSB), Albert Cabellos-Aparicio (UPC), Nina Miolane (UC Santa Barbara - UCSB), Abdelkader Lahmadi (LORIA, UL, RESIST)
arXiv Computation and Language
Oct 23

FusionEval: Benchmarking Model Fusion for Large Language Models

FusionEval is a new benchmark for evaluating model fusion techniques that combine multiple expert models into a single large language model. It provides controlled expert pools trained from Qwen3 and Llama on domains such as math, code, science, instruction following, and tool use, and evaluates capability retention, computational cost, runtime, and GPU memory across eight public benchmarks. Experiments show that training‑free fusion methods keep reasoning performance but hurt tool use, while MOPD maintains tool use and balances capabilities at the cost of extra data preparation and training.

By Shuo Cai, Yanggan Gu, Yuhang Liu, Su Lu, Yuanyi Wang, Yang Yu, Zhen Li, Zhitian Hou, Guanghao Zhu, Ming Li, Hongxia Yang