Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language
Oct 22

Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research

Quant-Arena is a new benchmark and public leaderboard for end‑to‑end quantitative research using real A‑share market data. Models build features and stock‑scoring code, receive feedback only on 2022–2023 data, and freeze a strategy for evaluation on the sealed year 2024. Scores are based on realized excess return, signal quality, and feature diversity, with GPT‑6 Astra currently leading but still not outperforming the equal‑weight market in 2024.

By Houde Qian, Zheng Jiang, Wenyi Li, Chenxi Xu, Xinjie Lin, Yuzhao Peng, Ruoyu Han, Youjie Zheng, Kaisen Yang, Qinhuai Na, You Fu, Calvin Xiao
arXiv Computation and Language
Oct 22

Where Should a Stuck Agent Spend Its Next Tokens? Causal Attribution of Inference-Time Resources

The paper introduces Banzhaf Resource Attribution (BRA), a factorial framework that quantifies how different inference-time resources—computation, information, and feedback—contribute to repairing failed coding agents. Experiments on six models across 194 function-level and 126 repository-level failures show that task-relevant specifications consistently improve repair, while public-test feedback offers modest gains and retrieved code offers little benefit. The study finds that as model capability increases, the value of extra computation diminishes, yet information remains useful, and that agents often prefer feedback even when it yields lower gains than random selection, revealing a mismatch between preferred and effective resources.

By Saiyue Lyu, Jerrold Huang, Yunbei Zhang, Janet Wang, Zhenting Wang, Yingqiang Ge, Haibo Zhang
arXiv Computation and Language
Oct 22

VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective

The paper introduces VeeReview, a tool that classifies build‑and‑verify pairs in pull requests using repository signals and a language‑model judge. Evaluated on 1,454 agent‑ and human‑authored pull requests, VeeReview found lower build‑only rates for agent‑authored changes but failed to meet calibration and agreement thresholds. The study concludes that a mechanically decided component and validated human reference are essential for reliable verification‑practice comparisons.

By Gabriel Dautovi, Jie JW Wu
arXiv Computation and Language
Oct 22

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

The paper introduces a reproducible framework that converts existing text-based tool‑calling benchmarks into audio‑based evaluations using text‑to‑speech, speaker variation, and environmental noise, while preserving original annotations. Experiments on seven omni‑modal models show that performance varies strongly by model and task, with notable gaps between text and voice results. The study also includes a stress test, a reference‑free LLM‑as‑judge protocol, and demonstrates that open‑source judges can match proprietary ones, aiding privacy‑preserving evaluation.

By Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN
arXiv Computation and Language
Oct 22

MemoProAgent: Toward Memory-Driven Proactive Assistance

MemoProAgent tackles memory‑driven proactive assistance by mining long‑horizon personal memories to decide when an assistant should intervene. The authors introduce MemoProBench, a benchmark pairing egocentric video with year‑long, distractor‑rich user memories and human‑verified trigger windows, and show that existing models perform poorly (≤15%). MemoProAgent uses a graph‑organized memory store and a lightweight RL stage to improve accuracy from <10% to 42.4% without training.

By Shenglai Zeng, Xiao Yang, Tiffany Ma, Xinyuan Zhang, Nanshu Wang, Deren Lei, Kai Sun, Xilun Chen, Tammy Stark, Xianhui Zhu, Yuetai Li, Rulin Shao, Tianxin Wei, Rakesh Wanga, Anuj Kumar, Jiliang Tang, Wen-tau Yih, Xin Luna Dong
arXiv Computer Vision
Oct 22

Expandable Modality Experts for Medical Vision-Language Learning

The paper introduces Expandable Modality Experts, a medical vision‑language framework that uses an expandable bank of modality experts and a Multiway Transformer to combine shared visual computation with deterministic routing and a separate text pathway. It employs a modality‑aware visual tokenizer for masked pretraining on 19.4 million unpaired image rows across seven modalities, followed by contrastive alignment on 3.23 million image‑text pairs. With a ViT‑Large backbone, the method achieves state‑of‑the‑art results in 24 of 25 benchmark settings across nine modalities, including zero‑shot classification, linear probing, and retrieval tasks, while preserving existing representations.

By DM BAPPY, Eunsu Kim, Hanjoo Lee, Sung Hak Lee, Sangjeong Ahn
arXiv Computer Vision
Oct 22

ViBMR: A Large-Scale Multi-Sequence Brain MRI Dataset and Comprehensive Evaluation for Vietnamese Radiology Report Generation

ViBMR is the first large‑scale multi‑sequence brain MRI dataset paired with routine Vietnamese clinical reports, containing 3,649 patients and 11,875 MRI series across T1, T2, FLAIR, and contrast‑enhanced T1 sequences. The authors establish a comprehensive evaluation framework for brain MRI report generation, testing various medical vision‑language models, multi‑sequence inputs, and feature aggregation strategies. Results highlight dataset characteristics, model behavior, and the importance of native low‑resource clinical datasets for reliable Vietnamese‑origin brain MRI report generation.

By Dai Lam Le, Minh Khoi Nguyen, Son Dinh Nguyen, Amir Reza Jafari, Viet Ha, Nguyen Son, Reza Farahbakhsh, Noel Crespi, Phi Le Nguyen
arXiv Computer Vision
Oct 22

Attn-Forcing: Learning Full-Context Attention Proxies for Streaming Video Generation

Attn-Forcing is a new framework for streaming video generation that preserves full-context attention contributions while operating under bounded memory and computation. It introduces learnable visual latents as attention proxies, updated via a Proxy Update Module to incorporate historical visual context, and employs an Attention Matching Loss to explicitly supervise the proxies using full-context attention. Experiments show that Attn-Forcing consistently improves video generation quality and outperforms existing baseline methods.

By Junyao Gao, Sibo Liu, Jiaxing Li, Yanan Sun, Hanxin Zhu, Weidong Zhang, Jun Zhang, Cairong Zhao
arXiv Computer Vision
Oct 22

DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

The paper introduces DONIC, a generalist bimanual tracking controller trained on diverse operation categories and human-motion data, and BiHOI, a multi‑view dataset of 1,575 sequences with 229 objects that captures sustained bimanual interactions. DONIC employs proximal policy optimization combined with behavioral cloning, an object‑centered per‑hand reward, and Clip‑Centered Advantages to learn coordinated manipulation from a single policy. The method achieves 86.79% success on unseen trajectories of training objects and 57.32% on held‑out objects, outperforming existing bimanual tracking baselines by large margins.

By Mingze Sun, Ziang Cao, Yuan Fei, Qiankuan Tao, Zihan Wang, Tianqi Liu, Junhao Chen, Haiwen Diao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu
arXiv Computer Vision
Oct 22

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

The paper introduces Multimodal Embodied Agents (MMEAs), goal‑directed systems that combine multimodal task reasoning with physical action and adapt decisions based on feedback. It proposes the PAPAV framework, which defines five core capabilities—Perceive, Anticipate, Plan, Act, Verify—common to computer‑use agents, robotic systems, and MMEAs. The authors analyze 64 benchmarks, noting that most evaluations focus on Act while Anticipate and Verify are rarely assessed, highlighting gaps and open challenges in designing reliable physical agents.

By Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Kevin Qinghong Lin, Mike Zheng Shou
arXiv Computer Vision
Oct 22

Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling

The paper introduces PLOVIS, a data‑efficient framework for semantic segmentation of 3D point clouds that tackles three simultaneous data shortages: few training scenes, sparse point‑level labels, and missing 2D image sequences. PLOVIS generates pseudo‑labels by applying an open‑vocabulary image segmentation model directly to 3D point clouds, then refines these labels through a two‑stage filtering process and a class‑balanced memory bank. Experiments on ScanNet, S3DIS, Toronto3D, and Semantic3D show that PLOVIS outperforms existing fine‑tuning and weakly supervised methods under realistic, data‑scarce conditions.

By Takahiko Furuya
arXiv Computer Vision
Oct 22

Goal Images Are Control Targets: Spatially-Grounded Goal-Image Synthesis for Robotic Manipulation

Goal images used as control targets in robotic manipulation are not best chosen by general image‑editing quality. The authors show that large proprietary editors, though top on editing‑quality leaderboards, produce poorer control targets than smaller fine‑tuned open models, especially on long‑horizon tasks. They introduce SGE‑Goal, a spatially‑grounded editing framework that predicts edit‑region masks before synthesis, enabling open base models to become the strongest goal generators on CALVIN and nearly as strong on SIMPLER‑Bridge, with superior performance on chained tasks.

By Riddhi Chatterjee, Ritish Shrirao, Kushagra Mehrotra, Viswanath Gopalakrishnan
arXiv Computer Vision
Oct 22

Painting Change Map as guidance for generating painting processes

The paper introduces the Painting Change Map, a spatio‑temporal control mechanism that tracks normalized structural changes across a 1,000‑timestep continuum for dynamic video generation of paintings. By encoding localized visual differences into an RGB space, it provides fine‑grained guidance for both forward painting and reverse unpainting processes. Experiments show that this approach outperforms existing control baselines and matches or exceeds state‑of‑the‑art painting methods, especially under oracle conditions.

By Markus Pobitzer, Chang Liu, Niccol\`o Biondi, Nicu Sebe
arXiv Machine Learning
Oct 22

A Novel Schur Decomposition-Based Weight Projection Method for Stable State-Space Neural-Network Architectures

The paper presents a new projection scheme that uses the Schur decomposition to enforce asymptotic stability in linear discrete-time state-space layers of neural networks. By projecting the quasi-triangular factor of the state matrix onto its nearest stable counterpart, the method maintains stable dynamics while keeping the weight count low. Experiments on synthetic linear systems show comparable accuracy and convergence to state-of-the-art techniques, with reduced overparameterization aiding training of deeper architectures.

By Sergio Vanegas, Lasse Lensu, Fredy Ruiz
arXiv Machine Learning
Oct 22

Attention-based Classification for Indirect Prompt Injection

The paper evaluates the Attention Tracker, a white‑box detection method for indirect prompt injection (IPI) attacks, in realistic agentic settings using the AgentDojo benchmark. It finds that the method’s claimed dataset‑agnostic detection fails in agentic environments, with near‑chance performance across most model families. The authors propose AttnDA, a Fisher Linear Discriminant Analysis classifier that replaces the scalar threshold, which improves detection across multiple benchmarks but still struggles to transfer to AgentDojo, highlighting the importance of training data from the target domain.

By Amelia Kawasaki, Stefan Lee
arXiv Machine Learning
Oct 22

SAGE: Semantic Audio Generative Encoder

SAGE, a 105‑million‑parameter semantic audio generative encoder, compresses music waveforms into compact latent representations by distilling embeddings from a pretrained audio‑text model. It achieves a three‑way balance—high reconstruction quality, rich semantic latent structure, and fast inference—running three times faster than Stable Audio Open while matching or surpassing larger models in listening‑test quality and objective metrics. Across all nineteen latent‑semantics probing tasks, SAGE sets new state‑of‑the‑art performance both within and outside its training domain.

By Francesco Brigante, Luca Cerovaz, Davide Marincione, Giorgio Strano, Luca Zhou, Emanuele Rodol\`a, Michele Mancusi
arXiv Computation and Language
Oct 22

Remembering Through Change: Provenance-Preserving Hypergraph Memory for Multimodal Agents

The paper introduces MHG‑Mem, a multimodal hypergraph memory that preserves fine‑grained visual evidence while tracking entity evolution across sessions. It represents entity mentions and observed facts as nodes, groups identity‑consistent mentions via occurrence hyperedges, and synthesizes aspect‑specific evolution states linked through evolutionary and evidential hyperedges. Hypergraph‑based agentic search enables adaptive traversal of the memory, allowing detailed visual verification and reasoning over cross‑session state changes, achieving 8.20‑9.98% performance gains on three benchmarks.

By Jielong Tang, Ling-An Zeng, Xujie Yuan, Xiao Dong, Zhiqian Xia, Qingfeng Li, Shixiong Kai, Kaichao Liang, Rong Yao, Jian Yin
arXiv Computation and Language
Oct 22

The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages

The paper investigates how token consumption varies across programming languages for coding agents. Five recent models were evaluated on Python, Java, Rust, and OCaml, revealing significant differences in token usage that persist across models. Analysis of agent trajectories shows that agents often generate non‑compiling solutions in unfamiliar languages, revise passing solutions, plan in code comments, distrust provided tests, and prototype in Python to sidestep unfamiliar languages.

By Zixuan Wu, Carolyn Jane Anderson, Arjun Guha
arXiv Machine Learning
Oct 22

A Comprehensive Evaluation on Quantization Techniques for Large Language Models

The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.

By Yutong Liu, Cairong Zhao, Guosheng Hu
arXiv Machine Learning
Oct 22

Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints

The paper introduces Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (FMER), a reinforcement learning framework that uses ODE-based flow matching to overcome limitations of SDE-based diffusion policies. FMER derives an analytic entropy objective to correct density distortion from a tanh transformation, establishes that minimizing an advantage-weighted conditional flow matching loss serves as a simulation-free surrogate for policy mirror descent, and dynamically tunes the mirror descent temperature to enforce a robust trust region. Experiments show FMER outperforms baselines on the sparse-reward FrankaKitchen task while remaining competitive on dense-reward MuJoCo benchmarks.

By Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, Elvin Isufi