Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 9

Neural Network Verification for Deep Joint Source-Channel Coding

The paper introduces the first bound‑propagation framework for verifying the decoder of Deep Joint Source‑Channel Coding (DeepJSCC), providing worst‑case reconstruction error guarantees over a specified wireless channel noise region. It extends existing DNN verification techniques to handle PReLU activations, transposed convolutions, and Rayleigh fading, and incorporates Lipschitz‑regularized global robustness training (GloRo) to tighten certification bounds. Experiments on image transmission models show significant improvements in certified safety and real‑world OFDM validation confirms the theoretical guarantees.

By Thanh Le, Hai Duong, Takeshi Matsumura, ThanhVu Nguyen
arXiv AI
Oct 9

Agentic-TTT: Training test-time policy for test-time training

Agentic‑TTT introduces a test‑time policy that decides when and how to apply test‑time training (TTT) to large language models. By treating TTT procedures as tools and learning from the utility gains of its decisions, the policy can trade off performance improvements against computational cost. On a benchmark, Agentic‑TTT nearly doubles the utility of the base model, adapts to unseen domains, and demonstrates autonomous self‑improvement capabilities.

By Jiahao Lu, Mohan Kankanhalli
arXiv AI
Oct 9

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

The paper introduces AgenticBBO-Bench, a cross‑domain benchmark for evaluating large language model agents on black‑box optimization tasks such as synthetic functions, hyperparameter tuning, database tuning, chip design, and molecular design. Experiments show that agentic BBO outperforms direct LLM‑based methods in all five domains and beats the best numerical optimizers in four. The study also analyzes the impact of optimization tools, task semantics, prior knowledge, and the LLM’s role, finding that numerical tools are not consistently helpful, task semantics are broadly useful, and numerical optimizers can capitalize on agent‑generated search trajectories.

By Ming Chen, Rong-Xi Tan, Ke Xue, Yu-Jie Zhou, Taiye Lu, Zhi-Xuan Gao, Peng Xie, Zijun Shen, Chen Lu, Haopu Shang, Chao Qian
arXiv Computer Vision
Oct 9

Accurate and Efficient Object Pose Estimation via the Aggregation of Diffusion Features

The paper introduces a new approach to 3D object pose estimation that leverages diffusion model features, such as those from Stable Diffusion, to improve performance on unseen objects. Three aggregation architectures—vanilla, nonlinear, and context‑aware—are proposed, with a confidence‑adaptive aggregation network that selectively uses discriminative features to balance speed and accuracy. The method achieves state‑of‑the‑art accuracy on unseen object benchmarks (97.7% vs. 93.5% on Unseen LM) and real‑time performance (68 FPS) while reducing computational cost.

By Tianfu Wang, Guosheng Hu, Hongguang Wang
arXiv Computer Vision
Oct 9

Sparse-GS2Mesh: 3D Gaussian Splatting Guided by Novel Stereo Views and 2DGS for Sparse View Surface Reconstruction

Sparse-GS2Mesh is a stereo-aware framework that improves 3D surface reconstruction from sparse camera views by combining 3D Gaussian Splatting (3DGS) with stereo matching techniques. The method begins by initializing 3DGS with epipolar depth priors to reduce overfitting, then applies adaptive baseline selection, stereo-matching‑guided fine‑tuning, and 2D/3D co‑regularization to refine the model. Experiments show a 15% performance gain over state‑of‑the‑art methods in low‑overlap scenarios and comparable results in high‑overlap settings.

By Younghyun Noh, Minje Kim, Tae-Kyun Kim
arXiv AI
Oct 9

DynaTE: Accelerating Diffusion LLMs via Dynamic Token Execution

DynaTE is a hardware–software co‑design that accelerates diffusion‑based large language models (dLLMs) by dynamically adapting execution to token states. It skips low‑utility token computations, uses a dimension‑reconfigurable processing array, refines locally dependent tokens with FLDD, and streams vocabulary to handle irregular token patterns. These techniques yield 2.05–2.78× speedup and 2.99–3.93× higher energy efficiency compared to existing dLLM accelerators, and outperform Jetson AGX Orin by 2.55× speedup and 6.07× energy efficiency.

By Minghan Jiang, Jiayi Wang, Shuaiting Li, Haibin Shen, Kejie Huang
arXiv AI
Oct 9

Characterizing Overconfident Failure in LLM-Based Code Generation

The paper investigates how large language models (LLMs) used for code generation can produce syntactically plausible but incorrect programs, a phenomenon termed overconfident failure. By evaluating four open-source code models across three execution-based benchmarks, the authors examine whether token-level uncertainty metrics can predict execution correctness, analyze overconfidence at both global and local levels, and test mitigation strategies such as selective generation and instruction tuning. Their findings show that existing uncertainty signals are only partially reliable, overconfidence persists across models, instruction tuning can increase certainty without improving correctness discrimination, and common mitigation techniques do not fully resolve the issue. "whyItMatters":"The study highlights that current uncertainty metrics and mitigation strategies are insufficient to reliably detect or prevent incorrect code generation, underscoring the need for better reliability signals in LLM-based code synthesis."

By Ravishka Rathnasuriya, Wei Yang
arXiv Computation and Language
Oct 9

SteerablePlex: Can We Steer Full-Duplex Models?

The paper introduces SimIF-Bench, a benchmark that tests whether full‑duplex conversational models can stay within a prescribed scenario and complete multiple goals in order. It shows that current open‑source models struggle with these constraints. The authors then propose a GDPO‑based training recipe that allows a full‑duplex model, named SteerablePlex, to follow textual instructions during conversation while preserving turn‑taking, and demonstrate that it outperforms existing open‑source models and GPT‑Realtime when used as a controllable user simulator.

By Haolong Zheng, Maike Z\"ufle, Dominik Mach\'a\v{c}ek, Peter Pol\'ak, Xulin Fan, Xavier Sumba, Siyin Wang, Ond\v{r}ej Klejch, Mark Hasegawa-Johnson
arXiv AI
Oct 9

MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

MetaEncoder is a bi‑encoder model that transforms a pre‑trained Muse‑Glimmer 30B decoder into an instruction‑following decision‑making encoder for multimodal systems. It represents both user requests and candidate options in natural language, leveraging image and video inputs, and uses unidirectional contrastive learning to align requests with candidates across both small closed‑set and large open‑set spaces. Extensive tests on 11 benchmark suites and 190 tasks show MetaEncoder surpasses state‑of‑the‑art multimodal encoders in many areas, while still facing challenges on reasoning‑intensive tasks.

By Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao
arXiv AI
Oct 9

SDPAD: A Fully Spike-Driven Pipeline for End-to-End Autonomous Driving

SDPAD is a fully spike‑driven end‑to‑end planning pipeline for autonomous driving that converts a pre‑trained ANN perception stack into integer‑spike form, lifts multi‑view images into a bird’s‑eye‑view space with a spike‑driven depth distribution, and plans via a spiking query transformer that fuses ego, agent, and map queries. On the nuScenes benchmark it matches strong ANN planners with an average L₂ error of 0.40 m and a collision rate of 0.12 %, while consuming only 69.9 mJ—less than 2 % of recent ANN baselines. In closed‑loop NAVSIM evaluation it achieves 86.3 PDMS, surpassing the previous SNN planner SAD by 4.3 points and matching mainstream ANN planners at a fraction of their energy.

By Chengjun Zhang, Yuhao Zhang, Jie Yang, Mohamad Sawan
arXiv AI
Oct 9

Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selection

The paper proposes a multilingual adaptation of an English quality classifier to select high‑quality training data for large language model pretraining across more than 100 languages. The method trains a small multi‑layer perceptron on Transformer embeddings, using machine‑translated text scored by the English classifier as labels. Experiments on 1B, 3B, and 8B parameter models show that the adapted classifier preserves benchmark performance and regional knowledge, and it generalizes to unseen languages by learning the original scoring criteria.

By Vinko Sabol\v{c}ec, Bettina Messmer, Yassine Turki, Martin Jaggi
arXiv Computation and Language
Oct 9

Fair-GPTQ: Bias-Aware Quantization for Large Language Models

Fair-GPTQ is a quantization method for large language models that incorporates explicit group‑fairness constraints into the quantization objective. By guiding the rounding operation, it reduces bias in generated text, particularly for occupational stereotypes and discriminatory language across gender, race, and religion. The approach maintains at least 90 % of baseline accuracy on zero‑shot benchmarks while preserving the memory and speed benefits of 4‑bit quantization.

By Irina Proskurina, Guillaume Metzler, Julien Velcin
arXiv Computation and Language
Oct 9

GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness

GUI-KV is a plug‑and‑play key‑value cache compression method designed for graphical user interface agents that use vision‑language models. It exploits spatial saliency and temporal redundancy to prune cache entries, achieving near‑full‑cache accuracy while using modest memory budgets. Experiments on standard GUI agent benchmarks show significant reductions in decoding FLOPs and improvements in step accuracy compared to existing compression baselines.

By Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai, Caiming Xiong, Chien-Sheng Wu
arXiv AI
Oct 9

Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing

The study examines how the rapid turnover of large language models (LLMs) undermines the effectiveness of AI‑detectors used to screen scientific manuscripts. By comparing 4,000 PNAS abstracts rewritten by 23 different LLM versions released between June 2023 and August 2026, the authors show that detectors trained on past versions can fail dramatically at generation boundaries, missing up to 96% of rewrites just after a sharp transition. Even detectors updated to newer versions can overlook rewrites of older models, and a commercial detector missed most rewrites of the newest version while flagging few human‑written abstracts.

By Kazuki Nakajima, Takayuki Mizuno
arXiv AI
Oct 9

Beyond Report Imitation: Clinically Aware Multi-Image Ultrasound Report Generation from Visible Evidence

The paper introduces CAMEO, a framework for generating ultrasound reports from multiple images that emphasizes clinical awareness and evidence grounding. It consists of three stages: learning visual-language primitives, cross-view evidence grounding using distilled image-report pairs, and clinically aware preference alignment with error-oriented pairs. Experiments on the USReport-Distilled benchmark show significant improvements over prior models in BLEU, ROUGE, METEOR, and ClinicalScore metrics.

By Yuchen Yang, Xin Wang, Lufan Wang, Yinghong Pan, Yujuan Feng, Yuqing Yang
arXiv AI
Oct 9

Neural Networks for Temporal Pattern Recognition and Dynamic Arm Gesture Speed Estimation for Robot Control

The paper benchmarks ten temporal pattern recognition tasks across eighteen neural network architectures, identifying BiGRU, TCN, Conv1D, and GRUReLU as consistently top performers with under 2,000 parameters. It then applies three of these models to estimate dynamic arm gesture speed from skeletal keypoint sequences, achieving mean absolute errors of 0.198 (peak-count), 4% (period-time), and 8% (mean spike spacing). The study demonstrates that compact neural networks can reliably infer gesture speed, enabling speed-aware gesture-controlled robotic systems.

By Mil\'an Zsolt Bagladi, L\'aszl\'o Guly\'as
arXiv Computer Vision
Oct 9

Learning to Retrieve: Internalizing Memory Retrieval for Video World Models

The paper introduces Learning-to-Retrieve (L2R), a method that embeds memory retrieval directly into video world models. By using the model’s internal state as a memory and employing a camera‑conditioned retrieval gate and a retrieval trigger supervised by a 3D re‑visibility signal, the approach learns when and what historical information to access. This eliminates the need for external memory systems and improves long‑term scene consistency across various base models and camera‑revisit benchmarks.

By JiaKui Hu, Tailai Chen, Yuqi Pan, Xuerui Qiu, Jialun Liu, Xiao Cao, Zhenxin Zhu, Guang Chen, Hangjun Ye, Bing Wang, Yanye Lu
arXiv AI
Oct 9

DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

The paper introduces DIAL-OPD, a token‑selection strategy for on‑policy distillation that weights reward magnitude by the logarithmic mean of teacher and student probabilities, controlled by a parameter β. Experiments on four teacher‑student pairs across seven mathematical reasoning benchmarks show that retaining only 40 % of tokens with DIAL‑OPD yields up to 5.25 % higher mean accuracy than vanilla OPD and doubles AIME25 Pass@16, outperforming even stronger full‑token baselines. The study demonstrates that careful supervision allocation can surpass gains from simply scaling teacher size.

By Anhao Zhao, Haoran Xin, Junlong Tong, Yingqi Fan, Xuan Lu, Ping Nie, Wenjie Li, Xiaoyu Shen
arXiv Computer Vision
Oct 9

Beyond Resolution: Object-to-Image Ratio Mismatch in Instance Retrieval

The paper investigates why visual instance retrieval often fails when the same object appears at different apparent sizes in query and gallery images. It identifies object-to-image (O2I) ratio mismatch—where the object occupies different fractions of the two images—as the dominant cause of performance degradation, rather than resolution loss. Experiments on a controlled benchmark with 3,021 Objaverse objects show that over 80% of cross-distance degradation is due to O2I mismatch for most pretrained backbones, and that multi-scale architectures only reduce resolution effects but remain vulnerable to O2I mismatch. The authors demonstrate that query-side scale augmentation and an OWLv2 crop reranker achieve state‑of‑the‑art results on ILIAS 100M without retraining the gallery, and that a LoRA fine‑tune can match these gains in a single forward pass, indicating that O2I robustness can be learned.

By Boaz Meivar, Ofir Kedem, Amit Edenzon, Gal Chechik, Shai Avidan
arXiv Computer Vision
Oct 9

Stop My Dancing! Understanding, Detecting and Attributing Motion-Aware Deepfake Videos

The paper introduces Motion‑Aware Deepfake (MAD) videos, a new class of deepfakes generated by pose‑guided diffusion models that can animate entire human figures. To study this threat, the authors build the first MAD‑specific benchmark with 1.5 million frames from 1,363 real and 30,122 synthetic videos, and analyze the subtle high‑frequency artifacts and spectral fingerprints that arise from limited input frames. Leveraging these insights, they propose MoDA, a defense framework that combines spatial semantics with steganalysis‑rich frequency features, achieving high detection and attribution accuracy across in‑distribution, cross‑dataset, and zero‑shot scenarios, and demonstrating robustness against adaptive attacks.

By Fazhong Liu, Yan Meng, Tian Dong, Guoxing Chen, Haojin Zhu