Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 21

ProMemAgent: Long-Horizon Proactive Assistance with Context-Aware Prospective Memory from Sensory Perceptions

ProMemAgent is a proactive agent that uses prospective memory to handle delayed user intentions over long horizons. It organizes past sensory perceptions into context-aware memory, generates prospective cues, and links earlier events to future developments for retrieval. The authors also introduce CPMBench, a benchmark of 1,000 instances across eight everyday categories, and show that ProMemAgent outperforms baselines in memory retrieval and proactive assistance tasks.

By Lilin Xu, Linyang He, Kaiyuan Hou, Yuang Fan, Minghui Zhao, Xuhai Xu, Xiaofan Jiang
arXiv AI
Oct 21

More Memory, Worse Agents: Error Reproduction and Anti-Persistence in LLM Agents

The paper investigates how large language model (LLM) agents can improve performance by storing and reusing experience in external memory. It identifies a problem called error reproduction, where mistakes in stored abstractions propagate through cycles of abstraction, execution, and re-abstraction, leading to persistent errors. To mitigate this, the authors propose Anti-Persistence, a memory design that retains raw trajectories and outcomes but reconstructs task-specific guidance on demand, discarding it after use. Experiments across multiple benchmarks show that Anti-Persistence yields higher held‑out success rates and lower LLM‑call costs compared to other memory designs, supporting the idea of separating experience accumulation from abstraction persistence.

By Ruhan Wang, Kishan Panaganti, Dongruo Zhou
arXiv Machine Learning
Oct 21

Bridging Digital Deception and Physical Fidelity: A Spatio-Temporal Attack on Cognitive Radars

The paper introduces the Segmented Spatio-Temporal Attack (SSTA), a new adversarial method for cognitive radars that addresses two key limitations of existing attacks: low cross-architecture transferability and sensitivity to physical channel distortions. SSTA uses a global‑local feature decoupling segmentation to break the dependence on long‑range sequence coherence, and incorporates differentiable spatio‑temporal operators into an Expectation over Transformation strategy to model physical distortions. Experiments on nine diverse models (CNNs, RNNs, Transformers) show a 10.36% improvement in average black‑box attack success rate, and simulations confirm that the crafted waveforms survive extreme distortions while maintaining acceptable signal quality.

By Tongrun Xing, Yongjun Dai, Xuan Liao, Zhiwen Zhang, Tong Zhou, Longlong Zhang, Tian Jiang
arXiv Machine Learning
Oct 21

LatticeMind: A Validation-Gated Agentic Framework for Density Functional Theory Workflows

LatticeMind is a validation‑gated, agentic framework that translates natural‑language requests into executable density functional theory (DFT) workflows. It uses workflow contracts to specify required stages, files, and dependencies, deterministic validators to check projects before solver launch, and bounded repair to fix failures, ensuring traceable reports linked to project files. On benchmarks spanning 44 RESCU and 19 VASP cases, LatticeMind reproduced all reference values, with 119 of 132 runs succeeding on the first validated plan and the rest after a single repair cycle, outperforming a conventional trial‑and‑repair baseline. whyItMatters":"The framework demonstrates that incorporating pre‑execution validation and repair can reliably prevent costly errors in multistage DFT workflows, making AI‑assisted scientific computing more trustworthy and efficient."

By Nathaniel M. Vegh, Yu Zhu, Rapha\"el J. Prentki, F\'elix Beaudoin, Shuaishuai Yuan, Hong Guo
arXiv Computation and Language
Oct 21

TRACE: Training-aware Robustness and Challenge Evaluation for Text-Vision Embedding Models

TRACE is a new framework that reorganizes 168 tasks for text‑vision embedding models into General, Adaptability, and Challenge categories. Experiments on two models show that adding general and adaptation data boosts Adaptability scores by 7.67 and 7.09 points while keeping General performance largely unchanged, yet Challenge performance stays below 5.0 with minimal gains from more data. The study highlights that data adaptation alone cannot overcome weaknesses in multilingual and visual‑document tasks, indicating a need for methodological advances.

By Pume Tuchinda, Tinnakit Udsa, Peerat Limkonchotiwat
arXiv Computation and Language
Oct 21

When Copying Is Hard: Copy-Constrained Decoding for Exact Span Reproduction

The paper introduces CopyGen, a lightweight copy-constrained decoding plugin for frozen large language models that improves exact span reproduction. CopyGen decomposes copying into three decisions—when to copy, how far to copy, and when to stop—using adaptive logit adjustments during verification to emit the longest valid prefix. Across three LLM backbones and four benchmark tasks, it boosts average exact match by 35% relative without finetuning and further improves by 4% with SFT, while also accelerating generation.

By Jinghui Zhang, Lang Gao, Zongfang Liu, Ruihong Zeng, Zirui Song, Rui Yan, Kentaro Inui, Xiuying Chen
arXiv Computer Vision
Oct 21

OmniVLoc: Topo-metric Visual Localization for Omnidirectional Cameras

OmniVLoc is a new visual localization system designed for omnidirectional cameras, leveraging a topo‑metric framework to overcome field‑of‑view limitations of traditional methods. It combines state‑of‑the‑art global place recognition with monocular depth estimation, solving a scale‑aware pose graph and applying precise spherical bundle adjustment to recover metric‑level poses. A hierarchical verification mechanism filters erroneous outputs, enhancing robustness, and extensive experiments on public and custom datasets demonstrate its superior accuracy and reliability.

By Zhuo Yu, Shuyang Zhang, Huihua Zhu, Wenjia Ouyang, Zuo Zuo, Jianhao Jiao, Yilong Zhu, Xiaopin Zhong, Zongze Wu
arXiv Computer Vision
Oct 21

VEOcc++: Learning Persistent Voxel Feature Maps for Embodied Online Semantic Occupancy Mapping and Offline Refinement

VEOcc++ is a voxel‑centric framework that unifies online incremental semantic occupancy mapping with offline global map refinement by maintaining a persistent sparse voxel feature map. During exploration the map expands and integrates local features with reliability‑guided updates, while a context‑aware decoder produces the current semantic occupancy map. After exploration, cached local features are consolidated and combined with the online representation for multi‑scale spatial refinement, achieving state‑of‑the‑art IoU and mIoU scores on EmbodiedOcc‑ScanNet and demonstrating zero‑shot transfer to unseen environments on a mobile robot.

By Ruoyu Wang, Jiahan Li, Yong Liu, Kexin Zhang
arXiv Computer Vision
Oct 21

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

The paper introduces Frequency-Aware Audio-Visual Segmentation (FAVS), a framework that treats audio-visual segmentation as a frequency-domain decomposition and recomposition task. It comprises a Frequency-Domain Enhanced Decomposer (FDED) that iteratively separates modality-specific semantics and structural features, and a Synergistic Cross-Modal Consistency (SCMC) module that uses a mixture-of-experts architecture to maintain semantic consistency and preserve modality-specific features. Experiments on three benchmark datasets show that FAVS achieves state-of-the-art performance, with qualitative visualizations supporting the effectiveness of both modules.

By Kai Peng, Yunzhe Shen, Miao Zhang, Jia Wang, Leiye Liu, Wei Ji, Jingjing Li, Yongri Piao, Huchuan Lu
arXiv Computer Vision
Oct 21

Efficient One-Step Diffusion Restoration Model with Compact Token Compression and Linear Attention

The paper introduces SANA‑SR, an efficient one‑step diffusion restoration framework for real‑world image super‑resolution. It uses a deep compression autoencoder to reduce latent tokens by a 32× ratio, preserving essential structures and textures, and applies a linear‑attention DiT with LoRA fine‑tuning for linear‑complexity token mixing. Experiments show competitive or superior performance on benchmark datasets, and the pruned model runs in 0.019 s with 407.95 G MACs and 344 M parameters, indicating strong potential for mobile deployment.

By Bingtian Qiao, Yue Shi, Yingjie Zhou, Yong Guo, Guangtao Zhai, Jiezhang Cao
arXiv Computer Vision
Oct 21

LADBench: A Benchmark for Logical Fault Detection in Images

LADBench is a new benchmark comprising over 1,000 synthetic images that contain logical anomalies across four domains—Residential, Urban, Collaborative, and Nature. It introduces a Tiered Prompting Protocol that progressively discloses information to test how much explicit help a model needs to locate and reason about these faults. Evaluation of leading foundation models shows that even the best only reaches 70.11% accuracy, indicating that implicit logical fault detection remains largely unsolved.

By Sahasra Kondapalli, Lara Radovanovic, Aadi Palnitkar, Mingyang Mao, Xiaomin Lin
arXiv Machine Learning
Oct 21

LightSplit: Practical Privacy-Preserving Split Learning via Orthogonal Projections (Full Version)

LightSplit introduces a lightweight orthogonal random projection at the split learning cut layer to reduce communication overhead and limit information exposure. By sending low‑dimensional projections instead of raw activations, the method preserves end‑to‑end differentiability while discarding irreversibly part of the representation, thereby mitigating reconstruction attacks. Experiments on seven benchmarks show up to 32× compression with minimal accuracy loss and strong resistance to state‑of‑the‑art inversion attacks.

By Mert Cihangiroglu, Alessandro Pegoraro, Phillip Rieger, Antonino Nocera, Ahmad-Reza Sadeghi
arXiv Computer Vision
Oct 21

CrossFit-GS: Out-of-Fold Pose Estimation for COLMAP-Free 3D Gaussian Splatting

CrossFit-GS is a new framework for pose-free 3D Gaussian Splatting that avoids the need for Structure-from-Motion by directly reconstructing scenes from unposed image sequences. The method partitions the input into interleaved subsets, maintaining two complementary submaps and cross-refining candidate poses against a frozen opposite submap to provide an independent geometric reference. Additional checkerboard validation and spatial coverage safeguards reject unreliable updates, and only verified poses are committed; after refinement, the final global representation is trained from scratch using the fixed refined poses, leading to reduced trajectory drift and improved camera pose estimation and novel-view synthesis across multiple benchmarks.

By Zijian Wu, Jinliang Wang, Mingfeng Jiang, Zidian Lin, Ying Song, Ziqian Lu