Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language
Sep 28

ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker

ZooWork-ShopRanker is a family of open e‑commerce rerankers (0.6B, 4B, and 8B) that align with human shopping preferences by using large language models as preference oracles to generate training pairs. The flagship 8B model serves as a teacher for the smaller 4B and 0.6B models, which are further refined on judged pairs. A new benchmark, ShopRank‑Bench, contains ~10,000 private‑traffic preference pairs and shows that all ZooWork models outperform the strongest open reranker baseline and their own un‑aligned versions.

By Siqiao Xue, Shuxuan Liu, Ning Hu
arXiv Computation and Language
Sep 28

Prompt Injection Detection for Email Agents Through Attack Chain Modeling

The paper introduces a prompt‑injection detection framework for email assistants that models attacks as a chain of stages. It combines a text detector, stage‑specific verifiers, rule‑based risk signals, user intent consistency checks, and a logistic decision policy. Experiments on five benchmarks show the framework outperforms pretrained detectors, achieving a mean F1 of 0.406 versus 0.216, and demonstrate that training on benign emails resembling attacks reduces false alarms.

By Ahmad Hashmi, Dhyey Patel, Yunting Yin
arXiv Computation and Language
Sep 28

Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering

Large language models (LLMs) are increasingly used for legal research, but their fixed training cutoffs and reliance on static knowledge clash with the evolving nature of statutory law. This study introduces a benchmark of 312 expert‑validated, time‑sensitive German statutory QA pairs that examine two temporal failure modes: post‑cutoff staleness and recency bias. Five LLMs were evaluated under four inference settings, and the results show that retrieval‑augmented approaches that enforce temporal validity significantly improve performance, while web search yields unstable gains and a pronounced recency bias.

By Max Prior, Andreas Schultz, Matthias Grabmair
arXiv Computation and Language
Sep 28

Generating Legal Commentaries from Case Databases via Retrieval, Clustering, and Generation

The paper introduces an automated pipeline that converts court decisions into legal commentaries for specific German Civil Code sections, using paragraph extraction, summarization, keyword clustering, and large language models to generate headings and citation-rich sections. The system processes 4,555 decisions from the German Federal Court of Justice, evaluates the output on relevance, heading match, citation faithfulness, cluster distinction, and logical ordering, and demonstrates that rapid, low-cost commentary generation is feasible while noting limitations due to source restrictions and legal reasoning norms.

By Max Prior, Niklas Wais, Matthias Grabmair
arXiv Computation and Language
Sep 28

From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMs

The paper investigates prompt injection attacks on 14 open‑source and 3 closed‑source large language models (LLMs), introducing a new metric called Attack Success Probability (ASP) that accounts for uncertainty in model responses. It demonstrates that a simple hypnotism attack can trigger objectionable behavior in models such as StableLM2, Mistral, Openchat, and Vicuna, achieving roughly 90% ASP. The study highlights that moderately well‑known LLMs are particularly vulnerable, underscoring the importance of public awareness and effective mitigation strategies.

By Jiawen Wang, Pritha Gupta, Eyke H\"ullermeier, Xiaoxue Gao, Nancy F. Chen
arXiv Computer Vision
Sep 28

ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models

ProCAP introduces a probabilistic cross-attentive prompt learning framework for vision-language models like CLIP, enabling improved cross-modal interaction without updating the backbone. It jointly learns visual and textual prompt tokens, linking them via stacked bidirectional multi-head cross-attention to refine each branch across prompt depth. The method incorporates Gaussian parameterization of prompt tokens, lightweight KL and L2 regularization, and a compact symmetric InfoNCE head to align image features with class-level text representations, achieving strong few-shot base-to-novel performance and competitive transfer results across multiple datasets and benchmarks.

By Hiwa Azeez Abbas, Fatemeh Daneshfar, Moloud Abdar
arXiv Computer Vision
Sep 28

Conditional Predictive Sufficient Statistics for Visual Representation Learning

The paper introduces Conditional Predictive Sufficient Statistics (CPSS) as a formal way to capture useful visual representations that preserve latent factors shared with future data while discarding noise. It shows that predicting the next image patch embedding with a cosine loss approximates maximum likelihood under a von Mises-Fisher model, and that stop‑gradient alone does not enforce sufficiency. Experiments on MNIST and CIFAR‑10 with small causal Transformers demonstrate that CPSS readouts outperform intermediate blocks, while removing stop‑gradient collapses embedding rank even when the pretext loss appears perfect.

By Yuzhou Hong
arXiv Computer Vision
Sep 28

LLPR: Location-aware learning and physics-based reconstruction for raindrop removal from a single image

The paper introduces LLPR, a framework that integrates location-aware learning and physics-based reconstruction to remove raindrops from single images. It replaces costly preprocessing masks with a learnable branch that can be discarded during inference, and reconstructs the background by first estimating a transparency matrix and raindrop layer using a physical model. The authors also present a new real-world dataset and show that LLPR outperforms existing state‑of‑the‑art methods.

By Zewei He, Xingyu Liu, Xing Luo, Guizhong Fu, Zixuan Chen, Yu Chen, Jinlei Li, Zhe-Ming Lu
arXiv Computer Vision
Sep 28

ManiVid: Unified and Explainable Forensic Analysis of Manipulated Videos

ManiVid introduces a unified forensic analysis framework for manipulated videos, combining forgery detection, artifact grounding, and anomaly explanation. The authors release ManiVid-38K, a large dataset of 19K real‑fake video pairs with authenticity labels, forgery masks, and explanations, and a benchmark ManiVidBench with 1K balanced pairs. ManiVidLens, the proposed model, outperforms existing methods in artifact grounding and anomaly explanation while matching state‑of‑the‑art detection accuracy.

By Hengrui Kang, Zhonghao Yan, Yuxuan Yang, Ruoyan Jing, Yuncheng Guo, Hao Chen, Kongming Liang, Zhanyu Ma, Conghui He, Weijia Li
arXiv Computer Vision
Sep 28

IDM-Net: A Lightweight Illumination-Decoupled Modulation Network for Low-Light Image Enhancement

IDM-Net is a lightweight Illumination-Decoupled Modulation Network designed for low-light image enhancement. It uses a dual-encoder architecture: a structure encoder extracts multi-scale appearance features from the RGB image, while a lightweight illumination encoder learns illumination priors from the decoupled luminance channel. An Illumination-Guided Modulation module injects these priors into the decoder via spatially adaptive affine modulation, and a Feature Refinement Block progressively suppresses artifacts and recovers fine details, achieving competitive performance on standard benchmarks with good balance between quality and efficiency.

By Cheng-Yen Hsiao, Jing-Ming Guo
arXiv Computer Vision
Sep 28

PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation

The paper introduces PICO, an end-to-end trainable model for 6DoF surgical tool pose estimation that uses multi-task learning to predict segmentation, depth, and pose parameters. It incorporates two geometry-aware proxy tasks—a projection loss and a point-to-point loss—to enforce consistency in 2D and 3D spaces, improving accuracy and robustness. Evaluated on the SurgRIPE dataset, PICO achieves strong performance, ranking second in rotation accuracy and maintaining competitive translation results, especially under occlusion.

By Lucy Fothergill, Pietro Valdastri, Dominic Jones, Duygu Sarikaya
arXiv Computer Vision
Sep 28

TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking

TRACKGRAPH is an online open‑vocabulary 3D mapping system that tracks 2D masks in the image stream before fusing them into a class‑agnostic 3D segment layer within a hierarchical scene graph. It uses FastSAM and CLIP for sparse keyframes, DINOv3 for dense mask propagation, and compact multi‑view CLIP embeddings for open‑vocabulary retrieval. The method outperforms state‑of‑the‑art mapping techniques on Replica, ScanNet++, and HM3D, achieving higher synonym frequency, faster processing, and lower GPU memory usage, and has been deployed on quadruped robots and drones at real‑time rates.

By Peder Borge Hellesylt, Albert Gassol Puigjaner, Kostas Alexis, Annette Stahl
arXiv AI
Sep 28

LLM Parkinsonism: Executive-Control Failure, Token-Inefficient Persistence, and an Uncertainty-Aware Global Executive Control Architecture for Autonomous Language-Model Agents

The paper introduces the concept of LLM Parkinsonism, describing how large language models can persist in low‑value actions after completing their objectives. It proposes a Global Executive Control (GEC) architecture that separates action generation from project‑level oversight, achieving comparable success to candidate‑set control while significantly reducing token usage and complexity. Experimental results on a 24,000‑episode benchmark show GEC cuts mean token use by 36.4% and limits token consumption at the 40,000‑token ceiling by 18.7%, eliminating pre‑completion drift.

By Dongsheng Xiao, Zeyuan Wang, Xuzhe Xia, Bo Zhao, Yankai Cao
arXiv Computation and Language
Sep 28

ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning

ToolSearcher is a reinforcement learning framework designed to improve large‑scale tool selection for large language models. It introduces category‑constrained discrimination, event‑level search modeling, and trajectory‑aligned credit allocation to better distinguish similar tools, optimize multi‑turn search, and provide fine‑grained rewards. Experiments on large‑scale benchmarks show that ToolSearcher outperforms strong baselines in iterative search and complex tool composition scenarios.

By Zhenlong Dai, Xujie Song, Zitong Wang, Tong Niu, Jian liu, Weiqiang Wang, Xiu Tang, Sai Wu, Chang Yao, Jingyuan Chen
arXiv AI
Sep 28

Stealth Apart, Harm Together: Skill Cascading Attacks on Skill-Based Agent Systems

The paper introduces skill cascading attacks, where a malicious goal is spread across multiple seemingly benign skills, causing harmful outcomes when combined. It presents SkillCascade, an automated red‑teaming framework, and releases SkillCascade‑Bench, a benchmark of 213 validated cascading test cases across various agent systems and domains. Experiments show that these cascaded interactions reliably induce harmful behaviors while evading existing per‑skill scanners and runtime monitors, revealing a gap between component‑level integrity and system‑level safety.

By Zihao Zhu, Siwei Lyu, Adel Bibi, Baoyuan Wu
arXiv AI
Sep 28

Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

The paper investigates Bayesian optimization using information geometry, deriving a local sensitivity tensor from the Fisher information metric that bounds the gradient of reparameterizable acquisition functions. This framework explains vanishing-gradient issues in high-dimensional settings and unifies heuristics like RAASP and dimension-scaled lengthscales. Leveraging this insight, the authors introduce FITR, a trust‑region BO method that replaces lengthscale scaling with local pullback‑Fisher weights, achieving competitive performance on GP benchmarks and extending naturally to non‑isotropic surrogates.

By Saksham Kiroriwal, Julius Pfrommer, J\"urgen Beyerer
arXiv AI
Sep 28

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

JevAdvBench introduces the first adversarial benchmark for reinforcement‑learning‑based calibrated decision (RLCD) models, providing 812 typed questions across 66 scenarios and a black‑box attack suite of 9,744 single‑edit variants. The benchmark evaluates attacks by comparing each perturbed decision to the model’s own clean decision and to an identical re‑run, revealing that rewording changes decisions by only 1.2 percentage points while certain injected opinions can flip 12.1% of decisions and lower confidence below 0.8 in 38% of cases. These findings demonstrate that RLCD models can be significantly misled by seemingly innocuous input edits, underscoring the need to treat the state as untrusted in applications.

By Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, Leo Yu Zhang