Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 27

The Agent in the Middle: How AI-Mediated Access Reshapes Web Vulnerability Exploitation

The paper examines how AI agents that interact directly with web service backends alter the exploitation of existing web vulnerabilities. Using the VELA benchmark, which includes 44 vulnerabilities across twelve application families, the authors evaluate three commercial agents and find that exploitability is largely determined by the interface design rather than the agent itself, with 91% of vulnerabilities remaining exploitable under the reference configuration. Interface restrictions can reduce attack success, yet the agents generally do not introduce new failures, and in some cases even complete attacks that users omitted.

By Ruozhao Yang, Jiongchi Yu, Ming Hu, Xiaofei Xie
arXiv AI
Oct 27

Towards a Unified Framework of Optimal Input-Side Dimensional Configuration for Deep CTR Prediction

The paper introduces OptIDC, a unified framework for optimizing input-side dimensional configuration in deep click‑through rate (CTR) prediction models. OptIDC jointly learns sample, feature, and embedding‑unit selections and employs a selection‑weighted Optimal Transport mechanism to coordinate decisions across levels, reducing conflicts while preserving semantics. Experiments on four real‑world datasets show that OptIDC consistently outperforms existing input‑side selection methods in terms of AUC.

By Chaohua Yang, Fuyuan Lyu, Xing Tang, Dugang Liu, Xiuqiang He, Zhong Ming
arXiv AI
Oct 27

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

The paper introduces objective‑conflict coefficients to measure how competing goals affect learning efficiency in multi‑objective reinforcement learning. It proposes a routed ensemble policy algorithm that keeps a small set of basis policies and routes trajectories to the most promising one, and a kernel surrogate trajectory attribution method that estimates how feedback from one trajectory influences learning on others. Experiments on Meta‑World and MO‑Gymnasium show routed ensembles boost average performance by 4.3% and the kernel attribution improves correlation with leave‑one‑out analysis by 50%, achieving a 27% reward gain over baseline attribution methods.

By Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. Zhang
arXiv AI
Oct 27

Do Similarity Measures Measure Similarity? How User Activity Creates Phantom Correlation in Recommender Systems

The paper investigates how standard item-item similarity measures in recommender systems can produce misleading associations, termed phantom correlation, due to variations in user activity levels rather than genuine shared preferences. By modeling independent consumption with differing activity volumes using a Rasch framework, the authors derive how item popularity influences these spurious associations for various similarity metrics. Empirical comparisons with real-world datasets reveal that many popular similarity measures are significantly affected by phantom correlation, impacting the interpretation and use of item co-occurrence in collaborative filtering.

By Marco Favier, Noah Dani\"els, Jens Leysen, Sander Marinus, Bart Goethals
arXiv AI
Oct 27

CAFE: Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning

CAFE is a new framework for automated feature engineering that treats the task as a causally‑guided sequential decision process. It first learns a sparse directed acyclic graph to identify causal relationships among features and the target, then uses a multi‑agent deep Q‑learning system to select causal groups and transformation operators while managing feature complexity. On 15 public benchmarks, CAFE outperforms leading AFE baselines, converges faster, and shows markedly lower performance drops under distribution shifts.

By Arun Vignesh Malarkkan, Wangyang Ying, Yanjie Fu
arXiv AI
Oct 27

A systematic evaluation of vision-language models for observational astronomical reasoning tasks

The paper introduces AstroVLBench, a benchmark of 4,100 expert‑verified instances covering optical imaging, radio interferometry, multi‑wavelength photometry, time‑domain light curves, and optical spectroscopy. Six leading vision‑language models were evaluated, revealing that performance varies strongly with modality and is not predicted by general benchmark rankings. Mechanistic ablations identified three bottlenecks: prompt grounding depends on model and task, presenting one‑dimensional data as tables rather than plots improves light‑curve classification, and a significant fraction of correct predictions are supported by physically inconsistent reasoning.

By Wenke Ren, Hengxiao Guo, Wenwen Zuo, Xiaoman Zhang
arXiv AI
Oct 27

Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability

The paper introduces Synergistic Faithfulness (𝔽_syn), a new metric based on the Shapley Interaction Index that isolates the joint contribution of visual and textual modalities in Vision‑Language Models (VLMs). It demonstrates that existing explainers overemphasize visual salience and fail to capture true cross‑modal synergy, and provides a scalable evaluation framework for auditing VLM reasoning. The authors evaluate nine XAI methods across three VLM architectures and datasets, revealing significant shortcomings in current approaches.

By Jo\"el Roman Ky, Salah Ghamizi, Maxime Cordy
arXiv AI
Oct 27

5W1H+Which: Context-Valid Semantic Indexing with Progressive Ontology Binding

The paper proposes 5W1H+Which, a semantic indexing framework that separates source‑grounded content extraction from progressive ontology binding. It organizes content into independently addressable units based on the 5W1H questions, while the "Which" component records links to versioned ontology elements, mapping relations, applicability scopes, and validation states. The design tracks business‑valid time, system knowledge time, and operational traces to enable revalidation and incremental maintenance, and illustrates its use with cloud migration and temporal policy examples.

By Yaxiao Liu (PwC China AI Center), Pengbo Liu (PwC China AI Center), Yiwen Liu (PwC China AI Center), Yihua Guan (PwC China AI Center), Jiaxing Song (Tsinghua University)
arXiv AI
Oct 27

JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

JRDB-AVR is a new benchmark for embodied agents that tests active visual reasoning in real-world environments. It requires agents to answer questions by requesting limited observations in time and view, and evaluates both the final answer and the visual evidence that supports it. The benchmark includes diverse tasks such as temporal search, viewpoint selection, and human-oriented compositional reasoning, and introduces a reference agent that builds an observation‑grounded graph world model.

By Zhixi Cai, Fucai Ke, Sukai Huang, Maria Garcia de la Banda, Peter J. Stuckey, Gholamreza Haffari, Hamid Rezatofighi
arXiv AI
Oct 27

FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents

FinAutoRubric is a system that lets experts define reusable evaluation guidance for financial research agents, which the agents then use to automatically generate, review, and validate task‑specific rubrics. The approach replaces costly fixed rubrics with query‑specific ones that incorporate expert standards and can be updated as new information becomes available. On three expert‑authored finance benchmarks, the automatically generated rubrics closely match human scoring and are preferred by in‑house analysts in blind reviews.

By Hoyoung Lee, Suyeol Yun, Jack Haverty, Yunju Cho, Meesong Kim, Daekyung Park, Sumin Kim, Jihoon Kwon, Jasmine Jia Geng, Andrew Chin, Yin Luo, Edward Tong, Yu Yu, Zach Golkhou, Minkyu Kim, Igor Halperin, Young Cha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee
arXiv AI
Oct 27

Can Multi-modal (reasoning) LLMs work as deepfake detectors?

The paper investigates whether state‑of‑the‑art multi‑modal large language models (LLMs) can detect deepfake images. Twelve recent LLMs were benchmarked against traditional detectors on multiple datasets, including real‑world deepfakes, and prompt tuning was used to improve performance. Results show that the best LLMs can match or exceed conventional methods on out‑of‑distribution data, though many LLMs perform poorly; larger model size helps, but newer versions and reasoning abilities do not necessarily improve detection.

By Simiao Ren (Dennis), Yao Yao (Dennis), Kidus Zewde (Dennis), Zisheng Liang (Dennis), Tsang (Dennis), Ng (Dennis), Ning-Yau Cheng, Xiaoou Zhan, Qinzhe Liu, Yifei Chen, Hengwei Xu
arXiv AI
Oct 27

Benchmarking Classification Tasks of Chest X-ray AI Across Multinational Public and Private Datasets: a retrospective diagnostic accuracy study

The study evaluated ten chest X‑ray AI classification models on 12 datasets from six countries, focusing on image‑level classification and measuring specificity at 95% sensitivity. MAVL and MedKLIP achieved the highest specificity (≈0.37) but still produced many false positives, with performance dropping on institutional cohorts compared to public benchmarks. No single model excelled across all findings, countries, or lesion types, highlighting limited generalisability.

By Qinmei Xu, Yiheng Li, Wenya Zhao, Zihao Zhao, Chuanjun Xu, Ahmet Gorkem Er, Xianghao Zhan, Brittany Dashevsky, Mohammed Alawad, Marc von der St\"uck, Marvin Gazibari\'c, Daniel Truhn, Shunsuke Shibata, Ryosuke Horino, Mio Ito, Akio Hiwatashi, Suyu Li, Mengya Yang, Ya Liu, Zhiyuan Sun, Changsheng Zhou, Haruka Itakura, Xiao Li, Olivier Gevaert
arXiv AI
Oct 27

Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generation

The paper introduces ParallaxRAG, a multi-view retrieval-augmented generation framework that addresses multi-hop reasoning challenges in large language models over knowledge graphs. It identifies that transformer attention heads specialize in distinct semantic relations across reasoning stages, forming a hop-aligned relay pattern, and proposes decoupling queries and KGs into head-specific semantic spaces to enforce relational diversity. Experiments on WebQSP, CWQ, and BioASQ show state‑of‑the‑art performance, reduced hallucination, and strong generalization.

By Jinliang Liu, Jiale Bai, Shaoning Zeng
arXiv AI
Oct 27

How well are open sourced AI-generated image detection models out-of-the-box: A comprehensive benchmark study

The study evaluates 16 AI‑generated image detection methods in a zero‑shot setting, testing 23 pretrained variants on 12 datasets with 2.6 million images from 291 generators. Results show no single detector consistently outperforms others, with accuracy ranging from 37.5 % to 75.0 %. The study highlights that training data alignment and generator type significantly affect performance, especially against modern commercial models, and identifies systematic failure patterns across datasets.

By Simiao Ren (Neo), Yuchen Zhou (Neo), Xingyu Shen (Neo), Kidus Zewde (Neo), Tommy Duong (Neo), George Huang (Neo), Hatsanai (Neo), Tiangratanakul (Dennis), Tsang (Dennis), Ng (Dennis), En Wei, Jiayu Xue
arXiv AI
Oct 27

Multi-Agentic AI for Fairness-Aware and Accelerated Multi-modal Large Model Inference in Real-world Mobile Edge Networks

The paper proposes a Multi-Agentic AI framework to enable fairness‑aware and accelerated inference of large, heterogeneous multi‑modal language models on mobile edge networks. It introduces a Global Planning Agent, a Prompt Scheduling Agent, and multiple on‑node LM Deployment Agents that jointly optimize prompt routing and model deployment using natural language reasoning over telemetry and historical data. Experiments on a city‑wide testbed show a 42.9% reduction in end‑to‑end latency versus a state‑of‑the‑art baseline and an 80.6% reduction versus a full‑activation baseline, while preserving high fairness (Jain's index 0.925).

By Haiyuan Li, Hari Madhukumar, Shuangyi Yan, Yulei Wu, Dimitra Simeonidou
arXiv AI
Oct 27

AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation

AffordSim is a scalable data generator and benchmark that integrates open‑vocabulary 3D affordance prediction into simulation‑based trajectory generation for robotic manipulation. Given a natural‑language task description, it synthesizes task‑relevant scenes, queries affordances, grounds them on object surfaces, samples region‑conditioned grasps, and selects executable candidates with motion planning, while randomizing pose, texture, lighting, noise, and backgrounds for sim‑to‑real transfer. The benchmark includes 50 tasks, five robot embodiments, and over 500 objects, achieving 93% of the success rate of manual contact annotations on affordance‑critical tasks and 89% on hard composite tasks, with policies trained on AffordSim transferring zero‑shot to a real Franka FR3 with 24% average success.

By Mingyang Li, Haofan Xu, Haowen Sun, Xinzhe Chen, Sihua Ren, Liqi Huang, Xinyang Sui, Chenyang Miao, Jiawei Ye, Qiongjie Cui, Xingyu Chen, Zeyang Liu, Xuguang Lan