arXiv AI

ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud

ScamWorld is an interactive environment that recreates reported scam websites across various scenarios, fraud mechanisms, and languages, allowing researchers to observe fraud indicators before a web agent’s final action. The platform enables controlled, repeatable evaluation of web agents, which are increasingly used to browse, shop, and manage accounts on the open web. A study of six leading LLM-powered agents revealed deception rates between 11% and 35%, with agents more likely to proceed with payments than to provide credentials or wallet information, and some agents continuing after recognizing suspicious conditions.

arXiv Machine Learning
Oct 23

ReDence: A Hybrid Agentic and Decision-Model Framework for Rapid, Evidence-Grounded Detection of Modern Social Engineering Scams

ReDence is an open‑source hybrid framework that combines a multimodal agentic Investigator model with a specialized Decider model to detect modern social engineering scams. It automates investigative planning, evidence interpretation, and decision‑making across multiple scam categories, achieving 96% accuracy and operating at least four times faster than comparable agentic models. The system outputs structured, evidence‑grounded reports that enable analysts to audit decisions and support mitigation efforts.

By Jaishnoor Kaur, Josef Hor\'a\v{c}ek, Ian Waskom, Sayak Saha Roy
arXiv AI
1d ago

Beyond Risk Cues: A Mechanism-Level Benchmark and Verifiable Debate for Scam Video Detection on the Web

The paper introduces MechScam, a benchmark of 6,614 videos that covers diverse scam mechanisms and non‑scam content, providing evidence‑supported explanations and annotations of scam elements. It also proposes ScamCourt, a debate framework where prosecution and defense debaters argue over verified citations, and a judge decides based on the transcript. Experiments show that current large multimodal language models struggle to differentiate scam videos from similar non‑scam content, while ScamCourt reduces false positives and better resists misleading arguments.

By Yifan Wang, Junyu Lu, Shun Zhang, Chaozhuo Li, Fanliang Bu, Rong Yin, Lingbin Bu, Changrong Min, Peng Chen
arXiv AI
Jun 16

MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attacks

arXiv:2602. 09222v2 Announce Type: replace-cross Abstract: Large language model (LLM) based web agents are increasingly deployed to automate complex online tasks by directly interacting with web sites and performing actions on users' behalf.

By Georgios Syros, Evan Rose, Brian Grinstead, Christoph Kerschbaumer, William Robertson, Cristina Nita-Rotaru, Alina Oprea
arXiv AI
Sep 10

Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning

arXiv:2609.06027v1 Announce Type: cross Abstract: Search-augmented LLM agents are increasingly used for consumer decisions, making them vulnerable to Generative Engine Optimization (GEO) poisoning. E...

By Zhongan Bi, Qiwen Wang, Jianrong Jiang, Jigang Ding, Wenwen Xiong, Changhua Meng, Xuanang Gao, Kepeng Lin, Changjiang Jiang, Yiang Chen, Huan Yao, Wei Wang, Zhenyu Ma, Wenhui Dong
arXiv Computation and Language
Oct 27

"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents

arXiv:2606.00497v2 Announce Type: replace-cross Abstract: Deceptive web content, widely instantiated across the internet and commonly known as \textit{social-engineering attacks}, manipulates autonom...

By Soham Roy, Sarthakbrata Halder, Arya Bharaty, Vaibhav Bhaskar, Yash Sinha, Dhruv Kumar, Srikant Panda, Murari Mandal
arXiv Machine Learning
Sep 25

A Corpus of Real Scam- and Spam-Call Conversations from an Active Voice-Agent Honeypot

The paper introduces a dataset of 10,015 real scam and spam phone calls collected over 53 days using an active voice‑agent honeypot. Each call is recorded, transcribed, and automatically labeled, yielding 328,869 turn‑level transcripts and 895 hours of audio from 5,665 distinct numbers. The corpus distinguishes between predatory‑but‑legal lead generation and outright scams, with labels validated by human review and technical checks on realism.

By Ethan Traister, Dennis Tsang Ng, Siyu Zhang, Huaiyu Guo, Tommy Duong, Tyler Wu, Yuchen Zhou, Xingyu Shen, Jiaqi Wu, Simiao Ren
arXiv Computation and Language
Oct 12

When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

The paper introduces MisVisAgentBench, a paired benchmark with 280 task instances that compare clean versus misleading chart evidence for web agents. It evaluates how deceptive visualizations affect agent execution, providing a diagnostic framework to separate misleading-induced failures from baseline difficulty. Results show that misleading charts reduce task success by over 34 percentage points and tend to redirect agents toward targeted misleading actions.

By Yisheng Li, Dongming Han, Hongkun Pan, Bingcheng Mao, Zijian Wang, Yunlong Mao, Wei Tong, Sheng Zhong
arXiv Machine Learning
Aug 26

Anatomy of a Scam Call: What 10,000 real scam and spam calls reveal about how phone scammers operate

The study analyzes 10,211 real scam and spam calls collected by an AI voice‑agent honeypot, revealing that scammers operate on a templated, office‑hour schedule and use disposable numbers to recycle scripts. Callers predominantly seek identity anchors such as home addresses and dates of birth, and the amount of conversation increases with the target’s age, though the requested information remains unchanged. Early detection is feasible, with escalation predictability reaching 0.87 ROC‑AUC by the eighth line using simple bag‑of‑words models.

By Ethan Traister, Ankit Raj, Jiaqi Gan, Xingyu Shen, Tyler Wu, Yuchen Zhou, Tommy Duong, Kidus Zewde, Siying Chen, Simiao Ren
arXiv AI
Jun 12

Who Pays the Price? Stakeholder-Centric Prompt Injection Benchmarking for Real-world Web Agents

arXiv:2606. 13385v1 Announce Type: cross Abstract: Web agents driven by large language models (LLMs) are increasingly deployed in real-world environments, where they operate over untrusted web content and execute actions with direct consequences.

By Zihao Wang, Yiming Li, Yutong Wu, Zheyu Liu, Kangjie Chen, Fok Kar Wai, Pin-Yu Chen, Vrizlynn L. L. Thing, Bo Li, Dacheng Tao, Tianwei Zhang