arXiv Machine Learning

Anatomy of a Scam Call: What 10,000 real scam and spam calls reveal about how phone scammers operate

The study analyzes 10,211 real scam and spam calls collected by an AI voice‑agent honeypot, revealing that scammers operate on a templated, office‑hour schedule and use disposable numbers to recycle scripts. Callers predominantly seek identity anchors such as home addresses and dates of birth, and the amount of conversation increases with the target’s age, though the requested information remains unchanged. Early detection is feasible, with escalation predictability reaching 0.87 ROC‑AUC by the eighth line using simple bag‑of‑words models.

arXiv Machine Learning
Sep 25

A Corpus of Real Scam- and Spam-Call Conversations from an Active Voice-Agent Honeypot

The paper introduces a dataset of 10,015 real scam and spam phone calls collected over 53 days using an active voice‑agent honeypot. Each call is recorded, transcribed, and automatically labeled, yielding 328,869 turn‑level transcripts and 895 hours of audio from 5,665 distinct numbers. The corpus distinguishes between predatory‑but‑legal lead generation and outright scams, with labels validated by human review and technical checks on realism.

By Ethan Traister, Dennis Tsang Ng, Siyu Zhang, Huaiyu Guo, Tommy Duong, Tyler Wu, Yuchen Zhou, Xingyu Shen, Jiaqi Wu, Simiao Ren
arXiv Computation and Language
Oct 9

Black-Box Forensics for Conversational LLM Agents

The paper presents methods for black-box forensic analysis of conversational large language model (LLM) agents, focusing on two tasks: attribution—identifying the base model powering a chatbot—and fingerprinting—detecting identical system prompts across different endpoints. Attribution classifiers achieve 98% accuracy using only a few non‑adversarial conversation turns, while a cross‑encoder fingerprinting approach attains an AUC of 0.768 and an F1 of 0.703 on unseen prompts, improving to 0.943 AUC when aggregating 50 interactions. These techniques enable investigators to trace AI‑enabled scams back to their underlying models and link related scams through shared system prompts.

By Isadora White, Yasaman Jafari, Taylor Berg-Kirkpatrick
arXiv Computation and Language
Sep 17

TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

TeleAntiFraud 2.0 is a monthly‑frozen, audio‑based benchmark for telecom fraud detection that incorporates newly observed scam patterns while preserving earlier test sets. It uses a Mixed‑Tree Anti‑Fraud Generation Pipeline to create profile‑grounded scenarios, expands them into mixed‑tree dialogues, and renders validated speech for 900 Chinese calls (600 fraud, 300 near‑domain non‑fraud) each month. Experiments show that classifiers perform well against unrelated negatives but drop significantly against near‑domain negatives, highlighting the need for near‑domain construction and collapse‑aware reporting in realistic evaluation settings.

By Huiyuan Liu, Zhiming Ma, Yanxing Liu, Shun Zhang, Qifan Wang, Di Liu, Yifan Wang, Yuyang Deng, Haoyang Meng, Yijin Zhou, Yuxi Zhao, Chengxian Hu, Peidong Wang, Peng Chen
arXiv Computation and Language
Oct 27

"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents

arXiv:2606.00497v2 Announce Type: replace-cross Abstract: Deceptive web content, widely instantiated across the internet and commonly known as \textit{social-engineering attacks}, manipulates autonom...

By Soham Roy, Sarthakbrata Halder, Arya Bharaty, Vaibhav Bhaskar, Yash Sinha, Dhruv Kumar, Srikant Panda, Murari Mandal
arXiv AI
Sep 30

Agentic Commerce Bench: Measuring Fraud Detection for Agents That Spend Money

The paper introduces the Agentic Commerce Bench (ACB), a benchmark for measuring fraud in AI agents that autonomously spend money. It presents a taxonomy of agentic commerce fraud, a dataset of twenty fraud classes derived from real production data, and an open‑source detector stack called gordonguard for auditing and replaying hostile counterparties. The study shows that current reasoning layers and security scanners perform poorly on many classes, highlighting the need for better detection mechanisms.

By Ankit Srivastava, Debjyoti Paul
arXiv AI
6d ago

Joint Detection of Fraud and Concept Drift inOnline Conversations with LLM-Assisted Judgment

The paper presents a two‑stage framework for detecting fraud in online conversations. First, an ensemble classifier flags suspicious dialogues, then a One Class Drift Detector isolates concept drift within those flagged exchanges. A large language model evaluates whether detected drift signals fraudulent manipulation or a legitimate topic shift, improving accuracy and interpretability compared to a Dual LLM baseline.

By Ali \c{S}enol, Garima Agrawal, Huan Liu
arXiv AI
1d ago

ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud

ScamWorld is an interactive environment that recreates reported scam websites across various scenarios, fraud mechanisms, and languages, allowing researchers to observe fraud indicators before a web agent’s final action. The platform enables controlled, repeatable evaluation of web agents, which are increasingly used to browse, shop, and manage accounts on the open web. A study of six leading LLM-powered agents revealed deception rates between 11% and 35%, with agents more likely to proceed with payments than to provide credentials or wallet information, and some agents continuing after recognizing suspicious conditions.

By Yutao Wu, Xiao Liu, Ming Wen, Hanxun Huang, Jun Wang, Xingjun Ma
arXiv AI
Sep 2

Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Models

The paper presents a cumulative turn‑based risk assessment framework for detecting financial scams targeting older adults, which aggregates conversational turns and updates risk estimates at each step. A multi‑turn dialogue dataset covering investment, charity, and tech support scams is created, with annotations for risk level, score, rationale, and safety recommendation at every cumulative stage. Four small language models (Phi‑4, LLaMA‑3.2, DeepSeek‑R1, Qwen3) are fine‑tuned; Phi‑4 and LLaMA‑3.2 outperform others in turn‑aware risk estimation, demonstrating that compact models can effectively support incremental scam detection in resource‑constrained, privacy‑aware deployments.

By Parviz Ghafariasl, Weimin Fu, Xiaolong Guo, Shing I. Chang