arXiv AI

Do Web Agents Investigate Before They Decide?

The paper introduces MIRAGE, a benchmark of 750 multi‑step decision tasks designed to test autonomous web agents’ investigative abilities across Wikipedia Forensics, Shopping Admin adjudication, and Reddit Moderation. Each task contains a misleading visible context and a hidden context that holds decisive evidence, allowing performance to be broken down into Investigation, Reasoning, and Decision Accuracy, with an added Investigative Hallucination Rate. Evaluation of eight LLM agents reveals three consistent patterns: agents often reach relevant pages but fail to extract decisive evidence, procedural hints improve investigation but not decision accuracy on Wikipedia tasks, and 12.6% of trajectories include fabricated facts.

arXiv Machine Learning
Oct 23

ReDence: A Hybrid Agentic and Decision-Model Framework for Rapid, Evidence-Grounded Detection of Modern Social Engineering Scams

ReDence is an open‑source hybrid framework that combines a multimodal agentic Investigator model with a specialized Decider model to detect modern social engineering scams. It automates investigative planning, evidence interpretation, and decision‑making across multiple scam categories, achieving 96% accuracy and operating at least four times faster than comparable agentic models. The system outputs structured, evidence‑grounded reports that enable analysts to audit decisions and support mitigation efforts.

By Jaishnoor Kaur, Josef Hor\'a\v{c}ek, Ian Waskom, Sayak Saha Roy
arXiv Computation and Language
Oct 12

SEEDS: Preview-Guided Evidence Admission for Search Agents

The paper introduces SEEDS, a preview-guided evidence admission framework that allows search agents to selectively admit retrieved documents into their reasoning process. SEEDS exposes lightweight previews of candidates and lets the agent decide which and how many to fully inspect based on its current reasoning state, enabling arbitrary subset selection rather than fixed or rank-based policies. Experiments on nine knowledge-intensive QA benchmarks show that SEEDS improves search effectiveness and reduces full-evidence consumption compared to traditional retrieval prefixes.

By Jiajin Liu, Chenxi Xu, Dongzhe Fan, Yuanfu Sun, Qiaoyu Tan
arXiv AI
Sep 10

Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning

arXiv:2609.06027v1 Announce Type: cross Abstract: Search-augmented LLM agents are increasingly used for consumer decisions, making them vulnerable to Generative Engine Optimization (GEO) poisoning. E...

By Zhongan Bi, Qiwen Wang, Jianrong Jiang, Jigang Ding, Wenwen Xiong, Changhua Meng, Xuanang Gao, Kepeng Lin, Changjiang Jiang, Yiang Chen, Huan Yao, Wei Wang, Zhenyu Ma, Wenhui Dong
arXiv Computation and Language
4d ago

BlindToChange: AI Agents Can Find Changes When Asked But Do They Look When Needed?

BlindToChange is a new benchmark that tests whether AI agents can proactively discover recent real‑world events that affect their responses, even when those events are not explicitly mentioned in user requests. The benchmark uses automatically extracted events from Wikidata to create recommendation tasks where an obvious candidate is no longer applicable due to a recent change. Experiments show a large performance gap—up to 27.5 points—when agents fail to verify recent changes, revealing that many failures stem from not directly searching for updated facts.

By Anirudh Phukan, Tanya Goyal, Danish Pruthi
arXiv AI
Oct 12

LongHarness: Advancing Agentic Reasoning over 100M-Token Corpora via Stateful Evidence Management

LongHarness is a model‑agnostic harness that explicitly manages the interaction between acquired evidence and evolving conclusions for long‑context reasoning over corpora larger than a model’s context window. Its Evidence‑Conclusion Coevolution mechanism uses unresolved conclusions to steer evidence retrieval, while new evidence refines those conclusions, and its Evidence Lifecycle Management keeps evidence current and traceable. Evaluations on OfficeQA Pro, AA‑LCR, and LongBench v2 show substantial performance gains over direct reasoning and state‑of‑the‑art agent harnesses, with GLM‑5.3‑Flash exact match improving from 15.8% to 60.2% on V1 and from 4.4% to 61.1% on V2.

By Songtao Huang, Zhijie Zhong, Xiaohan He, Shuaiyu Zhang, Zhiyin Yu, Li Kang, Heng Zhou, Yutao Fan, Xiangchao Yan, Bo Zhang, Shiyang Feng
arXiv AI
Oct 19

Testing the Planning Claims of LLM-Based Agents against Formal Semantics

The paper evaluates the planning abilities of large language model (LLM) agents by comparing their performance against formal semantics defined in hidden PDDL models. Across five domains, four LLMs, six prompting strategies, and three agent architectures, the best setup solved 78% of solvable tasks offline but only 40% online, with limited recognition of unsolvable tasks and no clear benefit from more elaborate prompting. The study concludes that claims of LLM planning competence require validated action traces rather than end‑to‑end success rates.

By Oriol Miro-Lopez-Feliu, Oscar Mir\'o L\'opez-Feliu, Victor Gimenez-Abalos, Javier Vazquez-Salceda, Victor Carballo Araruna, Sergio Alvarez-Napagao