ATLAS-Bench is a new benchmark comprising 1,200 local search queries across three regions, designed to evaluate large language model agents on their ability to satisfy mandatory requirements, preferences, and factual versus subjective judgments using place attributes, reviews, and user profiles. Each query distinguishes constraints from preferences and includes evidence and criteria for verification, with validation through iterative assertion checks and cross-model review. The benchmark is paired with ATLASBox, a standardized evaluation environment that provides search tools and a fixed POI knowledge base, allowing agents to recommend evidence-supported options or report unmet requirements, and the study reports that GPT-6-Luna achieves the highest constraint satisfaction and factual accuracy, though GLM-5.3 returns acceptable recommendations for more queries.
whyItMatters":"The benchmark addresses gaps in existing local search evaluations by rigorously testing agents’ ability to meet mandatory constraints and preferences, thereby advancing the development of reliable LLM agents for everyday decision-making and planning."
By Keyun Zhang, Dapeng Yin, Fanfu Wang, Shuang Lang, Zhenyu Zhu, Yulin Huang, Chong Yang, Pengjie Wang
arXiv:2606. 12871v1 Announce Type: new Abstract: Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses.
By Jingxuan Han, Wei Liu, Mingyang Zhu, Youpeng Wang, Ziwen Wang, Lin Qiu, Xuezhi Cao, Xunliang Cai, Zheren Fu, Licheng Zhang, Zhendong Mao
The paper evaluates two strategies for training language‑model‑based world models—fine‑tuning and Retrieval Augmented Generation (RAG)—across five diverse environments. Fine‑tuning generally yields higher rewards, while RAG is more data‑efficient. The authors propose a counterfactual error estimation for RAG, improve retrieval with a hierarchical query reformulation, and combine both approaches into a hybrid system that outperforms each method alone.
By Dhananjay Ashok, Shantanu Agarwal, Vivek Datla, Jonathan May, Alfy Samuel
arXiv:2602. 22638v2 Announce Type: replace Abstract: Route-planning agents powered by large language models (LLMs) have emerged as a promising paradigm for supporting everyday human mobility through natural language interaction and tool-mediated decision making.
By Zhiheng Song, Jingshuai Zhang, Chuan Qin, Chao Wang, Chao Chen, Longfei Xu, Kaikui Liu, Xiangxiang Chu, Hengshu Zhu
The study examines how three context‑engineering techniques—semantic layers, orchestration, and compaction—affect data agents that answer questions over heterogeneous data. Using seven large language models on 1,514 long‑horizon tasks, the authors identify three agent behavior groups (investigators, premature answerers, early stoppers) and show that each technique benefits only the group it aligns with, while sometimes harming others. Combining the techniques yields smaller net gains than the sum of individual effects, leading to guidance on selecting techniques based on agent behavior, task demands, and accuracy‑cost trade‑offs.
By Zhivar Sourati, Marianne Menglin Liu, Zheng Wang, Rongguang Wang, Sai Ashish Somayajula, Tao Sheng, Kyu J. Han, Sujith Ravi, Dan Roth
arXiv:2606. 13904v1 Announce Type: cross Abstract: Exploratory question answering (EQA) over data lakes requires an LLM agent to discover relevant sources, analyze retrieved data, and adapt its actions based on intermediate results.
By Austin Senna Wijaya, Jiaxiang Liu, Haonan Wang, Eugene Wu
arXiv:2605. 27882v2 Announce Type: replace-cross Abstract: LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying, revealing a persistent evaluation-experience gap.
By Xiaohongshu Inc
arXiv:2610.20959v2 Announce Type: replace-cross
Abstract: Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this,...
By Zhivar Sourati, Marianne Menglin Liu, Zheng Wang, Rongguang Wang, Sai Ashish Somayajula, Tao Sheng, Kyu J. Han, Sujith Ravi, Dan Roth
arXiv:2607. 27056v1 Announce Type: new Abstract: Personalized agents are increasingly applied to assist users across a wide range of tasks.
By Lingyang Zeng, Guangze Chen, Kaichen Yu, Zhicheng Pan, Siyang Weng, Zirui Hu, Xiangyun Du, Hailin He, Rong Zhang, Chengcheng Yang, Kai Huang, Xuan Zhou
Geographic Information System (GIS) professionals rely on multi-step spatial analysis workflows to support decision-making in urban planning, disaster response, and environmental monitoring. The process is tedious, time-consuming, and error-prone.
arXiv:2610.19560v1 Announce Type: cross
Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users....
By Rishitosh Kumar Singh, Amir Saeidi, Shamanthak Hegde, Shrinidhi Kumbhar, Chitta Baral
arXiv:2608.27831v1 Announce Type: cross
Abstract: Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues--long, structured, a...
By Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee