ATLAS-Bench is a new benchmark comprising 1,200 local search queries across three regions, designed to evaluate large language model agents on their ability to satisfy mandatory requirements, preferences, and factual versus subjective judgments using place attributes, reviews, and user profiles. Each query distinguishes constraints from preferences and includes evidence and criteria for verification, with validation through iterative assertion checks and cross-model review. The benchmark is paired with ATLASBox, a standardized evaluation environment that provides search tools and a fixed POI knowledge base, allowing agents to recommend evidence-supported options or report unmet requirements, and the study reports that GPT-6-Luna achieves the highest constraint satisfaction and factual accuracy, though GLM-5.3 returns acceptable recommendations for more queries. whyItMatters":"The benchmark addresses gaps in existing local search evaluations by rigorously testing agents’ ability to meet mandatory constraints and preferences, thereby advancing the development of reliable LLM agents for everyday decision-making and planning."
arXiv:2606. 12871v1 Announce Type: new Abstract: Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses.
arXiv:2607.10198v2 Announce Type: replace Abstract: Search APIs expose ranked snippets, URLs, and metadata on which agents decide whether to answer, search again, or fetch pages. We evaluate these in...
arXiv:2606. 17698v1 Announce Type: new Abstract: As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked.
arXiv:2603. 27476v2 Announce Type: replace Abstract: AI-powered people search platforms are increasingly used in recruiting, sales prospecting, and professional networking, yet no widely accepted benchmark exists for evaluating their performance.
arXiv:2606. 10156v1 Announce Type: cross Abstract: As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace.
The paper presents a method for fine‑tuning a large language model (LLM) recommender to generate personalized, non‑harmful explanations for its recommendations. By training two LLM‑judge reward models and using constrained GRPO, the authors achieve a significant increase in the PASS rate for all three criteria, from 0.649 to 0.956 on their own judges and from 0.677 to 0.931 on an independent judge. The fine‑tuned model maintains its original recommendation performance, demonstrating that LLM‑based recommenders can be adapted to complex tasks without loss of effectiveness.
arXiv:2606. 17453v1 Announce Type: new Abstract: Large language model agents are increasingly integrated into map services.
QREF is a plug‑and‑play system that improves DeepSearch agents by refining search queries to match the current reasoning intent and filtering evidence to keep only what is relevant. The Query Refiner rewrites queries for complementary coverage across each search batch, while the Evidence Filter removes distracting retrieval results. Experiments show that QREF boosts DeepSearch performance on four benchmarks, achieving significant Pass@1 gains and dramatically reducing ineffective query batches without changing the agents’ architectures or retraining their reasoning policies.
Clarify-Then-Search is a benchmark that tests whether large language models can ask clarification questions to improve the usefulness of deep search results. It uses 518 real-world query pairs from Baidu, where each intent query is paired with an underspecified version. The evaluation involves a clarifier asking up to three questions, a user answerer providing only explicit information, and a rewriter generating a new query that is then searched; performance is measured by a weighted nugget-recall score.
CARTBench is a new benchmark for evaluating large language model shopping assistants on complex, ambiguous requests that involve timelines. It compiles 1,603 timestamped dialogues from 17 retail domains and pairs them with product records, then uses case‑specific rubrics based on Ideal Response Principles to score responses across Core, Excellence, and Safety criteria. In a full‑suite evaluation of ten commercial APIs and several open‑weight models, only three models scored above 90% on Core and none surpassed 45% on Excellence, revealing a gap between required behavior and high‑quality purchase advice.
MOSS (Model Sanity Service) is a production framework that embeds LLM-based relevance scoring into a continuous monitoring loop for recommendation, search, and offer surfaces, using targeted sampling, strategy-aware prompting, and structured alerting to diagnose semantic failures. It has been deployed for over six months at a large U.S. retailer, identifying two confirmed serving defects and quantifying a 76.4% reduction in alert rate for a backfill blending fix. Benchmark results show that LLM judges outperform non-LLM baselines on both recommendation and search tasks, demonstrating that a single LLM monitoring framework can deliver actionable quality signals across multiple product discovery surfaces.
arXiv:2609.22223v1 Announce Type: cross Abstract: Long-form factuality verification is commonly implemented as a static decompose-search-verify pipeline, with separately prompted modules processing c...