arXiv:2606. 17698v1 Announce Type: new Abstract: As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper's requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked.
arXiv:2607. 20536v1 Announce Type: new Abstract: Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.
arXiv:2606. 12871v1 Announce Type: new Abstract: Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses.
StartupBench is a new benchmark that evaluates general-purpose agents on end-to-end workflows derived from AI startup products that have proven market adoption. It translates real-world product workflows into deliverable-oriented tasks and assesses them with detailed rubrics. The study finds that even the best models complete only about 30% of these tasks, highlighting challenges such as complex instruction following and domain expertise.
RealWorldShop introduces a new benchmark for conversational shopping agents, featuring 3.28 million grounded products, structured shopping episodes, a profile‑grounded user simulator, and role‑play evaluation. Analysis reveals that existing systems generate locally plausible responses but struggle with state tracking, constraint updating, and grounded convergence, especially in ambiguous or multi‑intent scenarios. The authors propose REALSHOP_AGENT, a session‑control framework with explicit state management, shopping‑flow control, catalog‑grounded retrieval, and runtime guards, which consistently outperforms strong baselines on the benchmark.
StartupBench is a benchmark that evaluates general‑purpose agents on end‑to‑end workflows derived from real AI startup products that have proven market adoption. It translates these product workflows into deliverable‑oriented tasks and assesses them with detailed rubrics that capture complex requirements. Even the best current models complete only about 30% of the tasks, highlighting failures in instruction following and domain expertise.
arXiv:2606. 12924v1 Announce Type: new Abstract: We present a modular two-agent simulation framework for evaluating conversational shopping assistant architectures.
arXiv:2607. 03162v1 Announce Type: new Abstract: LLM-powered agents struggle with personalization when users issue raw, underspecified queries.
The paper introduces KNOWS, a benchmark for evaluating web agents that act as assistants by retrieving, synthesizing, and presenting information across complex, multi-step browser tasks. It outlines a task design rubric, evaluation protocol combining deterministic checks with LLM judgments, and reports that current agents achieve only modest success, with the best performing agent succeeding on less than 3% of tasks. The study highlights significant gaps in agents’ tool use, visual understanding, and long‑horizon reasoning.
arXiv:2607. 29002v1 Announce Type: new Abstract: Online shoppers increasingly turn to AI shopping assistants, using images and multi-turn dialogue to express and refine product needs that are difficult to articulate in text alone.
arXiv:2608. 14068v1 Announce Type: cross Abstract: Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims.
RPTune is an end‑to‑end framework that improves in‑context catalog search for small merchant businesses by learning to curate product catalogs and fine‑tuning large language models (LLMs) with catalog‑grounded supervision. It uses an encoder‑reorganizer curator to order and prune products based on LLM feedback, and then applies context‑relative rewards during LLM post‑training. Across seven real merchants and 100 complex conversational queries per merchant, RPTune boosts search accuracy by up to 31.4 percentage points from curation alone and an additional 10.3 points on average from post‑training.
arXiv:2606. 06462v1 Announce Type: new Abstract: Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance.