The paper introduces the concept of epistemic warrant to assess when users should rely on individual recommendations from large language models (LLMs). It defines a four-tier reliance certificate—unstable, context-dependent, locally supported, and broadly supported—based on the stability and scope of a model’s preference. Experiments show that these warrants align with expert judgments and crowd consensus, and they provide distinct information from confidence scores or decision difficulty.
By Shai Vardi, Jo\~ao Sedoc
The paper introduces epistemic warrant, a decision-level construct that assesses the stability and scope of a large language model’s preference for individual recommendations. It operationalizes this concept with a four-tier reliance certificate—unstable, context-dependent, locally supported, and broadly supported—applied to pairwise recommendations. Validation shows that stronger warrants align with expert and crowd consensus, provide distinct information from verbalized confidence, and are not merely a reflection of decision difficulty.
The paper introduces PROVE-REC, a two‑pass framework that generates verifiable preference proofs for large language model (LLM) recommendation systems. In Pass A, the model condenses user interaction histories into a compact proof of positive and avoidance claims linked to specific evidence entries. Pass B then uses only this proof and its evidence to predict the next item, ensuring the recommendation follows the reasoning path. Verification steps compare masked evidence and removed claims to confirm grounding and influence, while a ranking‑preservation objective retains useful historical information. Experiments on diverse real‑world datasets show PROVE‑REC outperforms strong baselines by up to 7.45%, producing claims that are both better grounded and more influential to recommendation quality.
By Yu Hou, Nathaniel Kang, Pengkai Wang, Hua Li
CARTBench is a new benchmark for evaluating large language model shopping assistants on complex, ambiguous requests that involve timelines. It compiles 1,603 timestamped dialogues from 17 retail domains and pairs them with product records, then uses case‑specific rubrics based on Ideal Response Principles to score responses across Core, Excellence, and Safety criteria. In a full‑suite evaluation of ten commercial APIs and several open‑weight models, only three models scored above 90% on Core and none surpassed 45% on Excellence, revealing a gap between required behavior and high‑quality purchase advice.
By Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen
RPCBench is a new benchmark designed to evaluate large language models’ ability to critique recommendation requests by detecting, diagnosing, and handling flawed premises. It includes evidence‑grounded test instances across five recommendation domains and ten types of premise failures, and introduces a fine‑grained evaluation framework covering detection, error localization, handling strategy, and evidence faithfulness. Experiments with 11 LLMs reveal that proactive detection is the main bottleneck, with models struggling most on underspecified‑premise errors and showing that optimal critique quality occurs at intermediate reasoning lengths.
By Zhongru Chen, Yuan Wu, Yi Chang
arXiv:2608.21243v1 Announce Type: cross
Abstract: Sequential recommendation predicts the next item from a user's interaction history, but not every interaction is equally informative. Real logs combi...
By Zichun Jin, Zihan Zhou, Yinan Liu, Bin Wang, Xiaochun Yang
arXiv:2607. 18240v1 Announce Type: new Abstract: Large language models (LLMs) can achieve strong fact-checking accuracy, yet forced binary decisions conceal a critical reliability problem: systems may issue confident verdicts even when supporting evidence is weak, sparse, or internally inconsistent.
By Dekun Yang
ATLAS-Bench is a new benchmark comprising 1,200 local search queries across three regions, designed to evaluate large language model agents on their ability to satisfy mandatory requirements, preferences, and factual versus subjective judgments using place attributes, reviews, and user profiles. Each query distinguishes constraints from preferences and includes evidence and criteria for verification, with validation through iterative assertion checks and cross-model review. The benchmark is paired with ATLASBox, a standardized evaluation environment that provides search tools and a fixed POI knowledge base, allowing agents to recommend evidence-supported options or report unmet requirements, and the study reports that GPT-6-Luna achieves the highest constraint satisfaction and factual accuracy, though GLM-5.3 returns acceptable recommendations for more queries.
whyItMatters":"The benchmark addresses gaps in existing local search evaluations by rigorously testing agents’ ability to meet mandatory constraints and preferences, thereby advancing the development of reliable LLM agents for everyday decision-making and planning."
By Keyun Zhang, Dapeng Yin, Fanfu Wang, Shuang Lang, Zhenyu Zhu, Yulin Huang, Chong Yang, Pengjie Wang
GraphEcho is a benchmark that examines how large language model agents navigate graph paths and handle evidence redundancy. It tests whether agents treat repeated encounters as additional corroboration by varying path counts and evidential origins while keeping evidence content constant. The study finds that redundant paths increase repeated walks, and that provenance-aware post‑training can reduce revisits but may limit source diversity, revealing a gap between efficient exploration and effective evidence use.
By Sikun Wang, Yixi Zhou, Lei Fan, Fan Zhang
arXiv:2607. 20950v1 Announce Type: new Abstract: BoN improves model outputs by sampling several candidates and selecting one with a proxy score, but it assumes that complete candidates can be evaluated reliably.
By Cenwei Zhang, Teng Fang, Yuxia Wang, Derek Li, Bryan Dai, Lei You
arXiv:2606. 11349v1 Announce Type: new Abstract: In hierarchical reasoning, failures often originate at intermediate decision points where the agent commits to a wrong branch without recognizing that it lacks critical information.
By Aijing Gao, Yiming Kang, Mengdie Flora Wang, Jae Oh Woo
arXiv:2607. 22574v1 Announce Type: new Abstract: Evidence-based clinical decision making requires specialists to identify, evaluate and synthesize relevant scientific literature.
By Adela Bara, Simona-Vasilica Oprea