The paper proposes a framework to evaluate the reliability of large language model (LLM) agents when they generate stakeholder preferences for multi‑criteria decision‑making in healthcare operations. It examines both explicit preferences—agents rating the importance of patient waiting time, provider workload, and operating cost in an outpatient scheduling scenario—and implicit preferences—agents’ pairwise choices between emergency department configurations. Experiments show that explicit preferences vary by role and operating conditions, but deliberation can homogenize them, while implicit preferences are heavily influenced by presentation order and only modestly improved by contextual models. Robust weighted‑sum optimization models that incorporate preference uncertainty often yield different schedules than those based on average weights, underscoring the need for auditing LLM‑generated preferences before using them in healthcare decision support.
arXiv:2607. 05055v1 Announce Type: new Abstract: Healthcare appointment scheduling remains a persistent operational bottleneck, driven by manual coordination, fragmented legacy systems, and high administrative overhead.
arXiv:2610.23062v1 Announce Type: new Abstract: Multi-objective optimization models are often deployed in settings where decision makers (DMs) must balance some conflicting criteria. In practice, how...
The paper proposes an empirical pipeline to estimate the preferences that a large language model (LLM) implicitly optimizes by combining the model’s probability distribution over unknowns with its chosen action, and fitting a discrete choice model to recover the underlying cost function. This revealed-preference framework enables rigorous assessment of whether LLMs act consistently toward a goal, can articulate objectives that align with their decision policy, and can be steered by prompting to follow a user-specified cost function. Experiments across four medical diagnosis domains and various frontier and open-source models show that while many LLMs exhibit internal coherence, they still struggle to accurately report or adopt preferences when guided by users.
arXiv:2507.21790v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are becoming widely used to support various workflows across different disciplines, yet their potential in discr...
The paper introduces the Preference-Behavior Model (PBM), which jointly infers Large Language Model (LLM) preferences and imperfect decision-making mechanisms from observed choices. By applying PBM to three social dilemma games—Public Goods, Collective-Risk, and Common-Pool Resource—the authors show that preferences derived from one dilemma can predict behavior in others, whereas traditional behavioral economic elicitation instruments fail to do so consistently. The study also evaluates how well inferred preferences align with persona instructions, revealing both broad agreement and specific divergences across dimensions, LLMs, and games.
arXiv:2605.25200v3 Announce Type: replace Abstract: Travel planning in the real world is overwhelmingly a \textit{group} activity, yet existing LLM travel-planning benchmarks reduce it to a single us...
arXiv:2608. 02617v1 Announce Type: cross Abstract: We evaluate whether clinician pairwise preferences provide a reliable signal of clinical safety in large language model (LLM) evaluation using expert feedback from MOOVE (Massive Open Online Validation and Evaluation), a clinician-led platform collecting blinded pairwise preferences alongside multi-criterion rubric ratings.
arXiv:2610.02270v1 Announce Type: cross Abstract: Medical vision-language model (VLM) evaluation is sensitive to workflow design, prompting strategy, and benchmark construction, yet most studies trea...
arXiv:2602. 09802v2 Announce Type: replace Abstract: As Large Language Models (LLMs) are increasingly deployed in applications such as travel assistance and purchasing support, they are often required to make subjective choices on behalf of users in settings where no objectively correct answer exists.
arXiv:2606. 22974v2 Announce Type: replace Abstract: Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure.
The paper introduces a test-time aggregation framework for multi-agent reasoning with large language models. It mitigates non-semantic biases by normalizing reasoning traces and injects bias to create decision instability, which signals reliability. The method improves accuracy and token efficiency on several mathematical reasoning benchmarks without requiring validation data or prior model strength knowledge.
arXiv:2512. 01241v4 Announce Type: replace-cross Abstract: Large language models (LLMs) and medical AI tools are routinely used by physicians and patients for medical advice, yet their clinical safety profiles remain poorly characterized.