GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space

GPAgentBench-2K is a new benchmark that evaluates large language model agents in primary‑care clinical decision‑making using a constrained Markov Decision Process (CMDP). It is built from expert‑validated records of real GP encounters and includes six foundational clinical actions, a workflow prior, and safety‑informed abstention as an outcome. Testing 16 state‑of‑the‑art LLMs shows that performance drops as the action space grows and that even the best models violate safety constraints in more than half of high‑risk cases, highlighting a gap between clinical quality and safety.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• agents
• reinforcement-learning
• benchmarks
• safety