arXiv AI

OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents

OpenGray is a new benchmark that evaluates radiotherapy planning agents on their decision-making rather than just dose outcomes. It extends the OpenKBP-Opt head-and-neck dataset with tasks that have known correct decisions, including contradictions with dose goals and a feasible control plan. Four systems—ranging from a rule-based script to full LLM agents—were tested on 30 patients, revealing that high decision accuracy can mask specific failures such as missed hard goals and inadequate stopping actions.

arXiv AI
Oct 7

RadOnc-Agent: An LLM-Orchestrated Framework for AI Workflows Across the Radiotherapy Care Pathway

RadOnc-Agent is an AI framework that organizes radiotherapy into four clinical phases and offers 26 callable functions via a conversational interface. A large‑language‑model controller maps clinical intent to schema‑constrained calls, maintains patient and workflow context, and routes requests to specialist services. In evaluations, the system achieved high accuracy in single‑function calls (98.79%) and cross‑stage workflow completions (96.50% scripted, 96.67% real‑patient), demonstrating technical feasibility of LLM‑orchestrated coordination across heterogeneous radiotherapy capabilities.

By Caiwen Jiang, Shuoyang Wei, Songlin Zhao, Junyu Li, Jingyuan Chen, Wei Liu
arXiv AI
Oct 23

An Evidence-Driven Multi-Agent Compound AI System for Head and Neck Proton Radiotherapy Planning

The paper presents an evidence-driven, multi-agent AI system for head and neck proton radiotherapy planning. It combines a DRL-based planning agent that generates multiple treatment plans with an LLM-based conversational agent that refines the chosen plan through natural language interactions. The system, trained on 21 patients, improves instruction-following and domain-specific reasoning, yielding more clinically appropriate trade-offs in patient-specific plan refinement.

By Qingqing Wang, Liqiang Xiao, Chang Chang
arXiv AI
Jul 28

Comparing Optimization Models for Radiotherapy Scheduling

arXiv:2607. 22539v1 Announce Type: cross Abstract: The Radiotherapy Scheduling Problem (RTSP) involves determining an optimal schedule for patients undergoing radiation treatments, a task that has a massive impact on clinical outcomes given the central role of radiotherapy in cancer care.

By C. C. Rambaldi Migliore, D. Stanicel, N. Musliu, G. Iacca, M. Roveri
arXiv AI
Aug 28

AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

AgentJudgeBench is a new benchmark that evaluates the reliability of large language model (LLM) judges on agentic tool‑calling tasks involving workflow directed acyclic graphs (DAGs). It contains 3,808 instances across six DAG topologies and three difficulty tiers, tested with five generators (3B–70B open‑weight models and GPT‑5.4) and six judges (20B to frontier scale) under both paired‑with‑and‑without‑ground‑truth conditions. The study finds that judge alignment degrades with task difficulty, ground‑truth exposure can sometimes hurt alignment, and structured evaluation rubrics provide modest improvements, revealing a structural ceiling that model capacity alone cannot surpass.

By Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian, Sai Harshitha Aluru
arXiv AI
Oct 27

RadOnc-Agent: An Integrated Framework for AI Workflows Across the Radiotherapy Care Pathway

RadOnc-Agent is an integrated AI framework that links 17 phase‑specific functions and 9 cross‑cutting utilities across four radiotherapy care phases via a conversational interface. A language‑model controller translates clinical intent into structured calls, preserves longitudinal context, and validates dispatch. In evaluations on 2,600 single‑function requests, 200 cross‑stage scenarios, and 120 patient workflows, the system achieved 98.79% accuracy on single‑function calls and over 96% completion on scripted and real‑patient workflows, demonstrating the feasibility of longitudinal integration in radiotherapy workflows.

By Caiwen Jiang, Shuoyang Wei, Songlin Zhao, Junyu Li, Jingyuan Chen, Yuzhen Ding, Daniel J. Ma, Samir H. Patel, Steven E. Schild, Sujay A. Vora, Wei Liu
arXiv AI
Sep 15

Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs

The paper introduces a new evaluation method called "same-input rerun" to assess the consistency of clinical language‑model agents across repeated runs. By replaying 1,000 MedAgentBench tasks with identical inputs, the authors find that action‑level outputs—such as test orders, medication requests, and referrals—vary significantly, even when benchmark scores remain unchanged. The study demonstrates that current benchmarks, which typically evaluate only a single run per task, can miss substantial behavioral divergence.

By Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han, Wenbin Zhang
arXiv AI
Oct 2

AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks

AstroAgentBench is a seven-family benchmark designed to evaluate agentic planning for space mission tasks such as scheduling, observation planning, constellation design, and relay support. The benchmark requires agents to submit planning artifacts that are verified for schema, timing, geometry, resources, and mission value, with results reported as validity and normalized scores compared to task-specific solver references. Across five LLM agent systems and 35 held-out cases, the strongest systems approach or exceed solver-reference scores on several families, while weaker systems often fail to produce high-value valid plans, and even strong systems lose quality on geometric, product-level, or design-heavy tasks. Failure analyses identify task-contract misformulation and weak solution construction as key issues, and successful runs demonstrate that agent calibration against verifier feedback and adaptive search can improve performance; ablation studies show that procedure injection and memory accumulation help when they provide the missing formulation, calibration, or search support.

By Weiyi Wang, Xinchi Chen, Jingjing Gong, Xuanjing Huang, Xipeng Qiu
arXiv AI
Sep 28

Demo: Generative AI helps Radiotherapy Planning with User Preference

The paper presents a generative AI model that predicts 3‑D radiation dose distributions based solely on user‑defined preference settings, allowing planners to balance trade‑offs between organs‑at‑risk and target volumes. This approach offers greater flexibility and personalization compared to traditional deep‑learning models that rely on institutional reference plans. The authors report that their method can outperform the Varian RapidPlan model in adaptability and plan quality in certain scenarios, and it is designed for easy integration with clinical treatment planning systems.

By Riqiang Gao, Simon Arberet, Martin Kraus, Han Liu, Wilko FAR Verbakel, Dorin Comaniciu, Florin-Cristian Ghesu, Ali Kamen
arXiv AI
Oct 16

Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

The paper introduces a benchmark of 400 synthetic healthcare workflow episodes to evaluate large language model (LLM) agents’ reliability in escalating care when evidence is missing or conflicting. It finds that while clean decision accuracy is high (0.97), models miss a large proportion of necessary escalations (32–97%) under uncertainty, and even the largest 9.7B model’s performance drops to 0.78 in conflict scenarios. A workflow-aware controller improves utility and reduces missed escalations, but its effectiveness is limited by lexical generalization failures.

By Zihao Zheng (Washington University in St. Louis), Junyi Yao (Washington University in St. Louis), Baichuan Li (Southern Methodist University), Jiayu Long (Washington University in St. Louis)