← Back to all news
arXiv AI By Alicem Koyun

Does Your Trial Simulator Know What the Drug Does? An Intervention-Blind Test of Interventional Validity on 7,207 Registered Trials

Read the original on arXiv AI →

The study evaluates the interventional validity of clinical trial simulators by creating an intervention‑blind reference model that withholds intervention identifiers. Using 9,996 within‑trial contrasts from 7,207 phase 2/3 trials, the authors measure how much a simulator’s accuracy improves when it knows the intervention. Results show that a registry‑trained gradient‑boosting model performs poorly on interventional skill, while a large language model reading only the protocol achieves positive interventional skill, and perturbation tests confirm that the model relies on the intervention information.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 1

The Intervention Gap in Latent World Models

The paper introduces the concept of intervention fidelity in latent world models, measuring whether a model’s open‑loop transitions align with actual environment interventions. Experiments on TD‑MPC2, Cheetah, and DreamerV3 show that high reward fit does not guarantee fidelity, and that self‑supervised models can outperform task‑anchored ones in preserving intervention effects. The authors propose a capture‑gated audit to localize failures and argue that fidelity must be directly audited on the model’s native interface.

By Donna Vakalis
robotics
More like this →
arXiv AI
Oct 16

Learning When to Withdraw: Curriculum Coverage and Rule-Specific Failures in Fine-Tuned Language Agents

The paper investigates how fine‑tuned language agents can selectively withdraw actions when evidence changes or authority is revoked. Using a synthetic NAQD‑Env environment, the authors compare an original curriculum with an enriched one across multiple system‑prompt framings and evaluate four LoRA checkpoints of Qwen2.5‑3B‑Instruct against instruction‑tuned baselines. Results show that fine‑tuned checkpoints achieve higher decision agreement and withdrawal recall, reduce incorrect withdrawals, and narrow prohibited combinations, though they also omit valid event reports and exhibit mixed preservation of authorized actions.

By Mohamed Abouzahra
agentsfine-tuningefficiencysafety
More like this →
arXiv AI
6d ago

Training language models in simulation for psychiatric intake

The study demonstrates that multi‑turn reinforcement learning (RL) in simulation environments significantly improves a language model’s performance on structured, text‑based psychiatric intake tasks. In a randomized, double‑blind crossover trial involving 60 matched scenario pairs, the RL‑trained agent outperformed its base model in 76.7% of cases, showing higher scores on clinical examination, patient‑centered communication, and differential diagnosis appropriateness. Across 127 validated simulated scenarios, the trained agent nearly doubled the base model’s clinical rubric score and reduced under‑triage of acute safety risks by more than half, indicating that simulation provides a controlled testbed for training and stress‑testing psychiatric conversational agents.

By Samuel Schmidgall, Tim Strother, Alex Bijamov, Min Woo Sun, Anil Palepu, Akshay Goel, Ulysse Marteau-Ferey, Th\'eo Moutakanni, Yubin Kim, Joseph Breda, Christina Chen, Justin Chen, Paul Jhun, Andrew Marshall, Sarah Montgomery-Taylor, Eun Kyoung Hong, Shashank Ravi, Kameron C. Black, David J. H. Wu, Austin Schoeffler, John Lyon Havlik, Elizabeth C. Stade, Jared Moore, Johannes C. Eichstaedt, Logan Pierce, Jeff Jopling, Johannes Thrul, Thomas H. McCoy, Neguine Rezaii, Mark S. Chen, Ellen E. Fitzsimmons-Craft, Shirley Wang, Desmond C. Ong, Katharina Schultebraucks, Torrey Creed, Bhavna Daryani, Aleksandra Faust, S. Sara Mahdavi, Caroline Figueroa, Eric Lin, Kate H. Bentley, Ernest Lim, Zhihan Jiang, Xuhai "Orson" Xu, Daniel McDuff, John Torous, Yun Liu, Po-Hsuan Cameron Chen, Mike Schaekermann, Joelle Barral, Alvin Rajkomar, Tao Tu, Lin Yang, Isaac Galatzer-Levy, David Racz, Valentin Li\'evin
llmsagentsreinforcement-learningsafety
More like this →