Trick-or-Treat: Adversarial Evaluation of Multimodal Medical Reasoning with Synthetic Images

Trick-or-Treat is an adversarial multimodal medical benchmark that pairs real chest radiographs with synthetic clinical photographs to test cross‑modal reasoning. Each condition has matched pairs where the radiograph and photograph suggest the same disease, and mismatched pairs where the photograph shows an alternative finding. In tests of three frontier models, hallucinations of clinically significant findings rose from 25‑37.5% for single photographs to 45‑57.5% when mismatched pairs were presented, with models often ignoring mismatches and generating findings that better aligned with the radiograph.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• multimodal
• benchmarks
• safety