MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction

MedBenchAgent is a multi‑agent framework that automates the entire process of constructing medical vision‑language model benchmarks, from deriving the benchmark specification to generating and auditing evaluation items. It uses a Benchmark Intermediate Representation (BIR) to encode task definitions, evidence mappings, evaluation protocols, and item specifications, separating planning from instantiation. The system achieves a Task‑Space F1 of 90.9% and demonstrates high audit pass rates, outperforming existing task induction methods and showing portability across specialized medical domains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• multimodal
• benchmarks