Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation

The paper presents a two‑stage multimodal framework for chest X‑ray interpretation that incorporates radiologist gaze data from the MIMIC‑Eye dataset. Stage 1 introduces a gaze‑token classifier that fuses image patches, bounding‑box masks, transcription embeddings, and fixation maps, and a curriculum‑scheduled loss that improves accuracy and spatial alignment, yielding a 4.4% AUC gain and 13.3% F1 improvement. Stage 2 translates classifier predictions into region‑specific diagnostic sentences using confidence‑weighted keywords, an expert dictionary, and a prompted large language model, boosting clinical‑term BERTScore and ROUGE over keyword baselines.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• rag
• multimodal
• benchmarks
• safety