Solving the Needle-in-a-Haystack Problem in Mammography Vision-Language Model with Differentiable Subset Sampling

The paper introduces TopKSigLIP, a vision‑language model tailored for mammography that tackles two key challenges: high‑resolution imaging and homogeneous radiology reports. It replaces standard CLIP training with a TopK‑Patch module that selects sparse high‑resolution patches likely to contain lesions, and a Sup‑sigmoid loss that uses soft labels from structured data instead of contrastive loss. TopKSigLIP outperforms existing open‑source mammography and general medical VLMs on zero‑shot tasks such as density assessment, BI‑RADS classification, finding subtyping, and cancer prediction, while also providing better lesion localization than Grad‑CAM.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• llms
• multimodal
• benchmarks