arXiv Computation and Language
Sep 23

Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation

The paper introduces modality‑gated deep adapters, a parameter‑efficient method for adding new modalities to a frozen multimodal embedding language model without altering its existing outputs. These adapters are bottleneck modules attached to each decoder layer, grouped into modality‑specific packs that activate only during encoding of their own modality, ensuring exact preservation of the base model’s computation graph. Experiments on a 2B base model show significant gains in audio‑to‑text and thermal‑to‑text retrieval metrics, and the authors release the audio and thermal packs along with training and evaluation code.

By Abdul Basit Tonmoy, Kazi Fardinul Hoque, Md. Shahrier Islam Arham, Arman Luthra