Local Prototype Reconstruction for Text-Compatible Speech-to-LLM Bridge Pretraining
Read the original on arXiv Computation and Language →The paper introduces Local Prototype Reconstruction (LPR), a lightweight regularizer that encourages bridge embeddings in speech‑to‑LLM systems to stay close to the frozen LLM’s input‑embedding neighbourhoods. By measuring lexical manifold compatibility with a fixed, head‑free, timestamp‑free diagnostic, the authors show that next‑word prediction and sentence‑level contrastive pretraining do not fully capture token‑level lexical compatibility. Experiments on multilingual ASR and speech translation demonstrate that LPR improves transfer, especially for translation and low‑resource adaptation, and that the diagnostic correlates with downstream gains across objectives.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.