arXiv AI
Aug 25

From Isolation to Alignment: Unified LoRA for Efficient Multi-Task Learning

The paper introduces Align‑LoRA, a unified LoRA framework for multi‑task learning that replaces complex, isolated adapter designs with a single‑adapter model enhanced by a higher rank and an explicit alignment loss. It demonstrates that a router‑free, multi‑head model with high inter‑head redundancy can outperform more elaborate baselines, and that a unified LoRA can achieve competitive performance while enabling weight merging and zero inference latency. Extensive experiments and theoretical analysis confirm that Align‑LoRA surpasses prevailing approaches, offering a simpler, production‑friendly paradigm for parameter‑efficient fine‑tuning of large language models.

By Jinda Liu, Yi Chang, Yuan Wu
arXiv Machine Learning
Oct 3

Learning to Predict Distributions over Weight Updates for Test-Time Adaptation

The paper introduces query‑conditioned hypernetworks that predict distributions over LoRA weight updates for large language models. By learning a distribution rather than a single point estimate, the method allows sampling multiple adapted models for the same query, improving performance over deterministic hypernetworks and token‑sampling baselines. The study also shows that these learned updates can transfer across different queries, indicating reusable adaptation patterns.

By Azal Ahmad Khan, Keshav Ramji, Tahira Naseem, Ali Anwar, Ram\'on Fernandez Astudillo