arXiv AI
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
4d ago
MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichment
arXiv:2609.37574v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used to enrich user queries in information retrieval (IR) so that a standard retriever such as BM25 can...
arXiv AI
Aug 19
DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval
The paper introduces DEPT, a method that trains a single decoder-only large language model to both expand queries and encode documents for retrieval. By preserving document embeddings close to their initial cached values while allowing gradients to flow through the generator, DEPT stabilizes retrieval targets and enables efficient index reuse and online hard‑negative mining. Experiments on the BEIR benchmark with Qwen3‑4B‑Instruct‑2507 and LLaMA‑3.2‑3B‑Instruct show that DEPT outperforms training‑free, independently trained, and staged unified baselines, with ablations confirming the benefits of preservation, whitening, end‑to‑end expansion training, and online negatives.
arXiv Machine Learning
Aug 28
Modular Expert Merging for Biomedical Retrieval
The paper proposes a method called Modular Expert Merging for Biomedical Retrieval, which combines independently trained domain‑specialized experts instead of large mixed‑domain training. Experiments across four decoder‑only LLM families (0.6B‑7B) and twelve retrieval tasks from MTEB show that merging experts consistently outperforms mixed‑domain training. The authors also introduce a Synthesize‑Train‑Merge (STM) framework that generates hard negatives with a top‑tier LLM, fine‑tunes experts via LoRA, and merges them, achieving strong biomedical retrieval performance while retaining competitive general‑domain results.
arXiv Computation and Language
3d ago
DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
arXiv:2603.26164v2 Announce Type: replace-cross Abstract: Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters...
arXiv AI
Aug 6
Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains
arXiv:2608. 05138v1 Announce Type: cross Abstract: Modern Greek is absent from NVIDIA's Nemotron retrieval models and from major multilingual retrieval benchmarks, despite being important for retrieval-augmented generation (RAG) in legal, energy, financial, and medical applications.
arXiv AI
Jun 16
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
arXiv:2606. 16456v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.