The paper introduces COUNT, a method that builds n‑gram embeddings without adding learned parameters by storing precomputed n‑gram counts on host memory. By weighting the language model’s own unembeddings with corpus frequencies, COUNT can be added as an extra input embedding during pretraining, improving validation loss equivalent to a 57% increase in compute for a 1B‑parameter model. Additionally, COUNT enables efficient GPU memory usage and allows domain‑specific n‑gram tables to be swapped in at inference time for training‑free adaptation.
By Ian Magnusson, Noah A. Smith, Pang Wei Koh
ULME is a post‑training framework that unifies text embedding and language modeling within a single causal decoder‑only LLM by inserting learnable slots for semantic and reasoning embeddings. It trains three objectives—contrastive loss for semantic alignment, language‑modeling loss for answer generation, and a stop‑gradient predictive loss linking reasoning to the answer—so that generation actively shapes the embedding. Across Qwen2.5 and Qwen3 models (0.5B–8B), ULME improves MTEB scores by 3.3–4.1 points, tightens BRIGHT and AdvBench‑IR, and matches generation‑only fine‑tunes on standard accuracy benchmarks.
By Yang Liu, Zhiqiang Yuan, Shu-Dong Huang, Jiancheng Lv, Liwei Yang, Qiwei Yan, Kailin Lv, Jinchao Zhang
ULME introduces a post‑training framework that unifies text embedding and generation within a single causal decoder‑only LLM by inserting learnable slots for semantic and reasoning embeddings. It trains three objectives—contrastive, language‑modeling, and stop‑gradient predictive—so that generation actively shapes the embedding space. Across Qwen2.5 and Qwen3 models (0.5B–8B), ULME improves MTEB scores by 3.3–4.1 points, tightens BRIGHT and AdvBench‑IR, and matches generation‑only fine‑tunes on standard accuracy benchmarks.
arXiv:2608.17050v3 Announce Type: replace-cross
Abstract: Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access...
By Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji
arXiv:2610.15715v1 Announce Type: new
Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
By Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Pi\k{e}kos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yanning Dai, Yuhui Wang, J\"{u}rgen Schmidhuber
arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.
By Hippolyte Pilchen, Edouard Grave, Patrick P\'erez
The paper investigates how Engram-style hashed memory can be transferred between different language model backbones. By freezing a memory table trained on a source model and attaching it to a target model with only a lightweight reader, the authors find that both the memory content and correct addressing are important, but the reader must be aligned to the target to make the memory useful. In question‑answering experiments, a dual‑layer, four‑branch reader nearly matches same‑model performance, and when the reader interface is directly compatible, the frozen memory alone provides substantial benefit, with optional reader adaptation offering further gains.
By Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji
Lngram v2 introduces a latent N‑gram memory system that decouples memory routes, memory dimension, and backbone width, enabling scalable memory capacity for transformers. It employs context‑aware grouped‑query attention, a zero‑value sink, and counterfactual surrogate gradients to improve readout selectivity and routing trainability while preserving hard discrete addressing. Experiments on vision‑language models up to 30B parameters show consistent performance gains, reduced memory parameters, and stable semantic structure in the discrete IDs.
By Yunao Zheng, Bin Wen, Xiaojie Wang
External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence. However, existing memory paradigms represent each memory item in raw text and image forms, so retrieval-based systems must pass the retrieved text or images to the generation LLMs/VLMs, resulting in high token consumption and storage pressure, making it unaffordable for resource-constrained applications.
arXiv:2606. 10572v1 Announce Type: new Abstract: External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence.
By Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee
V-LUMEN introduces a visual lookup memory for vision‑language models by discretizing visual features with a frozen codebook, creating spatially aware memory keys via hashing, and modulating them with text‑conditioned relevance signals. The retrieved embeddings are fused into a frozen Qwen3‑VL‑2B backbone through a gated residual pathway, yielding substantial improvements on five reasoning‑intensive benchmarks and zero‑shot transfer to three disjoint image datasets. At a matched parameter budget, reallocating capacity from LoRA or MoE adapters to V‑LUMEN preserves accuracy on two backbones, demonstrating that external visual memory offers an independent capacity axis.
whyItMatters":"The paper shows that adding a non‑parametric visual lookup memory can significantly boost vision‑language model performance without increasing per‑token compute, offering a new way to scale such models."
By Miso Choi, Daekeun Kim, Eunji Kim, Jungbeom Lee
arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.
By Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Yike Guo, Sirui Han