arXiv:2608. 20202v1 Announce Type: new Abstract: Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions.
By Mengru Wang, Haozhe Luo, Zhenqian Xu, Zhixiang Cui, Haoming Xu, Qu Yang, Jizhan Fang, Junfeng Fang, Ningyu Zhang
Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved, while largely overlooking how retrieved memories reshape model reasoning and affect performance on the current task.
The paper introduces Ladder Side Tuning (LST), a parameter‑efficient fine‑tuning method that adds a lightweight side network to large language models. LST matches QLoRA’s compute scaling while halving peak memory usage, enabling 7B‑parameter models to be fine‑tuned on a single 12 GB GPU with 2k‑token contexts without gradient checkpointing. The authors also present xLadder, a depth‑extended variant that increases effective depth through cross‑connections, allowing deeper reasoning without extra memory overhead.
By Estelle Zheng, Nathan Cerisara, S\'ebastien Warichet, Emmanuel Helbert, Christophe Cerisara
arXiv:2606. 06574v1 Announce Type: new Abstract: Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers.
By Ziyue Li, Yang Li, Tianyi Zhou
arXiv:2506. 21833v2 Announce Type: replace Abstract: Forward-mode automatic differentiation (FmAD) and zero-order (ZO) optimization are increasingly proposed as memory-efficient, backpropagation-free alternatives for large language model (LLM) fine-tuning, yet their benefits are typically evaluated only against standard backpropagation (BP), omitting memory-efficient variants such as activation checkpointing.
By Kunjal Panchal, Sunav Choudhary, Yuriy Brun, Hui Guan
RetroThinker is a multi-stage post‑training framework that enhances SpeechLLMs by enabling them to self‑verify and forward‑correct Chain‑of‑Thought reasoning steps during inference. It combines supervised fine‑tuning on curated retrospective thinking data with length‑based direct preference optimization to improve reasoning while the user speaks. On the GSM8K benchmark, RetroThinker achieves an 11% absolute accuracy gain over non‑retrospective baselines while maintaining comparable latency.
By Yi-Jen Shih, Puyuan Peng, Abdelrahman Mohamed, David Harwath