← Back to all news
arXiv Machine Learning By Simon Dennis, Kevin Shabahang, Hao Guo, Rivaan Patil

Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures

Read the original on arXiv Machine Learning →

arXiv:2607. 21612v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning methods like LoRA have become the default for adapting large language models, succeeding across instruction following, style transfer, and factual adaptation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 15

LoRA Fine-Tuned Models for Control Systems Course Q\&A: A Multidimensional Evaluation of Model Scale and Rank Effects

The paper presents a study on fine‑tuning open‑source instruction‑tuned language models for a Linear Control Systems course using LoRA. A dataset of 360 system‑user‑assistant conversations was created, and LoRA was applied to Qwen2.5‑3B‑Instruct and Qwen2.5‑7B‑Instruct with ranks r=4, 8, and 16. Evaluation with ROUGE, BERTScore, and structured‑output metrics showed that LoRA improved similarity and stability, with the 7B‑r16 model achieving the highest scores and r=8 offering a good trade‑off between performance and parameter efficiency.

By Shaowen Lu, Chengxu Liu, Ping Zhou, Tao Yang
llmsfine-tuning
More like this →
arXiv Machine Learning
5d ago

New LoRA Skills Should Read but Never Write

The paper introduces READ, a method for composing low‑rank adapters (LoRA) in large language models. By rewriting each adapter into a balanced canonical form and enforcing a one‑directional coupling, READ allows new skills to read but never write into the output subspaces of existing skills, eliminating interference. Experiments on four benchmark suites and two model families show that READ consistently outperforms existing baselines, improving SuperGLUE scores by over twenty points and domain suite scores by more than seven points.

By Zeyan Li, Panqi Yang, Qirong Guo, Shengda Zhuo, SIyuan Qiu, Hu Xu, Chun Li, Jianfeng Xu
llmsfine-tuningbenchmarks
More like this →
arXiv AI
Jun 6

LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents

arXiv:2606. 06087v1 Announce Type: cross Abstract: Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext.

By Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo, Rong Shan, Zhihui Fu, Jun Wang, Weiwen Liu, Yong Yu, Weinan Zhang, Jianghao Lin
llmsagentsfine-tuning
More like this →