← Back to all news
arXiv AI By Bingqing Liu, Wei Liu, Jun Wang, Xiaobo Sun, Yuhua Li, Ruixuan Li

Editing Large Language Models with Geometry-Aware Regularization

Read the original on arXiv AI →

The paper introduces GEdit, a geometry-aware framework for editing large language models that leverages a Low-Rank Plus Diagonal factor model to capture the low-rank structure of Transformer FFN hidden representations. GEdit applies low‑rank regularization to protect core knowledge and adaptive regularization to adjust constraints based on the difficulty of each edit, allowing harder edits to succeed while preserving easier ones. Experiments on GPT2‑XL, GPT‑J, and LLaMA‑3‑8B with up to 10,000 continuous updates demonstrate that GEdit outperforms state‑of‑the‑art baselines in both editing success and knowledge preservation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 30

Generalizable Lifelong Model Editing via Preference Optimization

The paper introduces GLIME, a lifelong model editing framework that integrates knowledge editing with preference optimization to handle continual updates in large language models. GLIME employs replay-based editing and a gradient constraint to prevent overfitting to target prompts and preserve previously edited knowledge. Experiments demonstrate that GLIME enhances knowledge generalization while maintaining editing performance and overall model capabilities.

By Dahyun Jung, Suhyune Son, Heuiseok Lim
llms
More like this →
arXiv AI
Sep 25

ALOE: Semantically Addressed Low-Rank Operators for Knowledge Editing

The paper introduces ALOE, a method for knowledge editing that learns semantic addresses from paraphrases and hard negatives, aligns them with autoregressive hidden states, and embeds a gated low‑rank operator within a single MLP layer. This design allows the edited model to run in one forward pass without external retrievers or routers. Experiments on CounterFact, ZSRE, and KnowEdit show high efficacy (0.955–0.999) and locality (0.981–1.000) across 7–8B model families, with analyses indicating effective separation of edits and suppression of out‑of‑scope activation.

By Zeyan Li, Hu Xu, Jianfeng Xu
More like this →