The paper introduces FOVEATED, a plug‑and‑play framework that improves atomic‑fact recall in unstructured knowledge editing (UKE) for large language models. By randomly shifting Rotary Position Embedding (RoPE) positions during editing, FOVEATED creates focused views of each sentence, counteracting the context‑reliance problem where edited LLMs reproduce passages but fail to recall individual facts. Experiments show consistent gains across five editors, two LLM backbones, and three benchmarks.
By Ding Wu, Ye Zhang, Haoyu Wang, Tianci Liu
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
The paper investigates whether adding Abstract Meaning Representation (AMR) data to large language models (LLMs) improves performance on downstream tasks. By reproducing recent studies and applying a consistent hyperparameter protocol, the authors find that text-only baselines match or surpass AMR-augmented models. A perplexity-based probe shows that AMR does not provide LLMs with additional relational knowledge, suggesting no clear benefit from AMR augmentation.
By Hoa Quynh Nhung Nguyen, Jacopo Staiano, Michael Sullivan
arXiv:2601.03199v2 Announce Type: replace-cross
Abstract: Diffusion language models (DLMs) have shown strong potential for general natural language tasks with in-context examples. Existing In-Context...
By Yang Li, Han Meng, Chenan Wang, Zhenyu Bi, Xuan Wang, Haipeng Chen
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens.
The study investigates how augmenting training data with conceptually related alternatives—beyond nouns to include verbs and adjectives—affects large language models trained with next-token prediction. Results show that while concept augmentation consistently improves perplexity on news data regardless of part of speech, verbs yield the most reliable downstream performance gains. Representational analyses reveal that the conditions producing the largest downstream improvements do not always correspond to the greatest changes in internal representations.
By Laya Iyer, Chen Shani
arXiv:2604.16656v2 Announce Type: replace
Abstract: All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictates the cost and...
By Maitrey Mehta, Nishant Subramani, Zhichao Xu, Ashim Gupta, Vivek Srikumar
The paper investigates how adjectival modifiers affect the semantic plausibility of events, using the Adept benchmark of 16,000 English sentence pairs that differ by a single adjective. Experiments show that sentence transformers, despite being conceptually suited to the task, underperform compared to models like RoBERTa. The authors provide an error analysis and discuss the implications of their findings for future work on balancing training and test data.
By Anna Golub, Beate Zywietz, Annerose Eichel
Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results....
iFlip is an iterative refinement method for generating counterfactual examples using large language models. It incorporates three feedback types—model confidence, feature attribution, and natural language—to guide successive edits. Experiments show iFlip outperforms five state‑of‑the‑art baselines, achieving a 57.8% higher validity rate and improving model performance through counterfactual data augmentation.
By Yilong Wang, Qianli Wang, Nils Feldhus
arXiv:2601.13734v2 Announce Type: replace-cross
Abstract: In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. A...
By Chenyu Hui
Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different...