Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

28,685 stories · RSS feed

arXiv Computation and Language
Sep 30

Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

The study demonstrates that large language models (LLMs) can produce highly reliable labels under a single experimental setup, yet their outputs vary significantly when researchers alter task designs or model choices. By evaluating seven LLMs across 12 task designs and 3,000 tweets for offensive language and hate speech, the authors found that agreement dropped from a median Fleiss' κ of 0.91 to a median Cohen's κ of 0.76 when task designs changed. This design sensitivity inflates prevalence estimates by up to 110.6 times compared to sampling variance alone, and confidence scores fail to mitigate the issue.

By Thomas Reiter, Christoph Kern, Fedor Miasnikov, Sofiia Nikolenko, Rob Chew, Stephanie Eckman, Frauke Kreuter
arXiv Computation and Language
Sep 30

The Canonical Order Problem: When Large Language Models Are Unreliable Knowledge Bases for Multi-Valued Relations

The paper investigates how large language models (LLMs) handle multi‑valued relations when used as knowledge bases. It identifies a "canonical order problem," where LLMs internally organize such relations according to a fixed ordering (e.g., alphabetical or chronological). The authors analyze the generation process in three phases—retrieval, internal sorting, and selection—and show that deviating from the internal canonical order reduces the reliability of LLMs in producing complete sets for multi‑valued relations.

By Timo Pierre Schrader, Annemarie Friedrich, Simon Razniewski, Lukas Lange
arXiv Computation and Language
Sep 30

Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories

The paper investigates whether large language models (LLMs) that store persistent memories can truly forget information upon user request. It shows that existing LLMs cannot delete such memories even when they claim to have forgotten them, and that simply removing matching messages is ineffective due to message dependencies. The authors introduce DeLLM, a framework that builds relevant context dynamically and uses a provenance graph to identify which messages must be removed, achieving a high deletion rate while preserving utility.

By Olga Ohrimenko
arXiv Computation and Language
Sep 30

When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain

The paper introduces a holistic framework for large language model self‑evolution that uses learnable information gain to assess the novelty of each training round. Information gain is theoretically linked to the Kullback‑Leibler divergence and entropy change between successive data distributions, and practically estimated by fitting a small language model and scoring new data with negative log‑likelihood. The proposed ATRI method reweights samples within a round and stops training across rounds when information gain is low, and experiments on popular datasets show its effectiveness.

By Chenxu Wang, Chaozhuo Li, Xinze Shi, Songyang Liu, Kyrie You Wu, Ziluowen Luo, Shun Zhang, Chenxi Li, Litian Zhang
arXiv Computer Vision
Sep 30

DirectUV: Image-Conditioned UV Texture Generation with Surface-Aware Positional Encoding

DirectUV is an image-conditioned UV texture diffusion framework that generates high-quality textures directly in UV space using a pretrained image VAE and a Diffusion Transformer. It introduces Surface-Aware Positional Encoding (SAPE), which replaces standard 2D-grid positional encoding with encodings derived from 3D surface coordinates, enabling attention to operate based on surface proximity rather than UV-grid distance. A multi-level extension assigns different attention heads to progressively finer subdivisions of the UV patch, allowing the model to reason about surface structure at multiple granularities, resulting in sharper and more globally consistent textures, especially in occluded and view-unseen regions.

By Jiantao Lin, Yingjie Xu, Mingzhi Sheng, Yangkai Wei, Hao Chen, Ying-Cong Chen