arXiv Machine Learning

Do Language Models Consistently Encode the Current Year?

arXiv:2608. 15507v1 Announce Type: cross Abstract: A consistent concept of the current time is important for temporal reasoning, yet how language models represent the current time is not well understood.

arXiv AI
Jul 14

PRISM Edit: One Vector for All Temporal Answers

arXiv:2607. 11327v1 Announce Type: cross Abstract: Model editing keeps large language models (LLMs) up to date without retraining, but temporal facts expose a limitation of the prevailing locate-and-edit paradigm: an update is not always a replacement.

By Chen Huang (Tsinghua University), Qi Zheng (Tsinghua University), Ruiqin Zheng (ByteDance), Long Zeng (Tsinghua University), Yuantong Xu (ByteDance)
arXiv AI
Jul 15

Scaling Point-in-Time Language Models

arXiv:2607. 11889v1 Announce Type: cross Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences.

By Bryan Kelly, Semyon Malamud, Johannes Schwab, Teng Andrea Xu
arXiv AI
Sep 1

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

The study investigates how post‑training of large autoregressive language models (ARMs) into masked diffusion models (MDMs) affects their internal computation. Across two 7B ARM‑MDM families and four diagnostic tasks, the authors find that MDMs retain much of the ARM’s high‑attribution pathways on prefix‑dominant tasks, but reorganize computation toward earlier layers on globally constrained tasks. Component‑level probes reveal that ARMs depend on sharply specialized components, whereas MDMs show weaker specialization and more diffuse output‑space alignment.

By Injin Kong, Hyoungjoon Lee, Yohan Jo
arXiv Computation and Language
Sep 2

When Tokenization is Secretly Output Supervision

The paper argues that tokenization in language models should be viewed as an output supervision decision rather than merely input preprocessing. In autoregressive models, the granularity of the tokenizer determines the supervision signal the model receives, influencing learning difficulty, internal representations, and task performance. Experiments on numeric reasoning show that output tokenization, rather than input tokenization, drives differences in performance and training dynamics, and a survey of recent CL papers reveals that tokenization choices are rarely reported or acknowledged.

By Tanja Baeumel, Josef van Genabith, Simon Ostermann