arXiv:2606. 28127v1 Announce Type: cross Abstract: The AI community has framed the relationship between large language models (LLMs) and world models as a dichotomy: LLMs predict tokens; world models simulate reality.
The AI community has framed the relationship between large language models (LLMs) and world models as a dichotomy: LLMs predict tokens; world models simulate reality. Yann LeCun argues in 2022 that reaching general intelligence requires abandoning autoregressive token prediction in favour of latent-space architectures.
arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.
arXiv:2606. 09936v1 Announce Type: cross Abstract: World models are now built on substantially different computational substrates.
arXiv:2609.38806v1 Announce Type: cross Abstract: Next-token prediction has driven remarkable progress in large language models, yet a growing body of evidence suggests that they can struggle on prob...
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight, procedurally generated grid‑world testbed for evaluating predictive models, particularly Transformers, on tasks such as per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction. It compares byte‑level Transformer baselines with two memory‑augmented architectures, showing that a pseudo‑video spatial‑memory Transformer achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, while a generic auxiliary latent‑memory Transformer does not consistently help. The study highlights that structured, task‑aligned working memory can be more effective than merely increasing latent capacity, and positions the benchmark as a compact setting for testing architectures that couple textual interfaces to learned structured state.
NCP-ArchPreview is a latent‑space language model that extends standard next‑token prediction (NTP) with a Next Concept Prediction (NCP) objective, allowing the model to predict discrete concepts spanning multiple tokens. The architecture builds a product‑quantized concept vocabulary from hidden states, uses a dedicated Concept Module to forecast future concepts, and feeds these predictions back to guide token‑level generation, all trained jointly end‑to‑end. Trained on 5.73 T tokens with 8.9 B parameters, it achieves the final pretraining loss of OLMo‑3‑7B using only 51.3 % of the tokens, outperforms OLMo‑3‑7B on downstream tasks (including a 5.99‑point GSM8K gain), and demonstrates that the learned latent space enables lightweight domain adaptation and improved drafting performance.
The paper introduces LS-AR, a dual‑channel architecture that separates continuous goal steering from discrete token decoding using FiLM conditioning. It employs a Static Goal Encoder for long‑term macro‑objective retention and a Dynamic State Tracker for recurrent latent updates. Experiments show LS‑AR achieves perfect target recall on long‑horizon retrieval tasks, improves throughput by ~35%, reduces VRAM usage by 52.8%, and maintains higher completion rates in Blocksworld planning under perturbations.
arXiv:2606. 29604v1 Announce Type: cross Abstract: We aim to discover diverse, generalizable perturbations of LLM internals that can surface hidden behavioral modes.
arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight testbed for evaluating how Transformers and other models learn world dynamics. The benchmark, built from procedurally generated grid worlds, includes per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction, with source‑maze‑disjoint training and validation splits to test transferable action‑conditioned dynamics. Experiments show that a pseudo‑video spatial‑memory Transformer, which initializes and updates a two‑dimensional latent workspace from the input map and action history, achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, outperforming byte‑level and unstructured‑memory baselines and suggesting that structured, task‑aligned working memory is more effective than additional latent capacity alone.
arXiv:2609. 04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation.
arXiv:2511. 02748v2 Announce Type: replace-cross Abstract: We argue that sixth-generation (6G) intelligence is not fluent token prediction but the capacity to imagine and choose -- to simulate future scenarios, weigh trade-offs, and act with calibrated uncertainty.