The paper introduces spatial induction heads, a two‑layer gather‑and‑match mechanism that enables transformers to perform in‑context learning on multidimensional cellular automata. By reconstructing local spatial neighborhoods and matching them against prior occurrences, the method achieves Bayesian counting and can approximate the Bayesian posterior for stochastic rules or predict deterministically for deterministic rules. Experiments show that trained transformers generalize to unseen rules in one and two dimensions, achieving near‑perfect rollouts and KL divergences below 0.005 nats from the Bayes‑optimal predictor, with attention patterns supporting the proposed computation.
arXiv:2603. 16689v2 Announce Type: replace Abstract: Next-token predictors often appear to develop internal representations of the latent world and its rules.
arXiv:2507. 01414v2 Announce Type: replace Abstract: We introduce a new family of toy problems that combine features of linear-regression-style continuous in-context learning (ICL) with discrete associative recall.
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight testbed for evaluating how Transformers and other models learn world dynamics. The benchmark, built from procedurally generated grid worlds, includes per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction, with source‑maze‑disjoint training and validation splits to test transferable action‑conditioned dynamics. Experiments show that a pseudo‑video spatial‑memory Transformer, which initializes and updates a two‑dimensional latent workspace from the input map and action history, achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, outperforming byte‑level and unstructured‑memory baselines and suggesting that structured, task‑aligned working memory is more effective than additional latent capacity alone.
arXiv:2610.07562v1 Announce Type: cross Abstract: Learning an ensemble of GFlowNets to sample from a discrete target distribution has become a common approach for achieving better state space explora...
The paper introduces the AGI Maze Prediction Datasets and Benchmark, a lightweight, procedurally generated grid‑world testbed for evaluating predictive models, particularly Transformers, on tasks such as per‑step transition prediction, fixed‑horizon state prediction, and sequential textual‑observation prediction. It compares byte‑level Transformer baselines with two memory‑augmented architectures, showing that a pseudo‑video spatial‑memory Transformer achieves perfect validation accuracy on selected tasks and improves sequential text‑trace prediction, while a generic auxiliary latent‑memory Transformer does not consistently help. The study highlights that structured, task‑aligned working memory can be more effective than merely increasing latent capacity, and positions the benchmark as a compact setting for testing architectures that couple textual interfaces to learned structured state.
The paper introduces Partial GFlowNet, a method that partitions a large state space into overlapping partial state spaces to accelerate convergence of Generative Flow Networks. By restricting the actor’s exploration to these smaller regions and using a heuristic to switch between them, the approach enables efficient identification of high‑reward subregions. Experiments on popular datasets show that Partial GFlowNet converges faster, produces higher‑reward candidates, and improves diversity compared to existing methods.
arXiv:2609.39604v1 Announce Type: new Abstract: Although conventional world models - auto-regressive or diffusion models based on transformers or convolutional networks - may learn surface statistics...
arXiv:2607. 03200v1 Announce Type: new Abstract: Origin-destination (OD) flow prediction is central to urban analytics, yet deep models trained on raw counts remain vulnerable to distribution shift.
Hierarchical Cooperative GFlowNets (HCG) dynamically partition the search space to coordinate multiple GFlowNets, preventing them from collapsing onto the same high‑reward regions. HCG imposes an ordering where the top model explores the full space while lower‑ranked models focus on uncovered areas, using a shared classifier and a normalizer‑weighted aggregation rule to maintain reward‑proportional sampling. Experiments on multimodal distributions, sequence generation, combinatorial optimization, and biological sequence design demonstrate that HCG improves mode coverage and target approximation, enabling broader complementary exploration.
MetaLearnNCA introduces a decentralized few‑shot meta‑learning framework that uses interacting Neural Cellular Automata (NCAs) to adapt to new tasks without backpropagation. The method splits adaptation into an Active‑NCA that performs task inference on a 2D spatial memory grid, and a learned Meta‑NCA that diffuses error residuals locally to update this grid. Experiments show competitive in‑distribution performance (96.12% on Omniglot) and superior out‑of‑distribution transfer on MNIST, KMNIST, and Fashion‑MNIST compared to canonical meta‑learners.
The paper introduces Nomad, a transfer-and-ground framework for generating human mobility trajectories without target-city trajectory data. It learns relative transitions from source cities using POI attributes and then grounds these transitions onto a target city’s POI map via a behavior graph and exploration–return walk. Experiments across ten cities show Nomad improves trajectory fidelity and downstream utility by roughly 15% and 3% respectively over adaptation baselines.
arXiv:2608. 00270v2 Announce Type: replace Abstract: Neural Combinatorial Optimization (NCO) techniques have emerged as a highly efficient alternative to traditional exact algorithms for solving routing problems such as the Traveling Salesman Problem (TSP).