The paper introduces two new mathematical task ecosystems, IntWorld and PermWorld, that consist of related integer and permutation tasks. These procedurally generated datasets allow researchers to control the number of tasks and study how task diversity affects neural network training. Initial experiments suggest that greater task variety leads to richer internal representations in models.
arXiv:2609.25438v1 Announce Type: new Abstract: Diverse pretraining has been shown to be an effective method for learning reusable, domain-aware representations that provide a starting point for fine...
arXiv:2512. 15011v3 Announce Type: replace Abstract: Artificial intelligence (AI) increasingly generates the very content used to train future AI systems.
The paper introduces a method for training large language models to generate diverse, high‑quality scientific ideas grounded in input documents. By using a set‑level reinforcement learning objective that rewards ideas for being distinct from earlier ones, the authors train an 8B model that outperforms Claude Opus 4.8 and GPT‑6 Astra on a contamination‑free benchmark. Human experts rate the model’s idea sets as equally promising while containing slightly more unique ideas, and automated experiments show comparable executability.
arXiv:2606. 01811v1 Announce Type: cross Abstract: Measuring the diversity of creative outputs is central to evaluating post-training mode collapse, comparing decoding strategies, and quantifying creative behavior in both AI and human writing.
arXiv:2505. 20161v2 Announce Type: replace-cross Abstract: Effective generalization in language models depends critically on the diversity of their training data.
arXiv:2410. 12341v4 Announce Type: replace-cross Abstract: As AI-generated content increasingly populates the web, generative AI models are at growing risk of being trained on their own outputs, a process known as AI autophagy.
The paper argues that open‑world AI requires three core capabilities: inference‑time learning, rich representations that capture more than IID generalization, and predictive disentanglement aligning inputs and outputs into low‑dimensional groups. It derives these needs from modeling tasks as sharing latent factors and shows that language models meeting these criteria improve with related data, while others degrade. The study tests these claims on up to 8B‑parameter language models, focusing on qualitative properties rather than quantitative performance.
arXiv:2602. 03542v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are trained and tested extensively on symbolic representations such as code and graphs, yet real-world user tasks are often specified in natural language.
arXiv:2604. 06501v2 Announce Type: replace Abstract: Analogical reasoning is a hallmark of human intelligence, enabling us to solve new problems by transferring knowledge from one situation to another.
The paper introduces TPR-Attention, an attention mechanism that operates over tensor‑product representations to embed structured inductive bias into deep learning models. Experiments on compositional tasks demonstrate that TPR‑Attention outperforms existing architectural components in achieving combinatorial generalization. The results suggest that incorporating explicit compositional structure into neural attention can improve systematic generalization.
The paper introduces adVersarial Parameter Decomposition (VPD), a method that splits a language model’s parameters into simple subcomponents, each responsible for a small part of the model’s algorithm. VPD optimizes these decompositions so that the model’s input‑output behavior remains intact even when many subcomponents are removed, including adversarially chosen ones. This approach enables a new form of circuit analysis by tracing how sequences of parameter subcomponents interact to produce outputs for specific inputs.
arXiv:2604. 18827v2 Announce Type: replace-cross Abstract: Scaling data and artificial neural networks has transformed AI, driving breakthroughs in language and vision.