← Back to all news
arXiv Computation and Language By Anssi Moisio, Mathias Creutz, Mikko Kurimo

Type Diversity Enables Transformers to Generalise Compositionally

Read the original on arXiv Computation and Language →

The paper investigates why Transformers struggle more with structural than lexical compositional generalisation. It argues that this disparity stems from low structural type diversity rather than an inherent limitation of Transformers. By creating linguistically diverse variants of the COGS and SLOG datasets, the authors show that type diversity correlates equally with generalisation in both lexical and structural cases, challenging previous explanations of the difficulty.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Aug 28

Compositional Generalization via Structural Identification in a Category-Theoretic Framework

The paper proposes a new way to evaluate compositional generalization by examining which structural or lexical identifications allow held‑out COGS examples to be considered admissible based on training data. Sentences are modeled as functors from syntactic addresses to lexical tokens, and selective collapses induce Kan extensions that propagate observed associations. Across 21 COGS generalization types, admissibility follows distinct identification profiles, while residual failures highlight unsupported structural templates, providing data‑side diagnoses of what the training corpus licenses without training a predictive model.

By Akihiro Maeda, Thomas Seiller, Yohei Oseki
More like this →