The paper introduces a framework for learning feature transformations on tabular data that addresses three limitations of prior generative methods: neglect of hierarchical feature relationships, bias from order-sensitive embeddings, and reliance on gradient-based search. It combines a permutation‑invariant hierarchical module using self‑attention pooling to capture interactions across features, operations, and abstraction levels, with a policy‑guided multi‑objective reinforcement learning strategy that starts from strong seeds and optimizes both predictive accuracy and transformation efficiency. Experiments on diverse tabular benchmarks show the approach outperforms strong baselines, and the authors provide public code and data.
arXiv:2607. 23286v1 Announce Type: new Abstract: Automatic feature engineering (AutoFE) for tabular learning can be naturally formulated as a program synthesis problem, where the objective is to discover predictive feature transformations from an exponentially large search space.
The paper introduces TOPOFE, a framework that treats automatic feature engineering for tabular data as a graph-structured multi-island evolutionary search. Each island explores a semantically coherent family of transformations using LLM-guided mutation and crossover, while a Prompt Adaptation Memory steers proposals based on accept/reject feedback. TOPOFE dynamically learns a directed topology graph to coordinate cross-island transfer, enabling the discovery of compositional feature programs that outperform state‑of‑the‑art methods on 29 datasets and produce lower redundancy and higher representational coverage.
arXiv:2505.09716v4 Announce Type: replace Abstract: Out-of-distribution (OOD) generalisation through composition requires a system to discover invariant properties from input-output associations and...
The paper presents an iterative framework that uses large language models (LLMs) to automatically extract interpretable, schema‑bound categorical features from unstructured text for use in tabular prediction models. A generator LLM proposes semantic definitions, an extractor LLM materializes the features, and a downstream tabular model evaluates their predictive performance, with error‑driven natural‑language feedback guiding the search. Across three public datasets, the error‑driven loop speeds up feature discovery up to three times and the resulting features outperform any subset when combined with TF‑IDF and dense embeddings, while also providing instance‑level interpretability through SHAP importance rankings and a semantic audit trail.
arXiv:2602. 11626v3 Announce Type: replace-cross Abstract: Learning solution operators on arbitrary geometries remains a central challenge in scientific machine learning, especially for many-query simulation, physics-informed learning, and evolving geometries requiring accurate, geometry-aware predictions at arbitrary spatial locations.
OrionMoE-v1.0 is a compact tabular foundation model that uses a dual‑axis Transformer and a sparse mixture‑of‑experts architecture to perform in‑context learning on labeled support tables. It selectively activates experts along the feature axis, guided by both individual feature representations and overall dataset context, enabling adaptive computation across diverse tabular problems. Pretrained on synthetic classification tasks, it achieves competitive performance on multiple benchmarks, especially on small‑context, high‑dimensional, and class‑imbalanced datasets.
arXiv:2604. 00086v2 Announce Type: replace-cross Abstract: The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks.
arXiv:2510. 18315v2 Announce Type: replace-cross Abstract: We investigate how embedding dimension affects the emergence of an internal "world model" in a transformer trained with reinforcement learning to perform bubble-sort-style adjacent swaps.
arXiv:2602.03753v2 Announce Type: replace Abstract: While representation alignment with self-supervised models has been shown to improve diffusion model training, its potential for enhancing inferenc...
arXiv:2605. 18714v2 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture.
LoopICL is a transformer architecture that loops a single block to address tabular tasks. It separates parameter count from computational depth by using a cell stream for per‑cell features and a row stream for in‑context examples, refined via within‑column and cross‑column attention. During pre‑training, varying loop counts and a learned exit‑gate allow the model to adjust inference depth at test time, achieving competitive performance with TabICLv2 while using about 90% fewer parameters.
arXiv:2607. 24804v2 Announce Type: replace-cross Abstract: Recommendation systems have undergone significant transformations in the past years.