The paper argues that many AI tasks can be split into a deterministic core and a fuzzy residue, and that decomposing tasks rather than expanding models yields a property called parity by construction—where model‑size no longer affects performance on the deterministic sub‑tasks. Across nine models ranging from 494 M to Opus‑class size, the authors show that once the deterministic context is resolved, operation selection and selector resolution scores become identical, while a non‑decomposable task (labelling) remains sensitive to model size. They present a general method for identifying decomposition points, constructing deterministic surfaces and grammars, and ensuring safe residues, and demonstrate the approach in physics, motion retargeting, object‑model editing, and a Web‑CLI tool.
whyItMatters":"The study demonstrates that task decomposition can eliminate the need for model scaling or fine‑tuning, offering a systematic design discipline applicable across diverse AI domains."
By Tejaswi Gowda
A recurring design choice in building AI systems is whether, when a task is too hard for a model, to expand the model (make it larger, better-prompted, fine-tuned), or to decompose the task (extract t...
The paper investigates when in‑context learning (ICL) can succeed even if the instruction or target task is not covered by a model’s pretraining data. It introduces a theoretical framework that distinguishes two types of distribution shift—unsupported instructions and unsupported target tasks—and shows how demonstrations can mitigate these shifts. The authors further extend the analysis to long‑horizon tasks that can be decomposed into pretraining‑supported subtasks, providing error bounds that explain how ICL can handle tasks beyond the model’s original support.
By Xuhan Tong, Yuchen Zeng, Jiawei Zhang
The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table. It finds that a table’s usefulness is largely determined by its number of features rather than instances, and that fine‑grained column‑level preprocessing improves downstream performance while dataset‑level filtering does not. The authors propose that tabular in‑context generalization is primarily retrieval‑based, with models learning to identify and aggregate relevant examples from the provided context.
By Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini
arXiv:2506. 20699v2 Announce Type: replace Abstract: Learning in non-stationary and multi-context environments requires more than ordinary within-task generalization.
By Xin Li
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
By Ningyuan Shi, Zhipeng Zhou, Hao Wang, Chunyan Miao, Peilin Zhao
The paper demonstrates that a hypernetwork can generate tiny specialist models for individual transformations in the ARC-1D benchmark directly from a few demonstrations. These generated parameters form a structured weight space, enabling partial compositional generalisation and improved performance on unseen transformations when explicit task identifiers are omitted. The study provides a proof of concept that few‑shot task context can be compiled on‑the‑fly into compact executable model parameters that support reuse and generalisation beyond known functions.
By Fabio J. Fehr, Philip Torr
The paper introduces Retro, a tabular foundation model that uses retrospective inference to revisit and recombine intermediate representations during inference. Retro employs Attention Residuals to select which past information to reuse and query-conditioned Gated Attention to shape the contextual update for each query. Experiments on TabArena, TALENT, and RelArena show Retro achieves top‑three performance and lies on the Pareto frontier, demonstrating that reusing intermediate representations can better exploit depth in tabular foundation models.
By Hao-Run Cai, Si-Yang Liu, Zi-Jian Cheng, Kun-Yang Yu, Jin-Hao Sheng, Guo Yu, Chonghan Liu, Zhi Zhou, Jun-Peng Jiang, Lan-Zhe Guo, Han-Jia Ye
The paper investigates how Tabular Foundation Models (TFMs) can achieve strong transfer learning by self‑supervised pre‑training on a single real table, rather than large synthetic or real datasets. It finds that a table’s usefulness for downstream tasks is mainly determined by the number of features, not instances, and that fine‑grained column‑level preprocessing improves performance while dataset‑level filtering does not. The authors propose a task‑centric, retrieval‑based view of in‑context generalization, suggesting that effective TFMs identify and aggregate relevant examples from the provided context.
The paper introduces SCX Router, a lightweight GLiClass-based model selector that assigns suitability scores to inference-time language models without autoregressive generation. It uses a 0.6B-parameter Qwen3 decoder with a shallow bidirectional scorer, preserving a text-only key–value cache across sessions and predicting task attributes such as type, difficulty, and expected output length. The authors build a comprehensive task ontology with 23 families, 115 types, and 1,173 synthetic examples, generating 150,000 verifier-scored tasks to train the router, which outperforms baseline models on LiveBench subsets with a top‑1 score of 0.707 versus 0.696 for the strongest fixed model.
By Ihor Stepanov, Aleksandr Smechov, Mykhailo Shtopko, Dmytro Vodianytskyi, Oleksandr Lukashov
arXiv:2606. 01520v1 Announce Type: new Abstract: A single action-conditioned latent predictive architecture can in principle be trained on the structured state of a driving scene, a robot workspace, or a financial order book.
By Shayan Shokri
The study demonstrates that large language models (LLMs) can perform numerical prediction on tabular data by leveraging semantic grounding—matching variable names to the values they describe—without task-specific training. Across 20 datasets from nine scientific domains, five LLMs matched or outperformed seven supervised machine learning models, especially when training examples were scarce. Controlled experiments with physical formulas reveal that LLMs combine domain knowledge (as a prior) with in-context examples (likelihood) in a Bayesian‑like manner, adjusting predictions as contradictory examples accumulate.
By Xin-Rui Ma, Shi-Zhi Chen, Hai-Yan Jiang, De-Cheng Feng, Michael Beer