← Back to all news
arXiv AI

Multitask Interactions in Recurrent Dynamics across 100 Cognitive Tasks

The study trains hierarchical Almost-Linear RNNs on 100 diverse cognitive tasks to investigate how tasks interact when learned jointly. It finds that adding nonlinearity and task-feature dimensionality improves performance, and that a hierarchical model outperforms both individually trained models and a single unstructured joint model. The authors also derive task similarity measures from the trained model that relate to task complexity, solvability, transfer learning, and adaptation.

Related stories

arXiv Machine Learning
Sep 25

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

Transformers can learn broad families of tasks during pretraining and adapt to unseen tasks from a short prompt, but a rigorous understanding of this capability is limited. This paper studies how shared cross‑task structure influences the sample complexity of in‑context learning (ICL) by characterizing task‑space complexity through covering numbers, yielding a set of anchor functions that localize unseen tasks and predict responses. The authors construct a Transformer with Softmax attention to approximate this procedure and derive an error bound that separates the effects of pretraining tasks and prompt length, showing that once enough tasks are available the dependence on prompt length becomes dimension‑free.

By Zhongjie Shi, Rongjie Lai, Alexander Cloninger, Wenjing Liao
llms
More like this →
arXiv AI
Oct 12

Neural networks are more modular than single neurons suggest

The paper argues that neural networks can exhibit modular organization even when individual neurons are not specialized. It introduces a nested hierarchy of modularity—unit-level, orthogonal, and linear—and shows that many vision, language, and RNN models lack unit-level modularity but still display clear functional modularity at the subspace level. The authors demonstrate that these subspaces can be causally validated through ablation and align with brain data, revealing distinct content/style or syntax/semantics separations.

By Shreya Saha, Benjamin Bergen, Meenakshi Khosla
llmsbenchmarkssafety
More like this →
arXiv AI
Oct 12

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

NeuronSpark is a self‑attention‑free language model that uses input‑dependent spiking membrane dynamics, allowing each neuron’s decay rate, gain, and firing threshold to adapt to the current token. This selective spiking approach enables billion‑parameter pretraining from random initialization without relying on backbone transfer, conversion, or distillation, achieving competitive zero‑shot performance and a fast–slow hierarchy. On a 1.1B‑parameter, 2.1B‑token training run, NeuronSpark attains a 40.0 eight‑task zero‑shot average, surpasses comparable‑budget Pythia/Mamba checkpoints, and demonstrates significant energy and cycle reductions on FPGA implementations.

By Zhengzheng Tang
llmsefficiency
More like this →
arXiv AI
Oct 7

JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)

JIVEAdapter is a multi-task additive low-rank adapter that decomposes weight updates into a shared Joint component and a per-task Individual component, inspired by Joint and Individual Variation Explained (JIVE). It penalizes Individual structures to remain near-orthogonal to the Joint, ensuring interpretable separation of shared and task-specific signals, and allocates rank adaptively between shared and individual pools. The Joint component is learned once and frozen, then reused as a prior for new tasks, achieving competitive performance on GLUE and SuperGLUE with DeBERTaV3-base without additional modules.

By Sara Abdali, Pashmina Cameron
fine-tuning
More like this →
arXiv AI
Sep 30

AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models

AdaKerNet is a task‑adaptive neural kernel decoder that operates on frozen multimodal representations from large foundation models, without requiring access to the models’ parameters. It learns Lipschitz‑controlled multimodal features, a reference kernel providing a soft structural prior, and a lightweight nonlinear predictor that deforms this structure. Experiments on four multimodal large language models and diverse input modalities show consistent improvements over baseline decoders, achieving up to 41% error reduction in scarce‑label settings.

By Konstantinos D. Polyzos, Eleni Oikonomou, Tara Javidi
llmsragfine-tuningmultimodal
More like this →
Hugging Face Trending Papers
Jul 2

DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning

Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or subnetworks, yet our analyses suggest that classification supervision alone does not sufficiently preserve task-agnostic shared backbone representations over long incremental sequences. We identify two intertwined challenges: cross-task confusion from sequential training on predominantly current-task data, which biases decision boundaries toward recent tasks; and under-optimized shared representations in the backbone that cap long-term discriminability as tasks accumulate.

llms
More like this →