The paper introduces a new sequence modeling framework based on nonlinear time-delay systems that dynamically estimates effective delay using semantic similarity between current and past inputs. It proposes a differentiable semantic lag estimator for content-aware soft attention, a diagonal state space with O(1) per-step complexity, and an interpretable fuzzy semantic modulation mechanism that balances immediate processing with historical recall. Experiments on text classification and generation show that this approach outperforms state‑of‑the‑art methods.
arXiv:2608. 10120v1 Announce Type: new Abstract: Modern sequence models, from Transformers to State Space Models, have enabled powerful generative modeling across diverse domains, yet they are typically trained to predict what happens while treating when it happens as a secondary concern.
arXiv:2608. 02347v2 Announce Type: replace Abstract: Recurrent linear attention models (RLAs) such as Mamba offer efficient linear-time sequence modeling as an alternative to Transformers, yet their fixed-capacity recurrent states limit long-sequence modeling.
arXiv:2606. 09862v1 Announce Type: cross Abstract: The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios.
arXiv:2608. 03463v1 Announce Type: new Abstract: Long-term memory is essential for LLM-based agents to sustain interactions and reliably leverage distant history.
arXiv:2607. 25357v1 Announce Type: cross Abstract: Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory.
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
The paper introduces Temporal Semantic Memory (TSM), a framework that improves how large language model agents manage memory by addressing two key shortcomings: temporal inaccuracy and temporal fragmentation. TSM constructs a semantic timeline instead of a dialogue timeline, consolidating temporally continuous and semantically related information into durative memory. During retrieval, it aligns the query’s temporal intent with the semantic timeline, enabling the use of temporally appropriate durative memories and yielding up to a 12.2% accuracy boost over existing methods.
arXiv:2606. 12364v1 Announce Type: new Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost.
arXiv:2606. 18986v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have given rise to time-series question answering (TSQA), which formulates time-series analysis as natural-language question answering.
arXiv:2608. 07663v1 Announce Type: cross Abstract: When videos extend from hours to days, directly processing them end-to-end becomes impractical for current Multi-modal Large Language Models (MLLMs).
arXiv:2606. 06479v1 Announce Type: new Abstract: Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations.
The paper introduces Interdomain Attention (IA), a hybrid architecture that blends softmax attention with recurrent memory within a single layer, leveraging a kernel regression perspective to compress keys and values into a state‑space model (SSM) memory. IA reconstructs the attention matrix implicitly from memory states, preserving the computational efficiency of SSMs while achieving performance gains. Experiments on models ranging from 125 M to 1.3 B parameters show IA surpasses an S4D baseline at matched memory size, outperforms standard Transformers on perplexity and common‑sense reasoning, and matches or exceeds SSM‑based models on genomic downstream tasks.