arXiv Machine Learning
Aug 6
Attention-based representations for multi-task computation
arXiv:2608. 04243v1 Announce Type: new Abstract: Multi-head attention layers produce vector representations that support multiple downstream tasks.
Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic
arXiv:2502. 01015v5 Announce Type: replace Abstract: Task arithmetic, representing downstream tasks through linear operations on task vectors, has emerged as a simple yet powerful paradigm for transferring knowledge across diverse settings.
Related stories
arXiv AI
Aug 12
TACTICL: Task-Aware Compression of Tabular ICL Models
arXiv:2608. 10837v1 Announce Type: cross Abstract: The strong performance of foundation models for tabular tasks comes at substantial inference costs.
arXiv Machine Learning
Jun 18
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
arXiv Machine Learning
6d ago
Computational Algebra with Attention: Transformer Oracles for Border Basis Algorithms
arXiv:2505. 23696v2 Announce Type: replace Abstract: Solving systems of polynomial equations, particularly those with finitely many solutions, is a crucial challenge across many scientific fields.
arXiv AI
Jun 30
HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression
arXiv:2606. 28831v1 Announce Type: cross Abstract: Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.
arXiv Machine Learning
Aug 4
Gram-Space: Structure-Preserving Codebook Compression for Memory-Efficient Neuro-Symbolic AI
arXiv:2608. 01528v1 Announce Type: new Abstract: Vector symbolic architectures (VSA) are widely used for reasoning in neuro-symbolic (NeSy) AI, yet high-dimensional codebooks often create severe memory bottlenecks that limit scalability and deployment.
arXiv Machine Learning
Jun 30
Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging
arXiv:2512. 01461v2 Announce Type: replace Abstract: Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training.
arXiv Machine Learning
Jun 29
Accelerating Attention with Basis Decomposition
arXiv:2510. 01718v2 Announce Type: replace Abstract: Attention is a core operation in large language models (LLMs).
arXiv AI
1d ago
FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy
arXiv:2608. 15602v1 Announce Type: cross Abstract: While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads.
arXiv AI
Jul 29
At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference
arXiv:2607. 25504v1 Announce Type: cross Abstract: Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models.
arXiv Machine Learning
Jun 24
ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs
arXiv:2510. 04767v2 Announce Type: replace Abstract: While most autoregressive LLMs are constrained to one-by-one decoding, diffusion LLMs (dLLMs) have attracted growing interest for their potential to dramatically accelerate inference through parallel decoding.
arXiv AI
Jun 26
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
arXiv:2606. 27229v1 Announce Type: cross Abstract: Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored -- the gate sees only the arriving token, not the memory it is about to modify.