arXiv AI
Jul 7
Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines
arXiv:2607. 02630v1 Announce Type: cross Abstract: Hardware accelerators now sit on the critical path of online serving.
MADAR: An Address-Free Processor
arXiv:2606. 15535v1 Announce Type: cross Abstract: In a modern processor, computing is the cheap part.
Related stories
arXiv AI
Aug 14
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
arXiv:2608. 13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU.
arXiv Machine Learning
Sep 15
Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...
arXiv AI
Jul 21
Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents
arXiv:2607. 05690v2 Announce Type: replace Abstract: Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn.
arXiv Machine Learning
Jul 7
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
arXiv:2607. 04206v1 Announce Type: cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation.
arXiv AI
Jul 8
Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents
arXiv:2607. 05690v1 Announce Type: new Abstract: Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn.
Hugging Face Trending Papers
Jul 5
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others.
arXiv AI
Sep 12
DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Management
The paper proposes DCO, a dynamic cache orchestration scheme for multi-core AI accelerators that uses application-aware policies and dataflow information to guide cache replacement, bypass decisions, and thrashing mitigation. Using a cycle-accurate simulator, the authors demonstrate up to 1.80× speedup over conventional cache architectures and validate the approach with an analytical model and RTL implementation. The design occupies 0.064 mm² on a 15 nm process and operates at 2 GHz, showing that a shared system-level cache can simplify programming while boosting performance for large language model workloads.
arXiv Machine Learning
Jun 8
Hermes: Accelerating Long-Latency Load Requests via Perceptron-Based Off-Chip Load Prediction
arXiv:2209. 00188v4 Announce Type: replace-cross Abstract: Long-latency load requests continue to limit the performance of high-performance processors.
arXiv AI
Jun 9
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
arXiv:2606. 09613v1 Announce Type: cross Abstract: Multi-turn LLM agents interleave model calls with external tool invocations, shifting serving from stateless request processing to stateful program execution.
arXiv AI
Jul 7
Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes
arXiv:2607. 04668v1 Announce Type: cross Abstract: On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously.
arXiv Machine Learning
Aug 7
Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
arXiv:2608. 05944v1 Announce Type: cross Abstract: We report operational experience full-fine-tuning a 32.