PriorityFEC is a deadline‑aware streaming forward erasure correction scheme designed to hand off key–value (KV) cache data between agents in edge multi‑agent large language model systems. It prioritizes high‑utility KV blocks in the transmission stream and adapts repair packet scheduling based on receiver feedback to meet strict time‑to‑first‑token deadlines. Experiments with Qwen2.5‑7B‑Instruct and Llama‑3.1‑8B‑Instruct show that PriorityFEC improves task quality compared to other transport schemes, especially under tight deadline constraints.
arXiv:2609.32259v2 Announce Type: replace Abstract: Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires ea...
arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.
TOPAS is a Task‑Oriented Prefix‑Aware Scheduler designed for multi‑agent large language model serving. It jointly decides which agent prefixes to retain in a shared key‑value cache and which requests to schedule, balancing the reduction of each task’s longest remaining service path against the benefit of downstream prefix reuse while accounting for movement and preemption costs. Experiments on synthetic DAGs and MetaGPT software‑development workflows show that TOPAS can reduce mean and p99 job completion times by up to 39.8%/49.4% and 22.0%/26.6% respectively compared to the best baselines.
arXiv:2609.34060v2 Announce Type: replace Abstract: Prompt-specialized multi-agent systems enable multiple agents to share a model while performing complementary roles to solve complex tasks. However...
arXiv:2609.17193v1 Announce Type: new Abstract: Large language model (LLM)-powered agentic AI services increasingly demand low-latency inference, motivating the deployment of LLMs across distributed...
arXiv:2608. 12371v1 Announce Type: new Abstract: Stream-processing systems increasingly operate across heterogeneous mobile edge--cloud infrastructures, where workload volatility, resource contention, and stringent quality-of-service (QoS) requirements complicate decentralized scheduling.
arXiv:2608. 10545v1 Announce Type: cross Abstract: Edge LLMs must preserve inference continuity when a user hands over between edge nodes, requiring key-value (KV) cache transfer to the target node.
arXiv:2610.15509v1 Announce Type: cross Abstract: KV caches make long-context serving efficient by reusing past computation, but model switching breaks this reuse and forces the receiver to prefill t...
arXiv:2609.34054v2 Announce Type: replace Abstract: Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the grow...
SparKV is an adaptive framework for on-device large language model inference that intelligently decides whether to stream or compute each key-value (KV) cache chunk, overlapping communication and computation to cut latency. It dynamically adjusts offline-generated schedules at runtime to cope with changing wireless connectivity and edge resource availability. Experiments show SparKV can reduce Time-to-First-Token by 1.3×–5.1× and lower per-request energy consumption by 1.5×–3.3× with minimal impact on response quality.
HAServe introduces a horizon-aware scheduling and memory co-design framework for efficient large language model (LLM) serving. By modeling request admission and key-value (KV) retention as a unified problem using the HorizonSchema abstraction, it selects joint admission and KV-retention candidates that minimize predicted latency across varying prefix structures and arrival rates. Experiments on Qwen3-4B-Instruct-2507 and Llama2-13B with eight H100 GPUs demonstrate up to 5.90× P99 TTFT speedups, 1.44× throughput gains, and a 20.59 percentage point increase in KV-cache hit rate over the strongest baseline.
arXiv:2608. 16336v1 Announce Type: cross Abstract: Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing.