arXiv AI
FlowWise: Stateful Fast-Slow Model Serving for Streaming Traffic Intelligence
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 16
Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools
The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.
arXiv AI
Jun 2
Lodestar: An Online-Learning LLM Inference Router
arXiv:2606. 00946v1 Announce Type: cross Abstract: Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization.
arXiv AI
Jun 11
INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration
arXiv:2606. 11440v1 Announce Type: new Abstract: Existing multi-agent LLM orchestration methods, ranging from brute-force ensembles to learned routers, select models and topologies based on task and model features.
arXiv AI
Jun 15
STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming
arXiv:2606. 13968v1 Announce Type: cross Abstract: Researchers and practitioners working with large language models face a fragmented landscape: local models are free and private but hardware limits the model size and context windows a researcher can use; institutional HPC centers offer powerful GPU resources at no marginal cost and keep data within institutional boundaries, but operate behind firewalls and are designed for batch jobs rather than interactive use; commercial cloud APIs provide frontier-model quality on demand but impose significant cost and data retention policies unsuitable for sensitive research data.
arXiv AI
Jun 10
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.
arXiv AI
Oct 7
DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching
arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models...