arXiv:2610.07348v1 Announce Type: cross Abstract: Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraint...
arXiv:2502. 11007v5 Announce Type: replace Abstract: Compared to traditional machine learning models, recent large language models (LLMs) can exhibit multi-task-solving capabilities through multi-modal data sources and multi-turn conversations.
TokenRouter is a new serving system designed for token‑level routing of large language models (LLMs). It adopts a request‑centric programming model and launches a dedicated subserver for each LLM, using delayed‑batching schedulers whose hyperparameters are derived from a throughput model. Across various routing algorithms, workloads, and model pairs, TokenRouter delivers 2.01‑64.15× higher decoding throughput than existing systems, markedly improving the efficiency of token‑level LLM inference.
arXiv:2608. 13076v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands.
arXiv:2604. 26508v2 Announce Type: replace-cross Abstract: Deploying Vision-Language Models (VLMs) on edge devices remains challenging due to their substantial computational and memory demands, which exceed the capabilities of resource-constrained embedded platforms.
The paper proposes a Multi-Agentic AI framework to enable fairness‑aware and accelerated inference of large, heterogeneous multi‑modal language models on mobile edge networks. It introduces a Global Planning Agent, a Prompt Scheduling Agent, and multiple on‑node LM Deployment Agents that jointly optimize prompt routing and model deployment using natural language reasoning over telemetry and historical data. Experiments on a city‑wide testbed show a 42.9% reduction in end‑to‑end latency versus a state‑of‑the‑art baseline and an 80.6% reduction versus a full‑activation baseline, while preserving high fairness (Jain's index 0.925).
arXiv:2608.29291v1 Announce Type: new Abstract: Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generati...
arXiv:2610.08268v1 Announce Type: cross Abstract: Pervasive intelligent applications are increasingly deployed on mobile and Internet of Things (IoT) edge devices. Consequently, Large Language Models...
arXiv:2609.10714v1 Announce Type: cross Abstract: The ambitious requirements of sixth-generation (6G) networks are driving communication systems from reliable bit delivery toward meaning-aware and ta...
arXiv:2607. 20481v1 Announce Type: new Abstract: Local-cloud collaboration is a practical way to deploy large language models under resource constraints, but existing methods often rely on trained routers or collaboration-aware finetuning that tie routing behavior to a particular operating regime.
The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.
arXiv:2606. 09514v1 Announce Type: new Abstract: Large language models (LLMs) incur high inference cost due to their depth and parameter scale.