DART-FL is a multitask federated learning framework designed for edge devices that must balance online inference and model training under limited resources. It dynamically allocates resources between inference and training based on current inference backlog and service capacity, then distributes remaining training capacity among tasks using a queue‑aware scheduler that adjusts loss weights. Experiments on image classification datasets with synthetic and real workloads show that DART‑FL adapts to bursty inference demand, improving accuracy for high‑demand tasks while preserving overall multitask performance.
arXiv:2609.17193v1 Announce Type: new Abstract: Large language model (LLM)-powered agentic AI services increasingly demand low-latency inference, motivating the deployment of LLMs across distributed...
arXiv:2606. 27743v1 Announce Type: cross Abstract: Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph regardless of the runtime environment.
arXiv:2502. 11007v5 Announce Type: replace Abstract: Compared to traditional machine learning models, recent large language models (LLMs) can exhibit multi-task-solving capabilities through multi-modal data sources and multi-turn conversations.
arXiv:2602. 06932v5 Announce Type: replace Abstract: Speculative decoding can significantly accelerate LLM serving, yet most deployments today disentangle speculator training from serving, treating speculator training as a standalone offline modeling problem.
AsyncFlow is an asynchronous streaming reinforcement learning framework designed to improve the post‑training phase of large language models. It introduces a distributed data storage and transfer module that enables panoramic data management and fine‑grained scheduling, allowing automated pipeline overlapping and dynamic load balancing. The framework also employs an asynchronous producer‑consumer workflow to reduce computational idleness by deferring parameter updates within staleness thresholds, and it is architecturally decoupled from training and inference engines, providing modular, customizable user interfaces. Experiments show an average throughput improvement of 1.59× over the state‑of‑the‑art baseline.
MANE is a distributed inference framework that uses a multi‑path tail architecture to allow dynamic accuracy–throughput trade‑offs during edge onloading of deep neural networks. It introduces a novel multi‑path model, a three‑stage training scheme with Joint Head Network Distillation loss, and a hysteresis‑based scheduler with an equitable device‑fallback policy. The system achieves over 80% SLO satisfaction and 6pp higher accuracy than on‑device alternatives while supporting up to 40 concurrent devices.
arXiv:2510. 03243v3 Announce Type: replace-cross Abstract: Efficient scheduling of large language model (LLM) inference tasks is critical for achieving low latency and high throughput, a challenge that is becoming increasingly acute with the rise of reasoning-capable LLMs whose generation lengths are highly variable.
arXiv:2607. 05272v1 Announce Type: cross Abstract: Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic.
The paper introduces COMLLM, a generative framework that combines Group Relative Policy Optimization with a Look‑Ahead Collaborative Simulation to enable multi‑turn reasoning for task offloading in Mobile Edge Computing. By performing multi‑step Monte Carlo rollouts that jointly model server queue dynamics, COMLLM incorporates long‑term system evolution into its reward design, achieving near‑optimal latency and improved load‑balancing fairness. The framework demonstrates zero‑shot scalability to larger network topologies, outperforming supervised fine‑tuning, deep reinforcement learning, and heuristic baselines without requiring retraining.
TALLOP is a method for federated learning that jointly optimizes local training workload and communication compression within a fixed per‑round time budget. It predicts achievable training loss from early mini‑batch loss trajectories (LENS) and estimates compression‑induced degradation using a compression‑error model (CLEN). By combining these predictions, TALLOP selects a training‑communication allocation that minimizes post‑compression loss, achieving competitive or improved time‑to‑accuracy in heterogeneous federated learning environments.
arXiv:2606. 11440v1 Announce Type: new Abstract: Existing multi-agent LLM orchestration methods, ranging from brute-force ensembles to learned routers, select models and topologies based on task and model features.
Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic. We investigate whether reinforcement learning (RL) can learn adaptive batching and routing policies that outperform these heuristics, training REINFORCE and PPO agents on a discrete-event simulator validated against queuing theory and production traces (Azure Functions, BurstGPT).