← Back to all news
arXiv AI By Xiaoyang Zhang, Yuejun Yan, Qingang Zhang, Yonggang Wen

Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

Read the original on arXiv AI →

The paper introduces CarbonRouter, an online framework that jointly selects the model, GPU type, and region for each query in multi‑turn LLM web services to minimize carbon emissions. It incorporates a cache‑aware carbon cost model that accounts for the history recomputation cost when a conversation is routed to a different deployment, and a lightweight gate that keeps a session on its cached deployment when a move back is likely. Experiments on two real‑world conversation datasets show that CarbonRouter achieves the lowest carbon footprint among evaluated policies, reducing CO₂ emissions by 16.1 % on average and up to 23.1 % for longer sessions, while also improving predicted answer quality and estimated latency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
3d ago

Multi-Turn Semantic Caching for Low-Cost LLM Serving

The paper introduces two algorithms—CUCB‑SC‑MT and CLCB‑LS‑MT—for multi‑turn semantic caching in large language model serving. These methods learn a "reuse risk" metric from judge feedback to decide which conversation states to cache, balancing avoided generation cost against the risk of unacceptable reuse. Experiments on math‑tutoring and movie‑ticketing dialogues show that the online CLCB‑LS‑MT algorithm reduces generation cost by 35–41% and estimated latency by 19–35% compared to prior caching approaches.

By Baran Atalar, Selly Beyene, Xutong Liu, Carlee Joe-Wong
llmsrag
More like this →
arXiv Computation and Language
Oct 26

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

arXiv:2606.27457v2 Announce Type: replace-cross Abstract: Serving open-weight language models on premises forces a trade-off between accuracy and computational cost. Hosting models of different sizes...

By Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas, Solmaz Panahi, David Lynch, Zhuangzhuang Nie, Alexandros Agapitos, Aleksandar Milenovic, Hongmeng Song, Yucheng Shi, Yue Pan, Patricia Buffini, John D. Kelleher
llms
More like this →
arXiv AI
Sep 15

Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems

The paper presents a carbon‑aware routing framework for function‑calling in large language models that distributes queries across a three‑tier edge‑cloud architecture. A lightweight k‑NN predictor estimates accuracy, delay, and power for each edge tier, and real‑time grid carbon intensity is used to route queries to the lowest‑emission tier that can execute them. Experiments on state‑of‑the‑art benchmarks show the framework matches cloud‑level accuracy while cutting operational carbon emissions by an average of four times.

By Aikaterini Maria Panteleaki, Varatheepan Paramanayakam, Spyros Tragoudas, Iraklis Anagnostopoulos
llmsragagentsbenchmarks
More like this →
arXiv AI
Oct 16

Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools

The paper introduces BEA, a fine‑grained estimator that predicts the time‑to‑first‑token (TTFT) for language model queries by replaying token‑batch formation based on current workload snapshots. BEA achieves about 5% error with 0.1 ms prediction time, enabling the latency‑aware router BEAR to balance accuracy, cost, and latency across heterogeneous LLM pools. BEAR outperforms seven baselines, improving utility of on‑target responses by 63% on average and by 30.5% at each arrival rate.

By Shivam Patel, Akaash R. Parthasarathy, Akash Dhasade, Ankur Mallick, Gauri Joshi
llms
More like this →