arXiv AI By Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur, Abhinav TK, Ashwin Hegde, Anand Gautam

Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy

Read the original on arXiv AI →

The paper describes the Cosmos LLM Gateway, a policy‑driven, multi‑vendor gateway built on Envoy that manages enterprise LLM traffic at PayPal. It separates latency‑sensitive control decisions into an external service while handling wire‑format translation and token metering in‑proxy via a Rust WebAssembly filter, ensuring minimal overhead. Governance is declarative, managed through Open Policy Agent, and includes features like data‑loss‑prevention, budget scopes, and a control plane for role‑scoped authoring and approvals. The gateway currently serves over 12,000 developers, processes roughly 200 billion tokens per day across 100+ models, and maintains 99.999% availability with only a few milliseconds of added latency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
5d ago

Cross-Interface Authorization for AI Computer-Use Agents

The paper introduces a protocol for cross‑interface authorization of AI computer‑use agents, ensuring that policy decisions remain consistent across graphical interfaces, command lines, APIs, scripts, and Model Context Protocol servers. It defines a profile‑bounded system that enumerates all effect‑bearing routes, normalizes requests to canonical effects, issues prepare tickets, and re‑verifies all relevant state at commit time, requiring a sink receipt for a committed result. Extensive testing across two runtime mappings produced thousands of traces, with rigorous checks confirming no semantic regressions, mutant rejections, and correct outcome matching.

By Genliang Zhu, Chu Wang
arXiv AI
Jun 15

STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming

arXiv:2606. 13968v1 Announce Type: cross Abstract: Researchers and practitioners working with large language models face a fragmented landscape: local models are free and private but hardware limits the model size and context windows a researcher can use; institutional HPC centers offer powerful GPU resources at no marginal cost and keep data within institutional boundaries, but operate behind firewalls and are designed for batch jobs rather than interactive use; commercial cloud APIs provide frontier-model quality on demand but impose significant cost and data retention policies unsuitable for sensitive research data.

By Anas Nassar, Steve Mohr, Leonard Apanasevich, Himanshu Sharma
arXiv AI
Aug 19

PACE: Policy-Attested Contract Execution for Safe AI Agents in Decentralized Finance

The paper introduces PACE (Policy‑Attested Contract Execution), a framework that sits between large‑language‑model (LLM) based autonomous AI agents and on‑chain DeFi operations. PACE defines typed transaction intents, a deterministic policy verifier, and signed Policy Decision Records (PDRs) that cryptographically bind an approved intent, policy, and simulation report to the exact on‑chain execution bytes, providing replay and expiration protection. In evaluations across 40 tasks and six baselines, PACE achieves zero unsafe executions and zero false positives, outperforming unguarded agents by a large margin.

By Rabimba Karanjai (Larry), Yang Lu (Larry), Richard Williamson (Larry), Hemanth Hm (Larry), Prakhar Mehrotra (Larry), Lei Xu (Larry), Weidong (Larry), Shi