The paper evaluates 20 third‑party API relays that provide access to large language models such as GPT‑6 Astra and Claude Opus 5. It finds that many low‑price relays exhibit answer‑behavior differences, some return tool calls even when disabled, and several report inflated input‑token counts leading to higher charges than the official API. The study demonstrates that advertised models and prices alone do not guarantee reliable service, highlighting the need for repeated, detailed evaluations.
By Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing Ni
arXiv:2609.00052v1 Announce Type: cross
Abstract: Commercial LLM APIs advertise a specific foundation model, but the served backbone may be silently substituted, quantized, or wrapped, for example to...
By Xun Wang, Bihe Zhao, Michael Backes, Franziska Boenisch, Adam Dziedzic
The paper introduces API-TAIL, a method for detecting partial substitution of large language model APIs. It aligns batch-level temperature effects to standardize surprisal scores, then flags anomalous tail responses that persist under the claimed model. Experiments on commercial APIs show API-TAIL controls false alarms while detecting low-rate substitution more effectively than existing audits.
By Haocheng Mei, Yuchuan Tian, Ye Yuan, Chao Xu, Allen Chen, Kai Han, Yunhe Wang
The paper introduces Distributional Fingerprint Auditing (DFA), a statistical method for auditing large language model (LLM) relay services. DFA uses only textual responses from suspect and trusted endpoints, repeatedly querying simple finite-choice probes to build categorical response distributions that serve as fingerprints. By comparing these fingerprints probe‑by‑probe and aggregating evidence with the Cauchy combination test, DFA produces an approximate endpoint‑level p‑value for audit decisions at a chosen significance level, and further supports model attribution and substitution ratio estimation through a likelihood‑based framework.
By Fangran Miao, Shengjie Niu, Jian Huang, Ting Li
arXiv:2607. 10252v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly consumed through opaque serving chains - API aggregators, resellers, and inference providers - in which the client has no technical means to confirm that the model answering is the model advertised, and recent audits show that a substantial fraction of commercial endpoints deviate from the vendor's reference weights.
By Tomas Bruckner
The study evaluates whether prompt‑token counts can reliably identify the lineage of large language models served via APIs. Using a frozen‑threshold approach on 24 labeled endpoint pairs, the authors find that token‑count consistency perfectly separates development pairs but only half of the holdout pairs meet the strict repeatability criteria, yielding moderate accuracy and perfect specificity. The results confirm token‑count consistency as a fingerprint of shared tokenization stacks but reject it as a standalone test for model‑family attribution.
By Bo Chen