← Back to all news
arXiv Machine Learning

One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions

arXiv:2607. 10252v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly consumed through opaque serving chains - API aggregators, resellers, and inference providers - in which the client has no technical means to confirm that the model answering is the model advertised, and recent audits show that a substantial fraction of commercial endpoints deviate from the vendor's reference weights.

Related stories

arXiv Computation and Language
Sep 1

Token Counts Are Not Model Lineage: A Frozen-Threshold Holdout Study of Black-Box LLM API Fingerprinting

The study evaluates whether prompt‑token counts can reliably identify the lineage of large language models served via APIs. Using a frozen‑threshold approach on 24 labeled endpoint pairs, the authors find that token‑count consistency perfectly separates development pairs but only half of the holdout pairs meet the strict repeatability criteria, yielding moderate accuracy and perfect specificity. The results confirm token‑count consistency as a fingerprint of shared tokenization stacks but reject it as a standalone test for model‑family attribution.

By Bo Chen
llmsnlp
More like this →
arXiv AI
Aug 11

Targeted Counterfactual Fingerprinting for Black-Box LLM Ownership Verification

arXiv:2608. 08195v1 Announce Type: cross Abstract: Large language models (LLMs) are high-value assets that can be derived through redeployment, fine-tuning, quantization, or further alignment.

By Yutong Wu, Xiaofan Bai, Shixin Li, Pingyi Hu, Ziqi Zhou, Zilong Wang, Xiaojing Ma, Songfeng Lu, Yuhong Li, Jin Xuan, Yi Wang, Dongmei Zhang, Bin Benjamin Zhu
llmsfine-tuningefficiencysafety
More like this →
arXiv Machine Learning
Jul 1

Surrogate Fidelity: When Can Open LLMs Explain Closed Ones?

arXiv:2606. 32008v1 Announce Type: new Abstract: Mechanistic interpretability (MI) requires full access to model internals, yet the APIs for most widely deployed language models at best expose log-probabilities over output tokens.

By Philippe Chlenski, Zachariah Carmichael, Ayush Warikoo, Chia-Tse Shao, Yingxiao Ye, Aobo Yang, Vivek Miglani, Nehal Bandi
llmssafety
More like this →
arXiv Computation and Language
Aug 28

An Embarrassingly Simple Detector for Model Extraction Attacks in Large Language Model API Traffic

The paper presents a simple detector for model extraction attacks on large language model APIs. It frames detection as a benign‑calibrated traffic‑window distribution test, embedding queries into a semantic space and using maximum mean discrepancy (MMD) to compare against historical benign traffic. Evaluated on fourteen attacker‑normal query pairs across four extraction scenarios, MMD achieves near‑perfect true‑positive rates while maintaining a 0.3% false‑positive rate, outperforming several existing baselines.

By Shuze Liu, Qianwen Guo, Yushun Dong
llms
More like this →
arXiv AI
Sep 25

Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.

By Tapan Parikh
llmsagents
More like this →