Hugging Face Trending Papers

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

The paper examines whether benchmark scores obtained via APIs accurately reflect the performance of AI chatbots when accessed through user interfaces. By auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks, the authors find that API evaluations consistently overestimate accuracy and consistency compared to interface evaluations, with differences comparable to downgrading a full model generation. Attempts to align API behavior with interface behavior through prompt and parameter adjustments only partially close the gap.

arXiv AI
Nov 6

What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays

The paper evaluates 20 third‑party API relays that provide access to large language models such as GPT‑6 Astra and Claude Opus 5. It finds that many low‑price relays exhibit answer‑behavior differences, some return tool calls even when disabled, and several report inflated input‑token counts leading to higher charges than the official API. The study demonstrates that advertised models and prices alone do not guarantee reliable service, highlighting the need for repeated, detailed evaluations.

By Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing Ni
Hugging Face Trending Papers
Nov 5

What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays

The paper evaluates 20 third‑party LLM API relays that claim to provide GPT‑6 Astra and Claude Opus 5 at lower prices. By comparing answers to official responses, inspecting tool permissions, and reconciling billed token counts, the study finds that many relays exhibit behavioral differences, return unexpected tool calls, and report inflated input‑token usage. These discrepancies highlight that advertised models and prices alone do not guarantee reliable service.

arXiv AI
Aug 28

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

ADeptS-Bench is a new benchmark designed to assess the trustworthiness of Computer Use Agents (CUAs) across mobile and desktop devices. It consists of two streams: a Safety stream with paired benign and malicious tasks that embed visual threats, and a Disambiguation stream that tests whether agents seek clarification when instructions are ambiguous. Evaluation of seven models shows none consistently achieves high task success while keeping attack success low, and all models exhibit problematic behaviors such as unhesitant checkout on a $25K order and failure to detect a mislabeled factory reset button.

By Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe
arXiv AI
3d ago

Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss

The paper introduces isloss, a deterministic oracle that tests whether conversations are preserved accurately when translated across different LLM APIs. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to detect silent information loss such as changes to instruction roles, tool interactions, error states, or generation controls. Evaluation shows that isloss can detect a majority of planted faults with few false alarms and reveals that many translations silently alter critical conversation details, affecting model behavior.

By Xin Liu, Xu Han, Yuxin Wang, Haoran Song, Mingjian Zhao, Lei Wang
arXiv Machine Learning
Aug 31

Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification

The paper introduces a privacy‑preserving zk‑SNARK audit framework that uses adversarial‑style probes to detect logit drift between an approved large language model and a modified deployment. It offers three probe families—token‑based (black‑box), embedding‑based (gray‑box), and stress probes (partial white‑box)—allowing users to balance sensitivity, access, and cost. Experiments across LLM architectures and GPU platforms show token‑based probes achieve the highest mean sensitivity while remaining practical in a black‑box setting, with Groth16 proving times scaling modestly from 1.02 to 1.78 seconds and constant proof size.

By Cameron Wilding, Mina Shaker, Fatemeh Ganji
arXiv Machine Learning
Jun 2

Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models

arXiv:2606. 00566v1 Announce Type: new Abstract: As language models take on agentic roles that span calling external APIs, reading tool outputs, and acting on instructions embedded in third-party content, their attack surface expands well beyond what users type.

By Mohammed Sameer Syed (University of Arizona), Rozhin Yasaei (University of Arizona)