The paper examines whether benchmark scores obtained via APIs accurately reflect the performance of AI chatbots when accessed through user interfaces. By auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks, the authors find that API evaluations consistently overestimate accuracy and consistency compared to interface evaluations, with differences comparable to downgrading a full model generation. Attempts to align API behavior with interface behavior through prompt and parameter adjustments only partially close the gap.
arXiv:2608. 06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness.
By Ro Encarnaci\'on, Tina Behzad, Emma Lurie, Dana\'e Metaxa
ADeptS-Bench is a new benchmark designed to assess the trustworthiness of Computer Use Agents (CUAs) across mobile and desktop devices. It consists of two streams: a Safety stream with paired benign and malicious tasks that embed visual threats, and a Disambiguation stream that tests whether agents seek clarification when instructions are ambiguous. Evaluation of seven models shows none consistently achieves high task success while keeping attack success low, and all models exhibit problematic behaviors such as unhesitant checkout on a $25K order and failure to detect a mislabeled factory reset button.
By Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe
arXiv:2607. 02587v2 Announce Type: replace-cross Abstract: Trust-benchmark scores reported on a chat-LLM release line are often carried across several checkpoints of the same line, as if the underlying model had not shifted between releases.
By Zhichao Fan, Yanhang Li, Zexin Zhuang, Xian Sun, Yingshuo Wang
arXiv:2609.21293v1 Announce Type: new
Abstract: Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessari...
By Xiuhui Zhang, Yi Chen, Shusheng Xu, Fan Li, Huan Wang, Tongkai Yang, Binhang Yuan
arXiv:2606. 00566v1 Announce Type: new Abstract: As language models take on agentic roles that span calling external APIs, reading tool outputs, and acting on instructions embedded in third-party content, their attack surface expands well beyond what users type.
By Mohammed Sameer Syed (University of Arizona), Rozhin Yasaei (University of Arizona)
arXiv:2609.16590v1 Announce Type: new
Abstract: People increasingly use frontier AI models for health advice, but via different access modes (e.g., ChatGPT, ChatGPT Health, APIs) with varying setting...
By Yuan Pu, Yewon Chang, Furong Jia, Xunjian Yin, Jessica Ma, Ayman Ali, Monica Agrawal
arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.
By Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam, Prasaanth Balraj, Nakul Jain
The paper introduces a privacy‑preserving zk‑SNARK audit framework that uses adversarial‑style probes to detect logit drift between an approved large language model and a modified deployment. It offers three probe families—token‑based (black‑box), embedding‑based (gray‑box), and stress probes (partial white‑box)—allowing users to balance sensitivity, access, and cost. Experiments across LLM architectures and GPU platforms show token‑based probes achieve the highest mean sensitivity while remaining practical in a black‑box setting, with Groth16 proving times scaling modestly from 1.02 to 1.78 seconds and constant proof size.
By Cameron Wilding, Mina Shaker, Fatemeh Ganji
arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.
By Hiskias Dingeto, Will Leeney
arXiv:2609.18729v1 Announce Type: cross
Abstract: Consumers increasingly use AI chatbots for advice on what to buy. With companies like OpenAI and Google monetising their AI through advertising, this...
By Lucas G. Uberti-Bona Marin, Thales Bertaglia, Giovanni Astante, Bram Rijsbosch, Gijs van Dijck, Anik\'o Hann\'ak, Gerasimos Spanakis, Konrad Kollnig
arXiv:2609.06059v1 Announce Type: new
Abstract: As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to a...
By Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, Yanbing Liu, Pei Fu