arXiv:2609.08861v1 Announce Type: new
Abstract: Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assum...
By Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo
arXiv:2608. 06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness.
By Ro Encarnaci\'on, Tina Behzad, Emma Lurie, Dana\'e Metaxa
The paper evaluates 20 third‑party API relays that provide access to large language models such as GPT‑6 Astra and Claude Opus 5. It finds that many low‑price relays exhibit answer‑behavior differences, some return tool calls even when disabled, and several report inflated input‑token counts leading to higher charges than the official API. The study demonstrates that advertised models and prices alone do not guarantee reliable service, highlighting the need for repeated, detailed evaluations.
By Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing Ni
The paper evaluates 20 third‑party LLM API relays that claim to provide GPT‑6 Astra and Claude Opus 5 at lower prices. By comparing answers to official responses, inspecting tool permissions, and reconciling billed token counts, the study finds that many relays exhibit behavioral differences, return unexpected tool calls, and report inflated input‑token usage. These discrepancies highlight that advertised models and prices alone do not guarantee reliable service.
ADeptS-Bench is a new benchmark designed to assess the trustworthiness of Computer Use Agents (CUAs) across mobile and desktop devices. It consists of two streams: a Safety stream with paired benign and malicious tasks that embed visual threats, and a Disambiguation stream that tests whether agents seek clarification when instructions are ambiguous. Evaluation of seven models shows none consistently achieves high task success while keeping attack success low, and all models exhibit problematic behaviors such as unhesitant checkout on a $25K order and failure to detect a mislabeled factory reset button.
By Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe
arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.
By Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam, Prasaanth Balraj, Nakul Jain
arXiv:2609.06059v1 Announce Type: new
Abstract: As large language models evolve from question-answering systems into general-purpose agents, evaluation must move beyond static answer correctness to a...
By Yu Liu, Zhilin Liu, Zhiwei Yang, Shaojie Zhang, Zheyuan Deng, Tingwei Huang, Zhenbo Luo, Lei Jiang, Yanbing Liu, Pei Fu
arXiv:2607. 02587v2 Announce Type: replace-cross Abstract: Trust-benchmark scores reported on a chat-LLM release line are often carried across several checkpoints of the same line, as if the underlying model had not shifted between releases.
By Zhichao Fan, Yanhang Li, Zexin Zhuang, Xian Sun, Yingshuo Wang
The paper introduces isloss, a deterministic oracle that tests whether conversations are preserved accurately when translated across different LLM APIs. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to detect silent information loss such as changes to instruction roles, tool interactions, error states, or generation controls. Evaluation shows that isloss can detect a majority of planted faults with few false alarms and reveals that many translations silently alter critical conversation details, affecting model behavior.
By Xin Liu, Xu Han, Yuxin Wang, Haoran Song, Mingjian Zhao, Lei Wang
arXiv:2609.21293v1 Announce Type: new
Abstract: Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessari...
By Xiuhui Zhang, Yi Chen, Shusheng Xu, Fan Li, Huan Wang, Tongkai Yang, Binhang Yuan
The paper introduces a privacy‑preserving zk‑SNARK audit framework that uses adversarial‑style probes to detect logit drift between an approved large language model and a modified deployment. It offers three probe families—token‑based (black‑box), embedding‑based (gray‑box), and stress probes (partial white‑box)—allowing users to balance sensitivity, access, and cost. Experiments across LLM architectures and GPU platforms show token‑based probes achieve the highest mean sensitivity while remaining practical in a black‑box setting, with Groth16 proving times scaling modestly from 1.02 to 1.78 seconds and constant proof size.
By Cameron Wilding, Mina Shaker, Fatemeh Ganji
arXiv:2606. 00566v1 Announce Type: new Abstract: As language models take on agentic roles that span calling external APIs, reading tool outputs, and acting on instructions embedded in third-party content, their attack surface expands well beyond what users type.
By Mohammed Sameer Syed (University of Arizona), Rozhin Yasaei (University of Arizona)