arXiv:2609.08861v1 Announce Type: new
Abstract: Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assum...
By Jennifer Wang, Joachim Baumann, Daniel E. Ho, Sanmi Koyejo
arXiv:2608. 06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness.
By Ro Encarnaci\'on, Tina Behzad, Emma Lurie, Dana\'e Metaxa
The paper evaluates 20 third‑party API relays that provide access to large language models such as GPT‑6 Astra and Claude Opus 5. It finds that many low‑price relays exhibit answer‑behavior differences, some return tool calls even when disabled, and several report inflated input‑token counts leading to higher charges than the official API. The study demonstrates that advertised models and prices alone do not guarantee reliable service, highlighting the need for repeated, detailed evaluations.
By Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li, Jianbing Ni
The paper evaluates 20 third‑party LLM API relays that claim to provide GPT‑6 Astra and Claude Opus 5 at lower prices. By comparing answers to official responses, inspecting tool permissions, and reconciling billed token counts, the study finds that many relays exhibit behavioral differences, return unexpected tool calls, and report inflated input‑token usage. These discrepancies highlight that advertised models and prices alone do not guarantee reliable service.
ADeptS-Bench is a new benchmark designed to assess the trustworthiness of Computer Use Agents (CUAs) across mobile and desktop devices. It consists of two streams: a Safety stream with paired benign and malicious tasks that embed visual threats, and a Disambiguation stream that tests whether agents seek clarification when instructions are ambiguous. Evaluation of seven models shows none consistently achieves high task success while keeping attack success low, and all models exhibit problematic behaviors such as unhesitant checkout on a $25K order and failure to detect a mislabeled factory reset button.
By Joy Chen, Alejandro Castillejo Munoz, Pierluca D'Oro, Yuxuan Sun, Chloe Evans, Joseph Tighe
arXiv:2605. 28508v2 Announce Type: replace Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape usability as much as model quality.
By Aakash Pant, Kavya Shah, Apoorv Agnihotri, Sneha Nikam, Prasaanth Balraj, Nakul Jain