The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper introduces DIB-Bench, a benchmark that exposes deployment‑integrity blindness in large language model (LLM) evaluation by testing 7,560 scored examples across six tamper types and a range of evaluators from regex scorers to GPT‑5‑nano. It shows that stronger evaluators are more likely to accept tampered outputs—mean acceptance rises from 88.6% for rule‑based scorers to 94.1% for GPT‑5‑nano—and that certain trigger‑conditional tampering can produce harmful outputs in 48–100% of cases, increasing with model capability. The study also demonstrates that output‑only evaluation cannot detect tampering with zero output shift, while a manifest‑hash identity channel can detect it with zero samples, and proposes a graded deployment‑identity descriptor for evaluation reports.
arXiv:2609.35889v1 Announce Type: cross Abstract: Tool-using language-model agents select and execute third-party artifacts. Different implementations can return the requested output while producing...
arXiv:2604. 16706v2 Announce Type: replace Abstract: Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation.
arXiv:2607. 19449v1 Announce Type: cross Abstract: Evaluation frameworks for tool-augmented LLM agents focus overwhelmingly on capability metrics or explicit tool crashes, leaving silent infrastructure failures and HTTP 200 responses with empty, null, or malformed payloads largely unaudited.
LITMUS is a benchmark for testing the safety of large language model agents in real operating system environments, focusing on behavior jailbreaks that can cause irreversible OS-level harm. It includes 819 high-risk test cases organized into a harmful seed subset and six attack-extended subsets covering jailbreak speaking, skill injection, and entity wrapping, and employs a semantic-physical dual verification mechanism with OS-level state rollback. Evaluation of frontier agents shows that even strong models can execute dangerous operations, exhibit execution hallucination, and are vulnerable to skill injection and entity wrapping attacks.
The paper investigates whether automatic safety judges evaluate the content of a model’s reply or merely its style. By keeping the reply content fixed and adding various style wrappers—such as educational disclaimers, fake reasoning blocks, or token refusals—the authors show that many judges flip their verdicts, indicating that style can influence safety judgments. The study evaluates over 600 jailbreak examples across multiple judges, revealing that some judges are highly susceptible to style-based manipulation while others remain robust.