The Flow has not summarised this story yet — read it at arXiv Computation and Language.
arXiv:2608. 20047v1 Announce Type: cross Abstract: Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements.
Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually.
The paper introduces a user‑centric audit framework that separates the users who create benchmark content from those whose needs the evaluation results are meant to represent. It operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures, and applies the framework to 16 resources. Findings show that American‑centered conventions dominate many text benchmarks, geographic visibility in image‑text resources correlates strongly with national GDP, and documentation reveals a gap between source‑user and target‑user contexts. Paired scoring experiments demonstrate that meaning‑preserving answer variants can drastically lower accuracy metrics, highlighting the sensitivity of evaluation scores to answer phrasing.
The paper discusses the EU AI Act’s requirement for generative AI providers to embed detectable watermarks in their outputs, noting that Anthropic’s Claude models and Google’s Gemini use SynthID‑Text by default. It critiques the lack of verifiability of claims about watermark quality, privacy, and robustness, and evaluates the open‑source SynthID‑Text implementation on two open‑weight models, finding minimal impact on prose and modest correctness loss on code. The authors argue that the real governance issue is the inability to verify these assertions and outline necessary steps—such as output release, configuration disclosure, accredited audits, shared evaluation protocols, and interoperable detection—to address the gaps.
arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.
arXiv:2506.22343v2 Announce Type: replace-cross Abstract: Text watermarks in large language models (LLMs) are an increasingly important tool for detecting synthetic text and distinguishing human-writ...