arXiv:2608. 20047v1 Announce Type: cross Abstract: Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements.
Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually.
The paper introduces a user‑centric audit framework that separates the users who create benchmark content from those whose needs the evaluation results are meant to represent. It operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures, and applies the framework to 16 resources. Findings show that American‑centered conventions dominate many text benchmarks, geographic visibility in image‑text resources correlates strongly with national GDP, and documentation reveals a gap between source‑user and target‑user contexts. Paired scoring experiments demonstrate that meaning‑preserving answer variants can drastically lower accuracy metrics, highlighting the sensitivity of evaluation scores to answer phrasing.
The paper discusses the EU AI Act’s requirement for generative AI providers to embed detectable watermarks in their outputs, noting that Anthropic’s Claude models and Google’s Gemini use SynthID‑Text by default. It critiques the lack of verifiability of claims about watermark quality, privacy, and robustness, and evaluates the open‑source SynthID‑Text implementation on two open‑weight models, finding minimal impact on prose and modest correctness loss on code. The authors argue that the real governance issue is the inability to verify these assertions and outline necessary steps—such as output release, configuration disclosure, accredited audits, shared evaluation protocols, and interoperable detection—to address the gaps.
arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.
arXiv:2506.22343v2 Announce Type: replace-cross Abstract: Text watermarks in large language models (LLMs) are an increasingly important tool for detecting synthetic text and distinguishing human-writ...
arXiv:2510.18019v3 Announce Type: replace Abstract: Multilingual watermarking aims to make large language model (LLM) outputs traceable across languages, yet current methods still fall short. Despite...
The paper introduces a unified evaluation framework for assessing the trustworthiness of large language models, agentic AI, and multimodal systems. It connects output-level, trajectory-level, and cross-modal assessments across eight dimensions—capability, robustness, safety, fairness, transparency, governance, oversight, and efficiency—while preserving system-specific metrics and providing uncertainty estimates. A meta-evaluation layer checks the validity, reliability, and reproducibility of the evaluation itself, and the framework aligns with governance standards and regulatory requirements.
arXiv:2602. 09611v2 Announce Type: replace-cross Abstract: Watermarking has emerged as a pivotal solution for content traceability and intellectual property protection in large vision language models (LVLMs).
arXiv:2609.40031v1 Announce Type: cross Abstract: Digital image watermarking is increasingly critical in media contexts, as emerging regulations and industry practices require marking AI-generated co...
The paper audits 51 recent multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented; low‑resource languages face a task equity gap. Representativeness is compromised by heavy reliance on English translations, and rigor is weakened by benchmark contamination and parallel‑corpus overlap. The authors propose concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination‑aware, and community‑aligned.
MarkTune is a model‑embedded watermarking framework that improves the trade‑off between generation quality and detectability. By regularizing toward a high‑quality reference distribution, it decouples quality preservation from proximity to the base model and uses the GaussMark detection statistic as a reward during on‑policy fine‑tuning. Empirical results show that MarkTune enhances the quality‑detectability frontier, maintains downstream task performance, and remains robust against text‑editing and open‑weight attacks across diverse datasets.