arXiv AI
Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 21
Auditing Cross-Lingual Fairness in Language Model Watermarking
arXiv:2608. 20047v1 Announce Type: cross Abstract: Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements.
arXiv Machine Learning
Aug 5
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
arXiv Machine Learning
Oct 8
Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs
The paper introduces the Adversarial Surface-Form Robustness Dataset (ASRD), which contains 2,100 prompts across seven surface‑form families to test large language models on non‑canonical inputs. Five open‑weight models were evaluated on 10,500 responses, and a Quad‑State Evaluation Rubric categorized each response as harmful compliance, safe response, comprehension failure, or indeterminate. Results show that emoji and invisible Unicode variations produce few comprehension failures, while leetspeak, encoded wrappers, and hybrid transformations lead to high comprehension failure rates and varied harmful compliance percentages.
arXiv Computation and Language
Oct 7
Are Language Models Script-Aware?
arXiv:2610.08037v1 Announce Type: new Abstract: Language models frequently generate outputs in unintended languages or scripts, a phenomenon known as off-target generation. While existing research ha...
arXiv AI
Oct 2
The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages
arXiv:2605. 27901v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) monitoring has been proposed as a promising safety mechanism for detecting misaligned behavior in large language models.
arXiv AI
Aug 13
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
arXiv:2608. 11694v1 Announce Type: cross Abstract: A benchmark score comes from a single phrasing of each problem.