arXiv AI
Jul 10
Validity of LLMs as data annotators: AMALIA on authority
arXiv:2607. 08731v1 Announce Type: cross Abstract: A national language model offers a linguistic community its own instrument for measuring what its citizens say and value.
One Scale, Four Texts: Gaps, Overlaps and Silent Repairs in Published Grading Schemes
The paper investigates how different published texts for diabetic retinopathy grading—specifically four versions of an international scale and an Italian guideline—lead to gaps, overlaps, and silent repairs when interpreted literally. By transcribing these texts into a unified rule language and exhaustively enumerating over 1.4 billion finding profiles, the authors show that each text leaves some eyes uncategorized, that the 2017 tabulation contains an empty proliferative class and a macular gap absent in the 2003 text, and that English and Italian texts can order the same eyes differently. Experiments on annotated photographs and language‑model responses further reveal significant disagreements in referral decisions and persistent silent repairs even when models are instructed to read literally.
Related stories
arXiv Computation and Language
Oct 15
Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
arXiv:2610.20461v1 Announce Type: new Abstract: Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER,...
Hugging Face Trending Papers
Oct 14
Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR
Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and...
Hugging Face Trending Papers
Jul 9
Validity of LLMs as data annotators: AMALIA on authority
A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size.
arXiv AI
6d ago
One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them
The study evaluates whether a single confidence threshold in typed decision interfaces treats all languages equally. Using 1,000 items in 51 languages across three vendors, it finds that English consistently achieves higher coverage—clearing the 0.9 threshold 8.6–11.7 percentage points more often than other languages—despite similar accuracy. Even when thresholds are re‑weighted or calibrated per language, the gap persists, and translating inputs into English reduces but does not eliminate it. The authors recommend reporting per‑language coverage at the operating threshold.
arXiv AI
Oct 9
Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora
The study investigates how exposing large language models to Arabic translations of English benchmark tasks can hide data contamination from standard English-only detection methods. By testing four instruction-tuned models with increasing Arabic exposure, the authors find that traditional probes like TS-Guessing and Min-K%++ fail to detect contamination, yet English performance still improves. They propose a new Translation-Aware Contamination Detection (TACD) method that uses cross-lingual consistency and choice reordering to reveal contamination effects that are otherwise invisible to English-only diagnostics.
arXiv AI
Aug 24
The Divergence Hypothesis: Unmasking Lexical Interference and Label Bias in Mental Health NLP
The paper introduces TSS (Triple-Stream Stress probe), a diagnostic framework that splits text into lexical, morpho-syntactic, and psycholinguistic style channels to analyze mental health NLP classifiers. Across four English datasets, TSS uncovers a lexical interference effect where adding lexical features harms performance on human-labeled data but not on auto-labeled data, and proposes the Degree of Divergence (DoD) statistic to audit label-source bias. The study demonstrates that style features largely remain effective even after masking content words, emphasizing that shortcut learning is label-source specific rather than clinically relevant.
arXiv Computation and Language
Sep 24
MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors
The paper introduces MWE‑ECL, a bilingual diagnostic framework that tests whether distant discourse anchors can override local lexical priors in multi‑word expression interpretation. It evaluates models on a 0‑128K context grid, finding that while retrieval of anchors is near perfect, the ability to change locally preferred readings varies, especially when the model’s default conflicts with the anchor. The study shows that explicit recoverability does not always translate into behavioral influence, with gaps differing across models and languages.
arXiv Computer Vision
Oct 27
Do Words Matter? What Text Contributes to Few and Full-Shot Vision-Language Anomaly Detection
arXiv:2610.34610v1 Announce Type: new Abstract: Vision-language anomaly detectors describe normal and abnormal appearance with text, and their gains are commonly credited to the semantic content of t...
Hugging Face Trending Papers
Sep 23
MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors
Long-context evaluations often test whether a model can recover distant evidence, but recoverability does not guarantee behavioral influence. We test the prediction that a distant discourse anchor can...
arXiv AI
Aug 11
The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora
arXiv:2608. 09093v1 Announce Type: cross Abstract: How a document's arrangement is written down, its notation, is a training variable that no dataset card records.
arXiv Computer Vision
Oct 12
Auditing VLM Benchmarks from Existing Evaluation Runs
The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.