arXiv Computation and Language October 19, 2026 By Adib Sakhawat, Syed Rifat Raiyan

Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms

Read the original on arXiv Computation and Language →

The paper investigates how literal interpretations of Bengali idioms influence the behavior of decision models that output probabilities over supplied options without generating text. By inserting an idiom’s literal gloss into the option list, the authors show that all three tested models—an encoder scorer, a decoder scorer, and a closed API—consistently shift their predictions toward the literal option, causing 65% to 96% of remaining errors to be attributed to it. The effect is markedly stronger in Bengali than in English, with the commercial model’s apparent robustness being a consequence of the test language rather than intrinsic capability, and the models’ calibration deteriorates across all arms, making confidence-based gating unreliable.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

safety

arXiv Machine Learning

Sep 16

Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?

arXiv:2507.23248v2 Announce Type: replace-cross Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across th...

By Shimanto Bhowmik, Tawsif Tashwar Dipto, Md Sazzad Islam, Sheryl Hsu, Tahsin Reasat

llms computer-vision nlp benchmarks

arXiv Machine Learning

Sep 18

Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models

The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.

By Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh

llms benchmarks

arXiv Computation and Language

Oct 23

Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

The paper investigates how to estimate confidence for activation oracles—language models that translate internal activations of other models into natural language. Five confidence estimation methods are compared across four oracles from Qwen and Gemma families (8B–27B parameters) on a secret‑word recovery task. Results show that when a hypothesis space is available, scoring candidate answers directly yields the best accuracy and confidence separation; when free‑text answers are required, sampling agreement over twenty outputs provides the most calibrated confidence, while a rescaled probability from a calibration set performs comparably with fewer generations. Asking the oracle to output a numeric confidence score proves ineffective.

By Federico Torrielli, Peter Schneider-Kamp, Lukas Galke Poech

llms fine-tuning

arXiv AI

Oct 31

One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them

The study evaluates whether a single confidence threshold in typed decision interfaces treats all languages equally. Using 1,000 items in 51 languages across three vendors, it finds that English consistently achieves higher coverage—clearing the 0.9 threshold 8.6–11.7 percentage points more often than other languages—despite similar accuracy. Even when thresholds are re‑weighted or calibrated per language, the gap persists, and translating inputs into English reduces but does not eliminate it. The authors recommend reporting per‑language coverage at the operating threshold.

By Junwei Yao, Yongxi Zhou, Zike Yang, Zhijun Wang, Jianan Liu, Yanxian Chen, Wenbo Ye

arXiv Computation and Language

Oct 26

Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation

arXiv:2610.32144v1 Announce Type: new Abstract: When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, an...

By Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim

llms benchmarks

arXiv AI

1d ago

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

The paper evaluates a 307‑million‑parameter multilingual cross‑encoder, Ru‑Jev, on Russian‑language tasks, comparing its original and an augmented release that presents routing labels in multiple surface forms. Experiments show that the augmentation improves accuracy on unfamiliar ordinary words, borrowed jargon, and office slang—up to 0.24 points—while providing no benefit for abbreviations or opaque codes. The study also highlights evaluation hazards and notes that increased accuracy often coincides with reduced risk‑coverage, especially with many candidate labels.

By Vladimir Beskorovainyi