arXiv Computation and Language
Are Language Models Script-Aware?
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 25
Script Choice in LLMs: Evidence for Late-Layer Commitment
The paper examines how large language models (LLMs) encode script knowledge across their layers using logistic regression probing and logit‑lens analysis. Findings show that both the input and instructed output scripts are represented in the earliest layers, but the model commits to the actual output script only in the final layers, with intermediate representations defaulting to Latin. This two‑stage process is consistent across methods and is more pronounced in larger models, suggesting a link between model depth and script commitment.
arXiv Computation and Language
Aug 28
Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
The paper introduces PuMVR, a benchmark of 1,000 Punjabi image‑text pairs spanning three scripts—Gurmukhi, Shahmukhi, and Roman—to evaluate Vision‑Language Models (VLMs). Testing ten state‑of‑the‑art VLMs reveals a significant Script Gap: models perform well in one script but poorly in another, with accuracy differences up to 16%. The authors propose the Script Consistency Rate (SCR) as a new metric, noting it can be as low as 24.8% on their benchmark, and argue that current multilingual VLMs are not truly multi‑script.
Hugging Face Trending Papers
Jul 16
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored.
arXiv AI
Aug 5
Interpreting Black-Box Large Language Models with Sentence-Level Energy Landscapes
arXiv:2608. 02879v1 Announce Type: new Abstract: The widespread adoption of proprietary Large Language Models (LLMs) accessed strictly through closed APIs has created a critical challenge for responsible deployment: a fundamental lack of interpretability.
arXiv AI
Sep 1
Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?
arXiv:2608.29921v1 Announce Type: cross Abstract: The output of a Language Model can be tampered with \emph{while} the model is writing it. A simple test can thus be constructed by evaluating the mod...
arXiv AI
Jul 17
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
arXiv:2607. 14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias.