arXiv Computation and Language

Same Entity, Different Script: Diagnosing Cross-Script Entity Matching Failures in Language Models

arXiv:2610. 31407v1 Announce Type: new Abstract: Multilingual language models can give different identity judgments when the same people's names are written in different scripts.

arXiv Machine Learning
Oct 16

Calibration Language Shapes Block-Influence Pruning in Multilingual Language Models

The study investigates how the calibration distribution used for Block Influence (BI) pruning of Transformer blocks affects multilingual language models. It finds that English-only calibration underestimates late‑layer importance for Indian languages, leading to suboptimal pruning decisions, while a balanced multilingual calibration mitigates perplexity loss in some models but not universally. The results highlight that pruning strategies must be validated for each architecture and language set rather than relying on a single calibration source.

By Vedant Pandhare, Piyush Dhoka
arXiv AI
Sep 24

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

The paper introduces a generation benchmark for culturally specific kinship terms, evaluating five open‑weight large language models (LLMs) on Hindi, Tamil, and Korean. Unlike prior multiple‑choice tests that treat kinship understanding as a recognition task, the study prompts LLMs to generate terms across two communicative tasks and compares results to a matched option‑supported baseline. Findings show that while models like GPT‑OSS120B and Llama‑3.370B can select correct terms in over 90% and 78% of cases respectively, they produce the correct term only 36% and 24% of the time, indicating a significant evaluation‑format gap and highlighting the difficulty of culturally specific kinship generation even when relationships are explicitly stated.

By Sahil Pardasani, Madhusudan Singh
arXiv Machine Learning
Oct 7

Making COMET Comparable Across Scripts: Diagnosis and Correction of Tokeniser-Induced Script Bias in Indic MT Evaluation

arXiv:2610.08159v1 Announce Type: cross Abstract: COMET reports translation quality as a single number, and that number is routinely compared across target languages written in different scripts. Suc...

By G. L. John Salvin (Indian Institute of Technology Palakkad), Swapnil Hingmire (Indian Institute of Technology Palakkad)