arXiv AI
MixLID: Token-Level Language Identification in Mixed-Language Text
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 11
IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
IndicTriMix presents a new benchmark and models for token‑level language identification in tri‑language code‑mixed text involving Hindi, Gujarati, and Bengali. The authors reformulate the task as sequence labeling and fine‑tune transformer models MuRIL and XLM‑RoBERTa, evaluating them on manually annotated test sets. They also introduce two code‑mixing generation methods using parallel sentences and release the datasets and fine‑tuned models for public use.
arXiv Computation and Language
Sep 11
What Language is This? Ask Your Tokenizer
The paper introduces UniLID, a lightweight language identification method that uses the UnigramLM tokenization algorithm to predict a string’s language by evaluating which language’s unigram distribution best explains the text. UniLID is data‑ and compute‑efficient, allows incremental addition of new languages without retraining, and can be integrated into existing tokenization pipelines. Experiments show competitive performance against baselines such as fasttext, GlotLID‑M, and CLD3, achieving 69% accuracy with five labeled samples per language and 89% with 25, and delivering significant gains on fine‑grained dialect identification.
arXiv Computation and Language
Sep 7
Improving Language Identification for Code-Switched Utterances with Integer Linear Programming
The paper addresses the challenge of identifying code‑switched utterances in language identification systems. It revisits the MaskLID approach, highlighting its overreliance on word‑level language association scores, and reformulates its optimization as an Integer Linear Program to incorporate clear, interpretable constraints. These enhancements significantly improve performance across ten diverse languages on code‑switching benchmarks, with the authors releasing code and data for reproducibility.
arXiv Machine Learning
Aug 4
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
arXiv:2604. 03532v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult.
arXiv Computation and Language
Oct 5
Automatic register identification for the open web using multilingual deep learning
arXiv:2406. 19892v5 Announce Type: replace Abstract: This article presents multilingual deep learning models for identifying web registers -- text varieties such as news reports and discussion forums -- across 16 languages.
arXiv AI
Jul 10
Less Is More: Reducing Token Counts Without Compromising Performance
arXiv:2506. 15138v2 Announce Type: replace-cross Abstract: Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.