The paper presents an end‑to‑end sequence‑to‑sequence approach for correcting Tamil spelling and grammar errors, leveraging progressively fine‑tuned transformer models (mT5‑small and mBART‑50). Using a synthetic corpus of 657,720 noisy‑clean sentence pairs across ten error categories, the authors introduce a four‑stage training schedule that targets surface noise, contextual grammar, single‑site sandhi, and multi‑site cross‑word sandhi. The best model, mBART‑50 v5, achieves 69.3% exact‑match accuracy on a balanced diagnostic set, with notable gains in sandhi (87.5%) and subject‑verb agreement (43.5%) accuracy, while also revealing a precision‑recall trade‑off for sandhi corrections.
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
arXiv:2609.21595v1 Announce Type: new Abstract: In-context learning using Large Language Models (LLMs) offers a compelling path to training-free post-OCR correction, yet its effectiveness for Devanag...
arXiv:2605. 20712v2 Announce Type: replace-cross Abstract: Automatic speech recognition replaces typing only when correction costs less than manual entry - a threshold determined by error types, not counts: fixing a misrecognized domain term costs far more than inserting a comma.
arXiv:2610.18420v1 Announce Type: cross Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, m...
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
arXiv:2609.36131v1 Announce Type: new Abstract: Sinhala Sandhi splitting recovers the constituent words or morphemes hidden inside a phonologically merged surface form. The task is important for Sinh...
The paper introduces a black‑box, inference‑time diagnostic for low‑resource Automatic Post‑Editing (APE) that distinguishes whether poor performance is due to insufficient training data or inconsistent training signals. By varying an edit‑distance penalty and analyzing the resulting TER‑vs‑λ curve and confidence‑based constraint ordering, the authors identify two failure modes—Binary Collapse and Confident Miscalibration—across multiple language pairs. The diagnostic also suggests practical next steps, such as applying a static constraint for immediate accuracy gains, and the authors release new English‑Sinhala and English‑Tamil APE datasets with accompanying code.
arXiv:2608.23120v1 Announce Type: cross Abstract: Pnar, an Austroasiatic language spoken by approximately 0.4 million people in the Jaintia Hills of Meghalaya, lacks the digital corpora and natural l...
The paper introduces a token‑level extension of Omni‑Temporal Classification (OTC) for automatic speech recognition, allowing unsupported tokens to be bypassed while preserving supervision for the rest of the word. Across 19 languages and three corpora, this token‑level OTC consistently outperforms standard CTC, achieving the lowest mean word error rate on every dataset and a 9.45% average relative WER reduction. A predictive‑entropy‑indexed schedule replaces epoch‑based relaxation, reducing training‑length dependence while maintaining performance.
Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and evaluation across tasks and language pairs on one footing. We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into \indicqe: $126{,}754$ instances over nine directional pairs, with up to four label types aligned on the same segment, a direct assessment, a human post-edit, word-level OK/BAD tags and an error explanation, and a test set stratified over four difficulty axes.
Pearl-Chat is an 8.5‑million‑parameter causal decoder‑only transformer built from first principles in JAX and Flax NNX, specifically engineered for the low‑resource Bantu language Luganda. It uses a byte‑level BPE tokenizer trained from scratch on curated Luganda corpora, achieving a two‑fold compression improvement over larger models like LLaMA‑3 and Gemma. The model is trained in two stages—pre‑training on diverse Luganda text and instruction fine‑tuning on dialogues—reaching a validation perplexity of 1.20 and high ROUGE‑L and BLEU scores on a held‑out diagnostic benchmark, with all weights, tokenizers, and code released openly.
arXiv:2610.15705v1 Announce Type: cross Abstract: India's Eighth Schedule recognises 22 languages, yet parallel data is uneven across them, and many language pairs, especially among lower-resource la...