arXiv:2609.14817v1 Announce Type: new Abstract: In Yor\`ub\'a, pitch alone separates \d{o}k\d{o} (husband, Mid), \d{o}k\d{\`o} (vehicle, Low), and \d{o}k\d{\'o} (hoe, High) -- the diacritics ARE the...
arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.
The paper shows that HuggingFace’s ByteLevel pre‑tokenizer, which treats a word as a sequence of Unicode letters, splits abugida scripts at every vowel sign, creating a training‑free lower bound on tokenizer fertility. Across 26 languages, all 17 abugidas exhibit increased token counts (up to 9×), while Latin, Cyrillic, Hangul, and Han remain unchanged. The authors demonstrate that correcting the character class reduces Nepali token counts, improves model performance, and that this issue is widespread in popular HuggingFace models.
arXiv:2607. 24276v1 Announce Type: cross Abstract: Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.
Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages.
arXiv:2609.14542v1 Announce Type: new Abstract: Neyshekar is presented as an open Persian read-speech corpus designed for coverage of both formal and informal language, named entities, and longer utt...
The study evaluates phonologically informed tokenization for German speech recognition using a wav2vec 2.0 backbone fine‑tuned with CTC. Three tokenizer families—pretrained multilingual characters, data‑driven BPE, and phonologically informed units from syllabification and grapheme‑to‑phoneme conversion—are compared across 40 fine‑tunes on three German test sets (in‑domain read, dialectal spontaneous, and standard spontaneous speech). Results show that phonologically informed tokenizers match BPE and character baselines in‑domain, but under domain shift their performance depends on vocabulary size rather than linguistic variation, with syllable‑aware tokenization benefiting dialectal and spontaneous speech at small vocabularies; a phoneme‑level confusion analysis indicates that the acoustic encoder dominates error patterns.
arXiv:2605. 20712v2 Announce Type: replace-cross Abstract: Automatic speech recognition replaces typing only when correction costs less than manual entry - a threshold determined by error types, not counts: fixing a misrecognized domain term costs far more than inserting a comma.
BanglaKontho is a newly released 20‑hour single‑speaker Bangla text‑to‑speech corpus derived from professional audiobook recordings, comprising 7,050 segmented utterances with verified transcripts at 24 kHz. The project also provides a reusable Bangla text normalizer that handles Bangladeshi‑style digit grouping, currency and date expressions, Danda punctuation, and Unicode normalization, along with the full preprocessing pipeline. A baseline MB‑iSTFT‑VITS model trained from scratch on this corpus achieves a 9.5 % WER and 4.46 naturalness MOS, outperforming the same architecture retrained on the smaller 12‑hour IndicTTS‑Bn corpus.
arXiv:2608.28916v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems are commonly evaluated with word error rate (WER), yet many voice workflows depend on exact written values f...
arXiv:2609.12960v1 Announce Type: new Abstract: Sanskrit fuses case, number, person and tense into word endings and chains clauses into compounds, so it is information-dense per word. Whether that de...
arXiv:2609.14967v1 Announce Type: new Abstract: Quranic text is distributed in two orthographic forms that are byte-level distinct: the Uthmani script used in every printed mushaf, and the Standard (...