arXiv Computation and Language

Dictionary-Constrained Grapheme-to-Phoneme for Unsegmented Languages from LLM-Annotated Data

The paper introduces a context‑aware neural grapheme‑to‑phoneme (G2P) system for unsegmented languages like Japanese, using a discriminative conditional random field over a word lattice built from dictionaries. It addresses data scarcity by generating over two million sentences with large language models. Experiments show the method surpasses traditional morphological analyzers and neural sequence models, achieving 99.62% target word reading accuracy and very low phoneme error rates on the Joyo‑Kanji‑Yomi benchmark.

Hugging Face Trending Papers
6d ago

Dictionary-Constrained Grapheme-to-Phoneme for Unsegmented Languages from LLM-Annotated Data

The paper introduces a context‑aware neural grapheme‑to‑phoneme (G2P) system for unsegmented languages like Japanese, combining a discriminative conditional random field over a dictionary‑based word lattice with large language model‑generated training data. By generating over two million synthetic sentences, the method addresses data scarcity and achieves superior performance compared to traditional morphological analyzers and neural sequence models. On the Joyo‑Kanji‑Yomi benchmark, it attains 99.62% target word reading accuracy, 0.32% target word phoneme error rate, and 0.14% sentence phoneme error rate.

arXiv Computation and Language
2d ago

Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining

arXiv:2609. 21362v1 Announce Type: new Abstract: Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and often requiring large vocabularies.

By Nghia Hieu Nguyen, Thai Bao Huynh, Binh-An Dinh-Le, Phu Gia Hoang, Dat Tien Nguyen, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen
arXiv AI
Jun 9

End-to-End Training for Discrete Token LLM based TTS System

arXiv:2606. 09234v1 Announce Type: cross Abstract: Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently.

By Changfeng Gao, Yong Ren, Jun Yuan, Ye Bai, Zhao You, ShiDong Shang
arXiv Computation and Language
Sep 15

To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

The paper proposes a modular tokenizer framework for multilingual large language models, allowing the creation of language‑specific subtokenizers that match monolingual compression quality. It introduces a pretraining strategy that samples these subtokenizers to limit predictions to relevant vocabularies, enabling efficient training and inference. This approach reduces memory usage and speeds up inference without compromising performance.

By Franck Signe, Hippolyte Pilchen, Fran\c{c}ois Yvon, \'Edouard Grave
Hugging Face Trending Papers
Jun 24

Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis

While large language model (LLM)-based text-to-speech (TTS) systems have achieved high-quality speech synthesis, most existing systems focus on English and Chinese. Japanese, however, remains under-explored, and its unique linguistic challenges, such as widespread context-dependent kanji polyphony, have yet to be adequately tackled.

arXiv Computation and Language
Aug 28

Scaling phoneme-based TTS augmentation for ASR: A unified pipeline and controlled study

The paper introduces a unified phoneme‑based TTS‑to‑ASR augmentation pipeline that uses a multilingual TTS model with language‑ID conditioning and incorporates grapheme‑to‑phoneme conversion, reference‑speech filtering, and candidate‑text selection. It proposes phoneme‑frequency‑guided selection (PFGS) to rank sentences based on phoneme frequencies from real ASR labels, and demonstrates that random augmentation and PFGS both improve ASR performance across Arabic, French, Italian, and Portuguese test sets, with PFGS yielding up to a 19.3% relative WER reduction. The study also shows that filtering reference speech can further lower WER by up to 0.59 points on certain datasets.

By Zhen Wang, TianRui Wu, RongQi Han, Hao Wu, Wei Liang