← Back to all news
arXiv Computation and Language By Tanay Agrawal

A Computational Approach to Measuring Semantic Change in Sanskrit Literature

Read the original on arXiv Computation and Language →

The paper evaluates whether diachronic word embeddings can track semantic change in Sanskrit, an ancient low‑resource language with complex phonological and morphological features. A 2.7‑million‑token corpus covering four canonical periods is processed with a neural sandhi splitter and lemmatizer, and per‑period embeddings are trained. Validation against a curated set of 21 historical shifts shows that 19 shifts align with philological expectations, supporting the method’s applicability to Sanskrit.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Jul 28

BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis

arXiv:2607. 23319v1 Announce Type: cross Abstract: Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages.

By Poornima Kumaresan, Pavithra Muruganantham, Lakshmi Rajendran, Santhosh Sivasubramani
llmsnlpbenchmarks
More like this →
arXiv Computation and Language
Aug 27

Padamitra: Grounded Glossary Generation for Classical Sanskrit

The paper introduces grounded glossary generation, a structured NLP task that asks models to recover semantically meaningful Sanskrit phrases and provide translation‑grounded meanings from a sloka‑translation pair, mirroring the traditional patha commentary practice. A benchmark of 31,316 sloka‑translation‑glossary triples from the Valmiki Ramayana and Srimad Bhagavatam is built, evaluated with Jaccard for phrase recovery and Meaning Faithfulness for semantic consistency. Experiments with Gemma‑3n‑E4B, Gemma‑3‑12B, Phi‑4, and Qwen3.5‑9B show that instruction fine‑tuning outperforms prompting, and explicit segmentation further improves results, though over‑segmentation of sandhi and samasa compounds remains the main error source, highlighting morphological modeling as a key bottleneck.

By Manoj Balaji Jagadeeshan, Sai Pragnaan Marala, Pawan Goyal
llmscomputer-visionnlpfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Aug 28

Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility

The paper shows that HuggingFace’s ByteLevel pre‑tokenizer, which treats a word as a sequence of Unicode letters, splits abugida scripts at every vowel sign, creating a training‑free lower bound on tokenizer fertility. Across 26 languages, all 17 abugidas exhibit increased token counts (up to 9×), while Latin, Cyrillic, Hangul, and Han remain unchanged. The authors demonstrate that correcting the character class reduces Nepali token counts, improves model performance, and that this issue is widespread in popular HuggingFace models.

By Sajal Regmi, Siddhartha Pudasaini, Chetan Phakami Pun
llmsnlp
More like this →