arXiv Computation and Language
LAMB: A Modern Contextual Language Model for Latin
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 16
The Functionalizer: Lossless Functional Decomposition for Subword Tokenization
arXiv:2609.15991v1 Announce Type: new Abstract: Standard subword tokenizers either treat every orthographic variation of a word (such as hello, Hello, HELLO, and H\'ello) as unrelated vocabulary entr...
arXiv AI
Sep 28
Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning
The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.
arXiv Computation and Language
3d ago
LangMAP: A Language-Adaptive Approach to Tokenization
arXiv:2606.23566v3 Announce Type: replace Abstract: Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. Using one with an existing p...
arXiv Computation and Language
Oct 7
TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
arXiv:2512.23065v4 Announce Type: replace Abstract: The introduction of BERT established encoder-only transformer models as a foundational paradigm in natural language processing. Encoder-only models...
arXiv AI
Sep 2
Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages
Inspicio is an open‑vocabulary pipeline that links tokens in historical or low‑resource languages to synsets in the Open English WordNet without needing a source‑language sense inventory. It uses an instruction‑tuned LLM to generate two English translations, candidate dictionary definitions, and English lemmas, then performs hybrid retrieval combining dense definition similarity, sparse lemma matching, and Maximal Marginal Relevance re‑ranking. Evaluated on Latin, Ancient Greek, PREMOVE, and Italian data, the best configuration achieves 96% Recall@50 on a perception‑verb test set and remains competitive in out‑of‑domain and cross‑lingual scenarios.
Hugging Face Trending Papers
Sep 1
Inspicio: Open-Vocabulary, LLM-Based Sense Retrieval for Historical Languages
Word Sense Disambiguation has advanced rapidly for English and a handful of well-resourced modern languages, but it continues to assume the existence of a sense inventory and a word-to-sense mapping i...