arXiv AI
Mimir: Large-scale Multilingual Concept Modeling
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 15
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
The paper proposes a modular tokenizer framework for multilingual large language models, allowing the creation of language‑specific subtokenizers that match monolingual compression quality. It introduces a pretraining strategy that samples these subtokenizers to limit predictions to relevant vocabularies, enabling efficient training and inference. This approach reduces memory usage and speeds up inference without compromising performance.
arXiv AI
Oct 27
Conceptizer: Next-Token Prediction Is Not All You Need
Conceptizer is a training framework that augments next-token prediction with a reward for meaning by using a fixed vocabulary of about 9,000 concepts derived from contextual synonyms. It maps each token distribution to a concept distribution, sharing the reward for the target meaning across its possible wordings, thereby providing a concept-level training signal without adding model parameters. Applied to 1B–8B parameter models, Conceptizer achieves lower perplexity at concept-target positions, assigns higher probability to contextual synonyms, and improves downstream performance on knowledge and sentiment tasks while remaining comparable on commonsense tasks.
Hugging Face Trending Papers
Sep 14
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding...
arXiv Computation and Language
Oct 27
LangMAP: A Language-Adaptive Approach to Tokenization
arXiv:2606.23566v3 Announce Type: replace Abstract: Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. Using one with an existing p...
arXiv AI
Aug 21
When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models
arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.
arXiv Computation and Language
Sep 22
English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
arXiv:2604.13286v2 Announce Type: replace Abstract: Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing...