← Back to all news
arXiv Machine Learning By Ionut Adrian Rotaru

Neurograf v1: An End-to-End Romanian-First Language Model Family

Read the original on arXiv Machine Learning →

Neurograf v1 is an experimental end‑to‑end Romanian‑first language‑model pipeline that includes a 635 GB Romanian corpus (RoSfera), an 80k byte‑level BPE tokenizer, and five decoder‑only models ranging from 160 M to 6 B parameters trained on 131 B tokens. The pipeline also produces three instruction‑tuned assistants and two contrastively trained embedding models, all trained on ELI‑NP/IFIN‑HH infrastructure. Evaluation shows consistent scaling of validation loss and improved Romanian factual and cultural knowledge with larger models, achieving 69.89 % accuracy on the 1,000‑question WWTBM benchmark for the 6 B model. "whyItMatters":"The study demonstrates that it is feasible to train Romanian‑first language models from scratch, highlighting practical challenges such as diacritic preservation and tokenization efficiency while providing a unified training pipeline for base, instruction‑following, and embedding models."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Aug 25

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.

By Julian Valline, Cedric Lothritz, Siwen Guo, Jordi Cabot
llmsnlpfine-tuningbenchmarks
More like this →
arXiv Computation and Language
Sep 1

Manac\'a-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation

Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.

By Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio Andre Machado Porto
llmsnlpbenchmarks
More like this →
arXiv Computation and Language
Sep 14

Parameter-Efficient Retrievers for Polish and European Languages

The paper introduces a three‑stage training pipeline that builds compact, efficient dense retrievers without requiring ground‑truth relevance labels. Using cross‑lingual alignment, relational knowledge distillation, and contrastive fine‑tuning, the authors develop PolDense (six Polish models ranging from 17 M to 1 B parameters) and EuroDense (a 435 M‑parameter model covering nine European languages). Extensive evaluation on 41 Polish and 150 multilingual tasks shows that PolDense‑1B outperforms larger retrievers up to 9 B parameters, while EuroDense leads in task‑averaged and language‑averaged performance among models below 1 B parameters.

By S{\l}awomir Dadas, Rafa{\l} Po\'swiata, Ma{\l}gorzata Gr\k{e}bowiec, Micha{\l} Pere{\l}kiewicz
llmsragfine-tuningefficiencysafety
More like this →