arXiv Computation and Language

Different Corpora, Different Skills? A Paired Study of Small Language Models

The study compares two small‑scale pretraining corpora—1911 Encyclopaedia Britannica and BabyLM 2026—using matched 10‑million and 37‑million‑word budgets. Models trained on BabyLM 2026 consistently outperform those trained on Britannica on linguistic acceptability tasks, while Britannica models excel at Entity Tracking, a synthetic state‑tracking benchmark. These divergent skill profiles appear across teacher and student models, both budgets, and persist even after controlling for training exposure, indicating that corpus choice can shape relative strengths in small language models.

arXiv AI
Oct 9

Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models

The paper reports that fine‑tuned small language models (SLMs) can outperform large frontier models in annotating grammar concepts from learner–tutor transcripts. By training a 0.8B Qwen3.5 model on teacher‑generated data and embedding the annotation rules into adapter weights, the authors deploy an efficient end‑to‑end grammar mastery tracker that matches or exceeds the performance of 4B reference models and GPT‑5.4/5.6 on two human‑curated benchmarks. The lightweight model also cuts serving costs by about 16× and boosts learner engagement and key business metrics in an online experiment.

By Marjan Celikik, Ana Peleteiro Ramallo, Javier Morales
arXiv Computation and Language
Aug 25

LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data

LuxIT is a monolingual instruction‑tuning dataset for Luxembourgish, created by synthesizing instruction‑answer pairs from native texts using the DeepSeek‑R1‑0528 model and a quality‑assurance LLM‑as‑judge process. The resulting 227,507 high‑quality pairs were used to fine‑tune 14 LLMs (≤15 B parameters), yielding an average accuracy increase of +5.37 percentage points on standardized Luxembourgish proficiency exams and improvements in macro‑averaged F1 on nine of the fourteen downstream NLP tasks. These findings demonstrate that synthetic monolingual data can effectively enhance LLM performance in low‑resource languages and reveal the complex relationship between exam performance and practical NLP gains.

By Julian Valline, Cedric Lothritz, Siwen Guo, Jordi Cabot
arXiv Machine Learning
Aug 5

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.

By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y
arXiv AI
Sep 18

QVAC Genesis III: A Large-Scale, High-Quality Open Synthetic STEM Corpus for Efficient Language Model Pre-Training

QVAC Genesis III is a 191.43 B‑token synthetic STEM corpus covering 19 domains and multiple difficulty levels, created through a dual generation strategy that uses a weak edge‑scale student model to generate corrective explanations and contrastive reasoning. The authors evaluate the corpus with an LLM‑as‑a‑parser protocol and demonstrate that 1.7 B‑parameter models trained on QVAC Genesis III outperform those trained on Cosmopedia‑v2 and the Cosmo‑1B model on ARC, GPQA Diamond, and MMLU STEM benchmarks, achieving up to +28.57% improvement on ARC‑E and a 99.45% valid answer rate.

By Davide Vitabile, N. Ranjan, Akshay Nambiar, Kamal K. Gupta, Amril Nazir
arXiv Machine Learning
Sep 16

Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT

The study examines whether domain‑adaptive continued pretraining (DAPT) on a learner‑writing corpus (EFCAMDAT) can enhance transformer‑based automated essay scoring (AES) for English proficiency tests. Researchers applied DAPT to BERT, RoBERTa, and DistilBERT and compared the adapted models with their original checkpoints on the FCE and IELTS datasets, evaluating both in‑domain scoring and few‑shot cross‑dataset transfer. Results show that full‑corpus DAPT yields mixed effects, while proficiency‑specific DAPT often outperforms full‑corpus DAPT and sometimes even the non‑adapted baseline, though benefits vary by proficiency composition and encoder architecture and do not consistently transfer across tests.

By Duy Anh Nguyen
arXiv Machine Learning
Jun 2

GottBERT: a pure German Language Model

arXiv:2012. 02110v2 Announce Type: replace-cross Abstract: Pre-trained language models have significantly advanced natural language processing (NLP), especially with the introduction of BERT and its optimized version, RoBERTa.

By Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, Martin Boeker
arXiv AI
Oct 9

Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

The study investigates how exposing large language models to Arabic translations of English benchmark tasks can hide data contamination from standard English-only detection methods. By testing four instruction-tuned models with increasing Arabic exposure, the authors find that traditional probes like TS-Guessing and Min-K%++ fail to detect contamination, yet English performance still improves. They propose a new Translation-Aware Contamination Detection (TACD) method that uses cross-lingual consistency and choice reordering to reveal contamination effects that are otherwise invisible to English-only diagnostics.

By Chaymaa Abbas, Nour Shammaa, Mariette Awad
arXiv Computation and Language
Oct 5

Writerslogic at the CLEF 2026 SimpleText Track: Multi-Candidate LLM Simplification and Stacked Complexity Spotting

The Writerslogic team participated in the CLEF 2026 SimpleText shared task, tackling both text simplification (Task 1) and complexity spotting (Task 2). For simplification, they built a multi‑candidate pipeline with GPT‑4o‑mini, selecting the best candidate via a reference‑free heuristic, and their Claude Sonnet 4 submission achieved a SARI of 47.43 and BLEU of 14.21, ranking third overall on the Task 1 leaderboard. For complexity spotting, they fine‑tuned a DeBERTa‑v3‑large NLI model on 350 K labeled pairs, achieving a macro F1 of 0.8081 (0.8085 in an ensemble) on binary over‑generation identification and 0.804 accuracy on multi‑class error classification, placing them second among unique teams.

By David L. Condrey