The paper introduces the Bashkir Web Corpus, a 71,567‑document collection (46.9 M tokens) and three labeled datasets for text classification, and benchmarks eight large language models (from 82 M to 8 B parameters) on binary, multiclass, and multilabel tasks. It evaluates three adaptation strategies—full fine‑tuning, QLoRA, and in‑context learning—alongside continual pretraining, finding that QLoRA on 3.8–8 B models outperforms encoder‑only fine‑tuning for binary classification, while continual pretraining’s impact varies by model and task. The study also reports tokeniser fertility differences, shows that Cyrillic tokenisers pretrained on Russian perform best, and demonstrates that LLM‑generated ratings correlate with native judgments for grammar and coherence but not for language correctness, underscoring the need for native evaluation.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv:2610.09152v1 Announce Type: new Abstract: Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation....
arXiv:2608.28809v1 Announce Type: new Abstract: Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, a...
arXiv:2610.33813v1 Announce Type: new Abstract: Fine-tuning a language model for classification involves many design decisions. Comparisons of low-rank adaptation (LoRA) with full fine-tuning are com...
The study evaluates the effectiveness of small, fine‑tuned open large language models (LLMs) for predicting bug severity in Eclipse bug reports, comparing them against traditional TF‑IDF classifiers and a fine‑tuned CodeBERT encoder. Using a balanced sample of 20,000 reports, eleven model configurations—including zero‑shot and QLoRA‑fine‑tuned Llama 3 8B and Qwen 2.5 7B—were tested on 2,000 held‑out reports, revealing that fine‑tuned LLMs achieve macro‑F1 scores around 0.52, comparable to CodeBERT, while zero‑shot LLMs lag behind TF‑IDF baselines. The analysis also highlights class‑specific performance differences and the impact of duplicate reports on model evaluation.
Pearl-Chat is an 8.5‑million‑parameter causal decoder‑only transformer built from first principles in JAX and Flax NNX, specifically engineered for the low‑resource Bantu language Luganda. It uses a byte‑level BPE tokenizer trained from scratch on curated Luganda corpora, achieving a two‑fold compression improvement over larger models like LLaMA‑3 and Gemma. The model is trained in two stages—pre‑training on diverse Luganda text and instruction fine‑tuning on dialogues—reaching a validation perplexity of 1.20 and high ROUGE‑L and BLEU scores on a held‑out diagnostic benchmark, with all weights, tokenizers, and code released openly.
Zuse is a family of decision models (4B, 9B, and 27B) that predict probabilities for a fixed set of options across 27 European languages by training a low‑rank adapter and a pointer head on frozen open‑weight bases. The 27B model achieves 87.5 % accuracy on a broad European test suite of 11 sources, 9 task groups, and 27 languages, with a pooled expected calibration error of 0.016. Post‑hoc analysis reveals that miscalibration is task‑dependent, that automated decision share aligns more with chance‑corrected skill than raw accuracy, and that performance on languages outside the base model’s documented list is consistently lower.
arXiv:2608. 05162v1 Announce Type: cross Abstract: Pooling is a consequential but under-examined design choice in decoder-only concept representation work: practitioners must collapse token-level hidden states into a passage-level vector, yet no shared protocol exists for comparing this choice across concepts, models, and tasks.
arXiv:2607. 03801v1 Announce Type: cross Abstract: We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices.
arXiv:2508.11857v3 Announce Type: replace-cross Abstract: Tokenization remains a persistent bottleneck in language modeling, especially when vocabulary learning is limited by whitespace boundaries. W...
arXiv:2605. 04539v4 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO), the efficient alternative to PPO-based RLHF, falls short on knowledge-intensive generation: standard preference signals from human annotators or LLM judges exhibit a systematic verbosity bias that rewards fluency over logical correctness.
The paper introduces CAFE, a method that confines Centered Kernel Alignment (CKA) feature distillation to language‑routed adapters in compact multilingual ASR models. By restricting the CKA gradient to adapters that English bypasses, CAFE mitigates the trade‑off where CKA improves non‑English WER at the cost of English performance. On the Open ASR Leaderboard, CAFE reduces average WER from 8.55% to 8.02% and outperforms an adapter‑only capacity control, achieving competitive results across 48 test sets in 24 languages with 254M parameters.