The paper reports on adapting the Qwen3 language model to Tajik using small LoRA adapters across sizes from 0.6B to 8B. While the adapters consistently lowered Tajik text loss by 0.60 to 1.45 nats per token, they did not improve Tajik reading comprehension on the Belebele benchmark; in some cases accuracy even dropped. A more extensive 8B training pipeline, including 375 million tokens of raw Tajik text and supervised fine‑tuning, produced a new model (e3) that outperformed Qwen3‑8B on Belebele by 11.4 points, though it slightly degraded English MMLU performance.
By Tohir Saidzoda, Madina Abdusalomzoda, Nasimjon Rajabov, Akhrorkhon Ikromov, Komron Giyosiev, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon Shodyeva
Saidzoda‑Tajik‑E5‑Large is a 560 M‑parameter Tajik text embedder fine‑tuned from multilingual‑e5‑large‑instruct. The authors generated 235,140 question‑passage pairs from Tajik laws, tax documents, Wikipedia and web text, trained for one epoch with hard negatives, and mixed in English and Russian pairs to preserve multilingual performance. On unseen Tajik law and tax passages the model retrieves the correct answer in the top ten for 93.6 % and 94.5 % of questions, respectively, outperforming the base model and matching larger models on similar tasks.
By Tohir Saidzoda, Nekruz Maxudov, Bedil Karimov, Abubakr Abdullozoda, Amin Jalilov, Jonibek Kishvarov, Rahmonali Sharipov, Mariyamkhon Shodyeva
The study evaluates the effectiveness of small, fine‑tuned open large language models (LLMs) for predicting bug severity in Eclipse bug reports, comparing them against traditional TF‑IDF classifiers and a fine‑tuned CodeBERT encoder. Using a balanced sample of 20,000 reports, eleven model configurations—including zero‑shot and QLoRA‑fine‑tuned Llama 3 8B and Qwen 2.5 7B—were tested on 2,000 held‑out reports, revealing that fine‑tuned LLMs achieve macro‑F1 scores around 0.52, comparable to CodeBERT, while zero‑shot LLMs lag behind TF‑IDF baselines. The analysis also highlights class‑specific performance differences and the impact of duplicate reports on model evaluation.
By Asset Kanatov, Talgar Bayan, Askar Boranbayev
arXiv:2609.16145v1 Announce Type: new
Abstract: We study a practical question: can a small correction module fix errors in a frozen language model's outputs without degrading its base capabilities? W...
By Gautam Kishore
The paper introduces SafeStart, a technique for Mixture-of-Experts (MoE) language models that mitigates the safety degradation caused by active expert reduction during serving. By running all experts for the prompt and the first eight generated tokens before switching to a reduced set, SafeStart largely restores the model’s refusal to harmful requests while preserving most of the speedup achieved by expert reduction. Experiments on several MoE models show that SafeStart maintains high offline decode speedups (≈85% of the original) and limits the increase in harmful responses to within a few percentage points, even under jailbreak prompts.
By Erfan Baghaei Potraghloo, Saeid Shokoufa, Seyedarmin Azizi, Arya Fayyazi, Souvik Kundu, Mehdi Kamal, Massoud Pedram
arXiv:2610.18420v1 Announce Type: cross
Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, m...
By Ramchand Kumaresan, Madhusudhanan G, Kaushik Kezendirane
MetroLLM-Bench is a 955‑case benchmark designed to evaluate language models as the policy layer of transit kiosks across six real metro systems, covering routing, fare calculation, disruptions, accessibility, and adversarial input. The benchmark includes 14 deterministic scoring components (Tier 1) and 8 semantic‑quality components (Tier 2), with a 75/25 split for training‑data generation and held‑out evaluation. Twenty‑six models from six vendors were tested, and a 4B Qwen 3.5 student fine‑tuned via PEFT outperformed GPT‑5.6 on Tier 1 and matched GPT‑5.4 on the combined score, while larger models offered no further improvement.
By Remco Hendriks (Continker)
Task-Aware Spectral Pruning (TASP) is a post‑training framework that tailors sparse masks to specific tasks by calibrating module‑level spectral descriptors against task‑specific ablation effects. It constructs masks that close grouped‑query‑attention and SwiGLU dependencies, routing each user turn to a single compiled mask that remains fixed during prefill and decoding. In experiments, TASP achieves a 43% active‑FLOP reduction while preserving 97.7% of the dense BF16 performance on Llama‑3‑70B, and delivers a 1.44× speedup on an A100 80GB with INT8‑weight/BF16‑compute, reducing decode latency from 45.2 to 31.3 ms/token.
By Ibne Farabi Shihab, Fariya Afrin, Sanjeda Akter, Anuj Sharma
Delta N-gram introduces a forward‑only write path into a language model’s native hashed N‑gram table, preserving the pretrained table while adding evidence through a sparse patch Δ. The method uses a 4.1 M‑parameter writer that emits gated targets per event and applies a block‑averaged delta rule to keep each row a convex combination of its old value and the new targets. Experiments on a 0.6 B prototype and Qwen3.8‑Flash‑Next show that the delta writer maintains more document‑specific memory and reduces drift compared to online gradient writes, achieving stable performance over thousands of writes.
whyItMatters":"The approach allows continual updates to a language model’s memory without altering its pretrained parameters, enabling efficient, stable, and document‑specific knowledge retention during inference."
By Hao Wu
arXiv:2608. 04160v1 Announce Type: cross Abstract: Multilingual evaluations report accuracy at a single output-token cap, but languages need different numbers of tokens to express the same content, so the cap is a hidden experimental variable.
By Ankit Goyal, Jaideep Ray
arXiv:2606.21803v2 Announce Type: replace
Abstract: Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test t...
By Xuan Ouyang, Zefan Cai, Junjie Hu
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations.
"whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
By Ye Chen, Weining Zhang