Kolibri is an English‑German mixture‑of‑experts transformer with 78.1 B total parameters and 3.46 B active parameters per token, released under the Apache 2.0 licence. It was trained on 24 T tokens, with German comprising over 20 % of the data, and includes supervised fine‑tuning and reinforcement learning on more than 1.2 M curated tasks. The model activates only 4.4 % of its parameters per token and, among open models evaluated, sits on the Pareto frontier of quality and serving cost for both English and German.
arXiv:2607. 09424v1 Announce Type: cross Abstract: We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English.
arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...
Mimir is a 1.78‑B parameter language model built on the Hierarchical Reasoning Model Text (HRM‑Text) architecture, which interleaves low‑level and high‑level transformer stacks. It is trained from scratch on a curated mix of 161 instruction, reasoning, tool‑use, multilingual, and synthetic datasets totaling 70.5 B tokens per epoch (103 B for the continued‑training version Mimir v1.5), using only data that meets permissibility criteria. Across 20 benchmarks in English, Danish, and Math & Code, Mimir v1.5 outperforms all other sub‑2 B models and matches or exceeds the performance of larger 9 B‑parameter models, while memorisation audits show low risk and inference‑time ablations reveal graceful degradation when adjusting low‑level stack applications.
arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.
Public institutions hold large volumes of sensitive documents and support tickets that cannot leave the premises, ruling out cloud-hosted language models entirely. We report on RAGAL, a retrieval-augmented assistant for the technical-support team of AFIR, the Romanian Agency for Financing Rural Investments, built and operated under three hard constraints: zero data egress (no external API calls, even for synthetic data), a read-only mandate (the assistant drafts, humans execute), and a single 8 GB consumer laptop as the only development and training machine.
arXiv:2607.09885v3 Announce Type: replace Abstract: We present Index-1.9B, a series of open small language models developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundatio...
arXiv:2601. 22146v2 Announce Type: replace-cross Abstract: Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data.
arXiv:2610.14428v1 Announce Type: new Abstract: Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpor...
arXiv:2608. 12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation.
The paper introduces a three‑stage training pipeline that builds compact, efficient dense retrievers without requiring ground‑truth relevance labels. Using cross‑lingual alignment, relational knowledge distillation, and contrastive fine‑tuning, the authors develop PolDense (six Polish models ranging from 17 M to 1 B parameters) and EuroDense (a 435 M‑parameter model covering nine European languages). Extensive evaluation on 41 Polish and 150 multilingual tasks shows that PolDense‑1B outperforms larger retrievers up to 9 B parameters, while EuroDense leads in task‑averaged and language‑averaged performance among models below 1 B parameters.
arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.
The paper reviews data strategies for large language models, emphasizing that data strategy—selecting, weighting, and scheduling training data—determines model capabilities within a fixed data budget. It treats high‑quality data as having training‑dependent utility, varying with the target objective, model, and training state. The authors organize existing methods by decision granularity, model dependence, and temporal adaptivity, comparing their computational costs and empirical support, and note limited evidence for transfer across model scales and end‑to‑end cost parity.