Zuse is a family of decision models (4B, 9B, and 27B) that predict probabilities for a fixed set of options across 27 European languages by training a low‑rank adapter and a pointer head on frozen open‑weight bases. The 27B model achieves 87.5 % accuracy on a broad European test suite of 11 sources, 9 task groups, and 27 languages, with a pooled expected calibration error of 0.016. Post‑hoc analysis reveals that miscalibration is task‑dependent, that automated decision share aligns more with chance‑corrected skill than raw accuracy, and that performance on languages outside the base model’s documented list is consistently lower.
arXiv:2610.17205v1 Announce Type: new Abstract: LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a ty...
arXiv:2609.37647v1 Announce Type: cross Abstract: Jev is a commercial System One model from TypeSafe AI that does not generate text: given a state and typed questions, it returns a choice from fixed...
The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.
arXiv:2610.09152v1 Announce Type: new Abstract: Multilingual LLM benchmarks omit Slovak, a morphologically rich West Slavic language of five million speakers, or cover it only by machine translation....
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
Manacá-1B is a 1.72‑billion‑parameter, open decoder‑only language model trained from scratch for Brazilian Portuguese, released with a fully containerized, reproducible training pipeline and complete logs. The authors evaluate it against nine open baselines on four Portuguese benchmarks, reporting standard errors and paired significance tests, and find that Manacá-1B outperforms smaller models on LAMBADA‑PT while remaining competitive on commonsense completion. They also uncover a tokenizer‑related evaluation pitfall that can drastically lower accuracy and provide a simple fix, releasing all code, logs, and corrected tokenizer for full reproducibility.
arXiv:2607. 09424v1 Announce Type: cross Abstract: We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English.
EDU-Jev is a family of single‑pass decision models (0.8B, 2B, and 4B parameters) designed for educational tasks that require selecting from a fixed set of options. The models output a probability distribution over predefined choices without generating text, using a pointer head to score each option. In twelve education benchmarks, EDU‑Jev 4B achieved 67.6% accuracy, outperforming five commercial decision APIs by 9.4 percentage points and the best open 4B model by 18.7 points, while the 2B version can process about 150 decisions per second on a single H100 GPU.
The study evaluates whether a single confidence threshold in typed decision interfaces treats all languages equally. Using 1,000 items in 51 languages across three vendors, it finds that English consistently achieves higher coverage—clearing the 0.9 threshold 8.6–11.7 percentage points more often than other languages—despite similar accuracy. Even when thresholds are re‑weighted or calibrated per language, the gap persists, and translating inputs into English reduces but does not eliminate it. The authors recommend reporting per‑language coverage at the operating threshold.
The paper evaluates confidence‑based routing of small‑to‑large language models across multiple tasks, languages, and model families, covering 72 configurations and over 42,000 predictions. It finds that confidence routing improves performance on reading comprehension and topic classification datasets, but offers little benefit on a knowledge‑reasoning benchmark where small models remain confidently wrong. The study also shows that global confidence routing can create large language‑specific performance gaps, which equal‑quota routing mitigates with minimal overall accuracy loss.
arXiv:2608. 15964v1 Announce Type: cross Abstract: Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt.
arXiv:2607. 20526v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in settings where fluent but incorrect answers can be costly.