The study compares two Norwegian language models—Borealis 2, further‑trained from Gemma 4, and Gemma 4 itself—on how often they provide outdated numeric values from Norwegian statutes and regulations. Using the closed norregel benchmark, the models were evaluated on 503 primary items; Borealis 2 was outdated on 10 items and Gemma 4 on 11, a difference that is not statistically significant and falls within the pre‑registered smallest effect of interest. The only secondary measure that differed after Holm correction was the share of answers giving a value that took effect after the question date, with Gemma 4 giving such answers more often.
By Eirik Botten Nicolaysen
The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.
By Atul Anand
NitiBench+ is a Thai legal question‑answering benchmark that spans 17 domains and contains 1,670 items derived from real court decisions. Each item includes both the law text that was in force at the decision date and the current text, along with a retrieval corpus of every dated version of every Thai statute section. The benchmark evaluates models using the Thai Bar Council rubric, Bar‑standard score, and citation overlap, and it reveals that reading outdated or missing law significantly degrades performance.
By Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn Chaksangchaichot
The paper investigates how different scoring rules affect the ranking of large language models (LLMs) on the IndiaFinBench benchmark, which contains 406 expert‑annotated items covering regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. Using twelve LLMs evaluated under strict string‑matching, a cross‑model judge, and a composite regime, the authors find that model rankings vary dramatically across regimes, with no significant rank correlation (Spearman ρ = -0.273). Item‑level analysis reveals that a substantial portion of items are answered identically by all models, while a higher‑discrimination band isolates more challenging cases.
By Rajveer Singh Pall
arXiv:2610.19608v1 Announce Type: cross
Abstract: Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replicatio...
By Dmitrij \.Zatuchin
The paper introduces a new taxonomy for benchmark contamination that categorizes leakage by the mitigation it defeats—direct, derivative, temporal, distributional, and acquired—covering both training‑time and evaluation‑time scenarios. It proposes a four‑field disclosure protocol to record contamination status alongside benchmark scores, and provides a JSON schema, validator, and examples. An empirical study of 41 documents using a pre‑registered instrument shows limited reporting of contamination types and variable reliability, highlighting gaps in current disclosure practices.
By Johanna Angulo, V\'ictor Yeste, Hector Espinos-Morato