arXiv Computation and Language
The paper introduces *norregel*, a preregistered benchmark that tests language models on numeric values in Norwegian legal texts, including amounts, percentages, and age limits, each tied to a specific date. The benchmark contains 916 items extracted from Lovdata, allowing answers to be classified as correct, outdated, wrong, or uninterpretable. Two preregistered runs were conducted: the first showed mixed results across five models, while the second, involving eight models, supported the hypothesis that outdated answers are more common for values introduced after a model’s cutoff, though the robustness of this finding varied across sensitivity analyses.
The study compares two Norwegian language models—Borealis 2, further‑trained from Gemma 4, and Gemma 4 itself—on how often they provide outdated numeric values from Norwegian statutes and regulations. Using the closed norregel benchmark, the models were evaluated on 503 primary items; Borealis 2 was outdated on 10 items and Gemma 4 on 11, a difference that is not statistically significant and falls within the pre‑registered smallest effect of interest. The only secondary measure that differed after Holm correction was the share of answers giving a value that took effect after the question date, with Gemma 4 giving such answers more often.
By Eirik Botten Nicolaysen
The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.
By Atul Anand
NitiBench+ is a Thai legal question‑answering benchmark that spans 17 domains and contains 1,670 items derived from real court decisions. Each item includes both the law text that was in force at the decision date and the current text, along with a retrieval corpus of every dated version of every Thai statute section. The benchmark evaluates models using the Thai Bar Council rubric, Bar‑standard score, and citation overlap, and it reveals that reading outdated or missing law significantly degrades performance.
By Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn Chaksangchaichot
The paper investigates how different scoring rules affect the ranking of large language models (LLMs) on the IndiaFinBench benchmark, which contains 406 expert‑annotated items covering regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. Using twelve LLMs evaluated under strict string‑matching, a cross‑model judge, and a composite regime, the authors find that model rankings vary dramatically across regimes, with no significant rank correlation (Spearman ρ = -0.273). Item‑level analysis reveals that a substantial portion of items are answered identically by all models, while a higher‑discrimination band isolates more challenging cases.
By Rajveer Singh Pall
arXiv:2610.19608v1 Announce Type: cross
Abstract: Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replicatio...
By Dmitrij \.Zatuchin
The paper introduces a new taxonomy for benchmark contamination that categorizes leakage by the mitigation it defeats—direct, derivative, temporal, distributional, and acquired—covering both training‑time and evaluation‑time scenarios. It proposes a four‑field disclosure protocol to record contamination status alongside benchmark scores, and provides a JSON schema, validator, and examples. An empirical study of 41 documents using a pre‑registered instrument shows limited reporting of contamination types and variable reliability, highlighting gaps in current disclosure practices.
By Johanna Angulo, V\'ictor Yeste, Hector Espinos-Morato
Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replication. The same 162 requests (54 questions in English,...
arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.
By Koyar Afrasyab
Fiorillo v0.5 is an open 4‑billion‑parameter model that reads a randomized trial article (cut to 6,144 tokens) and predicts whether an intervention significantly increased, decreased, or had no significant effect on an outcome. Fine‑tuned on 1,431 license‑clean training articles, it achieved an expected calibration error of 0.0168, log loss 0.8603 lower than the prior, and macro‑F1 of 0.9248 on a public test split of 1,218 prompts. The model’s release, licensed under Apache 2.0, met all four pre‑registered criteria on the Open Science Framework and was validated against the evaluated predictions.
By Johann Emmanuel Li
The study evaluates a label‑free rubric‑clarification procedure that rewrites only the most contested sub‑criteria of a rubric, aiming to reduce disagreement among large‑language‑model (LLM) judges. Across six rubric conditions and three datasets (SummEval, HealthBench, USR Topical‑Chat), the method increased inter‑judge agreement on SummEval by 0.069 Krippendorff’s α, but did not improve agreement with human expert ratings. In other datasets, neither inter‑judge nor human agreement met the preregistered effect criteria, and human agreement was actually lower after editing contested clauses compared to random edits.
By Sangdo Han, Moonsoo Kim
arXiv:2611.03171v1 Announce Type: cross
Abstract: On July 1, 2025, Vietnam reduced its 63 provinces to 34, abolished its 696 districts, and replaced its 10,035 commune-level units (wards) with 3,321....
By Phan Binh Nguyen Lam, Nguyen Thai Anh
arXiv:2608.31016v1 Announce Type: cross
Abstract: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the n...
By Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris