The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replication. The same 162 requests (54 questions in English,...
arXiv:2607. 10972v1 Announce Type: new Abstract: Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop.
Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast.
arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?
The paper introduces *norregel*, a preregistered benchmark that tests language models on numeric values in Norwegian legal texts, including amounts, percentages, and age limits, each tied to a specific date. The benchmark contains 916 items extracted from Lovdata, allowing answers to be classified as correct, outdated, wrong, or uninterpretable. Two preregistered runs were conducted: the first showed mixed results across five models, while the second, involving eight models, supported the hypothesis that outdated answers are more common for values introduced after a model’s cutoff, though the robustness of this finding varied across sensitivity analyses.
The study evaluates whether aggregated brand recommendation profiles can identify the language model that generated them. Using 6,475 responses from five deployed endpoints, a character‑n‑gram classifier accurately attributes single responses to the correct system (97.84% accuracy). However, when responses are aggregated into domain‑condition units, the classifier’s performance drops to 66.53%, and a forest model misclassifies all gift‑domain units, indicating that aggregated brand behaviour does not reliably reveal the underlying system.