arXiv:2610.19608v1 Announce Type: cross Abstract: Counts of which brands a large language model names are useful only if they hold long enough to act on. We test this with a pre-registered replicatio...
The paper introduces *norregel*, a preregistered benchmark that tests language models on numeric values in Norwegian legal texts, including amounts, percentages, and age limits, each tied to a specific date. The benchmark contains 916 items extracted from Lovdata, allowing answers to be classified as correct, outdated, wrong, or uninterpretable. Two preregistered runs were conducted: the first showed mixed results across five models, while the second, involving eight models, supported the hypothesis that outdated answers are more common for values introduced after a model’s cutoff, though the robustness of this finding varied across sensitivity analyses.
arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?
arXiv:2607. 10972v1 Announce Type: new Abstract: Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop.
arXiv:2609.22478v1 Announce Type: cross Abstract: Behavioural evaluations of hosted language models can vary because the evaluated service, the measurement instrument, or both differ across runs. We...
Appending a two-word confirmation tag to a decision question -- "Is X the better choice? " versus "X is the better choice, right?
Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast.
The study evaluates whether aggregated brand recommendation profiles can identify the language model that generated them. Using 6,475 responses from five deployed endpoints, a character‑n‑gram classifier accurately attributes single responses to the correct system (97.84% accuracy). However, when responses are aggregated into domain‑condition units, the classifier’s performance drops to 66.53%, and a forest model misclassifies all gift‑domain units, indicating that aggregated brand behaviour does not reliably reveal the underlying system.
The paper reports on a series of experiments examining how different forms of directives—such as record pointers, criteria, or combinations—affect an agent’s choice of archived source records when it inherits six one-line memories. Across twelve registered studies involving 14,760 attempts on a single instrument lineage, the authors measured the impact of various directive formats on six direct-provider models, nine OpenRouter-served models, and several Claude and Opus 5 models, noting differences in performance metrics and replication outcomes. The results are purely descriptive, detailing the effects of exact edits on fixed panels with registered intervals and no claim of underlying mechanisms.
The study audits large language model (LLM) outputs by measuring how well repeated queries recover a collected set of responses versus the full set of possible outputs. Using sample-based rarefaction on 4,500 responses from 50 buying questions across six configurations, the authors find historical-dictionary median recovery rates between 92.6% and 95.2%, which drop to 89.5%–94.7% after re‑adjudicating all candidate strings. Additional analyses with Gemini 3.1 Pro annotations and matched roster data confirm that recovery percentages vary with extraction methods, question selection, and the finite reference collection, underscoring the need for explicit measurement definitions and sensitivity analyses in LLM audits.
arXiv:2609.37499v1 Announce Type: new Abstract: Historical archives are an under-used source for extending the instrumental climate record backward in time, and LLMs offer a way to extract the indice...
The report audits a long‑term‑memory retrieval chain on 500 LongMemEval‑S development questions, evaluating reader performance under an adapted GPT‑4o rubric. Scores vary widely across reader lanes (93–479), with the strongest lanes scoring 479 and 475, and re‑judging the same pass‑1 answers changes three labels, yielding 478. Fixed‑answer knowledge‑update re‑scoring achieves 70/72 or 69/72 depending on the template, and a different‑family reader scores 474, just 1.0 percentage point below the headline pass. The study also reports that live reader request bodies were not retained, that 18 of 23 gains occurred where baseline packets lacked evidence, and that a negative control rejects a verifier that repairs some wrong drafts but breaks many correct ones. Overall, the findings do not establish a new leaderboard leader or a transferable memory advantage, and the released artifacts support packet inspection and re‑scoring but do not reconstruct the method.