The paper evaluates how well typed decision models, which output probability distributions, actually provide reliable guarantees. By comparing open-weight Laya and Jev 1.13 across 190 evaluations, the study finds that point‑estimate thresholds frequently break promised accuracy, while a Beta lower‑bound rule and Learn‑then‑Test approach maintain guarantees more consistently. The work also explores calibration across languages, the impact of noise, data requirements for fine‑tuning, and the limitations of confidence in predicting outcomes or providing faithful explanations.
By Pedro Carvalho Brom
The study investigates how many large‑language‑model (LLM)‑generated training rows a decision encoder, specifically Laya (ModernBERT‑large), requires to achieve high accuracy on a choice‑making task. Using 19 fine‑tunes on nested subsets of a 2,818‑row pool, accuracy improves from 69.4 % before fine‑tuning to 86.9 % with all rows, with diminishing returns as more data is added. The experiment also examines the impact of label noise, finding that 10 % symmetric noise has negligible effect, while 20 % and 40 % noise degrade performance by 3.4 and 8.7 points respectively, and that a single run with 10 % confusable‑pair noise reduces confidence.
whyItMatters":"The results quantify the data efficiency and robustness of Laya to noisy LLM‑generated labels, informing how much synthetic training data is needed for reliable decision‑encoding models."
By Mohammed Nayeem Khan, Aliyas Ahmad Misri
StrictPRMBench is a replayable evaluation benchmark designed to audit fixed‑score PRM aggregation rules. It standardizes mathematical reasoning pools, scorer evidence, and deletion interventions so that different reducers confront identical selection problems. Across seven checkpoints and two benchmarks, the benchmark reveals that product aggregation yields higher deletion‑induced selection instability than geometric mean, with statistically significant differences in most cells.
By Pengfei Zheng
arXiv:2606. 10154v1 Announce Type: new Abstract: Quantized checkpoints are often screened first with quality metrics and only later, if at all, with direct safety tests.
By Sahil Kadadekar
The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.
By Alden Do Rosario, Kirollos Atef
arXiv:2609.25052v1 Announce Type: new
Abstract: "An agent that writes its conclusions into a store it later retrieves from closes a loop usually reported as one-way contamination. Taking the loop to...
By Wenhui Chen, Jianlin Chen, Ziyao Lin, Chi Man Vong
The paper evaluates Laya, a small model that processes a multiple‑choice question, its options, and a text in a single 512‑token sequence. Experiments show that when the relevant evidence lies beyond the window, accuracy drops to about 26–27%, and confidence metrics fail to warn of truncation. Fine‑tuning on longer documents improves performance on visible evidence but not on cut evidence, and a larger window or a 312‑token scan can yield modest gains.
By Mohammed Nayeem Khan, Allan Kevin T
The paper presents a system for the MedReason 2026 challenge that tackles both multiple‑choice and open‑ended medical visual question answering using offline, containerized inference. Key findings include that comparing answer semantics rather than labels boosts retrieval‑only accuracy from 20.0 % to 57.5 % on a 200‑case holdout, and that varying the number of in‑prompt retrieved examples has minimal impact on final accuracy (93.5 %–94.0 %). The final system achieves 94.0 % MCQ accuracy on the development set and 93.20 % on the official pre‑evaluation, far surpassing the off‑the‑shelf baseline.
"whyItMatters":"The results demonstrate that semantic‑aware retrieval and careful adapter tuning can dramatically improve medical VQA performance, offering a practical approach for high‑accuracy, offline inference in clinical settings."
By Tristan Kirscher (ICube, Institut Strauss), Niklas C. Koser (CAU), Soren Pirk (CAU)
arXiv:2606. 26185v1 Announce Type: new Abstract: LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions.
By Hiroki Tamba
arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.
By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao
The paper investigates whether providing candidate solutions during test‑time aggregation improves or harms accuracy compared to a fresh solve that does not use any candidates. Using Qwen3‑4B on AIME‑2025 and HMMT‑2025, the authors find that conditioning on multiple correct candidates boosts accuracy (+0.290), while conditioning on an all‑wrong candidate pool reduces accuracy (−0.123); the effect for a single correct candidate remains unclear. The study also explores structured interventions and placebo controls, but the underlying mechanisms of these effects are not resolved.
By Guiv Farmanfarmaian
The study evaluates how runtime gates for agent tool calls stack, assuming their errors multiply. Using 1,119 labeled agent actions from three corpora, the authors compare a deterministic rule layer with four LLM judges, measuring the effective number of multiplication-equivalent layers (n_mult). Results show that two judges compose to about 1.2–1.4 layers, while a rule layer plus one judge composes to roughly 1.86–2.09 layers, with variations depending on miss definitions and corpora.
By Chenglin Yang