arXiv AI

Recalibrating a Typed-Decision Model in an Afternoon: A Hands-On Recipe for Laya's Confidence Scores

The paper presents a practical five‑step recipe for recalibrating a typed‑decision model (Laya) that assigns confidence scores to text classification options. Using 130 lines of NumPy, the authors demonstrate that per‑bucket temperature scaling significantly reduces both pooled and macro‑ECE on a Turkish model, outperforming global scaling and other calibration methods. The study shows that separating buckets and applying bucket‑specific temperature adjustments yields the largest gains in accuracy and confidence calibration.

arXiv Statistics ML
Oct 28

Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

The paper evaluates how well typed decision models, which output probability distributions, actually provide reliable guarantees. By comparing open-weight Laya and Jev 1.13 across 190 evaluations, the study finds that point‑estimate thresholds frequently break promised accuracy, while a Beta lower‑bound rule and Learn‑then‑Test approach maintain guarantees more consistently. The work also explores calibration across languages, the impact of noise, data requirements for fine‑tuning, and the limitations of confidence in predicting outcomes or providing faithful explanations.

By Pedro Carvalho Brom
arXiv AI
3d ago

How many LLM-labelled rows does a decision encoder need, and how wrong can the labels be?

The study investigates how many large‑language‑model (LLM)‑generated training rows a decision encoder, specifically Laya (ModernBERT‑large), requires to achieve high accuracy on a choice‑making task. Using 19 fine‑tunes on nested subsets of a 2,818‑row pool, accuracy improves from 69.4 % before fine‑tuning to 86.9 % with all rows, with diminishing returns as more data is added. The experiment also examines the impact of label noise, finding that 10 % symmetric noise has negligible effect, while 20 % and 40 % noise degrade performance by 3.4 and 8.7 points respectively, and that a single run with 10 % confusable‑pair noise reduces confidence. whyItMatters":"The results quantify the data efficiency and robustness of Laya to noisy LLM‑generated labels, informing how much synthetic training data is needed for reliable decision‑encoding models."

By Mohammed Nayeem Khan, Aliyas Ahmad Misri
arXiv Machine Learning
Nov 2

StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation

StrictPRMBench is a replayable evaluation benchmark designed to audit fixed‑score PRM aggregation rules. It standardizes mathematical reasoning pools, scorer evidence, and deletion interventions so that different reducers confront identical selection problems. Across seven checkpoints and two benchmarks, the benchmark reveals that product aggregation yields higher deletion‑induced selection instability than geometric mean, with statistically significant differences in most cells.

By Pengfei Zheng
arXiv Computation and Language
Oct 16

Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM

The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.

By Alden Do Rosario, Kirollos Atef
arXiv AI
3d ago

Where in the Input Does a One-Pass Decision Encoder Look?

The paper evaluates Laya, a small model that processes a multiple‑choice question, its options, and a text in a single 512‑token sequence. Experiments show that when the relevant evidence lies beyond the window, accuracy drops to about 26–27%, and confidence metrics fail to warn of truncation. Fine‑tuning on longer documents improves performance on visible evidence but not on cut evidence, and a larger window or a 312‑token scan can yield modest gains.

By Mohammed Nayeem Khan, Allan Kevin T
arXiv AI
Sep 15

Option-Aware Retrieval and Task-Specific VLM Adaptation for Medical VQA

The paper presents a system for the MedReason 2026 challenge that tackles both multiple‑choice and open‑ended medical visual question answering using offline, containerized inference. Key findings include that comparing answer semantics rather than labels boosts retrieval‑only accuracy from 20.0 % to 57.5 % on a 200‑case holdout, and that varying the number of in‑prompt retrieved examples has minimal impact on final accuracy (93.5 %–94.0 %). The final system achieves 94.0 % MCQ accuracy on the development set and 93.20 % on the official pre‑evaluation, far surpassing the off‑the‑shelf baseline. "whyItMatters":"The results demonstrate that semantic‑aware retrieval and careful adapter tuning can dramatically improve medical VQA performance, offering a practical approach for high‑accuracy, offline inference in clinical settings."

By Tristan Kirscher (ICube, Institut Strauss), Niklas C. Koser (CAU), Soren Pirk (CAU)
arXiv Machine Learning
Jul 1

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao
arXiv AI
Aug 20

Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation

The paper investigates whether providing candidate solutions during test‑time aggregation improves or harms accuracy compared to a fresh solve that does not use any candidates. Using Qwen3‑4B on AIME‑2025 and HMMT‑2025, the authors find that conditioning on multiple correct candidates boosts accuracy (+0.290), while conditioning on an all‑wrong candidate pool reduces accuracy (−0.123); the effect for a single correct candidate remains unclear. The study also explores structured interventions and placebo controls, but the underlying mechanisms of these effects are not resolved.

By Guiv Farmanfarmaian
arXiv AI
Oct 7

Evaluate the Stack, Not the Layer: Do Deterministic and LLM Gates for Agent Actions Fail Independently?

The study evaluates how runtime gates for agent tool calls stack, assuming their errors multiply. Using 1,119 labeled agent actions from three corpora, the authors compare a deterministic rule layer with four LLM judges, measuring the effective number of multiplication-equivalent layers (n_mult). Results show that two judges compose to about 1.2–1.4 layers, while a rule layer plus one judge composes to roughly 1.86–2.09 layers, with variations depending on miss definitions and corpora.

By Chenglin Yang