The paper evaluates how well typed decision models, which output probability distributions, actually provide reliable guarantees. By comparing open-weight Laya and Jev 1.13 across 190 evaluations, the study finds that point‑estimate thresholds frequently break promised accuracy, while a Beta lower‑bound rule and Learn‑then‑Test approach maintain guarantees more consistently. The work also explores calibration across languages, the impact of noise, data requirements for fine‑tuning, and the limitations of confidence in predicting outcomes or providing faithful explanations.
By Pedro Carvalho Brom
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations.
"whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
By Ye Chen, Weining Zhang
Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.
By Ran Li, Lei Chen
The paper introduces CASTER, a gradient‑free test‑time adaptation method that keeps the model frozen by storing source class statistics in a discriminative subspace and applying an affine transformation estimated from target‑batch moments. CASTER avoids backward passes, optimizer state, and large feature banks, outperforming k‑NN on frozen features in most backbone‑dataset settings while using far less memory. The authors also propose a residual‑to‑margin transportability certificate that flags when affine transport is unreliable, and demonstrate that gating based on this certificate can recover performance losses.
By Salim Khazem, Ibrahim Mohamed Serouis
arXiv:2607. 27031v1 Announce Type: new Abstract: Reports on how sparsification, compression, and lottery tickets change model behavior have been mixed in the prior literature, with beneficial effects observed in some studies and adverse effects in others.
By Bum Jun Kim
arXiv:2609.35793v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly trained with reinforcement learning from verifiable rewards (RLVR). An exact verifier can also support te...
By Xuan Liu, Jingbin Qian, Haosheng Chen
arXiv:2608.20873v1 Announce Type: new
Abstract: Every way of teaching a deployed language model something new -- full fine-tuning, adapter merging, model editing -- replaces the released checkpoint,...
By Zifeng Liu, Zhiyong Du, Yaxin Lu, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing
The study examines whether accuracy gaps between source and target tasks can certify the failure of scalar recalibration maps for large language model judges. Across thirteen judges, two generators, eight domains, and 1,176 transfers, the accuracy gap only provides a weak lower bound on target calibration error and can predict opposite outcomes. Even with a finite‑sample lower certificate, the method shows low power (0.13) and does not reliably indicate when recalibration will fail.
By Fariya Afrin, Ibne Farabi Shihab
The paper introduces loss‑conditioned state execution, a model‑agnostic technique that decides whether to apply a world model’s proposed state change or keep the current state based on whether the change reduces downstream loss. It formalizes state movability as the existence of a loss‑reducing feasible correction and constructs loss‑specific proposals from predictive distributions, executing them only when a groupwise lower confidence bound on loss improvement is positive. Experiments on forecasting and dynamics benchmarks show that the method accepts updates for a subset of cases, achieving lower bounded loss than persistence or always executing the proposal, and highlights that event predictability and loss‑based decisions must be evaluated separately.
By Jintao Xu, Zhengyu Chen, Ben Zhang, Yongzhi Qi, Jianshen Zhang
arXiv:2605.06908v2 Announce Type: replace-cross
Abstract: Adaptive test-time compute for LLM agents aims to invoke extra computation only when it improves performance. Existing methods typically use...
By Ziming Li, Jiatan Huang, Xiaoguang Guo, Guiling Wang, Chuxu Zhang
arXiv:2608. 19488v1 Announce Type: new Abstract: Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain.
By Sawan Dasari
arXiv:2508. 00042v2 Announce Type: replace-cross Abstract: Machine learning models deployed in non-stationary environments degrade silently, since as the input distribution drifts their accuracy decays without an error signal and without labels to reveal it.
By Athanasios Tziouvaras, Carolina Fortuna, George Floros, Kostas Kolomvatsos, Panagiotis Sarigiannidis, Marko Grobelnik, Bla\v{z} Bertalani\v{c}