arXiv Machine Learning
Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 7
Harmful SFT Leaves a Continuous Trace in LLM Checkpoint Updates
arXiv:2610.07518v1 Announce Type: cross Abstract: Safety auditing of post-trained large language models typically relies on model behavior, requiring model execution and depending on the coverage of...
arXiv Computation and Language
Sep 25
StepCOPS: Closed-Testing Lower-Tail Certificates for Language-Model Policy Selection
StepCOPS is a new method for selecting a language‑model policy from many checkpoints, prompts, and decoding rules by providing closed‑testing lower‑tail certificates. It uses an independent proposal split to nominate a lower‑tail floor for each candidate, applies exact binomial tests on a fresh certification split, and employs Holm’s step‑down procedure to certify a set of floors. In experiments across 24 configurations and 11 benchmarks, StepCOPS achieves 96.4% selected‑policy coverage, raises the certified floor by 1.5 points over prior methods, stays 0.6 points below a large‑reference jury oracle, and abstains in 2.4% of trials.
arXiv Machine Learning
Sep 30
Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG
arXiv:2605.29139v2 Announce Type: replace-cross Abstract: Question-answering services built on retrieval-augmented generation (RAG), in which a language model answers from retrieved documents, are in...
arXiv AI
Aug 24
UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations. "whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
arXiv AI
Sep 30
From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning
arXiv:2609.36569v1 Announce Type: cross Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...
Hugging Face Trending Papers
Jun 10
Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness
End-to-end task-success is the dominant way to evaluate LLM agents, but one aggregate number tells you that an agent regressed, not where. We present layer-isolated evaluation: a deployed ordering agent is decomposed into a fixed taxonomy of layers (ontology, intent, routing, decomposition, escalation, safety, memory, and cross-cutting envelope/defense), each exercised by its own assertion slice in a deterministic, no-LLM "pure" mode.