← Back to all news
arXiv Computation and Language By Ran Li, Lei Chen

Readout Stability in Prefill-Only Decision Models:Zero-Label Prediction and Inference-Time Compute Allocation

Read the original on arXiv Computation and Language →

Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
5d ago

Benchmarking System One decision models against trained classifiers and language models for automated decision gates

The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.

By Amir Rafe, Subasish Das
llmsfine-tuningbenchmarks
More like this →
arXiv AI
3d ago

Keeping JEPA World Models Plannable When Little of the Frame Moves

The paper introduces SLIM, a pushing benchmark that tests language‑based goal specification for latent world models. It shows that a vanilla LeWM fails almost entirely due to an action‑insensitive encoder, but adding an inverse‑dynamics auxiliary loss restores encoder sensitivity and dramatically improves success rates. With the repaired latent, a lightweight language‑goal head can plan from natural language sentences, achieving high performance on navigation and pushing tasks without retraining the world model.

By Florian Strohm, Patrick Wagner, Jannik Schwab, Marco Huber
benchmarks
More like this →