Process Reward Models (PRMs) are crucial for scaling large language model reasoning at test time, but they suffer from two types of uncertainty: PRM uncertainty from limited supervision and imperfect training, and policy uncertainty from mismatched data collection policies. The paper introduces BayesianPRM, which treats process rewards as random variables, discretizes the reward space with an ensemble, and applies two belief updates—reliability‑aware updating for PRM uncertainty and conservatism‑aware calibration for policy uncertainty—to form a Bayesian belief whose expectation yields a more accurate reward estimate. Experiments on multimodal reasoning benchmarks show that BayesianPRM improves reward accuracy and test‑time scaling performance.
arXiv:2607. 00164v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards can in principle train calibrated probabilistic forecasters, since a proper scoring rule such as the Brier score is computed from outcomes alone and is minimized in expectation by the true probability.
arXiv:2607. 01612v1 Announce Type: new Abstract: Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks.
arXiv:2604. 26360v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) systems face a compounding alignment challenge: not only are learned reward models uncertain about unseen state-action pairs, but the human preference annotations they are trained on are themselves inconsistent, context-dependent, and noisy.
arXiv:2606. 30335v1 Announce Type: new Abstract: Autonomous scientific discovery systems increasingly use large language models (LLMs) to propose new hypotheses, but many such systems condition primarily on experimental memory: archives of high-scoring candidates or heuristic summaries of recent trials.
arXiv:2604. 23333v2 Announce Type: replace Abstract: Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability.
arXiv:2606. 00680v1 Announce Type: new Abstract: Offline reinforcement learning (RL) aims to optimize policies from pre-collected datasets.
The paper introduces a decision‑theoretic framework that splits a large language model’s decision loss into belief formation and action selection components. Using a synthetic benchmark, it evaluates how reinforcement‑learning interventions on beliefs, decisions, or both affect these components across three domains. The study finds that targeting a single component improves that part but may not transfer to others, while jointly targeting both improves both only when training and evaluation formats match.
Aligned language models routinely misreport under non-evidential incentive pressure: they agree with a confident user or overstate certainty even when their internal belief is unchanged. We cast this as a failure of internal incentive-compatibility (IC) and present a method for learning and certifying counterfactual report mediators that hold a model's reports to a causal contract: invariant to forbidden influences (pressure, prestige, restyling) and responsive to licensed ones (genuine evidence).
arXiv:2607. 12985v1 Announce Type: new Abstract: Aligned language models routinely misreport under non-evidential incentive pressure: they agree with a confident user or overstate certainty even when their internal belief is unchanged.
The paper introduces the Belief-State Engine (BSE), an inference module that supplies a large language model (LLM) with a Bayesian posterior over hidden states in a partially observable Markov decision process (POMDP). By keeping the raw action‑observation log hidden from the LLM, the BSE ensures the agent behaves as a sound Markov policy on the belief MDP, thereby inheriting classical POMDP optimality guarantees. Experiments on the Tiger POMDP and a red‑team attack‑graph task show that BSE‑augmented agents outperform six baselines in task return, belief calibration, and decision consistency.
arXiv:2605. 24396v2 Announce Type: replace Abstract: Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps, limiting the gains from additional test-time compute.
arXiv:2606. 11211v1 Announce Type: cross Abstract: The ability of large language models (LLMs) to express calibrated uncertainty is important for safe deployment.