arXiv Computation and Language By Ning Li, Zixuan Guo, Yan Xu, Wenbo Fei, Yifan Niu, Chang Luo, Yasheng Wang, Weiwen Liu, Yong Yu, Weinan Zhang

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

Read the original on arXiv Computation and Language →

BALTO is a Balanced Token-level Policy Optimization framework designed to reduce hallucinations in large language models. It extracts factual claims, verifies them against reference context, and assigns token-level credit in a balanced way to shift probability mass toward faithful content. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show BALTO outperforms existing baselines in faithfulness metrics and improves the faithfulness–informativeness trade‑off.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Oct 26

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection introduces a multi-turn reinforcement learning framework that uses a teacher model to provide Socratic feedback, guiding reasoning trajectories during rollout. This approach addresses localization ambiguity, reward sparsity, and verbosity bias in detecting hallucinations in large language models. Empirical results on the RAGTruth dataset show that SOPO (4B) outperforms leading proprietary models and standard RL baselines, achieving state‑of‑the‑art performance on multiple benchmarks.

By Jiawei Dong, Zilong Bai, Pengtian zhu, Peng Zhou
arXiv Computation and Language
Oct 5

HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation

HARPO is a reinforcement learning framework that jointly optimizes faithfulness and creativity in language generation. It uses a Hallucination-Aware Generative Reward Model (HA‑GRM) to evaluate both faithfulness and writing quality, and a Selective Activation Mechanism (SAM) that applies writing rewards only to hallucination‑free outputs. Experiments on Qwen models show that HARPO improves faithfulness scores and reduces hallucination rates while boosting creative‑writing performance.

By Tiezheng Yu, Yuxin Jiang, Jinpeng Li, Shuning Sun, Fei Mi, Haoli Bai, Lifeng Shang
arXiv AI
3d ago

TIDE: Token-Level Credit Assignment via Entropy-Gated Bounded Deviation

TIDE (Token-level Importance via Deviation and Entropy) is a critic‑free reinforcement learning framework that replaces the volatile reverse KL divergence with a bounded Hellinger distance to safely quantify token‑level exploration. It introduces an entropy gating mechanism that scales deviation by policy uncertainty, redistributing sequence‑level advantage across tokens and amplifying exploratory steps while suppressing low‑entropy deviations. Experiments on the Qwen2.5‑32B backbone show TIDE achieving 60.0% and 46.0% Avg@32 accuracy on AIME 2024 and AIME 2025, outperforming strong critic‑free baselines such as DAPO.

By Qiaoman Zhang, Shunhua Min, Xiang He, Zhenghan Zhang, Yuting Zhang, Guangxian Li, Yunzhu Kuang, Rui Jiang, Jingyan Zhang, Chaoyue Li, Jiayu Ding
arXiv AI
Sep 25

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

The paper introduces DEEPO, a Dual-Entropy Enhanced Policy Optimization method designed to mitigate hallucination in multimodal large language models (MLLMs). It addresses two weaknesses in reinforcement learning: (1) hard queries with high semantic entropy produce uniformly wrong samples, erasing advantage signals, and (2) confident-but-wrong tokens become invisible to gradients as the policy sharpens. DEEPO combines semantic‑entropy‑triggered expert prefixes to inject grounded continuations and Renyi preconditioning to counter logit saturation, yielding significant hallucination reduction while maintaining accuracy and training stability.

By Yingxuan Zhuang, Miao Pan, Wangjie Gan, Jingxiao Yang, Fan Wang, Weiming Liu, Cheng Tan, Xuhong Zhang, Jintao Chen
arXiv AI
Oct 1

Trust the Critic More

The paper introduces Actor‑Critic with Action Chunking (AC2), a method that assigns credit to short action chunks instead of entire trajectories, enabling policy updates without waiting for terminal rewards. AC2 employs local readiness, reference solutions, and 10k‑token chunks to make critic‑based credit assignment reliable. Experiments on Qwen3‑4B with FineProofs‑RL show AC2 surpasses GRPO’s peak validation score while using 2.5× fewer decoding FLOPs and fewer training steps.

By Kaiyue Wen, Luke Bailey, Arvind Mahankali, Tengyu Ma