arXiv AI

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

The paper introduces DEEPO, a Dual-Entropy Enhanced Policy Optimization method designed to mitigate hallucination in multimodal large language models (MLLMs). It addresses two weaknesses in reinforcement learning: (1) hard queries with high semantic entropy produce uniformly wrong samples, erasing advantage signals, and (2) confident-but-wrong tokens become invisible to gradients as the policy sharpens. DEEPO combines semantic‑entropy‑triggered expert prefixes to inject grounded continuations and Renyi preconditioning to counter logit saturation, yielding significant hallucination reduction while maintaining accuracy and training stability.

arXiv Machine Learning
Sep 10

Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification

The paper introduces Stable-MM-R1, a framework that stabilizes reinforcement learning for multimodal reasoning by addressing training instability and entropy collapse. It proposes Potential‑Aware Query Mining (PAQM) to filter data toward high‑potential samples and Hybrid Stratified Replay (HSR) to restructure batches using path entropy and reward stratification, reusing stability anchors and hard negatives. The method demonstrates superior performance on complex reasoning tasks compared to strong baselines.

By Yimeng Ye, Shuang Chen, Wenxuan Huang, Manyuan Zhang, Kaituo Feng, Zhangquan Chen, Jiayu Chen, Yucheng Zhou, Yicheng Xiao, Zhiyuan Feng, Tianyu Shi
arXiv AI
3d ago

TIDE: Token-Level Credit Assignment via Entropy-Gated Bounded Deviation

TIDE (Token-level Importance via Deviation and Entropy) is a critic‑free reinforcement learning framework that replaces the volatile reverse KL divergence with a bounded Hellinger distance to safely quantify token‑level exploration. It introduces an entropy gating mechanism that scales deviation by policy uncertainty, redistributing sequence‑level advantage across tokens and amplifying exploratory steps while suppressing low‑entropy deviations. Experiments on the Qwen2.5‑32B backbone show TIDE achieving 60.0% and 46.0% Avg@32 accuracy on AIME 2024 and AIME 2025, outperforming strong critic‑free baselines such as DAPO.

By Qiaoman Zhang, Shunhua Min, Xiang He, Zhenghan Zhang, Yuting Zhang, Guangxian Li, Yunzhu Kuang, Rui Jiang, Jingyan Zhang, Chaoyue Li, Jiayu Ding
arXiv AI
Oct 26

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection introduces a multi-turn reinforcement learning framework that uses a teacher model to provide Socratic feedback, guiding reasoning trajectories during rollout. This approach addresses localization ambiguity, reward sparsity, and verbosity bias in detecting hallucinations in large language models. Empirical results on the RAGTruth dataset show that SOPO (4B) outperforms leading proprietary models and standard RL baselines, achieving state‑of‑the‑art performance on multiple benchmarks.

By Jiawei Dong, Zilong Bai, Pengtian zhu, Peng Zhou
arXiv Computation and Language
6d ago

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

BALTO is a Balanced Token-level Policy Optimization framework designed to reduce hallucinations in large language models. It extracts factual claims, verifies them against reference context, and assigns token-level credit in a balanced way to shift probability mass toward faithful content. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show BALTO outperforms existing baselines in faithfulness metrics and improves the faithfulness–informativeness trade‑off.

By Ning Li, Zixuan Guo, Yan Xu, Wenbo Fei, Yifan Niu, Chang Luo, Yasheng Wang, Weiwen Liu, Yong Yu, Weinan Zhang
arXiv AI
Jun 10

TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

arXiv:2509. 25760v2 Announce Type: replace-cross Abstract: While large language models (LLMs) have demonstrated strong performance on factoid question answering, they are still prone to hallucination and untruthful responses, particularly when tasks demand information outside their parametric knowledge.

By Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Jingxiang Chen, Mohammad Kachuee, Teja Gollapudi, Yiwei Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen-tau Yih, Xin Luna Dong
arXiv AI
Jun 19

Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning

arXiv:2606. 19771v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced Large Language Model (LLM) reasoning; however, it faces a fundamental optimization instability: uniform token updates precipitate entropy collapse, leading to premature convergence to suboptimal strategies, whereas excessive Shannon Entropy maximization can cause entropy explosion, driving blind exploration toward incoherent reasoning chains.

By Xuanzhi Feng, Zhengyang Li, Zeyu Liu, Haoxi Li, Yuming Jiang, Bing Guo, Jingcai Guo, Jie Zhang, Song Guo