arXiv AI By Yingxuan Zhuang, Miao Pan, Wangjie Gan, Jingxiao Yang, Fan Wang, Weiming Liu, Cheng Tan, Xuhong Zhang, Jintao Chen

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

Read the original on arXiv AI →

The paper introduces DEEPO, a Dual-Entropy Enhanced Policy Optimization method designed to mitigate hallucination in multimodal large language models (MLLMs). It addresses two weaknesses in reinforcement learning: (1) hard queries with high semantic entropy produce uniformly wrong samples, erasing advantage signals, and (2) confident-but-wrong tokens become invisible to gradients as the policy sharpens. DEEPO combines semantic‑entropy‑triggered expert prefixes to inject grounded continuations and Renyi preconditioning to counter logit saturation, yielding significant hallucination reduction while maintaining accuracy and training stability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Sep 10

Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification

The paper introduces Stable-MM-R1, a framework that stabilizes reinforcement learning for multimodal reasoning by addressing training instability and entropy collapse. It proposes Potential‑Aware Query Mining (PAQM) to filter data toward high‑potential samples and Hybrid Stratified Replay (HSR) to restructure batches using path entropy and reward stratification, reusing stability anchors and hard negatives. The method demonstrates superior performance on complex reasoning tasks compared to strong baselines.

By Yimeng Ye, Shuang Chen, Wenxuan Huang, Manyuan Zhang, Kaituo Feng, Zhangquan Chen, Jiayu Chen, Yucheng Zhou, Yicheng Xiao, Zhiyuan Feng, Tianyu Shi
arXiv AI
3d ago

TIDE: Token-Level Credit Assignment via Entropy-Gated Bounded Deviation

TIDE (Token-level Importance via Deviation and Entropy) is a critic‑free reinforcement learning framework that replaces the volatile reverse KL divergence with a bounded Hellinger distance to safely quantify token‑level exploration. It introduces an entropy gating mechanism that scales deviation by policy uncertainty, redistributing sequence‑level advantage across tokens and amplifying exploratory steps while suppressing low‑entropy deviations. Experiments on the Qwen2.5‑32B backbone show TIDE achieving 60.0% and 46.0% Avg@32 accuracy on AIME 2024 and AIME 2025, outperforming strong critic‑free baselines such as DAPO.

By Qiaoman Zhang, Shunhua Min, Xiang He, Zhenghan Zhang, Yuting Zhang, Guangxian Li, Yunzhu Kuang, Rui Jiang, Jingyan Zhang, Chaoyue Li, Jiayu Ding
arXiv AI
Oct 26

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection introduces a multi-turn reinforcement learning framework that uses a teacher model to provide Socratic feedback, guiding reasoning trajectories during rollout. This approach addresses localization ambiguity, reward sparsity, and verbosity bias in detecting hallucinations in large language models. Empirical results on the RAGTruth dataset show that SOPO (4B) outperforms leading proprietary models and standard RL baselines, achieving state‑of‑the‑art performance on multiple benchmarks.

By Jiawei Dong, Zilong Bai, Pengtian zhu, Peng Zhou
arXiv Computation and Language
6d ago

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

BALTO is a Balanced Token-level Policy Optimization framework designed to reduce hallucinations in large language models. It extracts factual claims, verifies them against reference context, and assigns token-level credit in a balanced way to shift probability mass toward faithful content. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show BALTO outperforms existing baselines in faithfulness metrics and improves the faithfulness–informativeness trade‑off.

By Ning Li, Zixuan Guo, Yan Xu, Wenbo Fei, Yifan Niu, Chang Luo, Yasheng Wang, Weiwen Liu, Yong Yu, Weinan Zhang