We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens.
arXiv:2512. 20661v2 Announce Type: replace Abstract: Transformer-based pre-trained language models (PLMs) excel in text classification but suffer from attention dilution and attention sink effects, forcing models to over-focus on task-irrelevant tokens.
By Yawei Liu
The paper introduces Token Importance-Aware Policy Optimization (TIAO), a reinforcement learning approach that improves text summarization by weighting token importance based on token dependency. TIAO reweights a trajectory’s advantage according to the overall dependencies of core tokens, addressing the limitation of previous methods that treat all tokens equally. Experiments demonstrate that a 7B foundation model enhanced with TIAO achieves performance comparable to GPT‑4 and GPT‑5‑nano on real‑world datasets.
By Qixiu Li, Chenlong Bao, Xiang Zhu, Xiaoyong Li, Ruixin Cao, Shukai Chen, Zhenxiong Zhou
arXiv:2609.28117v1 Announce Type: cross
Abstract: In this paper, we introduce a gradient-based head attribution strategy where the Token-level Max-Margin loss is backpropagated to the attention maps....
By Pawe{\l} M\k{a}ka, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis
arXiv:2503. 02368v4 Announce Type: replace-cross Abstract: While guided decoding, especially value-guided methods, has emerged as a cost-effective alternative for controlling language model outputs without re-training models, its effectiveness is limited by the accuracy of the value function.
By Zhenhua Liu, Lijun Li, Ruizhe Chen, Yuxian Jiang, Tong Zhu, Zhaochen Su, Wenliang Chen, Jing Shao
arXiv:2606.18389v2 Announce Type: replace
Abstract: Large language models (LLMs) have become an effective tool for synthetic data generation, including for low-resource languages, where generated dat...
By Jan Cegin, Daniil Gurgurov, Yusser Al Ghussin, Simon Ostermann