The paper introduces TRACE, a digital‑advertising diagnostic environment that uses simulated interventions to generate verifiable rewards for training reasoning agents. By injecting controlled interventions into a simulator, the hidden cause of anomalies becomes an oracle label, enabling agents to learn to identify root causes and affected segments through noisy, confounded evidence. Experiments show that reinforcement learning with these synthesized rewards outperforms large prompted baselines, achieving higher accuracy while using fewer tool calls.
By Rui Sun, Zhan Shi, Bing He
RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.
By Zhengkang Guo, Wenhao Liu, Mingchen Xie, Jingwen Xu, Zisu Huang, Muzhao Tian, Jianhan Xu, Yuanzhe Shen, Qi Qian, Muling Wu, Xiaohua Wang, Changze Lv, He-Da Wang, Hu Yao, Xiaoqing Zheng, Xuanjing Huang
The paper introduces CIG‑SFT, a supervised fine‑tuning method that uses conditional information gain to evaluate how much each demonstrated token reduces uncertainty about the task. By comparing token likelihoods from a frozen reference model with and without task input, the method derives a KL‑regularized projection of the current policy, yielding a scalar‑weighted SFT objective whose gradients match the projected target. Experiments on mathematical reasoning and code‑generation benchmarks show that CIG‑SFT outperforms standard SFT and adaptive fine‑tuning baselines, especially in generalizing to challenging evaluation distributions, without requiring extra data or online rollouts.
By Yuxuan Gu, Kequan Niu, Xiaocheng Feng, Yun Li, Yiqiao Yang, Bing Qin
arXiv:2608. 10209v1 Announce Type: new Abstract: Feedback signals used to train Large Language Models (LLMs) are the primary driver of their behavior and our main lever for instilling alignment with human values and objectives.
By Alec Harris, Kasey Corra, Archie Chaudhury, Yixiong Hao
The study investigates how language models acquire in-context learning (ICL) by distinguishing between copying a nearby demonstration and integrating information across a prompt. Across 17 models, roughly one-third default to copied answers, a few are undecided, and the remainder favor rule-consistent answers that yield higher accuracy and are guided by the requested value rather than the nearest conflicting example. Analyses show that models prioritize task-relevant prompt tokens before accuracy surpasses chance, and that synthetic or fine-tuned training can shift models toward compositional responses or improve accuracy when prompts reassign label meanings.
By Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He
arXiv:2608. 12426v1 Announce Type: new Abstract: Large language models are increasingly deployed in settings that require simultaneous adherence to multiple explicit constraints - reasoning structure, safety boundaries, output schemas.
By Mariya I. Vasileva