← Back to all news
arXiv Computation and Language By Dicong Yu, Wenhua Wang, Tian Wang

Stepwise-Querying: Distilling Reasoning without Access to Teacher Chain-of-Thought

Read the original on arXiv Computation and Language →

Stepwise-Querying is a method that extracts step-by-step explanations from a black-box teacher model without needing its internal chain-of-thought. By independently querying the teacher for intermediate derivations, validity checks, and corrections for each step, the gathered explanations are used to fine-tune smaller student models. Experiments on mathematics and code benchmarks show that this approach outperforms three baselines in 14 of 16 comparisons, and the introduced Stepwise Explanation Information Gain metric helps diagnose the usefulness of intermediate reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang
llmsreinforcement-learningfine-tuningefficiencybenchmarks
More like this →
arXiv AI
Oct 16

CoTranslator: Reverse to Forward Chain-of-Thought Translator for Reasoning Distillation

CoTranslator is a method that converts reverse chain-of-thought (CoT) responses into verified forward CoT responses, enabling efficient large-scale generation and verification of training data for reasoning distillation. Trained on a small set of paired reverse and rejection-sampled forward responses, it improves student model accuracy on mathematics benchmarks by up to 5.82 percentage points and retains 95.0% of rejection sampling’s accuracy while using only 35.9% of its compute. The translator also transfers to new datasets, teachers, and STEM problems, boosting average STEM accuracy for 4B and 8B students by 4.88 and 4.31 percentage points, respectively.

By Zinan Tang, Zhuoshi Pan, Kairui Chen, Ze Yan, Shaomian Zheng, Yukun Zhang, Gang Wang, Biqing Huang
efficiencybenchmarks
More like this →