Fine-tuning and adaptation

LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.

8,205 stories · RSS feed

arXiv AI
Sep 21

BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models

BoostAPR is a three-stage framework that improves automated program repair by using execution-grounded reinforcement learning with dual reward models. The approach first fine‑tunes a model on execution‑verified demonstrations, then trains a sequence‑level assessor and a line‑level credit allocator from execution outcomes, and finally applies PPO optimization where the line‑level model redistributes rewards to critical edit regions. Evaluated on SWE‑Gym and four benchmarks, BoostAPR achieves significant gains, including 40.7% on SWE‑bench Verified and 95.0% on QuixBugs, demonstrating strong cross‑language generalization.

By Yuanhao Li, Hongbo Wang, Xiaotang Shang, Xunzhu Tang, Yiming Cao, Xuhong Chen
arXiv AI
Sep 21

Do Personality-Tuned LLMs Make Better Social Agents?

The paper examines whether fine‑tuning large language models (LLMs) with personality‑labelled data improves their ability to act as socially interactive agents. Two small open‑weight LLMs were fine‑tuned on a corpus of personality‑labelled social media posts and dialogues, and the resulting models were evaluated in various social interaction scenarios by independent LLM judges. The findings show that the fine‑tuned models do not outperform their baseline counterparts in role‑playing personalities, though they offer comparable text quality and increased linguistic diversity for the Qwen models; low inter‑rater agreement limits confidence in the results, suggesting future work should focus on training data quality and domain alignment.

By Tim Krabbe, Xiaodan Shi
arXiv AI
Sep 21

SynthDemo-RL: Breaking the Zero-Reward Barrier in VLA Adaptation with LLM-Guided Synthetic Demonstrations

SynthDemo‑RL introduces a teacher‑student framework that uses an automated teacher to generate successful manipulation trajectories from simulator‑privileged state, which are then distilled into a Vision‑Language‑Action (VLA) student via supervised fine‑tuning. The student is further refined with PPO using binary task‑success rewards. On the LIBERO‑PRO benchmark, SynthDemo‑RL rescues all 27 previously unsolvable tasks and achieves near‑perfect success rates, matching performance that would otherwise require human demonstrations.

By Hiroaki Kingetsu, Hiroaki Kurihara, Kaoru Yokoo, Kenji Fukumizu, Manohar Kaul
arXiv AI
Sep 21

Potential-Field Action Representation for Reinforcement Learning in Contact-Rich Manipulation

The paper introduces PA‑RL, a reinforcement‑learning framework that uses artificial potential fields as the action representation for contact‑rich robotic manipulation. Instead of directly commanding motion, the policy adjusts potential‑field parameters, which a Cartesian impedance controller then executes, decoupling task strategy from low‑level control. In peg‑in‑hole experiments, PA‑RL achieved a 100% success rate in simulation, outperformed baselines in torque and acceleration variation, and transferred to a real robot without fine‑tuning.

By Xinyu Liu, G\"okhan Solak, Arash Ajoudani
arXiv AI
Sep 21

KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos

KnowDemo is a framework that generates diverse robot demonstrations from human videos by leveraging structured manipulation knowledge. It uses a vision‑language model to extract task requirements and permissible execution variations, then resolves these against target‑scene entities to guide candidate generation and screening before motion planning. The resulting demonstrations feature multimodal behavior, alternative contact strategies, and valid subtask orders, and have been shown to improve planning success and enable sim‑to‑real policy transfer across three tasks.

By Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar, Jeroen Sch\"afer, Michael Beetz
arXiv Computation and Language
Sep 21

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Draft-OPD introduces an on‑policy distillation method for speculative draft models, addressing the mismatch between supervised fine‑tuning and inference by letting the target model supervise the drafter on draft‑induced states. The approach uses target‑assisted rollouts for stable continuations and replays drafting from error positions exposed during verification, enabling the drafter to learn from both accepted and rejected proposals. Experiments demonstrate that Draft‑OPD achieves more than five‑fold lossless acceleration across diverse tasks, outperforming prior draft models such as EAGLE‑3 and DFlash by 23 % and 13 % respectively.

By Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng