Fine-tuning and adaptation

LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.

8,201 stories · RSS feed

arXiv Computer Vision
Oct 12

TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering

The paper introduces TerraAgent, a training‑free framework that enhances remote sensing visual question answering by integrating three modules: GDR for demonstration retrieval, MACE for evidence‑typed sub‑question decomposition, and CVA for counterfactual evidence auditing. It demonstrates significant improvements on multiple benchmarks, raising accuracy, visual‑dependency scores, and reducing prior‑only answers and hallucinations without fine‑tuning.

By Jingwen Huang, Yiming Fang, Hanxiao Wu, Boyuan He, Keyi Luo
arXiv Computer Vision
Oct 12

MeshBind: Point-to-Mesh Binding for Non-Rigid Scene Flow Automatic Annotation

MeshBind introduces an automatic annotation pipeline that generates per-point scene flow for non‑rigid objects such as pedestrians by reconstructing articulated mesh models from six surround cameras and registering them to LiDAR data. The method employs a local‑frame binding technique that attaches each point to a mesh via a per‑triangle tangent‑normal frame, and a three‑layer flow registration pipeline that compensates ego‑vehicle motion, performs box binding, and applies mesh binding to produce full‑scene flow. MeshBind is implemented in the SF‑Studio tool, and the authors release the SUScape‑Flow Benchmark (SFB), a large‑scale dataset with 41k frames of per‑point non‑rigid pedestrian annotations and a comprehensive evaluation protocol, demonstrating that fine‑tuning on SFB reduces endpoint error by 2‑3× and equalizes non‑rigid and rigid EPE.

By Yinqi Chen, E Li, Shuaijun Wang, Weiyuan Peng, Meiying Zhang, Qi Hao
arXiv Computer Vision
Oct 12

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

UniMergeTrack introduces a hierarchical model merging strategy for RGB-X visual object tracking, combining independently trained modality-specific trackers into a unified framework. The method merges transferable backbones using high‑pass‑filtered merging, aligns tracking heads with optimal transport‑guided channel alignment, and fine‑tunes with distribution alignment to integrate cross‑task knowledge while preserving modality cues. Experiments on multiple benchmarks show state‑of‑the‑art or competitive performance across RGB‑Thermal, RGB‑Event, and RGB‑Depth tracking tasks, with strong generalization to unseen datasets.

By Shiao Wang, Xiao Wang, Yuhe Ding, Jiarui Xu, Zikang Yan, Bo Jiang, Yonghong Tian, Jin Tang
arXiv Computer Vision
Oct 12

Complexity-Calibrated Reconstruction Discrepancy for No-Reference Point Cloud Quality Assessment

The paper introduces a no-reference point cloud quality assessment method that leverages a complexity‑calibrated reconstruction discrepancy. By reconstructing a distorted point cloud and comparing successive reconstructions with varying smoothing, the approach isolates a proxy for structural complexity, which is then used to calibrate the initial reconstruction error. The calibrated geometric and projected‑view discrepancies, measured with frozen pretrained perceptual representations, are fused into a final degradation score without requiring any PCQA‑specific training.

By Haocheng Wu, Xiangjie Sui, Xin Sun
arXiv Computer Vision
Oct 12

ELMA: Benchmarking Anaphoric Compositions for Long-Term Text-to-Motion Generation

The paper introduces ELMA, a new episodic long‑term motion dataset designed to benchmark anaphoric compositions in text‑to‑motion generation. ELMA contains densely annotated 30‑second clips extracted from in‑the‑wild videos, preserving cross‑segment dependencies that current models struggle to handle. Fine‑tuning state‑of‑the‑art models on ELMA improves both global physical realism and semantic alignment, establishing a robust baseline for long‑term motion synthesis.

By Shun Kato, Dixin Yang, Kotaro Amaya, Yuiga Wada, Kazumi Fukuda, Takashi Shibuya, Yuki Mitsufuji, Mariko Isogawa
arXiv Computer Vision
Oct 12

Splat2Act: From Captured Scenes to Personalized VLA Manipulation

Splat2Act is an automated real‑to‑sim‑to‑real pipeline that personalizes a pretrained vision‑language‑action policy for a specific indoor scene using only a handheld phone scan and images of the user’s objects. It constructs a photorealistic digital twin with 3D Gaussian splatting, discovers receptacles, generates over 68,000 pickup demonstrations via a simulation oracle, and fine‑tunes the policy in two stages—first on public objects, then on the user’s own items—achieving up to 62.5% pickup success in target scenes. The approach demonstrates that such personalization can be achieved without costly real‑world demonstrations, with a real Stretch 3 robot matching simulated success rates on a 12‑task subset.

By Xiaomeng Ye, Krishna Jaganathan, Gyubeum Lim, Gunjan Chhablani, Muhammad Zubair Irshad, Zsolt Kira
arXiv AI
Oct 12

OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models

OGPO is an Offline Goal‑conditioned Policy Optimization framework designed to improve Vision‑Language‑Action (VLA) models for robotic manipulation. It replaces traditional step‑wise action imitation with adaptive relabeling of trajectories using semantic key states and goal‑conditioned rewards, enabling policies to reach task‑progress states even from off‑demonstration states. Experiments on LIBERO, MetaWorld, LIBERO‑Plus, and the proposed LIBERO‑ReAct benchmark show that OGPO consistently outperforms standard supervised fine‑tuning in terms of robustness and recoverability under distribution shifts.

By Xule Gao, Xi Wang, Zehua Zang, Rui Wang, Changwen Zheng, Chuxiong Sun
arXiv AI
Oct 12

Hindsight Structured Advice Distillation: Steering Executor Agents with Advisors Trained on Their Failures and Task Characteristics

The paper introduces Hindsight Structured Advice Distillation (HSAD), a method that trains a small advisor model to generate structured guidance for a frozen executor language‑model agent. By analyzing executor failures on a training set, a teacher writes advice that would have helped, and the advisor learns to produce this advice from input alone, adding minimal latency. Experiments on the FrontierFinance benchmark show that an 8B advisor boosts the qualification rate of Samaya’s research agent from 52.9% to 56.6%, surpassing the agent’s high‑effort configuration while cutting cost per query by 37%, and the approach also improves performance on WebShop, BIRD text‑to‑SQL, and ExpertLongBench.

By O. Ozan Koyluoglu
arXiv AI
Oct 12

SWEAR: Section-Wise Efficient Audio Reasoning via Self-Rated Difficulty

The paper introduces SWEAR, a framework for audio-language models that uses a five‑dimensional difficulty score to guide reasoning. By learning difficulty signals during fine‑tuning and applying reinforcement learning, SWEAR allocates reasoning effort adaptively across stages. Experiments on MMAU‑test‑mini and MMAR demonstrate that SWEAR boosts accuracy while cutting the number of generated tokens, especially for easier cases.

By Yixuan Li, Minghao Lv, Xuenan Xu, Zihao Zheng, Mengyue Wu
arXiv AI
Oct 12

Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

PRISM (Population‑Retrieved Inference for Structured Multimodal Prognosis) is a multimodal language model that integrates structured measurements, physiological waveforms, longitudinal history, and population context for clinical prognosis. It uses per‑feature numerical tokenization, recency‑weighted aggregation of ECGs, and a dual‑modality population memory bank, retrieving mean‑pooled embeddings of similar encounters without including individual records in the prompt. On the 1,443‑task MDS‑ED benchmark, PRISM achieves high AUROCs across diagnosis, deterioration, ICU admission, and mortality, outperforming baselines and demonstrating the value of structured multimodal integration and supervised external adaptation.

By Hamza Shafiq, Jialu Tang, Ting Dang, Jun Hu, Aaqib Saeed
arXiv AI
Oct 12

Unsupervised Multimodal Reward Modeling

The paper presents an unsupervised multimodal reward model built on a frozen Qwen3‑VL backbone. By adding a linear head to an intermediate layer and training it to rank a web caption higher than a perturbed version of itself, the authors avoid human preference labels and achieve competitive performance—outperforming GPT‑4o on several benchmarks with far fewer parameters and compute. The approach demonstrates that a carefully chosen intermediate layer, simple perturbation-based training, and verifier‑driven fine‑tuning can yield effective vision‑language rewards without costly annotation.

By Yueying Li, Jingxuan Fan, Kiant\'e Brantley, Sham M. Kakade, Hanlin Zhang
arXiv AI
Oct 12

JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

JoyInside‑VLM‑Agent is a ReAct‑style multimodal agent that learns product‑conditioned policies for embodied robots and smart hardware. It uses a closed‑loop sandbox to generate interaction trajectories, supervised fine‑tuning, multi‑teacher on‑policy distillation, and reinforcement learning with DPER to align token‑level evidence and rewards. The agent achieves a BFCL v4 overall score of 77.44, surpassing baselines on JoyInside‑Devices and JoyInside‑Robotics, and demonstrates visually grounded search, replanning, and failure reporting in two real‑world quadruped case studies.

By Gangyi Li, LuYao Wang, Jingfu Sun, Jingyi Yuan, Hang Liu, Hao Zong, Yujia Hu, Tingting Gao, Jianyong Wang, Xu Yang
arXiv AI
Oct 12

MotionCoach: Constructing Multimodal Guidance for Video Professional Motion Editing

MotionCoach is a multimodal agent designed to generate precise guidance for professional motion editing in videos. It constructs grounded prompts and reference clips through multi‑turn tool use and self‑verification, leveraging a training pipeline that includes a 10k SFT dataset and a 2k RL dataset optimized with Group Relative Policy Optimization. Evaluation on the VPMEBench benchmark shows significant performance gains, raising overall scores from 55.1 to 65.6 on Seedance 2.0 and from 51.2 to 64.5 on MiniMax‑H3.

By Siyi Qian, Hanqing Yang, Weihuang Lin, Zihao Yang, Yichen Lu, Yuanhong Zheng, Sheng Li
arXiv AI
Oct 12

DeltaLLM: Delta-Key Sparse Attention for Edge LLMs

DeltaLLM is a training‑free framework that accelerates on‑device large language model inference by introducing delta‑key sparse attention and a vectorized structured sparse online softmax. It reuses delta keys across query heads, preserves exact attention near the diagonal while approximating non‑local regions, and exploits hardware‑friendly sparsity with a specialized vector‑processor architecture. Experiments on BitNet‑b1.58‑2B‑4T and LLaMA3.2‑1B‑Instruct show up to 60% delta‑key sparsity, 63% structured attention‑score sparsity, and a 1.62× speedup on the KV260 FPGA with minimal accuracy loss.

By Jiawen Qi, Jiayu Liu, Chang Gao, Zhaochun Ren, Qinyu Chen
arXiv AI
Oct 12

AnomalyAgent: Agentic Industrial Anomaly Synthesis via Tool-Augmented Reinforcement Learning

AnomalyAgent is an agentic framework for industrial anomaly synthesis that uses five tools—Prompt Generation, Image Generation, Quality Evaluation, Knowledge Retrieval, and Mask Generation—to iteratively refine anomaly samples. It employs a two‑stage training process combining supervised fine‑tuning and reinforcement learning, guided by a three‑part reward system that encourages task performance, self‑reflection, and trajectory adherence. On the MVTec‑AD dataset, AnomalyAgent outperforms zero‑shot state‑of‑the‑art methods, achieving superior anomaly generation scores and high classification and detection accuracy.

By Jiaming Su, Tengchao Yang, Ruikang Zhang, Zhengan Yan, Haoyu Sun, Linfeng Zhang
arXiv AI
Oct 12

First-Order In-Context Shapley: Scalable Data Auditing for LLM Alignment and Evaluation

The paper introduces First-Order In-Context Shapley, an inference‑only method that evaluates data utility for large language model alignment by computing the first‑order term of an in‑context cooperative game over semantic k‑NN neighborhoods in O(k) forward passes. It projects log‑likelihood shifts onto a directed influence graph, yielding three signals—Exemplifying Advantage, Exemplified Advantage, and One‑of‑a‑Kind Score—to identify high‑utility demonstrations and rare prompts. The approach reduces compute by 19× compared to Monte‑Carlo Shapley, filters 1–5% of records, and improves error detection and reward accuracy in DPO training and benchmark auditing. whyItMatters":"By enabling scalable, inference‑only data auditing, the method addresses label noise and safety inversions that hinder post‑training alignment of large language models, leading to measurable improvements in model reliability and reward accuracy."

By Yunting Song, Matthew Watson, Moustafa Farid Alzantot, Peter Grabowski, Jun Qin
arXiv Computation and Language
Oct 12

U4RASD at KnowledgeGraphEval-2026: Ontology-Constrained Encoder and LLM Systems for Arabic Relation Extraction

The paper reports the authors’ submission to KnowledgeGraphEval 2026 Subtask 2 on directed Arabic relation extraction. They evaluated both encoder‑based models—using typed entity markers and separate relation‑existence and relation‑classification heads—and large language model approaches, fine‑tuning Yehia and Qwen3.5 with QLoRA under ontology‑constrained prediction. Their ensemble of Yehia‑RE, Qwen‑RE, and TypedPair achieved a micro‑F1 of 94.06, securing first place in the subtask.

By Ahmad Abbas, Hussein Mazeh, Chadi Abou Chakra, Aya Jouni, Mohammad Ballout, Fadi A. Zaraket
arXiv Computation and Language
Oct 12

MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents

MADS (Multi-Agent Dialog Simulation) is a synthetic data generation pipeline that creates structured, diverse dialogue trajectories, including adversarial conversations that reflect realistic failure scenarios. The resulting MADS-Bench benchmark covers six domains, 55 personas, and 1095 conversations, with about 66% featuring failure-prone situations. Models trained with MADS are evaluated using a two-stage approach combining supervised fine-tuning and reinforcement learning with verifiable rewards to assess robustness.

By Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni, Mudit Arora, Hugo Lin, Justin Chiu, Alessandro Di Bari, Neha Gupta, Andreas Stolcke
arXiv Computation and Language
Oct 12

Evo$^2$: Evolving the Evolver through Weight-Harness Co-Evolution

Evo$^2$ is a framework that jointly evolves a language model’s weights and its harness—the set of prompts, tools, and feedback mechanisms—using a dual-role pipeline where a single policy acts as both a Proposer and a Reflector. The training pipeline EvoTIDE combines supervised fine-tuning with interleaved GRPO to refine both roles based on harness edits and execution outcomes. Experiments on coding benchmarks show Evo$^2$ outperforms existing self‑evolution methods, achieving higher scores on Polyglot and SWE‑bench Verified, and its evolved harnesses transfer across tasks and models.

By Chejian Xu, Shujian Zhang, Wenxuan Zhou, John Lambert, Beck Chen, Bo Li, Andrew Hard, Rajiv Mathews
arXiv Computation and Language
Oct 12

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

The paper evaluates open‑source large language models (LLMs) for both flat and nested named entity recognition (NER) tasks, comparing them to traditional encoder‑based models and decoder‑based LLMs using in‑context learning. It shows that with parameter‑efficient fine‑tuning and structured output formats (e.g., inline brackets or XML), LLMs can match or exceed the performance of conventional NER systems, and that their NER abilities arise from instruction following rather than memorization. Additionally, fine‑tuning for NER minimally affects, and can even improve, the LLMs’ general capabilities on other benchmarks.

By Qi Zhan, Yile Wang, Hui Huang