Fine-tuning and adaptation

LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.

8,201 stories · RSS feed

arXiv Computation and Language
Oct 28

DocSkill: Skill-Integrated Agentic Reasoning for Multilingual Multi-Hop Document Question Answering

DocSkill is an agentic framework that learns to orchestrate skills for adaptive evidence acquisition and reasoning in multilingual multi-hop document question answering. It uses bounded Monte Carlo Tree Search to generate skill trajectories for supervised fine‑tuning and preference optimization, and introduces the MMHop-DocBench dataset with 60K QA pairs across 10 languages and eight domains. Experiments show that DocSkill with Qwen3‑VL‑8B outperforms baselines on MMHop‑DocBench and performs competitively on related benchmarks.

By Jing Huang, Zhiya Tan, Changtao Miao, Shutao Gong, Yang Wang, Joey Tianyi Zhou, Jianshu Li, Ying Yan
arXiv Computation and Language
Oct 28

Latent Visual Contrastive Reasoning with Multi-Branch Exploration

The paper introduces Latent Visual Contrastive Reasoning, a branch-aware latent reasoning framework that generates multiple question-conditioned latent visual branches as soft hypotheses over image tokens. It trains these branches using bounding‑box evidence annotations and a soft contrastive branch‑evidence alignment objective, encouraging evidence‑relevant branches to align with annotated visual regions while contrasting against negative evidence. Experiments on perception‑intensive visual reasoning benchmarks demonstrate that the method improves performance over the corresponding backbone and single‑path latent reasoning baselines at both 3B and 7B scales.

By Kegeng Tang, Jingbo Wang, Yangming Zhang, Huanshu Zhang, Shaogang Ren, Zihao Wang
arXiv Computer Vision
Oct 28

Explicit Regional Representations For Video Quality Assessment

The paper proposes an explicit pixel‑level regional representation for video quality assessment called a Quality‑Relevance Heatmap (QRH), which fuses localized degradation evidence with perceptual priors guided by a vision‑language model, all without pixel‑level annotations. Two deterministic fragment‑sampling strategies are introduced: distribution‑preserving sampling approximates the original QRH‑score distribution without retraining, while quality‑aware sampling focuses on high‑QRH regions and is used with fine‑tuning. Experiments on the DOVER model across multiple benchmarks show that distribution‑preserving sampling closely matches predictions from random sampling, and quality‑aware sampling improves most evaluated metrics, demonstrating the utility of explicit regional representations for downstream VQA tasks.

By Fuyi Yang, Lei Yang, Andrew Liu, Chenyu Wei, Xiaoyi Guo, Ming Kai Hsu
arXiv Computer Vision
Oct 28

HoloRecon: Incremental Reconstruction of Long-Range Panoramas with Global Consistency

HoloRecon proposes a panoramic reconstruction framework that merges ORB-based sparse mapping with a fine-tuned feed-forward model to achieve accurate localization and globally consistent dense mapping over long sequences. By using predicted local geometry for 3D verification of feature matches, the method establishes long-range camera constraints even between distant revisits, calibrates depths with sparse landmarks, and optimizes a global pose graph. The authors also introduce PanoRecon, a seven-scene benchmark for evaluating global consistency, and report that HoloRecon achieves the lowest mean absolute trajectory error and high F‑1 Score among evaluated methods.

By Zhuoxiao Li, Xinyi Liu, Hongye Hou, Jing OU, Kang Du, Shuai Zhang, Dongli Wu, Taoyu Wu, Siyuan Feng, Xiaohui Zhu, Wufan Zhao
arXiv AI
Oct 28

V-LUMEN: Visual Lookup Memory for Embedding Scaling in Vision-Language Models

V-LUMEN introduces a visual lookup memory for vision‑language models by discretizing visual features with a frozen codebook, creating spatially aware memory keys via hashing, and modulating them with text‑conditioned relevance signals. The retrieved embeddings are fused into a frozen Qwen3‑VL‑2B backbone through a gated residual pathway, yielding substantial improvements on five reasoning‑intensive benchmarks and zero‑shot transfer to three disjoint image datasets. At a matched parameter budget, reallocating capacity from LoRA or MoE adapters to V‑LUMEN preserves accuracy on two backbones, demonstrating that external visual memory offers an independent capacity axis. whyItMatters":"The paper shows that adding a non‑parametric visual lookup memory can significantly boost vision‑language model performance without increasing per‑token compute, offering a new way to scale such models."

By Miso Choi, Daekeun Kim, Eunji Kim, Jungbeom Lee
arXiv Computer Vision
Oct 28

A note on data-efficient finetuning of vision foundation models

The paper evaluates data‑efficient finetuning methods for vision foundation models, focusing on Visual Prompting (VP) and comparing it with Linear Probing, Full Fine‑tuning, and Low Rank Adaptation across different model families (Attention, Convolution, Mamba). It provides an empirical overview of how these models and transfer paradigms interact, broadening previous studies by including more architectures and tuning hyperparameters. The authors also investigate regularization techniques to enhance VP performance, aiming to offer practical guidelines for users.

By Niclas Hergenr\"other, Antonio Orvieto
arXiv Computer Vision
Oct 28

From Slices to Volumes: Comparing 2D, 2.5D, and 3D Residual Networks for Pulmonary Nodule Malignancy-Risk Classification

The study compares 2D, 2.5D, and 3D residual networks for classifying pulmonary nodule malignancy risk using a retrospective LIDC‑IDRI cohort of 2,654 nodules. Models were trained with random initialization, momentum‑contrast pretraining, and mild data augmentation, and evaluated on patient‑disjoint splits. Augmented 2.5D models achieved the highest test Macro‑F1 (0.6537) and accuracy (0.6869), while 2D and 3D models performed lower, and pretraining generally reduced performance.

By Bahodir Nazarov, Zineb Gharib, Fatima Zazoul
arXiv AI
Oct 28

EvoIn: Bridging Evolution and Internalization for Agent Fine-Tuning

EvoIn is a fine‑tuning framework that improves agent decision‑making by evolving new procedures from execution traces, validating them in the harness, and then rewriting the traces to internalize the logic into the model. The approach removes reliance on the evolved harness at inference time, enabling stronger decision‑making that generalizes to unseen tasks. Experiments show consistent gains, raising in‑domain pass rates by 10.9 points and out‑of‑domain by 9.2 points, and the method works across different model families.

By Shihan Dou, Shaofan Liu, Zhonghang Lu, Jiahang Lin, Shichun Liu, Binghai Wang, Jiajie Jin, Guanting Dong, Tao Gui, Qi Zhang, Xuanjing Huang
arXiv Computer Vision
Oct 28

AWARE: Action-Entropy-Weighted and Attention-Guided Retrieval of Observation Evidence for Vision-Language-Action Models

AWARE is a plug‑in module for Vision‑Language‑Action models that retrieves observation evidence without extra training. It tracks uncertainty via Action Entropy, uses associative memory to fetch relevant features, and weights them with instruction‑guided attention, scaling the injection by entropy and depth. Experiments on LIBERO, SIMPLER, CALVIN, and real‑world robot tasks show consistent performance gains, especially on long‑horizon tasks, with negligible computational cost.

By Jiabing Yang, Yixiang Chen, Yuan Xu, Peiyan Li, Zichen Wen, Bowen Fang, Chenhang Cui, Tao Yu, Qisen Ma, Kai Wang, Ziheng He, Wen Ye, Xiangnan Wu, Yingda Li, Yan Huang, Liang Wang
arXiv Computer Vision
Oct 28

TALON: Token-Aligned Lightweight Adapters for 6-DoF Spacecraft Pose Estimation

TALON introduces Token‑Aligned Lightweight Adapters for 6‑DoF spacecraft pose estimation, inserting spatiotemporal 3D adapters before self‑attention layers of a frozen Vision Transformer. A patch‑token alignment loss grounds adapted features to keypoint structure, enabling the frozen backbone to reason over temporally enriched tokens with minimal added parameters. On the SPADES and SwissCube datasets, TALON cuts pose error by 50% and improves ADD‑0.1d accuracy by 21.8%, respectively, while also achieving significant gains in zero‑shot sim‑to‑real cross‑domain evaluation.

By Abid Ali, Nidhal Eddine Chenni, Arunkumar Rathinam, Djamila Aouada
arXiv AI
Oct 28

LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets

The paper introduces Functional Rank Allocation (FRA), a method for post‑hoc compression of LoRA adapters that balances rank allocation across layers and adapters within a shared budget. FRA constructs functional rank‑distortion frontiers for each adapter, uses dynamic programming to search for low‑risk truncations, and then applies minimax allocation to optimize overall fleet performance. Experiments on 78 adapters from three public fleets show that FRA outperforms uniform spectral compression, especially in constrained‑budget scenarios.

By Shenghui Li, Thiemo voigt
arXiv AI
Oct 28

AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation

AC-JEPA is a plug‑in for supervised fine‑tuning that predicts masked intermediate states in both directions of a cross‑modal translation task, combining a joint‑embedding objective with causal token‑level supervision. It is designed to be robust to imperfect paired data by learning from continuous latent targets derived from pseudo‑labels, rather than hard categorical supervision. In optical chemical structure recognition, AC‑JEPA outperforms plain supervised fine‑tuning, achieving 76.07 % accuracy on clean pairs and 80.23 % when self‑training with pseudo‑labels, whereas SFT alone drops to 63.47 % under the same conditions.

By Chaofei Hong, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang
arXiv AI
Oct 28

Learning to Formalize: Bridging Open-Ended Workloads and Fast Decision Models

The paper introduces a small candidate formalizer that bridges the gap between open-ended tasks handled by large language models and fast decision models that require a predefined candidate list. By generating explicit, validated alternatives from task context, the formalizer allows a frozen decision model to select an entry efficiently. Experiments on CALVIN-D show that using the formalizer improves success rates from 11.8% to 52.6% and reduces latency to 2.09 s compared to 3.67 s with a larger language model.

By Chengxuan Pei, Ziyao Wang, Teng Mei, Zhe Li
arXiv AI
Oct 28

Natural Hard-Negative Mining from Positives for Instruction-Following Retrieval

The paper introduces Natural Hard-Negative Mining (NHNM) for instruction‑following information retrieval, a method that turns original positive passages into hard negatives under query‑compatible instructions. NHNM uses a pipeline of Candidate Cleaning, Reversal Construction, and Geometry Filtering to generate a compact, high‑contrast training set called NHNM‑MARCO with nearly 7,000 instances. Models fine‑tuned on NHNM‑MARCO, ranging from 568 M to 4 B parameters, achieve strong instruction‑following performance on InstructIR and FollowIR, matching or exceeding larger retrievers and demonstrating that the gains stem from natural hard‑negative supervision rather than model size or data volume.

By Chaolang Li, Jingyu Li, Zhaocheng Du, Xuyang Xie, Gang Wang, Pengwen Dai
arXiv AI
Oct 28

KPPax: preprocess differentiable kinetic ODE for machine learning in atmospheric chemistry

KPPax is a new kinetic preprocessor that converts atmospheric chemistry mechanisms written in the Kinetic PreProcessor (KPP) language into differentiable, just‑in‑time compiled, vectorisable code in JAX. It evaluates rate laws in log space to maintain differentiability and integrates stiff systems using implicit solvers with forward‑ and reverse‑mode adjoints from the diffrax library, enabling exact sensitivities without hand‑derived adjoint code. Demonstrations with the POLLU benchmark show that KPPax can predict which rate coefficients are identifiable and improve neural‑network corrections of emission inventories, reducing errors by up to a factor of five.

By Wenqing Peng, Michael Boy, Zhi-Song Liu
arXiv AI
Oct 28

ReCoPII: A Relation-Grounded Framework for Building Contextual PII Masking Datasets

ReCoPII is a relation‑grounded framework that generates contextual PII masking datasets without assuming a single data subject. It constructs ontology‑constrained entity relations, writes insertion sentences with placeholders, and derives MASK/KEEP labels through automated annotation and LLM verification. Applied to Korean documents across six domains, ReCoPII‑Ko achieves high agreement with human annotators (88.42% on KEEP labels) and improves masking performance compared to SPY‑style augmentation.

By Taehwan Lee
arXiv AI
Oct 28

Generation and Verification: Human-Agent Collaboration for Safety-Critical Automotive Software Engineering

The paper introduces a coding assistant tailored for safety‑critical automotive software, offering generate, edit, and verify tools via the Model Context Protocol. It customizes a large language model through prompt engineering and fine‑tuning on a dataset of automotive production code linked to requirements and UML behavior. In a two‑week pilot, developers favored the verify tool and the prompt‑engineered model, noting fewer MISRA violations in C and better architectural conformance in C++.

By Elizaveta Zinovyeva, Aiham Taleb, Nikita Kozodoi, Daniel Schleicher, Amir Mahdi Namazi, Martin Kraus, Levent Kent, Brian Jensen
arXiv AI
Oct 28

Rollback: Experience Backtracking in World Models for Policy Improvement

The paper introduces Rollback, a method that uses action‑conditioned world models to backtrack to critical decision points, branch into alternative actions, and compare outcomes to attribute success to individual decisions. By turning world models into counterfactual engines, Rollback provides fine‑grained, action‑level supervision without a separate value model and integrates distributed world‑model search with asynchronous real‑robot execution. Experiments on block extraction, ball tossing, and tabletop curling show that Rollback improves success rates by over 10% compared to the strongest baseline and surpasses human performance on two tasks within eight hours of training.

By Jiayi Luo, Yang Zhang, Liliang Chen, Nan Wang, Chen Gao, Boxiang Qiu, Zhuoxuan Li, Baorui Peng, Chuankang Li, Yuxiang Yan, Si Liu, Qingyun Sun, Shuicheng Yan, Maoqing Yao, Jianxin Li, Guanghui Ren
arXiv AI
Oct 28

Agent or Human? Detecting Commits by AI Coding Agents in the Absence of Authorship Signatures

The study examines whether commits made by AI coding agents can be distinguished from human commits when authorship signatures are removed. Using nearly 8 million commits from over a thousand repositories, the authors find that detectors can achieve a ROC AUC of up to 0.859, largely driven by message verbosity and change size rather than code content. However, precision remains low (≈0.332) and detection performance drops on held‑out agents, indicating limited reliability without explicit attribution.

By Kosei Horikawa, Hao Li, Yutaro Kashiwa, Youssef Esseddiq Ouatiti, Mohammed Sayagh, Bram Adams, Hajimu Iida, Ahmed E. Hassan
arXiv AI
Oct 28

Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

The paper investigates fine‑tuning reinforcement learning agents when offline data cannot be retained. It shows that continuing online behavioral cloning after the offline dataset disappears erodes the pretrained policy, leading to representational collapse and loss of action diversity. By freezing the expressive prior and stopping online behavioral fitting, the authors achieve better performance across several algorithms on seven OGBench domains.

By Sami Nur Islam, Johan Obando-Ceron, Doina Precup, Aaron Courville, Pablo Samuel Castro