Fine-tuning and adaptation

LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.

8,201 stories · RSS feed

arXiv Machine Learning
Oct 16

LCT: Losslessly Compressed Tensors with GPU friendly execution for Training

The paper introduces Losslessly Compressed Tensors (LCT), a GPU‑friendly framework that repeatedly encodes and decodes tensors during training. LCT uses Huffman coding with precomputed codebooks to exploit redundancy in floating‑point data, achieving up to a 1.455× compression ratio. Experiments on models such as NanoGPT and Qwen3‑4B show memory savings of 24.6% and 23.8% respectively, enabling larger context lengths with only modest increases in step time.

By Max Zhu, Bohang Sun, Pietro Lio, Jialin Yu
arXiv Machine Learning
Oct 16

Topology-Inspired Fine-Tuning for Wind-Field Super-Resolution

The paper introduces topology‑inspired fine‑tuning for wind‑field super‑resolution, adding auxiliary losses based on persistence diagrams and merge‑tree distances to a pretrained CNN. A comprehensive study of 16 fine‑tuning setups shows that gradient supervision improves bottleneck persistence‑diagram metrics, while fixed‑pair supervision yields the most consistent gains in both persistence‑diagram and merge‑tree distances. The combined objective reduces key topological distances by up to 31% and improves PSNR by 1.30 dB compared to the original CNN.

By Arjun Dadhwal, Brian Summa
arXiv Machine Learning
Oct 16

CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty

The paper introduces Continuous Adversarial Training for Consistent Honesty (CATCH), a method that trains large language models to produce honest responses while preserving their overall capabilities. Using only a few hundred self‑reporting scenarios, CATCH is applied to Llama‑3.3‑70B‑Instruct and Qwen3‑3.6‑27B, and the resulting models show improved honesty across multiple benchmarks, including MASK, Among Us, and AuditBench. Compared to prompting, supervised fine‑tuning, and state‑of‑the‑art honesty steering, CATCH achieves superior honesty while maintaining utility through a novel choice of utility data.

By Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato
arXiv Machine Learning
Oct 16

Soft Sensor for Bottom-Hole Pressure Estimation in Petroleum Wells Using Long Short-Term Memory and Transfer Learning

The paper presents a machine‑learning soft sensor that estimates bottom‑hole pressure (BHP) in petroleum wells using wellhead and topside data. It employs a Long Short‑Term Memory (LSTM) model, compares it with Multi‑Layer Perceptron and Ridge Regression, and introduces transfer learning to adapt the model across different operational settings. Tested on offshore datasets from Brazil’s Pre‑salt basin, the LSTM achieved a mean absolute percentage error below 2%, outperforming the benchmarks.

By M. A. Fernandes, E. Gildin, M. A. Sampaio
arXiv Machine Learning
Oct 16

Amortized Factor Inference Networks for Posterior Inference

The paper introduces Amortized Factor Inference Networks (AFINs), a single inference network that can generalize across varying priors, likelihoods, latent dimensions, and observation counts. AFINs represent a model as a list of typed prior and likelihood factors and use an encode–merge–decode architecture to produce a Gaussian posterior approximation, with learned parameters independent of latent dimensionality or observation count. Experiments on synthetic and real datasets demonstrate that a single trained AFIN achieves competitive posterior approximations at significantly lower test‑time cost than NUTS and per‑task variational inference, with optional importance sampling further improving accuracy.

By Joohwan Ko, Justin Domke
arXiv Machine Learning
Oct 16

Multi-Modal, Multi-Temporal Geospatial Foundation Models Improve Global Above-Ground Biomass Estimation

The study evaluates 13 Geospatial Foundation Models (GFMs) for estimating above‑ground biomass (AGB) from satellite imagery, comparing them to a fully supervised state‑of‑the‑art baseline trained on the AGBD dataset. Most GFMs perform worse than the baseline, but TESSERA and AlphaEarth Foundations (AEF) features combined with specific prediction heads achieve superior results, with the best model attaining an RMSE of 50.79 Mg/ha. This AEF‑based model also generalizes better to unseen regions, reduces RMSE by up to 25% in excluded areas, and matches the ESA CCI biomass product when trained on limited regional labels.

By Ghjulia Sialelli, Linus Scheibenreif, Jan Dirk Wegner, Konrad Schindler
arXiv Machine Learning
Oct 16

Identifier-Free Code Embedding Models for Scalable Search

The paper introduces Identifier-Free Code Embedding Models designed to improve function association for binary reverse engineering. It formalizes the bidirectional association problem between source code and decompiled, stripped code, and evaluates embedding models on this task. By fine‑tuning a Qwen3‑Embedding model with contrastive learning, the authors achieve significant performance gains over existing baselines and demonstrate generalization to a constant‑algorithm association task.

By Eric Wolos, Michael Doyle
arXiv Computation and Language
Oct 16

PRISM: PRIvileged Sibling-guided Multi-Agent Topology Design

PRISM is a method for designing collaboration topologies in multi‑agent systems that use large language models. It leverages privileged information by having a teacher model read a sibling graph that also solved the same query, guiding a student model to match per‑decision targets during its own trajectory sampling. Experiments on six diverse benchmarks show that PRISM outperforms all evaluated topology‑design baselines, and matched‑teacher controls confirm the benefit of the privileged query information.

By Phuong Tuan Dat, Qi Li, Xinchao Wang
arXiv Computation and Language
Oct 16

What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective

The paper introduces a weighted in-context influence (wICI) framework for selecting instruction‑tuning data, aiming to reduce redundancy and improve sample quality. It investigates what makes data effective from an in‑context perspective, explores the relationship between sample difficulty and influence, and evaluates how influence predicts tuning performance. Experiments on various models and benchmarks show that wICI consistently outperforms existing baselines under limited data budgets and reveals a negative correlation between sample difficulty and in‑context influence.

By Guangzeng Han, Xiaolei Huang
arXiv Computer Vision
Oct 16

Trajectory Prediction Method for person under WSN with Hierarchical Envelope Adversarial Domain Adaptation

The paper introduces HEADA‑LTCTPM, a trajectory prediction method that uses a Hierarchical Envelope Adversarial Domain Adaptation framework with a loosely‑tightly coupled model. It constructs envelope samples into graph structures and applies both local and global adversarial domain adaptation to better capture diverse social interactions and reduce domain shift. Experiments on two public datasets show the model improves ADE and FDE by 17.6% and 19.1% over six baselines while maintaining practical time overhead.

By Wenqian Jiang, Qifeng Sun, Xiaosong Gao, Xin Zou, Xingrui Ni, Yuan Hu, Yongming Li
arXiv Computer Vision
Oct 16

AdaLoop: Thinking with the Vision Encoder

AdaLoop introduces a question‑conditioned visual recurrence framework for vision‑language models, allowing the vision encoder to adapt its computation to each question while keeping pretrained weights frozen. The method reuses encoder transformations across multiple iterations, improving performance over standard fine‑tuning and single‑pass baselines. Experiments on the VisRO dataset demonstrate gains across three VLM families on both in‑distribution and out‑of‑distribution examples.

By Xinran Nicole Han, Yiyang Lu, Katelyn Gan, Todd Zickler
arXiv Computer Vision
Oct 16

Teaching DINOv3 About Partial 3D Geometry: A Self-Supervised Geometry-Aware Approach

The paper introduces a self‑supervised, geometry‑aware method that refines DINOv3 features with LoRA to better handle partial 3D shapes. By training on partial observations, the resulting descriptors outperform traditional and other vision‑foundation features in partial shape matching tasks. When incorporated into existing pipelines, the approach achieves state‑of‑the‑art results on both partial shape matching and left‑right prediction benchmarks.

By Viktoria Ehm, Dongliang Cao, Riccardo Marin, Daniel Scholz, Weikang Wang, Florian Bernard, Daniel Cremers
arXiv Computer Vision
Oct 16

Seeing Both the Forest and the Trees: Reusing Holistic 3D Priors for Part-Decomposed Generation

STRUCTPARTS is a new framework that reuses holistic 3D priors to generate part‑decomposed 3D assets. It introduces a dual‑frame coordinate interleaving mechanism, interpreting each shape latent under both part‑local and object‑global reference frames and processing them with a global‑local inter‑leaved transformer. Using a segmented mesh scaffold as structural input, the method achieves detailed intra‑part geometry and coherent inter‑part structure with lightweight finetuning, outperforming prior part‑decomposed models on PartObjaverse‑Tiny while enabling part‑mesh refinement and editing.

By Yiqun Zhao, Binbin Huang, Haobin Duan, Zibo Zhao, Shenghua Gao
arXiv Computer Vision
Oct 16

Retrofitting Recurrent Vision-Language Models from Pretrained Backbones

The paper introduces a method to retrofit depth recurrence into pretrained vision‑language models, reducing the language backbone’s parameter count by pruning layers and looping over the remaining ones with linear adapters. A three‑stage recovery training—language‑only pretraining, vision‑language projector alignment, and full fine‑tuning—helps recover performance, with distillation from the non‑recurrent model further improving results. Experiments on TinyLLaVA‑1.4B show the retrofitted recurrent model outperforms the unpruned baseline while saving 23.3% of language parameters and performs better on reasoning‑oriented benchmarks.

By Madhav Iyengar, Andreas Geiger, Hilde K\"uhne
arXiv Computer Vision
Oct 16

MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents

MMSearch-OPD is a new approach for long‑horizon multimodal search agents that combines a controllable data refinement pipeline with joint reinforcement learning and on‑policy distillation. The refinement process expands reasoning chains from existing simple multimodal questions, allowing adjustable search depth and cross‑modal evidence requirements while reusing image–question pairs. Training uses answer rewards, behavior penalties, and a decaying distillation coefficient to guide the agent, and experiments demonstrate that both data refinement and joint training after supervised fine‑tuning improve long‑horizon multimodal search performance.

By Huanyao Zhang, Jiepeng Zhou, Jiaye Lin, Ontai Tang, Jingtian Hao, Mugeng Liu, Ru Peng, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang
arXiv Computer Vision
Oct 16

Gen4P: Video Generators are 4D Perceivers

Gen4P is a unified framework that transforms a pretrained video generator into a feed‑forward 4D perceiver, enabling recovery of 4D geometry and motion from monocular video without iterative denoising. It processes video latents in a single encoder pass and uses a lightweight cross‑attention decoder to predict requested 4D attributes, supporting flexible spatial sampling and continuous‑time queries. By integrating a frozen video‑depth LoRA as a geometry prior, Gen4P improves performance on 3D point tracking, video depth, and camera pose estimation, achieving state‑of‑the‑art or competitive results on benchmark tasks and demonstrating robust tracking of complex motion in in‑the‑wild videos.

By Yitong Xia, Zeren Jiang, Chuanxia Zheng
arXiv Computer Vision
Oct 16

Denoising Textures is a Different Ballgame Altogether!

The paper investigates how current image denoisers handle textures, finding that generic denoising models often over‑smooth fine patterns and that improving texture performance requires fine‑tuning on texture‑specific datasets. However, such fine‑tuning causes the models to lose their ability to denoise the original data, indicating a trade‑off. The authors argue that dedicated, texture‑aware denoisers are necessary for accurate texture reconstruction and for downstream tasks like generative modeling.

By Saptarshi Mandal, Oindrila Haldar, Chirayata Bhattacharya, Chandra Sekhar Seelamantula, Zhiwei Xu, Thrasyvoulos N. Pappas, Gaurav Sharma
arXiv Computer Vision
Oct 16

Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought

The paper introduces Numerical Visual Chain-of-Thought (NV‑CoT), a framework that expands multimodal large language models (MLLMs) to reason over images using continuous numerical coordinates instead of discrete tokens or fixed patches. NV‑CoT allows MLLMs to generate bounding‑box coordinates directly as actions with minimal architectural changes, supporting both supervised fine‑tuning and reinforcement learning via a Gaussian or Laplace policy. Experiments on three benchmarks show that NV‑CoT improves localization precision, answer accuracy, and training convergence compared to 11 visual reasoning baselines.

By Kesen Zhao, Beier Zhu, Junbao Zhou, Xingyu Zhu, Zhongqi Yue, Hanwang Zhang
arXiv Computer Vision
Oct 16

V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance

The paper introduces V‑GIFT, a lightweight method that enhances visual instruction tuning for multimodal large language models by incorporating a small set of visually grounded self‑supervised tasks expressed as natural language instructions. These tasks—such as rotation prediction, color matching, and cross‑view correspondence—are reformulated into image‑instruction‑response triplets, providing supervision that cannot be solved without visual evidence. The approach requires no human annotations, architectural changes, or extra training stages, and consistently improves vision‑centric performance across various models, regimes, and benchmarks when only 3‑10% of the training data is augmented with these instructions.

By Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc, Nicolas Thome, Spyros Gidaris
arXiv Computer Vision
Oct 16

Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

The paper introduces Locked Image Multiple Instance PreTraining (LIMIT), a method that aligns slide‑level pathology reports with patch‑level visual features by freezing a vision encoder and fine‑tuning a text encoder. LIMIT uses report embeddings as cross‑attention queries over bags of patch features, optimizing slide representations via InfoNCE. Applying LIMIT, the authors create CONCH‑Z, which achieves state‑of‑the‑art zero‑shot performance on 22 pathology tasks, outperforming vision‑language encoders trained on curated caption datasets and narrowing the gap with larger slide‑level foundation models.

By Daniel Shao, Dongmin Bang, Luca L. Weishaupt, Sophia J. Wagner, Nic G. Reitsam, Ming Y. Lu, Long P. Le, Richard J. Chen, Faisal Mahmood