Fine-tuning and adaptation

LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.

8,205 stories · RSS feed

arXiv Computer Vision
Sep 16

VPRef: A Cross-Domain Benchmark for Referring Remote Sensing Image Segmentation

The paper introduces VPRef, the first cross‑domain benchmark for Referring Remote Sensing Image Segmentation, containing 46,972 language‑image‑annotation triplets with a three‑tier linguistic hierarchy. It proposes a parameter‑efficient adaptation method based on the Segment Anything Model and Low‑Rank Adaptation, using pseudo‑label self‑training for visual drift and random multi‑granularity prompt mixing for textual drift. Experiments show the approach improves cross‑domain segmentation while altering only 1.08 % of the base model’s parameters, offering a strong baseline for future research.

By Quanwei Liu, Tao Huang, Jiaqi Yang, Wei Xiang
arXiv Machine Learning
Sep 16

A deep dictionary network-based foundation model for ultra-low-dose CT denoising

The paper introduces a deep dictionary network (DDN) foundation model designed for ultra‑low‑dose CT (ULDCT) denoising across multiple organs. By cascading convolutional sparse coding layers with iterative soft‑thresholding, the architecture offers inherent interpretability, while dynamic dictionary and threshold modules enhance representation. The model is pre‑trained on over one million normal‑dose CT images and fine‑tuned on multi‑organ ULDCT datasets, achieving state‑of‑the‑art performance that consistently outperforms existing methods.

By Baoshun Shi, Shuangyi Yang, Ke Jiang, Bin Zhu, Zhanli Hu, Huazhu Fu
arXiv Computation and Language
Sep 16

Enhancing Accessibility of Medical Texts through Large Language Model-Driven Plain Language Adaptation

The paper explores how Large Language Models (LLMs) such as GPT-4o-mini, Gemini-1.5-pro, and LLaMA can be used to automatically simplify complex medical texts into plain language, a process known as Plain Language Adaptation (PLA). It evaluates different prompting strategies, fine‑tunes models with QLoRA, and incorporates Mixture‑of‑Agents (MoA) techniques to improve adaptability and robustness. The results show that LLM‑driven PLA effectively makes healthcare information more comprehensible while preserving essential content.

By Ting-Wei Chang, Hen-Hsen Huang, Hsin-Hsi Chen
arXiv AI
Sep 16

The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis

The paper introduces MIMIC, a framework that uses executable code to generate rigorous reasoning data for large language models (LLMs). By converting algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation, MIMIC creates a Code-Instrumented Reward (CIR) that supplies dense, high‑fidelity supervision for reinforcement learning. Models trained with MIMIC’s synthetic dataset show significant, consistent improvements in general reasoning, complex mathematics, and fine‑grained deterministic tasks.

By Jinyang Zhang, Weibin Liao, Keqin Bao, Sihang Li, Shaobo Wang, Muyang Ye, Hongxin Ding, Yue Fang, Tianyi Tang, Fei Huang, Kexin Yang, Xingzhang Ren, Dayiheng Liu
arXiv Machine Learning
Sep 16

From Foundation Embeddings to Cropland Maps: Label Efficiency, Temporal Transferability and Independent Human Validation

The study evaluates the use of frozen geospatial foundation embeddings (AlphaEarth) for mapping cultivated versus non‑cultivated land in Maine. Using 192 spatially separated patches and USDA Cropland Data Layer labels, a lightweight classifier achieved 93.7% overall accuracy without fine‑tuning, and a nearest‑class‑centroid rule reached 90.2%. A balanced sample of 60,000 labeled pixels was nearly as effective as the full 8.6 million‑pixel pool, and classifiers trained in one year remained accurate across 2018‑2023. In a blind human validation of 385 points, the AlphaEarth‑plus‑random‑forest map matched 95.3% of the consensus, outperforming the CDL reference (91.7%).

By Mohammad Ammar Mughees, Giovanni Montefoschi, Zhongxin Chen, Maria Antonia Brovelli
arXiv Machine Learning
Sep 16

EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

EMODY Flow is a lightweight flow‑matching framework that generates synchronized full‑body motion and facial expressions conditioned on speech and emotion. It attaches to a frozen Qwen‑3 Omni model, reusing its audio codecs to drive two parallel DiT generators for SMPL‑X body pose and FLAME facial expressions. An auxiliary emotion classifier at training time restores emotion sensitivity, enabling EMODY Flow to achieve state‑of‑the‑art gesture quality on BEAT2 and zero‑shot facial animation on TFHP, with significant improvements in FGD, Beat Correlation, and Diversity metrics.

By Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin
arXiv AI
Sep 16

HoloAegis: Frozen Representation, Topological Inference --- Minimally Parametric Safety Manifolds and Their Capability Boundaries for LLM Guardrails

HoloAegis is a minimally parametric topological inference framework that uses frozen representations to map text onto the unit sphere and makes decisions via Gibbs‑Boltzmann free‑energy differences over pre‑computed anchor centroids. On a frozen three‑benchmark protocol, it matches WildGuard‑7B on toxicity, outperforms it on harmful behaviors, but underperforms on oversafety detection, while ShieldGemma‑2B fails on indirect harms. The study demonstrates that geometric guardrails can substitute for LLM judges in some cases and must defer to them in others, with anchor banks reducing score variance and boundary displacement.

By Tak Ho Alex Li, Kaijie Liu, Lik-Hang Lee, Kin Chung Ho, Ping Shum, Michael K. Ng
arXiv AI
Sep 16

KaiNinja: Extending Native 3D Generators to the Part Level

KaiNinja extends the native 3D generator TRELLIS.2 to produce part-level meshes by introducing a dual‑volume representation that overcomes the single‑sheet limitation of the O‑Voxel grid. It maintains TRELLIS.2’s speed and quality while eliminating the need for external segmentation, and is trained on diverse data including CAD models and assets created by an LLM‑driven agent. The method improves whole‑object fidelity and outperforms other part‑generation pipelines, reducing Chamfer distance by 40% and increasing strict part F‑score by 16%.

By Ruihan Yu, Lian Fu, Muyao Niu, Zheng-hui Huang, Yu-Ju Tsai, Sho Kuno, Fengbo Lan, Yonghao Yu, Erwin Wu, Ming-Hsuan Yang, Kaipeng Zhang, Zhixiang Wang
arXiv Computer Vision
Sep 16

FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

FLAT (Flexible‑Length Aligned Transmodal representations) is a joint multimodal pre‑training framework that learns a shared encoder for images and text, producing 1‑D continuous embeddings that can be directly used by downstream generative decoders. By combining contrastive alignment with bidirectional cross‑modal generative objectives, FLAT yields representations that are both discriminative and generative, enabling cross‑modal retrieval and generation with a single pre‑training stage. The model achieves strong performance on T2I generation (GenEval 71.1), image captioning (BLEU‑4 40.5, CIDEr 138.6), and retrieval tasks (Recall@5 86.8/75.8 on MS‑COCO, 98.3/93.6 on Flickr30K), and supports linear interpolation, latent space arithmetic, and zero‑shot composed retrieval.

By Guangyu Sun, Shlok Kumar Mishra, Wentao Bao, Robert Zhenheng Yang, Xiao Wang, Xiyuan Wang, Yujunrong Ma, Chen Yuan, Max Xiangjun Fan, Jun Xiao, Jianpeng Cheng
arXiv Machine Learning
Sep 16

Drift Field Net: Learning Ocean Lagrangian advection fields from in-situ and satellite observations

Drift Field Net (DFN) is a deep neural network that predicts ocean surface flow fields from satellite observations, trained via a two‑stage strategy combining simulated data pretraining and Lagrangian fine‑tuning with an advection‑consistent loss. DFN improves particle trajectory forecasts, reducing mean positioning error by 20 km over a 7‑day period compared to an operational physics‑based model, and further decreasing error by 10 km when the advection loss is applied. The study demonstrates that incorporating Lagrangian constraints into deep‑learning training enhances ocean surface flow prediction accuracy.

By Th\'eo Archambault, Pierre Garcia, Mattia Romero, Anastase Charantonis, Dominique B\'er\'eziat
arXiv Computation and Language
Sep 16

What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity

The study investigates how pruning affects large language models (LLMs) used for smart‑home tool calling. Researchers examined four LLMs—dense Transformer, dense hybrid, and mixture‑of‑experts (MoE) architectures—using depth, width, hybrid, and expert pruning, followed by supervised fine‑tuning. They evaluated over 19,500 instances from three smart‑home datasets, analyzing not only overall accuracy but also degradation in action components (operation, device, argument, value) and task complexity, finding that dense models suffer sharp performance drops after a narrow safe pruning range, while MoE models tolerate more pruning; aggressive pruning also leads to over‑refusal and loss of grounded specificity.

By Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem
arXiv Computation and Language
Sep 16

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

The paper investigates whether in‑context learning (ICL) can replace instruction tuning for multilingual language models, especially as model size varies. It highlights the difficulty of obtaining high‑quality instruction data in multilingual settings and compares the performance of ICL versus instruction‑tuned models. The findings show that a performance gap persists between the two approaches, suggesting the need for further research to close it.

By David Ponce, Thierry Etchegoyhen
arXiv Machine Learning
Sep 16

GrowMTP: Can RL Grow Its Own Draft Head?

GrowMTP is a method that trains a draft head entirely within the reinforcement learning (RL) loop, using supervision from the RL verification step and a rollout distribution that is narrower than pretraining. By detaching draft‑head updates from the policy backbone, it enables online training of the draft head from scratch. Experiments on Qwen3‑4B, MiMo‑7B‑SFT, and Qwen3.5‑4B‑Base show rollout speedups ranging from 1.36× to 2.13× and overall end‑to‑end speedups from 1.20× to 1.60×, making it a modular acceleration component for RL frameworks lacking pretrained draft heads.

By Minghua He, Lingzhe Zhang, Yuan Liu, Xiao Zhou, Aiwei Liu