Fine-tuning and adaptation

LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.

8,201 stories · RSS feed

arXiv Computation and Language
Oct 28

Parallel Universes for Temporal Evaluation and Knowledge Updating in LLMs

The paper introduces PARALLELEVENTS, a benchmark that simulates realistic future event trajectories to evaluate knowledge updates in large language models without contaminating data or conflicting with existing knowledge. It also presents SYNAPSE, a training framework that uses model-generated synthetic data for instruction tuning, enabling scalable integration of new knowledge. SYNAPSE achieves a 14.23‑point improvement in QA accuracy over previous methods, demonstrating the effectiveness of simulation‑based synthetic training for coherent knowledge insertion.

By Jonathan Zheng, Zirui Shao, Alan Ritter, Wei Xu
arXiv Computer Vision
Oct 28

CRLoMTL: Conflict-Resolution through Low-rank matrices for Multi-Task Learning

CRLoMTL is a parameter‑efficient multi‑task learning framework that mitigates task interference by operating in a low‑rank parameter‑update space rather than gradient space. It learns task‑specific LoRA updates and fuses them through a shared‑subspace resolver based on singular value decomposition, producing a single deployment‑time update that can be folded into the backbone. Experiments on Swin Transformer backbones across PASCAL‑Context, NYUD‑v2, and CelebA show that CRLoMTL improves accuracy over single‑task baselines while training only a small fraction of the parameters.

By Ahmed Agiza, Tasneem Shaffee, Rahil Badkul, Sherief Reda
arXiv Computer Vision
Oct 28

SO-101 Bench: Accessible Spatial Manipulation

SO-101 Bench is a simulated benchmark for language‑conditioned spatial manipulation on the low‑cost SO‑101 robotic arm, built using an Isaac Lab digital twin of a real setup. It features four tasks across five families, 138 objects with seen/unseen splits, 916 evaluation episodes, and 3,629 simulated demonstrations for policy fine‑tuning. The benchmark’s rankings mirror real‑world performance (Spearman ρ = 1.00) and reveals that while learned policies reach about 61% success, an LLM‑driven agent that writes code and delegates grasping achieves 83.2%.

By Truman Hickok
arXiv Computer Vision
Oct 28

Frugal Handwritten Text Recognition for Sub-Saharan Civil-Status Registers: A Smartphone-to-Records Pipeline with CPU-Only Inference

The paper presents a low‑resource pipeline for converting handwritten civil‑status registers in Sub‑Saharan Africa into searchable text using only a smartphone for capture and a laptop CPU for inference. It introduces a new corpus of 1,568 smartphone images from a Chadian municipal registry, and shows that a classical preprocessing stage plus a fine‑tuned Kraken recognizer can achieve a character error rate of 0.50% and a word error rate of 3.37% on a test set with writers seen during training. The recognizer is lightweight (61 MiB) and runs in under 6 s per image on a six‑core laptop, demonstrating that such a frugal system is feasible for local administrations lacking GPUs or reliable connectivity.

By Ousmane Hamadou, Jean Louis Kedieng Ebongue Fendji
arXiv Computer Vision
Oct 28

SAGE: Semantic-Augmented Geometric Experts for Point Cloud Registration

SAGE (Semantic-Augmented Geometric Experts) is a new point‑cloud registration method that incorporates frozen, pre‑trained semantic representations into the registration pipeline without extra sensors or fine‑tuning. It uses a Semantic‑Guided Mixture of Experts (SG‑MoE) to route data so that each expert specializes in specific semantic‑geometric patterns, thereby reducing mismatches that plague geometry‑only approaches. On the 3DLoMatch benchmark, SAGE achieves the highest registration recall and, with the same number of refinement iterations, also attains the best inlier ratio and recall on both 3DLoMatch and 3DMatch.

By Tomoyasu Shimada, Kazuhiko Murasaki, Shoichiro Takeda, Shogo Sato, Riku Inoue, Toshihiko Nishimura, Ryuichi Tanida
arXiv Computer Vision
Oct 28

Zhengrong-Spatial-Aware 1.0: Advancing 3D Multimodal Intelligence

The paper introduces two end‑to‑end multimodal adaptation pipelines for 3D‑aware visual question answering on a custom spatial dataset. One pipeline fine‑tunes HiSpatial‑3B, adding metric‑scale 3D point maps to RGB inputs, while the other adapts Qwen3‑VL using memory‑efficient QLoRA supervised fine‑tuning. The engineering stack includes dataset conversion, image‑annotation alignment, monocular geometry preprocessing, instruction construction, mixed‑domain training, checkpoint management, batch inference, and automated evaluation, achieving significant accuracy gains on both the target spatial benchmark and a styled split.

By Yu Zhang, Xiaogang Wang, Yuhang Cheng, Jingrui Li, Wenbo Chen
arXiv Computer Vision
Oct 28

LEVIRDet: A Million-Scale 159-Category Dataset and Foundation Model for Universal Remote Sensing Object Detection

LEVIRDet-159 is the largest remote sensing object detection dataset, featuring 159 categories, 2.56 million bounding boxes, and 700k fine‑grained annotations across a multi‑level taxonomy. It surpasses existing datasets by roughly 7× more images, 6× more object instances, and 4× more categories. Leveraging this dataset, the authors propose LEVIRDetNet, a scale‑hierarchy‑aware foundation model that predicts ground sampling distance, modulates queries accordingly, and employs a hierarchy‑aware detection head, achieving state‑of‑the‑art performance on nine external benchmarks without target‑domain training or fine‑tuning.

By Qinzhe Yang, Dongyu Wang, Haohan Niu, Jia Xu, Zhenwei Shi, Zhengxia Zou
arXiv AI
Oct 28

From Structural Feedback to Prompt Policies: Learning Faithful Text-to-Image Prompt Editors

The paper introduces StruPO, a structure‑aware prompt editor that uses a calibrated scene graph to guide prompt modifications while preserving requested scene attributes. By aligning token edits with graph‑based structural feedback, StruPO learns a reusable editing policy that improves faithfulness and relational fidelity across multiple generation backbones. Experiments on diverse test sets and human studies show consistent gains in image quality and preference, especially for complex relations.

By Wentao Ye, Yali Ye, Zhiqing Xiao, Ru Peng, Xinpeng Ti, Liyao Li, Zhanming Shen, Peng Lu, Jie Zhang, Haobo Wang
arXiv AI
Oct 28

OA-RAG: Ontology-based Agentic RAG for Faithful LLM Reasoning

OA‑RAG is an agent that combines formal ontology reasoning with large language model (LLM) reasoning to produce more faithful answers. For each response it lists the ontology observations used as evidence and assigns a grounding level—entailed, supported, related, or unrelated—to indicate the strength of that evidence. On benchmarks from three large ontologies and six real‑world medical QA tasks, OA‑RAG outperforms other ontology‑fine‑tuned and popular RAG methods, especially on reasoning‑intensive questions, and its grounding levels correlate with higher accuracy.

By Hui Yang, Jiaoyan Chen, Yongsheng Gao
arXiv AI
Oct 28

LM-DSB: A Lightweight Multi-agent System for Automated Daily Safety Bulletin Generation in Construction

The paper introduces LM-DSB, a lightweight multi‑agent system that automates the creation of daily safety bulletins (DSBs) for construction sites using a retrieval‑augmented generation approach with a locally deployable large language model. The system employs specialized planner, researcher, and writer agents coordinated in a hierarchical architecture, fine‑tuned on construction safety documents and grounded in 37 HSE documents. Validation in two phases showed that the multi‑agent RAG system outperformed alternatives in lexical alignment, coherence, and document completeness, and expert reviewers rated it highly for regulatory compliance and factual accuracy.

By Xianxiang Zhao, Advik Mehta, Falak Sethi, Brian Gue, Qipei Mei, Lingzi Wu
arXiv AI
Oct 28

PsyJev: Psychological Analysis via System-1 Judgment

The paper introduces PsyJev, a System‑1 model for psychological analysis that directly outputs distributions over answers conditioned on questions. It leverages the PsyJudgeBench benchmark, which contains 13 annotated datasets and 82 questions across six psychological domains, and employs a joint supervised training followed by Judgment‑Anchored RLCD to adapt logit‑space reward optimization. Experiments on Qwen3.5‑based models show that the 9B variant achieves an average test F1 of 67.56, outperforming the strongest external baseline by 2.54 points.

By Fangyuan Liu, Sirui Zhao, Xiaohao Wang, Guo Cheng, Peng Zhang, Enhong Chen
arXiv AI
Oct 28

GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents

GUI-JEPA introduces a decoder‑free policy for mobile GUI agents that uses a Joint‑Embedding Predictive Architecture (JEPA) to generate action candidates directly in latent space. By predicting a latent representation for each candidate action, the model scores all options in a single batched pass without token decoding, dramatically reducing computation. Experiments on AndroidControl show that GUI‑JEPA is 2.8–4.0× faster than a 7B‑scale VLM agent while retaining 86–92% of its accuracy with only 6% of the parameters.

By Bingchao Wang, Yongcheng Jing, Shunyu Liu, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao
arXiv AI
Oct 28

ASTROLABE: A 3D Particle-Level Surrogate for Cosmology Simulation with Physical Flow Matching

ASTROLABE is a flow‑matching generative emulator that operates directly on the 6N phase‑space coordinates of individual particles, avoiding voxel grids and allowing any observable to be measured from inference. It uses a field‑particle hybrid transformer so that compute and memory scale linearly with the number of particles, and incorporates physical flow matching by basing the prior on COLA‑derived coordinates with momentum and mass conservation enforced throughout the flow. In Lambda‑CDM tests, ASTROLABE reproduces realistic fields, accurate power spectra, halo mass functions, velocity profiles, and halo density profiles, and it generalizes to out‑of‑distribution dark‑acoustic oscillation initial conditions while also being fine‑tuned to emulate large‑scale simulations such as TNG50‑3‑dark at a fraction of the cost.

By Ibrahim Elsharkawy, David Curtin, Yonatan Kahn, Neal Weiner
arXiv AI
Oct 28

Conservative Dueling Bandits: Preference Learning under Baseline Constraints

The paper introduces conservative contextual dueling bandits, which enforce that each displayed item’s cumulative score remains above a specified fraction of a trusted baseline’s score. It presents algorithms—MaxInP-BC for linear utilities and BTIGW-BC for general utilities—that guarantee safety with high probability while achieving sublinear regret. Experiments show these methods satisfy the safety constraint more reliably than unconstrained counterparts, with only a moderate increase in regret.

By Rohan Deb, Mohammad Ghavamzadeh, Arindam Banerjee
arXiv AI
Oct 28

Robust LLM Data Selection via Content-Invariant Gradient Attribution

The paper introduces Content-Invariant Gradient Attribution (CIGA), a method for selecting training data for large language models that mitigates surface-form bias in gradient-based attribution. CIGA identifies layers whose gradients are both influential and stable under semantics-preserving perturbations, then fuses these signals with semantic representations to produce robust data rankings. Experiments show that CIGA yields more stable rankings, maintains downstream performance, and reduces attribution cost.

By Jia Gu, Liang Pang, Huawei Shen
arXiv AI
Oct 28

PerceptRL: Making Visual Perception Rewardable in MLLMs from Answers to Interactions

PerceptRL enhances visual perception in multimodal large language models by employing reinforcement learning that transforms dense image captions into verified QA pairs with rule‑based rewards. It uses a multi‑agent pipeline to link correlated errors, combine unrelated errors, and target high‑error domains, while a specialized multi‑turn inspection process corrects spatial inaccuracies. Across seven benchmarks, PerceptRL improves average accuracy to 71.6%, outperforming both the base model with tools and the leading evaluated VLM by significant margins.

By Tongkun Guan, Zhibo Yang, Mingkun Yang, Haocheng Wang, Wei Shen, Xiaokang Yang
Hugging Face Trending Papers
Oct 27

A note on data-efficient finetuning of vision foundation models

The paper evaluates data‑efficient finetuning methods for vision foundation models, focusing on Visual Prompting (VP) and comparing it with Linear Probing, Full Fine‑tuning, and Low Rank Adaptation across Attention, Convolution, and Mamba model families. It expands on prior work by testing a broader set of models, tuning hyperparameters, and applying regularization techniques to improve VP performance. The study aims to provide practical guidelines for users working under resource constraints.