LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.
arXiv:2609.07780v2 Announce Type: replace Abstract: Automated drone surveillance has become increasingly important for public safety, critical infrastructure protection,and restricted airspace monito...
The paper introduces OmicsBench, a new reasoning benchmark for multi‑omics sequences that includes 1,160 expert‑validated questions across DNA regulation, RNA processing, and protein function tasks, requiring traceable evidence chains. Evaluation of 17 large language models shows that scientific LLMs, while more accurate in classification, often lack valid evidence, suggesting shortcut learning. To address this, the authors propose tool‑augmented on‑policy distillation (TA‑OPD), a post‑training method that improves both evidence grounding and predictive performance across five Qwen3.5 models of varying sizes.
The paper introduces UniK, a universal knowledge perception platform designed to serve both digital AI—such as chatbots and agent workflows—and physical AI, which controls robots and autonomous systems. UniK handles the entire knowledge lifecycle—ingestion, enrichment, indexing, retrieval, and continuous evaluation—across diverse modalities including text, video, molecular data, and sensor telemetry, without task‑specific fine‑tuning. In five digital AI domains, UniK paired with a 70‑billion‑parameter model matches or surpasses larger proprietary LLMs, achieving high retrieval‑augmented generation accuracy on government data, medical QA, and chemistry tasks, and it also addresses similar data challenges in physical AI world‑model training.
The paper introduces Dial, a knowledge‑grounded framework for generating SQL queries that respect the specific dialect of heterogeneous database systems. Dial comprises a Dialect‑Aware Logical Query Planning module, a hierarchical intent‑aware knowledge base (HINT‑KB) containing canonical syntax, function, and constraint repositories, and an execution‑driven debugging loop that separates syntactic recovery from logic auditing. The authors also present DS‑NL2SQL, a benchmark of 2,218 dialect‑specific test cases across six major database systems, and report that Dial improves translation accuracy by 10.25% and dialect feature coverage by 15.77% over existing baselines.
Euston is an 8‑B parameter mathematical claim‑verification model that resists producing false derivations when presented with corrupted theorems. It was trained on 3,026 matched true/corrupted statement pairs generated by GraphSynth, a probabilistic factor‑graph generator, and fine‑tuned from DeepSeek‑R1‑8B using GRPO. On a balanced held‑out split, Euston’s balanced accuracy rose from 29.50 % to 63.75 %, and its discrimination gap improved from –0.5 % to +27.5 %, while maintaining comparable general mathematical ability and reducing response length and truncation rates.
The paper presents a smartphone‑compatible, retrieval‑augmented language model tailored to Bangladeshi statutory law. By distilling a 9‑billion‑parameter Gemma‑2 teacher into a 2‑billion‑parameter student using supervised fine‑tuning and QLoRA, the authors achieve significant gains in ROUGE‑L and BERTScore on an English benchmark while keeping the model lightweight (1.6 GB) and operable offline on a Pixel 6. The system retrieves from 36,029 statutory passages using a hybrid dense/BM25 approach, and cross‑lingual evaluation shows effective Bangla query handling against an English‑only corpus, with a practicing lawyer rating the responses highly in a pilot study.
This thesis explores how to select and adapt NLP models for global health literature when annotated data and computational resources are scarce. It compares skip‑gram word2vec models trained on increasingly large specialized corpora with BioWordVec for semantic tag discovery, finding that larger coverage does not always yield more useful domain associations. The study also evaluates convolutional spaCy models versus a RoBERTa transformer for named entity recognition, noting a trade‑off between higher F1 scores and longer inference time, and investigates MiniLM few‑shot versus BART‑MNLI zero‑shot classification for multi‑label topic classification, highlighting practical constraints of inference cost. "whyItMatters":"The work provides empirical guidance on balancing model accuracy and resource demands for building knowledge systems in low‑resource global health settings."
CoaG introduces a minimal geometric interface— a ground plane grid and a single cylinder per person—to control the placement and movement of people and the camera in generated videos. Users draw the grid, position cylinders, and animate them over 81 frames; the model then renders a photorealistic video that follows these layouts and camera paths, with appearance guided by a text prompt and a background image. To train the system, the authors automatically generate 2,000 captioned clips, reconstruct geometry from each clip, and fine‑tune a LoRA on Wan2.2‑Fun‑Control, enabling the model to respect user‑drawn layouts and camera motions on unseen clips.
The paper introduces a free, label‑free visual evidence signal that improves fine‑grained vision‑language reasoning. By selecting image crops that maximize the model’s answer distribution peak, the method locates answer‑bearing regions without training or annotations, boosting accuracy from 70 % to 85 %. The evidence gap also complements model confidence, enabling better correctness prediction and error flagging.
AnalogDepth adapts the Depth Anything 3 (DA3) visual geometry model to the spatially structured noise of analog video transmission (VTX) used in FPV drones. The method employs a parameter‑efficient training pipeline that uses student‑teacher knowledge distillation with Low‑Rank Adaptation (LoRA) on the DINOv2 backbone, and trains on a noise bank built from real FPV recordings rather than synthetic Gaussian noise. Experiments on six real FPV flight sequences across three indoor scenes show that this real‑noise training consistently lowers per‑frame depth RMSE and 3D reconstruction Chamfer distance compared to the pretrained DA3 baseline and Gaussian noise baselines.
AdaptiveCDM is a modular framework for source‑free few‑shot domain adaptation in cell detection, enabling a pretrained model to adapt to new imaging domains using only a handful of labeled target images and no source data. It combines Resolution‑Aware Augmentation (RAug) to balance scarce, class‑imbalanced samples while preserving cellular morphology, and Category‑Aware Representation Learning (CARL) to strengthen class‑consistent proposals for better localization and classification. Experiments on M5 and Raabin‑WBC datasets show that AdaptiveCDM achieves competitive or superior mAP scores compared to state‑of‑the‑art methods under their respective supervision settings.
The paper investigates selective on‑policy distillation, where a student model is trained only on token positions chosen by a selector. It demonstrates that the commonly used shared learning rate is not neutral: performance varies significantly with the learning rate for different selectors, leading to inconsistent comparisons. The authors attribute this selector‑rate entanglement to the selection process itself and recommend reporting the full arm‑by‑rate matrix for fair evaluation.
Merge++ is a post‑hoc refinement technique for model merging that synthesizes task‑representative images by inverting expert checkpoints and then distills expert knowledge into a merged model. It operates without any additional data beyond the checkpoints and can be applied universally across existing weight‑space merging algorithms. Experiments show consistent improvements, with average gains of +2 to +8 points and up to +25.9 on specific configurations.
The study investigates whether language models can explicitly report constraints they have learned through post‑training fine‑tuning. Using constrained recipe generation with five banned ingredients, the authors compare supervised fine‑tuning (SFT) and Group Relative Policy Optimization (GRPO) against an untrained baseline on a Constraint Awareness Benchmark. Both fine‑tuning methods increase behavioral compliance from 4% to about 90% but reduce explicit constraint reporting and erode retained third‑person knowledge, with GRPO showing more destructive effects. The results suggest that reward‑based signals may suppress constraints context‑independently, and that models fail to enumerate constraints on request even when they can avoid them internally.
Swiss-Knife is a framework that extends decode‑time alignment for frozen language models by treating the alignment specification as a runtime object. It introduces hot‑swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule, and characterises admissible aggregation operators with a representation theorem. In experiments, Swiss‑Knife paired with DPO‑LoRA blades and an uncertainty‑aware pairwise tournament outperforms six existing decode‑time methods, achieving a higher harmonic F1 score, lower refusal rate, and faster objective reconfiguration.
The paper introduces a cross-dialect Named Entity Recognition (NER) framework for Bangla, leveraging the ANCHOLIK-NER dataset that covers five major regional dialects. Using a Leave-One-Dialect-Out Cross-Validation strategy, eight transformer-based models were evaluated, with Multilingual-E5 Large achieving the best performance (F1 up to 97.26% on Mymensingh, 82.38% on Chattogram). Local Interpretable Model-agnostic Explanations (LIME) revealed that the models rely mainly on the surface form of entity words rather than surrounding context, suggesting a direction for future improvement.
The paper investigates time‑incremental continued pretraining (CPT) of large language models using web‑scale data that overlaps across snapshots. Across six open‑weight models, CPT improves factual recall without catastrophic forgetting, while the cost is negligible and data quality outweighs quantity. The study identifies optimal learning rates, shows LoRA can match full CPT, and demonstrates that CPT gains transfer to fine‑tuned models.
MinCU is a new benchmark for grounded minimal‑change understanding that presents pairs of near‑identical images differing by a single atomic variation in object category, attribute, count, or spatial position. Models are evaluated on their ability to describe the change, localize the changed region, and identify the changed entity. The authors also introduce SG‑ISA, a structured autoregressive method that decomposes the task into a Think‑Locate‑Describe sequence, showing that fine‑tuning with SG‑ISA improves both grounding accuracy and description quality while reducing reasoning‑token overhead.
The study evaluates the performance of the DINOv2 visual representation for classifying Trachomatous Inflammation-Follicular (TF) versus normal conjunctival images. Using 1,546 images processed by the OPTED pipeline, the authors compare six pretrained backbones and then test four lightweight adaptation methods on DINOv2 ViT-B/14. The best results—91.66% accuracy, 90.69% macro‑F1, and 96.06% AUC—were achieved with DINOv2 plus Efficient Channel Attention (ECA) and a focal‑plus‑center loss, though ECA’s benefit varied with the loss function.
The paper investigates how conditioned floor plan generation models perform when applied to datasets from different regions, revealing significant performance drops due to domain shift. To address this, the authors create a large synthetic training set that enforces physical constraints while deliberately reducing architectural realism, and show that pre‑training on this data boosts zero‑shot cross‑domain performance and speeds up fine‑tuning in low‑data scenarios.