The paper introduces Subspace Tuning (SST), a fine‑tuning technique that confines weight updates to the bottom singular subspace of model parameters, thereby avoiding changes along principal directions that cause catastrophic forgetting. SST is optimizer‑agnostic, works with existing fine‑tuning recipes, and includes dense gradient estimation and a refresh mechanism. Experiments show that protecting up to 99% of the weight subspace significantly reduces forgetting, improves retention, and maintains strong target‑task performance, especially in large‑model regimes, while also reducing KL drift by 2–4× compared to baseline methods.
Discrete image generators learn the distribution of tokens produced by an image tokenizer. These token representations should be easy for generators to model, not merely enable faithful image reconstr...
Surgical teams could benefit from assistants that watch the video for requested events, such as "tell me when the clip applier appears during Calot triangle dissection", and answer questions about wha...
EMG-FM-Bench is a benchmark that unifies 20 public electromyography datasets, totaling over one million EMG segments, to evaluate how foundation models transfer and adapt to EMG tasks. It tests nine pretrained models across four scenarios: frozen versus fully fine‑tuned performance, the benefit of pretraining versus training from scratch, generalization to new users with limited data, and performance across different EMG tasks. Results show that linear probing reveals useful pretrained representations, full fine‑tuning can significantly alter performance, pretraining benefits vary by model, performance drops on new users, and few‑shot adaptation improves macro‑F1 in most cases but only partially recovers lost performance.
By Tianhao Wu, Jiawei Yu, Xiangdong Hu, Xu Wu, Hongyu Wu, Sidharth Anupkrishnan, Amirmohammad Radmehr, Yi Chen Liu, Phuc Nguyen, Yi Wu, Jian Liu
ECLIPTICA is a framework that makes large language model alignment instruction‑driven and runtime‑controllable, allowing natural‑language alignment instructions to modulate behavior on the fly. It introduces Contrastive Instruction‑Tuned Alignment (CITA), which combines supervised fine‑tuning with contrastive preference optimization anchored to a reference model, ensuring stable policy updates within a shared neighborhood. The authors release an ECLIPTICA benchmark of 3,000 controlled cases and demonstrate that CITA achieves 86.7% instruction‑alignment efficiency on Llama‑3.1‑8B, outperforming DPO, GRPO, and PPO.
By Kapil Wanaskar, Gaytri Jena, Vinija Jain, Aman Chadha, Amitava Das
The paper introduces DRMOT, a new dataset and framework for Referring Multi-Object Tracking (RMOT) that incorporates RGB-D data. DRSet contains 2,076 referring expressions across 186 RGB-D sequences, pairing synchronized RGB images, depth maps, and language descriptions—including depth-dependent cues—to evaluate spatial‑semantic detection and tracking. The proposed DRTrack framework uses a Geometric‑Aware GRPO fine‑tuned Multimodal Large Language Model to perform precise detection and robust trajectory association from joint language, RGB, and depth inputs, with experiments demonstrating its effectiveness.
By Sijia Chen, Shaofeng Liang, Lijuan Ma, Yanqiu Yu, Nan Li, Wenshuo Chen, Runwei Guan, Dongming Wu, En Yu, Yutao Yue
The paper studies cross‑stage context passing in Graph Visual Question Answering for autonomous driving, focusing on the Perception, Prediction, and Planning stages. Two complementary approaches are evaluated on DriveLM‑nuScenes: an explicit prompt‑based conditioning strategy using a domain‑adapted 4B VLM that reduces NLI contradiction by up to 42.6%, and an implicit gated context projector that, with only ~0.5% of parameters updated, cuts planning‑stage NLI contradiction by 34% and boosts cross‑stage entailment by 50%. The explicit method offers a strong zero‑training baseline for surface consistency, while the implicit method delivers significant semantic gains, highlighting the potential of domain adaptation for further improvement.
By Gautam Kumar Jain, Carsten Markgraf, Julian St\"ahler
The paper introduces MUCS, a method for training data attribution (TDA) in diffusion models that uses mirrored unlearning and noise-consistent skew. By fine-tuning a second model with bounded mirrored gradient ascent and measuring its normalized skew relative to the original model using consistent noise samples, MUCS achieves significantly better performance than existing TDA methods across three datasets. The authors also explore design choices, overlap of influential instances, and the potential of ensembling TDA approaches.
By Joan Serr\`a, Dipam Goswami, Fabio Morreale, Wei-Hsiang Liao, Yuki Mitsufuji
The paper introduces GAC, a noise‑aware adaptive mixing controller that dynamically balances supervised fine‑tuning (SFT) and reinforcement learning (RL) gradients during hybrid post‑training of large language models. By deriving a closed‑form mixing weight that minimizes an MSE upper bound on the mixed gradient, GAC incorporates EMA smoothing, a schedule prior, and capped updates, all estimated online from training tensors. Experiments show that GAC outperforms rule‑based controllers, improving performance on diverse benchmarks by up to 3.8 percentage points, reducing KL‑drift by 28%, and cutting large mixing‑weight changes by over 70% with negligible overhead, with gains scaling from 1.5 B to 14 B parameters.
By Yuelin Hu, Wei Liu, Zhenbo Yu, Zhengxue Cheng, Li Song
FedGraph is a graph-based aggregation framework designed to defend federated large language model fine-tuning against backdoor attacks. It exploits the observation that malicious clients produce consistently similar updates, while benign clients generate diverse ones, by modeling client updates as nodes in a dynamically constructed graph and extracting structural features. The resulting compact client representations enable effective identification and suppression of malicious contributions, achieving high detection accuracy and low false positives in extensive experiments.
By Xi Chen, Chunyi Zhou, Rui Zeng, Xiaogang Xu, Zhe Liu, Shouling Ji
The paper introduces point‑in‑time language models that are trained only on text available up to each calendar date, thereby eliminating lookahead bias that plagues standard large language models. By scaling decoder‑only transformers to 4 billion parameters and training on 1 trillion chronologically filtered tokens from FineWeb, the authors produce monthly checkpoints from 2013 to 2024 that achieve performance close to unconstrained models of similar size on common‑sense reasoning and language understanding tasks. Instruction fine‑tuning with LoRA further enhances downstream usability, and the authors release the full pipeline—including dataset construction, training infrastructure, and evaluation code—for reproducible research.
By Bryan Kelly, Semyon Malamud, Johannes Schwab, Teng Andrea Xu
The paper investigates how to evaluate bounded model updates for ICU prediction models using the MIMIC‑IV database. After auditing the data for recording artifacts and cleaning it, the authors compare a bounded update that retrains only treatment‑context and fusion parameters with full fine‑tuning across two model families and two targets, finding that bounded updates are non‑inferior or superior in most cases but depend on target, model family, and learning‑rate tuning. The study also shows that audits should precede update evaluation and that the update boundary should be chosen per target.
By Fatema Ferdous Tamanna, K. M. Merajul Arefin, Md. Abdul Masud
The paper introduces MusicConstraintBench, a benchmark of 2180 items testing symbolic music generation under multiple explicit constraints. It shows that current language models, such as Llama-3.1-70B, perform poorly on multi-constraint tasks, achieving only 4.4% success on four-constraint items. The authors propose MusicRLVR, a reinforcement learning approach that uses verifier rewards without human annotation, which significantly improves performance—raising Qwen3-4B-Instruct-2507’s success from 16% to 80.7% on mixed constraints after 300 updates.
By Haoyue Liu, Xiaoying Tang
ESTHER is an end‑to‑end model for egocentric stereo hand reconstruction that integrates stereo geometry, temporal reasoning, and a suitable output representation for wearable cameras. It is trained on pseudo‑labels from a calibrated pipeline and introduces the ESTHER3D benchmark, which pairs a large in‑the‑wild training set with a motion‑capture test set of true metric ground truth. Experiments demonstrate state‑of‑the‑art accuracy, strong generalization, and robustness to missing views, dropped frames, and extreme lighting or motion blur, while preserving metric scale even when reduced to a single monocular view.
By Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Peng Yin
The paper introduces a self‑distillation method that leverages answer‑only data to improve large language models (LLMs) after training. By training an inverse reasoning LLM to generate latent chain‑of‑thought traces conditioned on the answer, the authors can perform offline distillation that significantly boosts performance on hard prompts for tasks like Countdown and GSM8K. Additionally, the inverse model serves as an accurate verifier, enhancing best‑of‑n selection accuracy.
By Eric O. Korman
The paper introduces genesis steering, a method that derives a steering vector from contrastive examples in a model’s own fine‑tuning data, and proposes using membership inference to build proxy datasets when such data are unavailable. Two theoretical results are presented on optimal displacement to the decision boundary and reduced proxy‑estimation bias. Experiments on T"ulu‑3‑8B and Llama‑3.1‑8B‑Instruct show that safety data from T"ulu‑3‑8B’s training mixture produce stronger steering than external datasets, and that membership‑based proxy construction can achieve competitive performance.
By Tora Bodin, Eric Aubinais, Ilaria Torre, Chih-Hong Cheng
The paper introduces DTDD, a Divergence-Triggered Dynamic Distillation framework that adapts on‑policy distillation by monitoring segment‑level student‑teacher divergence and initiating teacher takeover when divergence exceeds a threshold. During takeover, the student continues to receive standard OPD supervision on the prefix while the teacher generates a recovery suffix that the student learns via behavior cloning. Experiments across long‑horizon reasoning, base initialization, and multi‑turn agents show that DTDD reduces peak gradient norms, accelerates learning from a base initialization, and improves performance on ALFWorld tasks, with segment‑averaged triggers and behavior cloning proving most effective.
By Qijia He, Zequn Li, Spencer Luo, Tianle Chen, Pradyumna Narayana, Zixian Ma, Ranjay Krishna, Lan Nie, Sugato Basu
The paper challenges the common practice of selecting latent world models based on one‑step prediction loss, arguing that this metric poorly predicts planning performance. Using a frozen‑decoder protocol, the authors decompose one‑step error into encoder‑inherent and predictor‑added components and evaluate goal‑conditioned planning across three continuous‑control environments. Their experiments show that improvements in one‑step fidelity do not correlate with planning success, with correlation coefficients near zero and inconsistent exchange rates across tasks.
By Joshua Cheung, Zhang He
The paper introduces Selective Rank-Preserving Supervised Fine‑Tuning (SRP‑SFT), a method that applies a hinge‑based soft margin target to preserve selected local non‑target logit margins during teacher‑forced fine‑tuning. SRP‑SFT uses reference‑scaled lower bounds for pairs from the reference’s top‑K non‑target tokens, stopping once the bound is met. Experiments on Qwen3‑4B‑Base and UltraFeedback show that SRP‑SFT improves MATH‑500 and code‑generation scores compared to cross‑entropy and LP‑SFT baselines.
By Yuxuan Wang, Guangxiang Zhao, Xinyu Tan, Yahui Han, Xinxiang Yin, Zhonghai Wu, Xiangzheng Zhang, Lin Sun, Tong Yang
The paper presents Qwenfant, a Qwen3-based model trained on concise parallel text for the 2026 BabyLM Challenge’s Multilingual track. By leveraging LLM-summarized and translated data, a larger model size, and a character-level vocabulary—both fine-tuned with LoRA—the authors achieve a top ranking in the competition.
By Lukas Edman, Alexander Fraser