LoRA, PEFT, instruction tuning and domain adaptation — adapting a pretrained model without paying to train one.
arXiv:2610.02388v1 Announce Type: new Abstract: Existing 3D large language models (LLMs) compromise on two fronts: they compress shapes into latent codebook indices or coordinate text, which removes...
arXiv:2510.22276v4 Announce Type: replace-cross Abstract: Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing Eng...
arXiv:2609.33212v2 Announce Type: replace Abstract: Speaker verification systems achieve high accuracy but provide little account of the acoustic evidence behind their judgments. Making these systems...
arXiv:2610.03370v1 Announce Type: new Abstract: CLIP's visual encoder produces only global image representations, limiting its use in region-level tasks. Existing adaptations rely on visual prompting...
The paper presents a data‑efficient pre‑training framework for neural surrogate models solving PDEs on unstructured 3D geometries. It introduces a geometry‑driven strategy for steady‑state problems that uses intrinsic shape descriptors, and a physics‑driven approach for transient problems that generates synthetic PDE data online. Experiments show faster convergence, higher data efficiency, and improved accuracy during fine‑tuning, especially in low‑data settings.
Efficient Memory Crystallization (EMC) is a training‑free, test‑time framework that transforms each incoming graph domain into a compact, semantically faithful memory via a closed‑form distribution‑matching solution. By eliminating the need for auxiliary generative modules, EMC reduces redundant domain information and models inter‑domain dependencies through state‑evolving memories, yielding a tighter generalization error bound. Experiments show EMC outperforms state‑of‑the‑art baselines on non‑stationary graph data, cutting average runtime by 87.4% and GPU memory usage by 92.4% compared to a recent competitor.
SecJev is a family of Jev-like decision models specialized for security, ranging from 0.8 B to 9 B parameters. Built on Kev’s single‑pass candidate scorer, it learns Boolean, choice, and ordered decisions from text, telemetry, and observation histories, and is trained on the SecJev‑Corpus that unifies source‑label prediction and explicit‑policy evaluation across 14 tasks and eight sources. The security specialization improves every model, with SecJev‑0.8B outperforming general Kev‑9B by 20.51 percentage points in task‑macro accuracy and matching generative fine‑tuning in accuracy and latency while using less peak inference memory.
The paper presents a mechanistic defense for Vision‑Language‑Action (VLA) models against adversarial patches. By using a sparse autoencoder, the authors identify a feature whose activation correlates strongly with the presence of an adversarial patch and suppress this feature only when a linear probe detects an attack. This conditional intervention improves robustness on the LIBERO‑10 benchmark while avoiding the performance degradation that occurs with continuous suppression.
The paper investigates how reinforcement learning with verifiable rewards (RLVR) interacts with majority voting over sampled completions, a common test‑time scaling technique. It shows that training a single LoRA adapter with the full RLVR budget can improve single‑sample accuracy but often harms the overall majority‑vote performance because the policy’s errors become increasingly correlated. By splitting the same training budget across multiple adapters—each trained on a disjoint shard, forming an "adapter thicket"—the authors demonstrate that this approach consistently out‑votes the fully trained single adapter and preserves most of the single‑sample gains, especially when the number of votes is large.
TrueMuse is a new benchmark designed to evaluate data attribution in text-to-music models. It consists of a controlled dataset created by fine‑tuning three diffusion‑based models on curated attribution samples, providing known attribution targets. The benchmark covers four settings—melodic structure, timbral characteristics, artist‑level style, and genre‑level patterns—across 133 attributes, 648 models, and 95,456 generated samples, and is used to assess existing black‑box attribution methods along several dimensions.
The study evaluates whether biological reasoning models actually use their biological inputs by testing six models on DNA, protein, and single‑cell tasks. By perturbing one biological input while keeping others fixed, the authors find that many models (e.g., Evo2, ESM3, BioReason, BioReason‑Pro) rely primarily on textual information, with minimal impact from the biological representations. In contrast, models like ChatNT, Prot2Text‑V2, CellWhisperer, and Cell2Sentence‑Scale show greater dependence on their biological inputs, yet overall accuracy gains do not consistently reflect increased biological input contribution.
The paper introduces query‑conditioned hypernetworks that predict distributions over LoRA weight updates for large language models. By learning a distribution rather than a single point estimate, the method allows sampling multiple adapted models for the same query, improving performance over deterministic hypernetworks and token‑sampling baselines. The study also shows that these learned updates can transfer across different queries, indicating reusable adaptation patterns.
The paper introduces SimpleTimeBench, a diagnostic suite testing basic temporal primitives like monotonic trends, periodic signals, and leading indicator covariates. It finds that prominent multivariate Time Series Foundation Models (Chronos‑2, Moirai, and Toto) often produce suboptimal zero‑shot forecasts for these simple patterns, and that fine‑tuning can improve specific tasks while harming performance on other fundamentals. These failures persist in real‑world sensor forecasting, indicating that current TSFMs may lack the inductive biases needed to capture straightforward relationships, thereby limiting their practical reliability.
SeedFlood is a novel decentralized fine‑tuning method for large language models that scales to billions of parameters and hundreds of clients. It leverages the seed‑reconstructible structure of zeroth‑order gradients to reduce message sizes to near‑zero, enabling efficient flooding across the network. Experiments show SeedFlood outperforms standard zeroth‑order baselines in communication efficiency and generalization, and rivals first‑order gossip methods while incurring far less communication cost.
The paper investigates a phenomenon called forking, a type of generalization failure observed in NanoGPT and similar autoregressive models. Forking manifests as a sharp divergence between training and validation loss at epoch boundaries, especially when models employ over‑encoding n‑gram memory modules and data replay. The authors reproduce the effect in both vanilla NanoGPT and a DeepSeek-style model, attributing it to repeated updates that sharpen seen continuations while suppressing unseen ones, with low‑frequency contexts contributing most to the loss gap.
The paper introduces VANE, a new routing strategy for Mixture-of-LoRA-experts that scores experts based on the expected loss reduction of their updates rather than token similarity. By decomposing each expert into a reader and writer, VANE evaluates all reader–writer pairs using a low‑rank compass that predicts the descent direction, enabling efficient top‑k selection with additive gates. Experiments on Llama‑3 models show VANE outperforms twelve PEFT and MoE‑LoRA baselines while using fewer trainable parameters and providing router scores that better reflect expert usefulness.
The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.
The paper introduces ReMILP, a reformulation‑contrastive learning framework that uses self‑supervision from equivalent formulations of mixed‑integer linear programs (MILPs). By distinguishing re‑descriptions and substitutions, the method trains a graph neural network and a hypernetwork to predict how variable embeddings transform under changes of variables, achieving invariance and equivariance without solver‑derived labels. The learned representations prove useful for tasks such as binary solution, constraint activity, and integrality gap prediction, and serve as a strong initialization for fine‑tuning.
Deep Repurposing (DR) is a post‑hoc framework that adapts deep neural networks when parts of their output space become obsolete after deployment. DR estimates the latent geometry of obsolete and retained regions, removes components that support obsolete outputs, and reallocates retained-compatible evidence through an analytic repair map without gradient updates. The method yields repaired predictions and representations that eliminate obsolete outputs while preserving or improving retained accuracy, and it adapts up to 60× faster than competing unlearning methods.
The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.