The paper introduces query‑conditioned hypernetworks that predict distributions over LoRA weight updates for large language models. By learning a distribution rather than a single point estimate, the method allows sampling multiple adapted models for the same query, improving performance over deterministic hypernetworks and token‑sampling baselines. The study also shows that these learned updates can transfer across different queries, indicating reusable adaptation patterns.
By Azal Ahmad Khan, Keshav Ramji, Tahira Naseem, Ali Anwar, Ram\'on Fernandez Astudillo
The paper introduces LoopCD, a training‑free contrastive decoding framework that improves token selection in Loop‑Transformer models by comparing the final prediction with earlier recurrent passes. LoopCD operates either in logit space (LoopCD‑Logits) with a single extra output pass or in hidden‑state space (LoopCD‑Hidden) with no output overhead. Across multiple looped Transformer families, LoopCD yields significant performance gains—raising pass@1 scores on tasks such as AIME 2024 and HumanEval—while enabling a reduction in the number of recurrent loops and a corresponding decrease in inference FLOPs.
By Weihao Liu, Huangjie Zheng, Tianrong Chen, Rohit Dilip, Richard He Bai, Yizhu Jiao, Yuyang Wang, Ruixiang Zhang
The paper introduces a particle‑level generative model that uses residual‑quantized full‑event data to enable fast, ML‑based surrogate simulation for collider events. It demonstrates conditional generation from detector‑stable particles, explores scaling across dataset and model sizes, and shows that token‑level loss predicts downstream physical fidelity. The work offers an empirical framework for scalable collider full‑event generation using residual‑quantized representations.
By Dan Godi, Dmitrii Kobylianskii, Eilam Gross
AnchorPrompt is an adaptation technique for large audio‑language models that keeps the base model frozen and learns a single block of prompt vectors inserted at the decoder input. By training these prompts through self‑distillation on diverse audio and text perturbations, the method improves answer consistency and reduces hallucinations across multiple benchmarks. The approach is perturbation‑agnostic at inference, enabling zero‑shot transfer to unseen distortions such as reverberation and choice permutations.
By Pooneh Mousavi, Amir Ivry, Mirco Ravanelli, Cem Subakan
The paper introduces ONLINE LLM PICKER, a framework for active model selection of large language models in streaming settings. It selects the most informative prompts for annotation within a limited budget, enabling the identification of the best or near‑best model among many candidates. Experiments on 10 datasets and over 130 language models show up to 71.67% savings in annotation cost and a reduction in regret by up to 2.51× when using the chosen model for sequential generation.
By Alessandro Turrin, Patrik Okanovic, Torsten Hoefler, Nezihe Merve G\"urel
The paper introduces LLMLeak, a covert exfiltration technique that exploits Large‑Language‑Model (LLM) web‑fetching capabilities to transmit confidential data. By embedding a secret into a URL and presenting the corresponding website as part of a legitimate task, the malicious client tricks the LLM into fetching the URL, thereby leaking the secret to an attacker‑controlled server. Experiments on eleven open‑parameter models show a 79.7% success rate, and a real‑world case study confirms the attack’s practicality.
By Alessandro Pegoraro, Daryan Merx, Phillip Rieger, Ahmad-Reza Sadeghi
The paper introduces Universal Byte-Level Encoding (UBE), a dual‑alphabet tokenizer that routes 3‑4‑byte UTF‑8 characters through UTF‑16 while keeping 1‑2‑byte characters on the UTF‑8 path. This design lowers the worst‑case token‑budget disparity for high‑premium scripts without increasing costs for efficient English spans, and it preserves standard BPE merges and exact decoding. In extensive Unicode audits and multilingual language‑model experiments, UBE matches or improves token‑count efficiency and context usability compared to traditional byte‑pair encoding.
By Hyunsik Kim, Youngmoon Jung
SeedFlood is a novel decentralized fine‑tuning method for large language models that scales to billions of parameters and hundreds of clients. It leverages the seed‑reconstructible structure of zeroth‑order gradients to reduce message sizes to near‑zero, enabling efficient flooding across the network. Experiments show SeedFlood outperforms standard zeroth‑order baselines in communication efficiency and generalization, and rivals first‑order gossip methods while incurring far less communication cost.
By Jihun Kim, Dongyeop Lee, Namhoon Lee
The paper introduces dFlowGRPO, a reinforcement learning framework tailored for discrete flow models (DFMs). It generalizes previous work on diffusion large language models by supporting various probability paths and non-masked source distributions, and formulates denoising as a Markov decision process that leverages transition rates and posterior models. Experiments on the multimodal DFM FUDOKI show that dFlowGRPO outperforms existing GRPO methods on text‑to‑image generation and matches continuous flow models on multimodal understanding tasks.
By Zhengyan Wan, Yidong Ouyang, Panwen Hu, Qiang Sun
The paper introduces TANGO, a Token‑Aggregated Nonlinear Gating Operator that blends cross‑token mixing and token‑wise transformation in transformer architectures. By computing a nonlinear gate per source token and averaging these gates for each destination, TANGO forms a source‑conditioned linear operator that improves predictive performance. Experiments on web text, formal mathematics, and code show that full‑prefix TANGO achieves the lowest test negative log‑likelihood across 16 settings, while a narrower variant offers substantial throughput gains with only a modest increase in loss.
By Joshua Nunley
LumoTree is a verifier designed for hybrid language models that ensures a single coherent continuation across recurrent, convolution, and attention states. It executes recurrent paths in parallel, reuses state tiles, and coordinates replay, history gathering, and cache remapping through a shared logical tree. Experiments demonstrate exact candidate‑selection parity and recurrent agreement within error bounds, and a Qwen3.8‑27B deployment achieved 25.63 pooled tokens/s on a single NVIDIA DGX Spark.
By Zhiyuan Ma
Attention Manifolds introduce learned 2D B-spline surfaces that modulate each value dimension in transformer attention based on query-key interactions. Applied to LLaMA models, they reduce WikiText-2 perplexity by 2–2.5 points with minimal parameter overhead and alter greedy-decoded outputs in 69–83% of diverse prompts, especially on ambiguous inputs. The surfaces can be edited mechanically to steer or block value flow, offering a geometric tool for improving output quality and potentially enhancing safety.
By Naveen Mysore
Large Language Bayes (LLB) samples probabilistic programs from a language model, runs approximate inference on each, and averages them weighted by an exponentiated evidence bound. The authors demonstrate that this weighting is not invariant to reparameterisation, unlike the log marginal likelihood, leading to significant discrepancies in weights across different program formulations. These discrepancies can reach up to 31.9×, affect Bayes factors, and introduce controlled errors in posterior estimates.
By Jian Xu
CommunityKV is a new framework that treats sparse attention as a community detection problem, building a token graph from $QK^T$ scores and partitioning it into semantically coherent communities. It updates token communities in constant time during streaming decoding, avoiding costly global re‑partitioning. Experiments on Qwen3 and Llama‑3.1 show that CommunityKV can increase end‑to‑end generation throughput by up to 1.25×, and with query‑group graph aggregation up to 1.71×, while maintaining comparable accuracy.
By Joe McKenna, Anastasios Alexandridis, Nathan Susanj, Jing Liu
The paper introduces VANE, a new routing strategy for Mixture-of-LoRA-experts that scores experts based on the expected loss reduction of their updates rather than token similarity. By decomposing each expert into a reader and writer, VANE evaluates all reader–writer pairs using a low‑rank compass that predicts the descent direction, enabling efficient top‑k selection with additive gates. Experiments on Llama‑3 models show VANE outperforms twelve PEFT and MoE‑LoRA baselines while using fewer trainable parameters and providing router scores that better reflect expert usefulness.
By Priya Nair, Lukas Brenner, Maya Lindqvist, Daniel Whitmore, Wen-Hsuan Liu, Tom Saliencro, Amara Okonkwo, Rohan Desai
The paper introduces Simplex-Lattice Interpolation Merging (SLIM), a method that builds a quadratic surrogate model of aggregate performance over merging coefficients for large language models using a classical mixture design. By evaluating only individual experts and equal-weight pairs, SLIM identifies the surrogate with the fewest measurements needed and then optimizes it without further target-metric evaluations. Experiments on two model architectures show that SLIM accurately predicts unseen multi-expert mixtures and achieves competitive merge performance under limited evaluation budgets, outperforming random designs and regularized fitting.
By Seongcheol Jeong, Masahiro Suzuki, Yutaka Matsuo
Kernelized Activation Steering (KAS) introduces a training‑free method for controlling attributes of generative models by lifting activation steering into a reproducing kernel Hilbert space. It formulates steering as an optimization problem using only kernel evaluations, producing an implicit, activation‑dependent steering score that adapts locally to the geometry of the activation space. Unlike standard Difference‑in‑Means, KAS generates a nonlinear steering field and recovers DiM as a special case with a linear kernel, achieving comparable or superior performance on tasks such as jailbreaking LLMs and image style control.
By Laziz U. Abdullaev, Minh-Hieu Pham, Bach Do, Khoat Than, Tan M. Nguyen
Mixture-Trained Merging (MTM) is a method for creating unified language models that combine multiple objectives—such as mathematics, code, instruction following, and controllable thinking—into a single parameter set. Instead of sequentially post‑training on each objective, MTM trains each branch on a mixture of objectives, ensuring that the branches remain compatible in weight space and can be merged without degrading performance. The approach iteratively refines merge coefficients using low‑cost evaluations and multi‑objective Bayesian optimization, outperforming naive merging and preserving distinct behaviors across domains.
By SeongHyeon Kim, Chaeyun Jang, Seungyoo Lee, Jiyeon Ham, Yunju Bak, Boseop Kim, Juho Lee
QK-Wanda is a new pruning method that extends Wanda by scoring query and key weights together, using each projection’s deletion cost under an unmasked pre‑RoPE reconstruction objective. It augments each projection’s score with information from the opposite projection, enabling shared pruning budgets without requiring gradients or weight updates. Across 15 models ranging from 0.5 B to 72 B parameters, QK-Wanda reduces reconstruction error by 60 % at 50 % sparsity and 45 % at 80 % compared to Wanda, and yields notable downstream improvements on some models such as Llama 2 70 B.
whyItMatters":"The study demonstrates that coupling query and key pruning criteria can substantially lower reconstruction error and improve downstream performance, highlighting both the potential and limitations of local reconstruction as a predictor of model quality."
By Ivan Ilin, Peter Richt\'arik
Latent JEPA is a new framework that trains continuous latent thoughts to anticipate informative aspects of future solutions in chemical reasoning, without verbalizing every intermediate step. It combines autoregressive learning with joint-embedding prediction of one or more future views, using textual and molecular prediction objectives that link latent thoughts to subsequent reasoning and molecular outcomes. Experiments on ChemCoTBench demonstrate improvements in molecular optimization, editing, and reaction metrics, and representation analyses show that future prediction makes latent thoughts more informative about molecular outcomes and better aligned with chemical structure.
By Xinjian Zhao, Yaoyao Xu, Xuemin Chen, Xiaozhuang Song, Tianshu Yu