CommunityKV is a new framework that treats sparse attention as a community detection problem, building a token graph from $QK^T$ scores and partitioning it into semantically coherent communities. It updates token communities in constant time during streaming decoding, avoiding costly global re‑partitioning. Experiments on Qwen3 and Llama‑3.1 show that CommunityKV can increase end‑to‑end generation throughput by up to 1.25×, and with query‑group graph aggregation up to 1.71×, while maintaining comparable accuracy.
By Joe McKenna, Anastasios Alexandridis, Nathan Susanj, Jing Liu
The paper introduces VANE, a new routing strategy for Mixture-of-LoRA-experts that scores experts based on the expected loss reduction of their updates rather than token similarity. By decomposing each expert into a reader and writer, VANE evaluates all reader–writer pairs using a low‑rank compass that predicts the descent direction, enabling efficient top‑k selection with additive gates. Experiments on Llama‑3 models show VANE outperforms twelve PEFT and MoE‑LoRA baselines while using fewer trainable parameters and providing router scores that better reflect expert usefulness.
By Priya Nair, Lukas Brenner, Maya Lindqvist, Daniel Whitmore, Wen-Hsuan Liu, Tom Saliencro, Amara Okonkwo, Rohan Desai
The paper introduces Simplex-Lattice Interpolation Merging (SLIM), a method that builds a quadratic surrogate model of aggregate performance over merging coefficients for large language models using a classical mixture design. By evaluating only individual experts and equal-weight pairs, SLIM identifies the surrogate with the fewest measurements needed and then optimizes it without further target-metric evaluations. Experiments on two model architectures show that SLIM accurately predicts unseen multi-expert mixtures and achieves competitive merge performance under limited evaluation budgets, outperforming random designs and regularized fitting.
By Seongcheol Jeong, Masahiro Suzuki, Yutaka Matsuo
Kernelized Activation Steering (KAS) introduces a training‑free method for controlling attributes of generative models by lifting activation steering into a reproducing kernel Hilbert space. It formulates steering as an optimization problem using only kernel evaluations, producing an implicit, activation‑dependent steering score that adapts locally to the geometry of the activation space. Unlike standard Difference‑in‑Means, KAS generates a nonlinear steering field and recovers DiM as a special case with a linear kernel, achieving comparable or superior performance on tasks such as jailbreaking LLMs and image style control.
By Laziz U. Abdullaev, Minh-Hieu Pham, Bach Do, Khoat Than, Tan M. Nguyen
Mixture-Trained Merging (MTM) is a method for creating unified language models that combine multiple objectives—such as mathematics, code, instruction following, and controllable thinking—into a single parameter set. Instead of sequentially post‑training on each objective, MTM trains each branch on a mixture of objectives, ensuring that the branches remain compatible in weight space and can be merged without degrading performance. The approach iteratively refines merge coefficients using low‑cost evaluations and multi‑objective Bayesian optimization, outperforming naive merging and preserving distinct behaviors across domains.
By SeongHyeon Kim, Chaeyun Jang, Seungyoo Lee, Jiyeon Ham, Yunju Bak, Boseop Kim, Juho Lee
QK-Wanda is a new pruning method that extends Wanda by scoring query and key weights together, using each projection’s deletion cost under an unmasked pre‑RoPE reconstruction objective. It augments each projection’s score with information from the opposite projection, enabling shared pruning budgets without requiring gradients or weight updates. Across 15 models ranging from 0.5 B to 72 B parameters, QK-Wanda reduces reconstruction error by 60 % at 50 % sparsity and 45 % at 80 % compared to Wanda, and yields notable downstream improvements on some models such as Llama 2 70 B.
whyItMatters":"The study demonstrates that coupling query and key pruning criteria can substantially lower reconstruction error and improve downstream performance, highlighting both the potential and limitations of local reconstruction as a predictor of model quality."
By Ivan Ilin, Peter Richt\'arik
Latent JEPA is a new framework that trains continuous latent thoughts to anticipate informative aspects of future solutions in chemical reasoning, without verbalizing every intermediate step. It combines autoregressive learning with joint-embedding prediction of one or more future views, using textual and molecular prediction objectives that link latent thoughts to subsequent reasoning and molecular outcomes. Experiments on ChemCoTBench demonstrate improvements in molecular optimization, editing, and reaction metrics, and representation analyses show that future prediction makes latent thoughts more informative about molecular outcomes and better aligned with chemical structure.
By Xinjian Zhao, Yaoyao Xu, Xuemin Chen, Xiaozhuang Song, Tianshu Yu
The paper proves that deep residual self‑attention networks can universally interpolate between any two collections of sequences using only two fixed single‑head attention blocks with Gaussian‑initialized projections. The interpolation is achieved by varying the order, signs, and durations of these blocks, independent of the specific input and output sequences. The result holds for both continuous and finite depth, and the authors also extend the analysis to causal‑masked settings.
By Sibylle Marcotte, Joan Bruna
Faynt is a family of Transformer policies (10M and 75M parameters) that control all 26 characters in Super Smash Bros. Melee from a single checkpoint. After reinforcement learning, the 10M model wins 98.4% of same‑character games against fourteen specialist and multi‑character releases, and defeats a zero‑delay Slippi‑AI model in all 68 evaluated games. The work details architecture, scaling, hyperparameter transfer, supervised pretraining on 840,000 human replays, post‑training curricula, distillation, and efficient inference, and it releases the weights, benchmark suites, and a platform for automated model tournaments.
By Ali Janati, Nikita Kuzmin, Rohit Swamy, Charles Niu
The paper introduces ZFO, a lightweight framework that separates direction selection from step-size determination in large‑scale neural network optimization. ZFO uses a trusted first‑order optimizer to pick a search direction and then performs only two additional objective evaluations to build a local curvature‑aware model, selecting an adaptive step within a bounded interval. The authors provide theoretical guarantees for reliable curvature estimation, near‑optimal step selection, and convergence to a stationary point, and demonstrate that ZFO improves optimization and final performance over fixed‑step first‑order baselines on language‑model fine‑tuning tasks.
By Cristian McGee, El Houcine Bergou, Aritra Dutta
CAST (Cost‑Aware Speculative Trees) is a method that improves speculative decoding for large language models by packing multiple drafted token candidates into a tree and verifying the entire tree in a single target‑model pass, rather than only the top‑scoring chain. The tree width is adaptively chosen based on a latency measurement, ensuring that each added candidate’s expected gain outweighs its verification cost. Experiments across five domains, three GPU generations, and two model families show that CAST can be up to 43 % faster than the standard chain, while preserving the target model’s output distribution under both greedy and sampled decoding.
By Jungseob Lee, Sugyeong Eo
The paper introduces an unsupervised algorithm that separates content and style in language model outputs, enabling explicit style control. Applied to over 100K traces from nine teacher models, the method uncovers six recurring, imbalanced styles and fine‑tunes student models to follow them. This style conditioning improves Pass@$k$ on six math reasoning benchmarks and shows that style influences problem‑solving success, offering a new avenue for controlling and enhancing model performance.
By Ioana Marinescu, Eric Karl Oermann, Kyunghyun Cho
StabilityArc is a method that decodes protein sequence embeddings into generalizable stability landscapes. It uses a shared RoPE transformer to map frozen ESMC-600M residue representations into an Lx20 matrix of substitution effects, with a symmetric, contact-aware residual to predict epistasis. In extensive leave-one-protein-out tests on 134,794 ProteinGym variants, StabilityArc achieves a Spearman correlation of 0.7134, surpassing the best zero‑shot baseline, and further improves Kermut’s performance when used as a prior.
By Aaron L. Feller, Andrew D. Ellington, Claus O. Wilke
VERITYGATE is a four‑gate framework that checks whether LLM‑generated narrations adhere to a fixed schema of declared evidence IDs, entities, numbers, and claim types, rather than verifying every fact in the prose. In experiments with GPT‑4o‑mini, Llama‑3.3‑70B, and Claude Sonnet 4.6 on 900 grounded‑ungrounded pairs, the framework identified high failure rates (up to 80.3% for mini claims) and demonstrated that a single repair pass can improve claim survival rates. The authors also provide code, data, and preliminary human studies to validate the rules and highlight gaps between schema compliance and correct prose.
"whyItMatters":"The framework offers a systematic way to evaluate and improve the faithfulness of LLM explanations to structured evidence, revealing significant failure rates and the impact of repair strategies."
By Sachin Gupta
The paper investigates how sentence‑specificity scores can guide the selection of revisions in collaborative technical documentation. It compares two predictors—SpeciTeller and a target‑adapted model by Ko et al.—across Wikipedia and three technical corpora, finding that the predictors rank sentences differently and that SpeciTeller can improve direction‑valid selection rates in certain datasets. The study also shows that filtering and token‑length adjustments alter but do not reconcile these ranking differences.
By Rocker D'Antonio, Thomas Benton Townsend, Dimitrios Michael Manias
TRACE tackles real‑world dynamic resource assignment by combining evolutionary automatic heuristic design with an agentic knowledge‑extraction workflow. A Reasoner agent interprets system logs to hypothesize about underlying dynamics, while a Coder agent generates and runs schema‑specific code to validate these hypotheses, producing insights or executable tools for the evolved heuristics. Evaluations on a synthetic cloud benchmark and a 5G vRAN scenario show that TRACE outperforms existing AHD methods, delivering more auditable heuristics with less than 2% overhead.
By Jose A. Ayala-Romero, Andres Garcia-Saavedra, Xavier Costa-Perez
The paper introduces LAST, a Looped Audio Spectrogram Transformer that processes all tokens once and then reuses the same blocks to refine only the class token over fixed audio features, making subsequent passes inexpensive. On AudioSet, a ten‑pass LAST outperforms a twelve‑layer sequential transformer by 2.1% relative mean average precision while using 49.4% fewer parameters, 42% fewer MACs, and achieving 9.8% higher throughput. Increasing the pass count from two to ten improves accuracy with only a 1.2% increase in computation, and the model shows enhanced robustness to temporal masking and other auditory augmentations across music, environmental, and event sound classification tasks.
By Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina, N. Apurva Ratan Murty
ConQuR introduces a lightweight post‑training rotation calibration for large language model activation quantization. By learning orthogonal rotations that align normalized activations with the corners of an inscribed hypercube, the method distributes activation energy evenly and can be updated online without storing activations. Experiments on Llama‑2 and Llama‑3 models (3B–70B) show competitive or improved perplexity and reasoning performance while avoiding costly training or large offline storage.
By Chayne Thrash, Ali Abbasi, Soheil Kolouri
The paper proposes constraining the query and key projection matrices in Transformer attention to the Stiefel manifold and optimizing them with a Riemannian Adam optimizer. It demonstrates that this geometric constraint yields significant performance gains on a CIFAR‑10 patch benchmark, with the constrained model outperforming standard AdamW by up to +6.79 percentage points. The authors also show that weight decay has no effect on the constrained frames and that the improvement is driven by a scale‑free step size rather than the manifold projection or equivariance properties.
By Rub\'en Dar\'io Guerrero
The paper proposes a sensitivity‑aware residual‑stream pruning method for large language models that goes beyond minimizing activation reconstruction error. By using a second‑order approximation of output KL divergence, the authors derive a spectral upper bound that selects pruning subspaces based on both activation covariance and output sensitivity, enabling efficient eigendecomposition. Experiments on instruction‑tuned language models show that this approach consistently reduces calibration KL divergence, improves perplexity, and enhances downstream task performance across various compression levels.
By Chayne Thrash, Kevin Chen, Soheil Kolouri