The paper introduces TACO, a new optimizer for fine‑tuning large language models that drastically reduces optimizer state memory while preserving first‑order gradients. TACO selects the sign of the largest magnitude entry in each column of weight matrices, achieving a 174× reduction in persistent optimizer memory compared to AdamW8bit and a 2.9× decrease in peak training memory on OPT‑13B. This allows full‑parameter fine‑tuning of 30–32B‑parameter models on a single 80 GB GPU across multiple model families and tasks, with comparable accuracy and runtime to existing methods.
By Jichao Jiang (University of Central Florida), Cristian McGee (University of Central Florida), El Houcine Bergou (Mohammed VI Polytechnic University), Hanqin Cai (University of Central Florida), Aritra Dutta (University of Central Florida)
The paper introduces an agentic forecasting environment built on 2,100+ resolved Polymarket questions, where a language model (Qwen3.5-35B-A3B) learns to gather evidence during rollout via web search, page reading, and financial time series, all filtered to avoid post‑cutoff leaks. Training with single‑epoch GRPO and a Brier‑score reward improves calibration by 30‑40% and reduces search attempts, while the trained policy outperforms four frontier models in evidence‑based forecasting, achieving lower soft‑Brier scores at roughly 5% of the inference cost. The authors release the environment, dataset, and per‑rollout records as a reusable harness for temporal forecasting agents.
By Yusuf Afifi, Artur Kiulian, Anton Polishko, Mykola Khandoga, Hamudi Naanaa, Alina Krasnobrizha
The paper investigates the phenomenon of self‑repair in language models, proposing that it arises from a pre‑existing gain in components that act as counterweights when a component is ablated. By modeling interventions as points on a counterfactual axis, the authors derive an affine law for the causal repair response of fine‑grained units, showing that most downstream directions across several models follow this law. They further demonstrate that the slope of this law can be predicted from fixed weights, suggesting that self‑repair is a predictable, counterweight‑driven response rather than a noisy, unexplained effect.
By Areeb Ahmad, Pratinav Seth, Vinay Kumar Sankarapu
The paper evaluates Jev, a general-purpose decision model, for classifying network traffic using only the first ten packets’ sizes, directions, and inter‑packet times. Across 52,000 records, adding 40 labeled examples improves Jev’s accuracy from 9.80% to 28.42%, yet Random Forest and Extra Trees models trained on 8,000 records achieve higher accuracies of 69.95% and 66.80%. Even with 150 examples, Jev reaches only 34.50% accuracy, and in a paired 100‑record test, Jev’s 29% accuracy and 0.750 s median request time are lower than the generative OpenAI GPT‑5.6 Sol’s 37% accuracy and 6.036 s response time, though Jev incurs lower API charges.
By Shenghe Xu, Lifan Mei
The study evaluates 4‑bit quantization and low‑rank adapter fine‑tuning (QLoRA) on several large protein language models, finding that many model‑task pairs retain over 90% of full fine‑tuning performance while achieving up to 90% GPU memory savings. QLoRA preserves early‑layer representations and induces task‑specific changes in later layers, closely resembling full fine‑tuning with smaller representational shifts. For generative models, 4‑bit quantization largely maintains structural and sequence‑level properties, though token‑level analysis reveals model‑dependent changes in autoregressive output distributions.
By Ilan Yaniv Zeisler, Sebastian Clancy, Pouriya Bayat, Saaim Raad, Ivan Kraskov, Matthew Xie, Vivian White, Spencer Perkins, Serena Singh, Sepehr Bayat, Keith Pardee
SkillSpec is a two‑phase framework for evolving natural‑language skills in large language model agents. The first phase, consensus‑gated evolution, generates candidate skills from complementary editing intents and commits updates only when paired evaluations reach consensus on overall improvement and non‑negative aggregate gain. The second phase, representation specialization, uses signals from the optimization trajectory to choose an appropriate flat, graph, or hybrid structure for the skill, improving success rates by an average of 6.89% over SkillOpt across six benchmarks and three target language models.
By Huancheng Chen, Xiaodi Sun, Zhaoqiong Huang, Shenyang Huang Shreya Singhal, Jingwen Lu
The paper introduces entry-state sharpening, a data‑free pre‑training step that prepares a language model’s checkpoint in a sharper, lower‑entropy state before test‑time reinforcement learning (TTRL). By reducing policy entropy, the model can more efficiently use its limited adaptation budget, leading to higher endpoint conversion efficiency across tasks such as MATH, GPQA, and AMC. Experiments with different data‑free objectives (e.g., R‑Zero vs. SPIRAL) demonstrate that the choice of pre‑training objective strongly influences the checkpoint’s readiness for TTRL, and a label‑free self‑distillation intervention can further sharpen the entry state.
By Zhanming Zhang, Vinoth Selvendran
The paper introduces a supervised algebraic counting field (ACF) that predicts cryptic ligand‑binding pocket residues from apo protein structures by compiling geometric, physicochemical, and topological features into compact integer‑weighted lookup tables. ACF allows each prediction score to be reconstructed from feature values, training counts, table weights, and spatial aggregation, eliminating the need for sequence search, structural‑template transfer, or a protein language model during inference. In benchmark tests on CryptoBench and external datasets, ACF outperformed the P2Rank method in mean paired ROC‑AUC on one external set, though gains varied across datasets and were not universally confirmed.
By Shan Yu, Xuening Wu
ScaffoldM3C is a lightweight, multimodal, auto‑regressive framework that generates stable 3D block constructions by treating the task as a probabilistic next‑block generation problem. It incorporates text, image, and sketch conditioning, introduces a scaffold block token to aid intermediate stability, and uses Sequential Monte Carlo to explore multiple assembly sequences simultaneously. The model is four times smaller than existing baselines, achieving 5‑ to 20‑fold inference speedups while matching or surpassing state‑of‑the‑art construction quality and stability in both simulations and real‑world robot demonstrations.
By Gadiel Sznaier Camps, Chengyang He, Guillaume Sartoretti, Eduardo Montijano, Mac Schwager
RPTune is an end‑to‑end framework that improves in‑context catalog search for small merchant businesses by learning to curate product catalogs and fine‑tuning large language models (LLMs) with catalog‑grounded supervision. It uses an encoder‑reorganizer curator to order and prune products based on LLM feedback, and then applies context‑relative rewards during LLM post‑training. Across seven real merchants and 100 complex conversational queries per merchant, RPTune boosts search accuracy by up to 31.4 percentage points from curation alone and an additional 10.3 points on average from post‑training.
By Chuxuan Hu, Hejie Cui, Norman Huang, Shubham Kumar Bharti, Wang-Chiew Tan, Sercan \"O. Ar{\i}k
Chordonomicon is a new dataset of over 666,000 song-level symbolic chord progressions, each annotated with structural parts such as verse, chorus, and bridge, as well as genre and release date. The dataset was compiled by scraping user-generated progressions from multiple sources and shows strong similarity to established prior datasets. The authors also provide a reproducible benchmark suite for next chord prediction, evaluating RNN, GRU, and LSTM models across various context windows and data scales, and find that structural part annotations consistently improve prediction performance.
By Spyridon Kantarelis, Ioannis Liolitsas, Konstantinos Thomas, Vassilis Lyberatos, Edmund Dervakos, Giorgos Stamou
The paper extends mechanistic interpretability of large language models by modeling concepts as low‑dimensional non‑linear manifolds rather than linear subspaces. It introduces a concept‑based alignment (CBA) score to compare these manifolds across layers and models, revealing block structures in intermediate layers, a shift from syntax‑dominated to mixed syntactic‑semantic concepts, and training‑dependent multilingual sharing. The study also shows that alignment patterns differ across model families and training stages, with adjacent stages aligning more closely than distant ones.
By Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff
SyntheticHLS is a framework that uses large language models to create large-scale, diverse synthetic high‑level synthesis (HLS) datasets. It employs an iterative, feedback‑guided mutation loop that transforms seed designs into more complex, scalable ones, guided by quantitative metrics of design complexity and scalability. The resulting datasets outperform manually curated or zero‑shot generated ones in training deep learning models for HLS quality‑of‑results, offering broader coverage of design space and better generalization.
By Stefan Abi-Karam, Miaoyan Zhou, Callie Hao
arXiv:2610. 00436v1 Announce Type: new Abstract: Online batch selection fine-tunes a language model on the most useful part of each candidate batch.
By Hongyu Chen, Xinyi Luo, Ming Zhao, Lin Tang, Zihan Xu, Jing Li, Yuxuan Wang, Haoran Deng, Wei Zhang
The paper introduces Clock Diffusion, a framework for semi‑autoregressive continuous diffusion language models that incorporates position‑dependent noise schedules, efficient training, and sampling algorithms. It presents two generation modes—block and sliding window—and defines ClockDLMs, a family of Gaussian models that achieve state‑of‑the‑art diffusion likelihoods on OpenWebText and outperform continuous baselines on GSM8K while matching or exceeding discrete diffusion models. The authors also propose Cache Grab, a set of efficient samplers that adapt accelerated inference techniques from discrete diffusion to further improve model quality and efficiency.
By Yair Schiff, Omer Belhasin, Roy Uziel, Matan Rusanovsky, Ran Zilberstein, Marianne Arriola, Gilad Turok, Guanghan Wang, Volodymyr Kuleshov, Michael Elad
The study investigates how layer‑wise intervention responses in language models change over the course of pretraining, using single‑block identity bypass across multiple checkpoints and model‑domain combinations. It finds that while depth ordering of responses persists, their magnitudes shift, with nearby checkpoints showing stronger rank correspondence than distant ones and large changes occurring at positions that recur across samples and transfer across evaluation domains. Controlled experiments reveal that these longitudinal changes cannot be explained by a single downstream sensitivity and depend on perturbation strength and direction, indicating that layer sensitivity is structured but dynamic.
By Shengye Tao, Yinzhu Cheng, Haihua Xie
The paper introduces higher-order positional encodings that enrich graph representations by incorporating topological information from lifted incidence structures, without altering existing graph learning backbones. It theoretically shows that these encodings can mix graph Laplacian frequencies beyond what scalar spectral filters achieve, and demonstrates their effectiveness on Graph Transformers for datasets like ZINC and synthetic benchmarks. The approach bridges graph positional encodings and topological deep learning, enabling standard models to exploit higher-order interactions.
By Caleb Stam, Aagrim Hoysal, Sanjukta Krishnagopal
The paper introduces IDiom, an autoregressive protein language model trained on a large dataset of intrinsically disordered protein regions (IDRs) from AlphaFold, and demonstrates that it can generate sequences matching natural IDR composition, motifs, and disorder. It further presents RL‑SAE, a reinforcement learning approach that uses sparse autoencoder features to steer generation toward specific functional patterns, achieving high activation of targeted features and improved predicted subcellular localization and transcriptional activity. The combination of IDiom and RL‑SAE allows interpretable, composable IDR design by explicitly controlling function‑associated sequence features.
By Jason X. Liu, Sebastian Ibarraran, Frank Hu, Soojung Yang, Xinyu A. Feng, Abigail Park, Anagha Aneesh, Lacramioara Bintu, Alexander R. Dunn, Grant M. Rotskoff
The paper reports a large-scale, compute-controlled study of Chemical Language Models (CLMs) involving over 30,000 experiments across different molecular representations, tokenizations, model sizes, datasets, and architectures. It finds clear scaling trends in pretraining loss but shows that these improvements do not translate into proportional gains in goal-directed molecular design, with chemical syntax saturating early while semantic properties develop more slowly. The authors release a new suite of models, NovoMolGen, that achieves state-of-the-art results in drug discovery tasks, highlighting a disconnect between representation learning and downstream design and calling for new pretraining paradigms that target chemical semantics.
By Roshan Balaji, Kamran Chitsaz, Quentin Fournier, Nirav Pravinbhai Bhatt, Sarath Chandar
The paper evaluates System One decision models—typed models that output probabilities for branching decisions—against supervised classifiers and generative language models on automated decision gate tasks. Eight checkpoints from six families, including the hosted model Jev, were benchmarked on workflow, intent, and social‑science items, showing that small trained classifiers match or slightly outperform decision models on intent and workflow when labels are available, while decision models outperform zero‑shot classifiers when labels are absent. The study also explores calibration, risk thresholds, and cost‑efficiency trade‑offs, providing condition‑dependent design guidelines for automated decision gates.
By Amir Rafe, Subasish Das