The paper examines how large language models (LLMs) express uncertainty compared to humans, noting that humans use verbal markers like "possible" or "likely" to convey metacognitive awareness. By curating a corpus of human uncertainty markers and benchmarking LLMs against it, the authors find that LLMs encode these markers with numerical levels that differ substantially from human usage. They introduce METHODNAME, an optimization-based algorithm that learns an optimal uncertainty profile over verbal markers directly from LLM outputs, enabling a direct comparison of confidence semantics and revealing systematic disparities in verbal expressions.
By Jinhao Duan, Zicheng Liu, Zijie Liu, Kaidi Xu, Tianlong Chen
The paper investigates using short polynomial approximations to accelerate special‑function operations in large language models on NVIDIA Blackwell GPUs. By replacing native sigmoid, tanh, and SiLU with degree‑3 or degree‑4 bfloat16 programs, the authors achieve up to 2.19× speed‑ups in isolated FP16 benchmarks and modest training‑step throughput gains (2.7–8.0%) across four integration tasks. The study also evaluates model behavior, finding negligible training‑loss differences within 100 billion tokens.
By Robert Hu
EchoPress is a training‑free method for pruning key‑value caches in large language models. It approximates the reconstruction attention used by KVzip by leveraging queries and keys from standard prefill, reconstructing only the first chunk to calibrate importance scores for the rest of the context. Experiments on LongBench and RULER with Qwen3‑8B and Llama‑3.1‑8B‑Instruct show that EchoPress matches KVzip’s task accuracy across eviction ratios from 50% to 90%, while reducing compression overhead by 1.7–19.6× and total prefill time by up to 2.9×.
By Jiawei Lin, Saibo Geng, Thomas Bourgeat
The paper introduces attention kernels that replace the exponential function in transformer softmax to better handle operator learning on probability measures with heavy-tailed (polynomial) distributions. Two new benchmarks with closed‑form targets are constructed to evaluate how different kernel growth rates and data preprocessing affect performance. The study finds that slower‑growing kernels prevent ensemble collapse on heavy‑tailed tasks, while softmax with symlog preprocessing only succeeds on a subset of problems, and that all kernels perform similarly on Gaussian data.
By Kailen Hargenrader, Edoardo Calvello, Bohan Chen
The paper shows that the geometric patterns seen in language model embeddings—such as circles for months and saddle-shaped manifolds—arise from group-invariant statistics in word co‑occurrence data. By extending previous work on translation symmetry to arbitrary finite, compact, and homogeneous groups, the authors prove that embeddings correspond to matrix elements of the irreducible representations of the symmetry group. They validate this theory experimentally with the cyclic group <Z_{12}> for months, a dihedral group for musical chords, and a spherical‑harmonic embedding for celestial objects.
By Liam Storan, Andreas Tolias, Nina Miolane
The report introduces AnyJev, a method that extracts typed decisions from a pretrained instruction‑tuned language model by restricting the next‑token distribution to a set of option tokens. It corrects two defects—label bias and position bias—without any gradient updates by dividing out a label prior and averaging log‑probabilities over cyclic rotations of the option list. Experiments on two 20‑option tasks show that rotations reduce order‑flip rates and improve accuracy across all tested models, while a stopping rule reduces the number of required rotations and increases decision throughput.
By Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu, Liang Wu
AF‑Muon is an AdamW‑free extension of the Muon optimizer that retains Muon’s matrix update for hidden weights while applying a support‑aware finite‑cap linear minimization oracle to tied vocabulary tables and an RMS‑normalized update for one‑dimensional auxiliary parameters. This design eliminates second‑moment state, reducing optimizer‑state memory by about 20% compared to Hybrid Muon. Across nine tied‑token settings—including decoder‑only language models, T5‑style encoder‑decoders, and ImageGPT‑style variants—AF‑Muon consistently improves mean validation loss and perplexity over both Hybrid Muon and a SCION‑style Sign endpoint, with robust gains confirmed by long‑horizon runs and hyperparameter studies.
By Arash Lagzian, Paniz Halvachi, Junming Zhang, Zhouhan Lin, Dianbo Liu
The paper presents a method for distilling tabular foundation models (TFMs) into lightweight, dataset‑specific students. By using the full labeled training set as teacher context and training students on both observed and synthetic queries, the authors achieve significant performance gains over traditional supervised models on TabArena and TALENT benchmarks. The distilled students also provide substantial inference speedups, reducing the cost of repeated inference.
By Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
The paper investigates how treating the optimization objective as a design variable can improve Remaining Useful Life (RUL) prediction in predictive maintenance. Five model architectures are compared under single‑objective and multi‑objective hyperparameter optimization, with the latter using NSGA‑II and Entropy‑CRITIC weighting to balance accuracy and prediction timeliness. Results on NASA C‑MAPSS and BackBlaze datasets show that multi‑objective optimization reduces directional imbalance in predictions and can alter model rankings, highlighting the importance of objective choice in RUL modeling.
By Tugrul Cabir Hakyemez, Ener Uras Gokhan
FastCI is a new framework designed to make continuous integration (CI) for large language model (LLM) training frameworks more efficient. It uses runtime evidence to select and prune tests that run changed code in equivalent contexts, prioritizes high‑risk tests, and optimizes test workloads beyond each test’s intended validation scope. In evaluations on ByteDance’s LLM training framework, FastCI cut CI latency by 77.5 % and GPU usage by 63.9 %, while slightly improving modified code coverage retention.
By Tianshuo Qiao, Naiqian Zheng, Xiaopeng Liu, Shuguang Wang, Diandian Gu, Xuanzhe Liu, Xin Jin
The paper audits whether synthetic distractors in RLVR corpora act as shortcuts for learning policies. A classifier using only surface statistics barely outperforms chance, and manual inspection reveals that code distractors are almost identical to correct answers. Experiments with a paraphrase‑matched control show no exploitation advantage for the unmodified data, indicating that the detectable artifact was not used by the policy.
By Esther Xin
The study evaluates whether aggregated brand recommendation profiles can identify the language model that generated them. Using 6,475 responses from five deployed endpoints, a character‑n‑gram classifier accurately attributes single responses to the correct system (97.84% accuracy). However, when responses are aggregated into domain‑condition units, the classifier’s performance drops to 66.53%, and a forest model misclassifies all gift‑domain units, indicating that aggregated brand behaviour does not reliably reveal the underlying system.
By Dmitrij \.Zatuchin
The paper investigates how fine‑tuning large language models with a small number of harmful examples can erode their refusal behavior, and explores whether localizing safety‑related behavior to specific layers or directions can provide robust defenses. Experiments across six checkpoints from four model families show that harmful and benign prompts remain linearly separable after attack, and that patching clean hidden states or freezing layers up to a transition depth can restore refusal. However, attackers can bypass these defenses by spreading updates or targeting singular directions, indicating that adaptive fine‑tuning can defeat localized repairs and highlighting the need for multiple defensive checks.
By Jungseob Lee, Dongyub Jude Lee, Sugyeong Eo, Seongtae Hong, Seungyoon Lee, Heuiseok Lim
The paper introduces NEEDLE, a training‑free technique for removing backdoors from large language models. After a trigger is identified, NEEDLE estimates a backdoor direction and a refusal subspace using activation vectors, then applies sequential weight orthogonalisation to suppress the backdoor while preserving refusal‑related representations. The method requires no clean reference model or original poisoned data and achieves the lowest attack success rate and minimal impact on model performance across multiple model families and attack types.
By Minoo Kim, Vasileios Lampos, George Drayson
AIR-LLM is an edge inference architecture that broadcasts large language model (LLM) weights over radio, allowing edge devices to perform matrix-vector multiplications directly in the RF domain without storing or loading the weights. The system uses MIMO spatial multiplexing and an energy‑efficient precoder‑postcoder pair to reduce airtime and calibrate the wireless channel, enabling a single broadcast to serve unlimited users. Experiments on real urban channel models show that AIR-LLM achieves only a 4.0% perplexity loss on LLaMA‑3.1‑8B while saving energy by up to 157.7× compared to FP16 and reducing airtime by over 100× for 20 users.
By Zhihui Gao, Tingjun Chen, Dirk Englund
Leto is a fault‑tolerant training system for large language models that enables fast in‑place recovery on surviving hardware after hardware‑operable failures. It retains the working model state and reusable process state, while pre‑initializing remaining state in a shadow trainer, using two‑tier erasure protection and chunk‑level transactional updates to maintain consistency. Experiments on NVIDIA A100 clusters show Leto recovers 3.6–6.5× faster than checkpointing baselines and boosts productive training time by up to 13.7 percentage points, with simulations indicating over 95% productivity on a 131,072‑GPU cluster.
By Geon-Woo Kim, Joon Ha Kim, Daehyeok Kim
STEER is a sampling method for relational foundation models that reduces inference cost by focusing on the most relevant tables for a prediction task. It uses a large language model to rank foreign‑key edges in the database schema into relevance tiers, then assigns traversal probabilities based on these tiers. Evaluated on three state‑of‑the‑art RFMs, STEER cuts inference context size by roughly 40% on average while preserving or improving accuracy.
By Abdalla Mohamed, Ashraf Aboulnaga
The paper investigates why Group Relative Policy Optimization (GRPO) benefits from per‑prompt normalization by examining the local curvature of the sequence‑level policy gradient. It shows that standard deviation normalization acts as an adaptive gradient, yielding a provably faster convergence rate than unnormalized REINFORCE under mild conditions, with the improvement tied to the average within‑prompt reward standard deviation. The authors also propose IS‑GRPO, an importance‑sampling variant that maintains alignment with the full gradient and offers a tighter convergence guarantee, and empirically validate these theoretical insights on GSM8K and MATH datasets at 1.5B and 7B model scales.
By Cheng Ge, Caitlyn Heqi Yin, Hao Liang, Jiawei Zhang
GLoC-EHR is a multimodal language model that processes electronic health records by combining a fixed-size global memory of the entire patient trajectory with a local memory of selected events. It generates hospital-course summaries and masked concept descriptions, then is fine‑tuned to cite evidence before answering clinical questions, using group relative policy optimization to reward correct, evidence‑supported responses. On MIMIC‑IV outcome tasks, GLoC‑EHR achieves the highest macro AUROC among compared models when answering directly, and maintains strong performance with evidence‑cited reasoning while adding distinct supported findings from the local memory.
By Chaiho Shin, Kwangsoo Kim
arXiv:2610. 01172v1 Announce Type: new Abstract: We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models.
By Jimin Seo, Gyubok Lee, Yeonsik Jo, Kiwoong Yoo, Yeongoon Kim, Minhae Oh, Jin Woo Koo, Suhwan Kim, Nakyung Lee, Minsik Seol, Idris Nechnech, Jaehyeon Kim, Giho Lee, Jungwoo Lee