arXiv Machine Learning
Sep 29
Block Sparse Attention with Log-Linear Complexity
arXiv:2609. 31093v1 Announce Type: new Abstract: Scaling language models to long contexts is limited by the quadratic cost of self-attention.
Benchmarks and evaluation
Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.
arXiv Machine Learning
Sep 29
Encryptability As a Coordinate Choice: Depth-One Homomorphic Federated Learning of Quantum Neural Networks
The paper demonstrates that by choosing unit‑quaternion coordinates, encrypted updates for variational quantum circuits become bilinear, allowing depth‑one homomorphic federated learning without bootstrapping. This coordinate choice reduces encrypted rotation updates to a single multiplicative level and federated averaging to zero levels, eliminating the previously prohibitive cost of one round per gate. Experiments across two cryptographic backends and up to 20 clients show negligible aggregation error and no measurable loss in utility, with hardware validation on a 156‑qubit processor achieving near‑optimal fidelity.
arXiv Machine Learning
Sep 29
Complex-Valued Phase-Coherent Transformer
The paper introduces the Phase-Coherent Transformer (PCT), a complex-valued architecture that replaces traditional softmax attention with a real-valued, smooth gate applied to L2-normalised query-key similarities. PCT eliminates token competition, preserving phase information across layers, and demonstrates strong generalisation on a variety of mid-scale benchmarks, outperforming both standard softmax Transformers and other complex-valued counterparts. Experiments confirm that the gate design is essential: preserving negatively aligned phase components is crucial for performance, while violating these conditions leads to degradation or collapse on long-range tasks.
arXiv Machine Learning
Sep 29
To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationarity
arXiv:2609. 30501v1 Announce Type: new Abstract: Although bilevel optimization (BLO) has emerged as a powerful framework for addressing many complex and nested machine learning problems in recent years, most existing studies are confined to the lower-level strongly convex (LLSC) or lower-level generally convex (LLGC) settings (i.
arXiv Machine Learning
Sep 29
Progressive Memory Transformer: Memory-Aware Attention for Time-Series
The paper introduces the Progressive Memory Transformer (PMT), a transformer variant that adds writable, window‑aligned memory to expose mid‑range representations alongside token and sequence‑level outputs. PMT is trained with a hierarchical learning framework that applies separate objectives at local, mid‑range, and global scales, encouraging the model to capture fine‑grained variation, window‑level motifs, and overall sequence agreement. Experiments on seven UCR/UEA/UCI classification datasets, a cue‑retention probe, and forecasting tasks show that PMT achieves strong low‑label classification performance, competitive multi‑horizon forecasting, and evidence that its memory states encode mid‑range motifs.
arXiv Machine Learning
Sep 29
HiLiftAeroML: A High-Fidelity Computational Fluid Dynamics Dataset for High-Lift Aircraft Aerodynamics
HiLiftAeroML is the first open high‑fidelity CFD dataset focused on high‑lift aircraft aerodynamics, comprising 1,800 simulations across 180 variants of the NASA Common Research Model and ten angles of attack from 4° to 22°. Each case uses a GPU‑accelerated explicit wall‑modeled LES on grids of 300–500 million cells, covering attached, separated, and post‑stall flow conditions. The dataset, released under CC‑BY‑4.0, includes geometries, time‑averaged fields, integrated loads, validation data, and benchmark splits, and is accompanied by baseline models that perform well on interpolation and held‑out geometry tests but still struggle with high‑angle separated flow and out‑of‑distribution regimes.
arXiv Machine Learning
Sep 29
Electric Vehicle Charging Station Location Selection using Geospatial Artificial Intelligence (GeoAI)
The paper presents a GeoAI framework that uses a variational autoencoder and a graph convolutional network to analyze high‑dimensional geospatial data for electric vehicle charging station location selection. By compressing EV usage, land‑use, population, and traffic attributes into a latent space, the model predicts suitable future station sites and identifies 27 new candidates in Bryan‑College Station, Texas. Two policy scenarios—maximizing geospatial similarity versus minimizing travel distance—demonstrate how different strategic objectives influence station placement outcomes.
arXiv Machine Learning
Sep 29
NEMSim: Learning Control-Conditioned Multi-Event Physical Dynamics via Executable Event-Mechanism Priors
NEMSim is a neural simulator that incorporates executable event‑mechanism priors to model control‑conditioned multi‑event physical dynamics. By compiling predefined event‑attribute descriptions into a transition structure, it links control‑dependent event intensities, prior‑guided mechanism attribution, and state‑dependent responses. In a 3D KMC benchmark, NEMSim achieves 58.9%–81.3% lower RMSE than the strongest baseline and remains superior even with only 10% of training data, demonstrating that rule integration drives its performance gains.
arXiv Machine Learning
Sep 29
WEECFP-SuRGE: A Position-Aware Substructure Encoding Method for Molecular Property Prediction
WEECFP-SuRGE introduces a position‑aware substructure encoding method that combines tokenized hierarchical Morgan fingerprints with graph‑distance‑dependent rotations applied at the input and within transformer self‑attention. The approach captures local chemistry, long‑range interactions, and molecular topology without requiring external pretraining or 3‑D conformer generation. Benchmarks on MoleculeNet and the Therapeutic Data Commons ADMET datasets show competitive performance, and a reconstruction procedure correctly identifies constitutional isomers for 92.6% of a 4,200‑molecule library.
arXiv Machine Learning
Sep 29
Neural Parameter Estimation of RC Thermal Building Models for Model Predictive Control
The paper introduces the Estimator from Scratch, a neural network that embeds RC thermal building equations to estimate parameters, and its extension, the Pretrained Estimator, which is pretrained on multiple buildings to avoid initial‑guess dependence. Both methods outperform a genetic‑algorithm estimator and a fully black‑box neural network across simulated and real buildings, achieving superior prediction accuracy and lower, more consistent MPC costs. The Pretrained Estimator is highlighted as a robust, computationally efficient, and initial‑guess‑free alternative for RC parameter estimation, with potential applicability to other control‑oriented dynamical systems.
arXiv Machine Learning
Sep 29
BeatGraph: Self-Supervised Heartbeat Graphs for Infant ECG Representations from the Home Environment
BeatGraph is a self‑supervised model that represents infant ECG recordings as graphs of individual heartbeats rather than fixed‑length patches, allowing it to capture the higher heart rates and distinct waveform patterns of infants. The model uses a shared beat encoder, a Transformer for temporal ordering, and graph attention layers to produce a window embedding, which is pretrained on a large unlabeled infant ECG corpus and fine‑tuned for tasks such as sleep‑wake detection, infant‑state classification, activity‑source identification, and affect recognition. BeatGraph achieves state‑of‑the‑art performance on multiple infant‑specific benchmarks and transfers well to pediatric and adult ECG datasets, while also releasing the first public infant ECG corpus collected in diverse home and classroom settings.
arXiv Machine Learning
Sep 29
On the Limits of Univariate Deep Learning for Significant Wave Height Forecasting
The paper performs a systematic hyperparameter search across five deep learning architectures (DLinear, LSTM, PatchTST, ResAttLstm, and Mamba2) and nine context lengths for forecasting significant wave height (Hs) at a single buoy, then re‑evaluates the best configurations on a 47‑buoy, 37‑year dataset. Across all models, performance converges to a very small spread (SD = 0.0014 m², 0.8% of the grand mean), yet all models still outperform persistence, though none consistently outperforms the others. The study finds that persistence already captures the dominant linear‑inertial signal in univariate Hs, and that model‑class differences are dwarfed by cross‑buoy variance, suggesting diminishing returns for further architecture engineering under univariate input settings.
arXiv Machine Learning
Sep 29
ChemMLLM: Chemical Multimodal Large Language Model
ChemMLLM is a unified chemical multimodal large language model designed for molecule understanding and generation across text, SMILES strings, and images. The authors curated five multimodal tasks and benchmarked ChemMLLM against leading general MLLMs, chemical LLMs, and specialized models, finding it outperforms general-purpose MLLMs and matches specialized models on all tasks. The study demonstrates that a single foundation model can handle diverse cross‑modal chemical tasks, including image generation, enabling more intuitive visual human‑AI interaction.
arXiv Machine Learning
Sep 29
When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization
arXiv:2609. 30271v1 Announce Type: new Abstract: Adaptive optimizers are commonly parameterized by a fixed power of the second-moment estimate.
arXiv Machine Learning
Sep 29
Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model
The paper audits a commercial non‑generative System‑1 model on biosecurity‑relevant benchmarks, evaluating accuracy, calibration, error detection, selective prediction, and sensitivity to answer‑option order. It finds that the model’s accuracy varies strongly by task, is reasonably well calibrated when the vendor’s uncertainty field is interpreted correctly, and that option order can cause significant prediction changes—averaging across rotations improves accuracy. The study also shows that applying averaging only to low‑confidence items recovers most of the gain at a lower cost.
arXiv Machine Learning
Sep 29
Aurora-X: Built for Extreme Time Series Forecasting
Aurora‑X is a billion‑parameter time‑series foundation model designed for extreme forecasting tasks. It employs a progressive curriculum that starts with channel‑independent pretraining, then adds cross‑variable dependencies, variable context and horizon lengths, and optional future covariates during mid‑training. A variable‑resolution post‑training stage allows adjustable temporal spans per token at inference, while a pattern‑guided mixture‑of‑experts expands capacity through sparse activation and expert specialization. An implicit quantile network head predicts arbitrary quantiles, enhancing probabilistic forecasting flexibility. Experiments on GIFT‑Eval, TIME, FEV‑Bench, TFB, and DAG‑Bench show state‑of‑the‑art performance against both pretrained TSFMs and task‑specific supervised models.
arXiv Machine Learning
Sep 29
Differential Attention Unlocks Complementary EEG and Speech Fusion for Emotion Recognition
The paper introduces EmoSpeechBrain, a multimodal emotion recognition framework that fuses EEG and speech signals. It employs differential attention in the EEG encoder to cancel shared noise and an attention-based gating adapter to align modalities and weight their contributions. Experiments on PME4 and EAV datasets show up to 12.9% accuracy improvement over other EEG encoders and surpass unimodal baselines by up to 23.1%.
arXiv Machine Learning
Sep 29
Scaffold: Support Graph Theory Based Sparsification for Graph Neural Networks
Scaffold is a new unsupervised graph sparsification framework for graph neural networks that uses support graph theory preconditioners to jointly control dilation and congestion, thereby preserving short communication paths while avoiding bottlenecks. It achieves superior aggregate ranking across 19 homophilic and heterophilic benchmarks, recovering or closely approaching full‑graph GNN performance with only 10%–50% of the original edges. The method reduces memory usage to less than half and cuts end‑to‑end training time, including sparsification overhead.
arXiv Machine Learning
Sep 29
NEXT: Physics-Informed Neuro-Spectral Exponential Time Differencing Architectures
The paper introduces NEXT, a new physics-informed neural architecture that combines Neuro‑Spectral Architectures with high‑order exponential time‑differencing integrators. NEXT integrates the stiff linear part of PDEs exactly via matrix exponentials while a neural network models the remaining nonlinear dynamics, thereby overcoming spectral bias, causality issues, and numerical instability seen in previous PINNs and NeuSAs. Benchmark tests on stiff PDEs show NEXT remains stable and accurate, and it can also solve inverse problems by learning unknown parameters or boundary conditions from sparse data.
arXiv Machine Learning
Sep 29
MeshGraphNet-Transformer: Scalable Mesh-based Learned Simulation for Solid Mechanics
MeshGraphNet-Transformer (MGN‑T) is a new architecture that fuses Transformers’ global modeling with MeshGraphNets’ geometric inductive bias, keeping a mesh‑based graph representation. It replaces iterative message passing with a physics‑attention Transformer that updates all nodal states simultaneously, enabling efficient learning on high‑resolution meshes with diverse geometries, topologies, and boundary conditions. MGN‑T accurately models impact dynamics, self‑contact, plasticity, and multivariate outputs, outperforming state‑of‑the‑art methods on classical benchmarks while using far fewer parameters.