arXiv:2604. 26866v2 Announce Type: replace-cross Abstract: Large language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next token prediction.
By Dimitris Dimakopoulos, Shay B. Cohen, Ioannis Konstas
The study investigates whether unusually high‑gain parameters in transformer models—specifically gated feed‑forward network (gated‑FFN) rows—play a critical functional role across both text and genomic foundation models. By computing exact bilinear weight operators and testing structural extremeness, the authors find that high‑gain rows are enriched for functional importance but do not reliably predict causal effect size or severity. The analysis reveals model‑specific causal organizations, including super‑additive interactions in DNABERT‑2 and position‑localized dependencies in GENERator, indicating that structural prominence signals enrichment rather than calibrated criticality.
By Alexandros Tzanakakis, Aris Karatzikos, Ilias Georgakopoulos-Soares
The paper introduces IDiom, an autoregressive protein language model trained on a large dataset of intrinsically disordered protein regions (IDRs) from AlphaFold, and demonstrates that it can generate sequences matching natural IDR composition, motifs, and disorder. It further presents RL‑SAE, a reinforcement learning approach that uses sparse autoencoder features to steer generation toward specific functional patterns, achieving high activation of targeted features and improved predicted subcellular localization and transcriptional activity. The combination of IDiom and RL‑SAE allows interpretable, composable IDR design by explicitly controlling function‑associated sequence features.
By Jason X. Liu, Sebastian Ibarraran, Frank Hu, Soojung Yang, Xinyu A. Feng, Abigail Park, Anagha Aneesh, Lacramioara Bintu, Alexander R. Dunn, Grant M. Rotskoff
arXiv:2606. 07607v1 Announce Type: new Abstract: Advances in machine learning and computational power have unlocked the predictive potential of the human genome, yet biologists now demand that these models also elucidate the underlying biological mechanisms.
By Shasha Zhou, Mingyu Huang, Ke Li
Neural language models trained on large crowdsourced corpora frequently exploit spurious surface patterns tied to target labels without true linguistic or causal relevance, boosting benchmark performance while failing on adversarial or out-of-distribution inputs. Existing approaches either require manual specification of the feature vocabulary or automate discovery only partially, leaving the gap between dataset-level correlation and model-level exploitation unaddressed.
arXiv:2608. 09209v1 Announce Type: cross Abstract: Neural language models trained on large crowdsourced corpora frequently exploit spurious surface patterns tied to target labels without true linguistic or causal relevance, boosting benchmark performance while failing on adversarial or out-of-distribution inputs.
By Chidaksh Ravuru, Shashank Srivastava