← Back to all news
arXiv Machine Learning By Trevor McCourt, Ila R. Fiete, Isaac L. Chuang

Fault-tolerant foundation models

Read the original on arXiv Machine Learning →

The paper demonstrates that large‑language models can be trained to tolerate unreliable computer hardware, with their error resilience improving as the models grow. Using 40,000 GPU‑hours of training on simulated faulty hardware, the authors derive modified neural scaling laws that show models learn to compute within efficient error‑correcting codes. This suggests that, if formally fault‑tolerant, such models could enable AI inference on low‑energy, faulty hardware, potentially yielding significant energy savings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
1d ago

Analysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection Study

The paper presents the first instruction-level fault injection study of large language model (LLM) inference, focusing on how GPU soft errors affect these compute- and memory-intensive models. It examines the impact of model architecture, parameter scale, and task complexity on LLM vulnerability, revealing distinct reliability characteristics compared to earlier neural networks. The study offers new insights that can guide the development of more effective fault tolerance mechanisms for LLMs.

By Duo Chai, Zizhen Liu, Shuhuai Wang, Songwei Pei, Cheng Liu, Huawei Li, Shangguang Wang
llms
More like this →
arXiv Machine Learning
Oct 5

Exploring the Trade-Off Between Structured Pruning and Fault Tolerance in Deep Neural Networks for Space Applications

The study examines how reducing the width of deep neural networks through structured pruning affects their robustness to Single Event Upsets (SEUs). By iteratively pruning baseline models and conducting fault-injection tests at each stage, the authors find that while pruning increases per-inference sensitivity to bit-flips, the shorter execution time offsets this, lowering the overall SEU probability. Consequently, structured pruning can deliver energy and latency savings without compromising reliability for space-based AI systems.

By Toon Vinck, Na\"in Jonckers, Jaro De Roose, Jeffrey Prinzie, Peter Karsmakers
efficiency
More like this →
arXiv AI
Jul 7

From Arithmetic to Logic: The Resilience of Logic and Lookup-Based Neural Networks Under Parameter Bit-Flips

arXiv:2603. 22770v2 Announce Type: replace-cross Abstract: The deployment of deep neural networks (DNNs) in safety-critical edge environments necessitates robustness against hardware-induced bit-flip errors.

By Alan T. L. Bacellar, Sathvik Chemudupati, Shashank Nag, Allison Seigler, Priscila M. V. Lima, Felipe M. G. Fran\c{c}a, Lizy K. John
efficiencybenchmarkssafety
More like this →
arXiv AI
Sep 10

Limits of Reliability and Scaling in Language Models

The paper argues that large language models cannot achieve perfect reliability for any task, even with unlimited scale. It establishes that each generative task has an inherent reliability ceiling set by how much output uncertainty can be resolved from observable context, with a resolvable part that can be improved by more context and a subjective part tied to task ambiguity. The authors derive a scaling law showing that performance is limited by the scarcer resource—either training data or model capacity—and explain how this law explains phenomena such as retrieval augmentation and catastrophic forgetting.

By Subhabrata Majumdar
llms
More like this →