RetryTrigger: intelligent inference duplication for enhancing LLM resilience to hardware transient faults

Abstract

Large language models (LLMs) have exhibited outstanding performance across a wide range of natural language processing (NLP) tasks. However, the rising prevalence of hardware transient faults has made silent data corruptions (SDCs) in LLMs increasingly problematic, severely degrading output quality and user experience. State-of-the-art protection schemes primarily rely on hardware-assisted algorithm-based fault tolerance (ABFT) or boundary-setting-driven online fault tolerance (FT2) for selective layers, yet these solutions suffer from strict hardware dependencies, substantial overhead, or incomplete coverage. To address these limitations, we propose RetryTrigger, a novel hardware-free fault-aware inference methodology capable of handling all potential faults. During LLM inference, RetryTrigger dynamically collects runtime output features (e.g., maximum probability, top-k probability gaps, output entropy, logits statistics, and inference latency), which are used to train a LightGBM meta-model. This meta-model accurately predicts whether duplicate inference should be performed, thereby effectively mitigating faults while preserving efficiency without additional hardware dependence. Extensive experiments on seven representative LLMs (including T5-Small, RoBERTa, BioMedBERT, Qwen2.5-Coder-0.5B/7B, MiniMind, and Opt) demonstrate that RetryTrigger reduces SDC rates by up to 95.33% (on average 92.97%) and achieves a minimal performance overhead of 2.4012% (on average 4.1167%), offering a superior balance between reliability and efficiency compared to state-of-the-art solutions.

1. Introduction

Large language models (LLMs), primarily built upon Transformer-based architectures, excel at modeling long-range dependencies through self-attention mechanisms. Leveraging these capabilities, LLMs have achieved remarkable success across a diverse spectrum of natural language processing (NLP) tasks, ranging from sentiment analysis and text generation to language translation and code synthesis (Radford et al., 2019; Zhang et al., 2023; Jiang et al., 2024). The massive-scale pre-training of LLMs enables them to rapidly learn complex linguistic patterns, thereby facilitating their widespread adoption in both academic research and industrial applications.

However, the rapid growth in model size, coupled with prolonged computational workloads, has made LLMs increasingly vulnerable to hardware transient faults. Such faults can be triggered by cosmic ray strikes, voltage fluctuations, or thermal instability, and have become one of the predominant threats to system reliability (Baumann, 2005). Unlike permanent hardware defects, transient faults cause temporary bit upsets that may corrupt intermediate computations and even lead to system crashes. When occurring within LLM architectures during the inference phase, these faults can result in silent data corruptions (SDCs), thereby degrading output quality and diminishing user experience. The impact is even more critical in safety-sensitive domains, such as medical diagnosis (Zhou et al., 2025) and legal document analysis (Battaglini-Fischer et al., 2025), which necessitate non-negotiable resilience against hardware transient faults.

Fault-tolerance research for LLMs spans the training and inference phases. Training-stage defenses often include system-level solutions such as optimized checkpointing and elastic scheduling (Li et al., 2025; Wan et al., 2025), as well as algorithm-based fault tolerance (ABFT) and fault-aware training (Cavagnero et al., 2022; Liang et al., 2025). While effective for long-running distributed training, these methods are rarely tailored for the strict latency constraints of inference. Consequently, inference-stage protections have received increasing attention to mitigate SDCs caused by fault amplification. These works are often classified into the following three categories: (1) Boundary setting policy. Activation clipping and range restriction (Hoang et al., 2019; Chen ZT et al., 2021; Mousavi et al., 2024; Roquet et al., 2024) limit intermediate activation magnitudes to suppress fault propagation. Although effective for small deep neural networks (DNNs), in LLMs, they often require costly offline profiling (Sun et al., 2025) and still leave partial coverage gaps. An illustrative extension of activation clipping for LLMs is first-token-inspired online fault tolerance (FT2) (Sun et al., 2025), which achieves online fault tolerance via a token-boundary setting mechanism. These bounds are derived from the first token's statistics and applied to subsequent tokens.