FastCheck: fast checkpointing and recovery for DNN training via parallel transmission and compression

Abstract

Training large-scale deep neural networks (DNNs) is prone to software and hardware failures, with critical failures often requiring full-machine reboots that substantially prolong training. Existing checkpoint–recovery solutions either cannot tolerate such critical failures or suffer from slow checkpointing and recovery due to constrained input/output bandwidth. In this paper, we propose FastCheck, a checkpoint–recovery framework that accelerates checkpointing and recovery through parallel transmission and tailored compression. First, FastCheck partitions checkpoints into shards and leverages multiple nodes for parallel checkpointing and recovery. Second, it further reduces checkpoint size and overhead with delta compression for weights and index compression for momentum. Third, FastCheck employs lightweight and consistent health status maintenance that accurately tracks node health, preventing checkpoint transmission to failed nodes. We implement FastCheck in PyTorch and evaluate it on multiple DNN models against two baselines. Experimental results show that FastCheck reduces the checkpointing time by up to 78.42% and the recovery time by up to 77.41%, while consistently improving efficiency across different training stages.

1. Introduction

In recent years, with the rapid evolution of deep neural networks (DNNs), especially the pretrained large-scale language models (commonly known as LLMs), the number of model parameters has continuously increased. For instance, PaLM (Chowdhery et al., 2023) has 560 billion parameters, which is 360 times the size of GPT-2 (Radford et al., 2019). Such an immense number of parameters entails the use of massive machines during DNN training. A large-scale training cluster induces frequent hardware and software failures. For instance, training a trillion-parameter model on the Sunway platform with 100,000 graphics processing units (GPUs) results in hardware or software failures once an hour on average. In Microsoft’s GPU training clusters, the mean failure duration over a 2-month period reached 45 minutes (Jeon et al., 2019). The 54-d training of LLaMA 3.1 (Meta AI LLaMA Team, 2024) on 16,000 GPUs encountered 419 failures, with failure occurring every 3 h (Lian et al., 2025).

To achieve quick training resumption instead of restarting from the beginning, checkpointing is universally adopted in DNN training. This involves periodically saving both model weights and optimizer momentum to persistent storage. When failures occur, the checkpoints are read to continue training, which reduces the recovery time. However, some naive checkpointing solutions are ineffective, and high failure frequency leads to substantial recovery overhead and low efficiency; for instance, the OPT-175B model (Zhang S et al., 2022) suffered an extra overhead of 178,000 GPU hours due to training failures. Existing studies indicate that checkpointing in DNN models significantly hinders training progress: especially during failure recovery, it substantially incurs wasted training time, which can slow the process by up to 43% (Maeng et al., 2021). This inefficiency occurs due to the time overhead during recovery and checkpointing. Current solutions struggle to (1) effectively reduce the training cost associated with failures and (2) reduce the recovery overhead. As model sizes grow, the storage space and checkpointing time become substantial burdens.

Taking a DNN model with a trillion parameters as an example, each checkpoint needs to store approximately 12 TB of model parameters. The massive size of the checkpoint significantly increases checkpointing time. CheckFreq (Mohan et al., 2021) adaptively adjusts checkpoint frequency and implements asynchronous checkpointing; however, it still performs poorly due to the huge size of the checkpoint and the low bandwidth of persistent storage. Moreover, traditional compression techniques such as GZIP (Deutsch, 1996) are ineffective for checkpoints, due to the significant randomness of the ending mantissa bits (Hu Z et al., 2020).