A Detailed Explanation of Distributed Training Fault Tolerance: Checkpoints, Resume Training, and Fault Recovery

Distributed training fault tolerance is a mechanism for recovering as much progress as possible after training is interrupted. Through checkpoints, resume training, and fault recovery, it reduces repeated computation and helps clarify the boundaries of code adaptation, weight verification, and job restart.

What Is Distributed Training Fault Tolerance?

Distributed training fault tolerance refers to restoring a training task as much as possible from the saved training state after an interruption or failure occurs, rather than restarting training completely from scratch. Its goal is not to eliminate all failures, but to reduce the loss of training progress after a failure occurs so that the training business can continue moving forward.

In large model training or long-cycle training, the training process often takes a long time. If one failure causes all progress to be lost, it will lead to significant repeated computation. Resume training from a breakpoint is a key capability in distributed training fault tolerance: after training is interrupted, training continues from the latest saved intermediate weights, thereby reducing repeated computation and improving training efficiency.

A checkpoint can be understood as a “periodic archive” during the training process. After a failure occurs, the training system does not recover out of thin air. Instead, it relies on these saved states to connect training back as close as possible to where it was before the interruption.

Key point: Distributed training fault tolerance focuses on “how to recover the training business after a failure and reduce losses”; resume training from a breakpoint focuses on “how to continue training from a saved checkpoint or intermediate weights.”

What Is the Relationship Among Checkpoints, Resume Training, and Fault Recovery?

Checkpoints, resume training from a breakpoint, and fault recovery often appear together, but the problems they solve are not exactly the same. Understanding the relationship among the three helps determine to what extent a training task can actually be recovered after a failure.

ConceptMain functionPosition in the recovery process
CheckpointSaves intermediate states that can be used to recover training; in the context of this article, it mainly refers to intermediate weights or checkpoints saved during trainingProvides a loadable recovery point for subsequent resume training
Resume training from a breakpointContinues training from a saved checkpoint or intermediate weightsAfter the task runs again, reconnects training to the saved state
Fault recovery mechanismEnables the training business to run again after a failureCommon methods include restarting the process or rebuilding the job

When training starts for the first time, if the checkpoint directory is empty, training will randomly initialize weights according to the normal process. Later, if a checkpoint has already been saved, training can be restored from the checkpoint saved in the directory when resuming training.

It is important to note that fault recovery does not mean that all training progress is automatically retained. Fault recovery can enable the training business to run again by restarting the process or rebuilding the job, but whether it can recover to a reasonable progress point still depends on whether an available checkpoint exists and whether the training code has already been adapted for resume training.

Basic Flow from Initial Training to Resume Training

Resume training from a breakpoint is usually not an isolated action, but a complete process that runs through training startup, state saving, task interruption, and recovery loading.

Starting Training for the First Time

When training starts for the first time, if the checkpoint directory is empty, training will normally initialize weights randomly. This shows that an empty directory does not prevent initial training from starting, but it also means there is currently no historical training state available for recovery.

Saving Intermediate States During Training

During training, intermediate weights or checkpoints are saved. This saving action determines the latest saved state to which training can be restored after a failure.

If the task is interrupted after saving, subsequent resume training can restore training from the saved checkpoint. This can reduce repeated computation caused by training from scratch.

Re-entering Resume Training After Interruption

After training is interrupted, the recovery process needs to read the existing checkpoint and allow training to continue from that state. At this point, the most critical check is: whether the loaded weights truly correspond to the weights at the time when training was interrupted.

If the wrong checkpoint is loaded, training can still continue running, but the continued state may not meet expectations, which may then affect training progress judgment and subsequent result analysis.

Key point: It is normal for the checkpoint directory to be empty during initial training. In a resume-training scenario, whether the checkpoint exists, whether it is correctly loaded, and whether it corresponds to the state at the time of interruption are the key points.

How Does Training Recover After Fault Recovery Is Triggered?

The role of the fault recovery mechanism is to allow the training business to run again after a training failure occurs. Common recovery paths mainly include restarting the process or rebuilding the job.

Restarting the Process or Rebuilding the Job

When the training process exits abnormally or the job cannot continue execution, the fault recovery mechanism may recover the training business by restarting the process or rebuilding the job. This action solves the problem of “getting the training program running again.”

However, after the training program runs again, it does not naturally know where to continue from. To recover training progress, the training code also needs to have resume-training logic, be able to read the specified checkpoint after startup, and enter the resume-training process.

Working Together with Resume Training

A more complete recovery process can usually be split into two layers:

LayerProblem solvedDependencies
Fault recoveryEnables the training process or training job to run againThe platform or scheduling layer has recovery capabilities such as restart and rebuild
Resume training from a breakpointContinues training from a saved stateAn available checkpoint exists, and the training code has been adapted for resume-training logic

Therefore, before enabling fault recovery, you should confirm that the code has been adapted for resume training. Otherwise, even if the training job is pulled up again, it may still be unable to correctly load the checkpoint, causing training progress to be lost and computing power to be wasted.

Graceful Fault Tolerance in Specific Hardware Scenarios

In some supported Ascend NPU scenarios, graceful fault tolerance can first attempt to recover the faulty chip when a chip-level failure occurs, and implement process-level restart of the training task.

This type of capability is a recovery method under specific environments, and whether it is available depends on the actual hardware, platform, and support scope. Even with process-level restart capability, whether training can continue to a reasonable state still needs to depend on checkpoints and resume-training adaptation.

What Training Benefits Can a Fault Tolerance Mechanism Bring?

The direct value of distributed training fault tolerance is reducing training losses caused by failures. This is especially important for training tasks that take a long time and require high resource investment.

Training problemRole of the fault tolerance mechanismExpected value
Starting from scratch after training is interruptedContinues training from the last saved intermediate weightsReduces loss of training progress
Completed computation is executed repeatedlyUses checkpoints to restore to a saved stateAvoids repeated computation
Insufficient stability for long-cycle tasksRuns the training business again after a failure through the recovery mechanismImproves training efficiency and resource utilization stability

Resume training from a breakpoint does not make failures completely impact-free. Instead, it limits the impact of failures to a recoverable range. The better checkpoints can meet recovery needs and the more correctly the training code can load the recovery state, the more easily the benefits of the fault tolerance mechanism can be reflected.

What Are the Key Boundaries for Code Adaptation and Weight Verification?

Distributed training fault tolerance cannot be judged only by whether the platform provides fault recovery capability. It also depends on whether the training code can correctly handle recovery logic. Otherwise, restarting the process or rebuilding the job can only make the task run again, but cannot guarantee that training progress is correctly reconnected.

Code Needs to Be Adapted for Resume Training

Before enabling fault tolerance recovery, you should confirm that the training code has already been adapted for resume training. In other words, after the code restarts, it needs to be able to identify the recovery scenario, read the specified checkpoint, and continue training from the saved state.

If the code has not been adapted, two types of problems may occur:

  • After the job restarts, weights are randomly initialized again, causing historical training progress to be lost;
  • The job loads a file, but does not enter the resume-training process as expected, resulting in wasted computing power.

Weight Verification Is Required Before Recovery

Resume training from a breakpoint requires checking whether the loaded weights are the weights corresponding to the time when training was interrupted. Weight verification is not a formalized step, but a key boundary for recovery correctness.

It is recommended to confirm at least the following issues:

  • Whether the currently loaded checkpoint comes from the target training task;
  • Whether the loaded weights correspond to the expected recovery point before and after interruption;
  • Whether the resume-training entry point actually reads this checkpoint rather than reinitializing;
  • Whether the recovered training state is consistent with the expected recovery process.

Fault Recovery Does Not Mean Recovering to the Exact Moment of Interruption

A common misconception is that as long as fault recovery is enabled, no training progress will be lost. In fact, fault recovery is usually responsible for making the training business run again, while the progress to which it can be recovered depends on checkpoints, code logic, and the recovery method.

If the most recent available checkpoint is far from the point where the failure occurred, training may still need to recompute this interval. If the checkpoint is unavailable or loaded incorrectly, the recovery effect will also be affected.

Practical Checklist for Training Job Recovery

When designing or troubleshooting a distributed training fault tolerance process, you can perform checks around checkpoints, the recovery entry point, the weight correspondence relationship, and the recovery strategy.

Check itemWhat needs to be confirmedRisk reminder
checkpoint directoryIs this initial training or resume training? Is it expected that the directory is empty?Do not mistake an empty directory for a recoverable state
Recovery entry pointAfter restarting the process or rebuilding the job, will the code read the specified checkpoint?A job resuming running does not mean it has entered resume training
Weight correspondence relationshipDo the loaded weights correspond to the state when training was interrupted?Loading the wrong checkpoint will cause the recovery state to be inconsistent with expectations
Code adaptationHas the training code been adapted for resume training from a breakpoint?Lack of adaptation may lose training progress and waste computing power
Recovery strategyDoes the current environment use process restart, job rebuilding, or support finer-grained fault tolerance recovery?Different recovery paths depend on the platform and scenario

The recovery process can be simplified into the following sequence of judgments:

  1. First confirm whether the training business can run again by restarting the process or rebuilding the job.
  2. Then confirm whether the training code will enter resume-training logic.
  3. Next confirm whether the checkpoint exists and can be read.
  4. Finally confirm whether the loaded weights correspond to the expected state at the time of interruption.

Only when these conditions work together can distributed training fault tolerance truly reduce progress loss, rather than merely restarting a failed job once.