What is a model checkpoint? Saving, resuming, and releasing training states
Understand model weights and optimizer states through interrupted training, including the files needed for inference versus resuming a run.
Understand the concept
A model checkpoint is a saved training-state snapshot, often containing weights and possibly optimizer state and progress. It can support resuming or version selection, but contents depend on configuration. Weights alone may not restore an interrupted run fully, and a file named best is not necessarily best for every task.
Weights differ from a complete training state
PyTorch supports saving a model state_dict and describes broader checkpoints including optimizer state. Weights load learned parameters; broader states support more complete recovery. Required fields depend on the training script rather than the filename extension.
If a long run stops halfway, loading weights with a newly initialized optimizer can change subsequent updates. Data order and random states may also matter. Saving a file does not establish that recovery has been tested.
Resuming training and inference require different files
Text generation usually also requires matching configuration and tokenizer rather than handing weights to an arbitrary runtime. Resuming needs the states expected by the framework. Transformers supports checkpoint resumption, but save options determine what is available.
Changing architecture or vocabulary can prevent direct loading. Inspect missing keys and shape mismatches instead of suppressing every warning merely to start the program. After loading, run a minimal input and inspect its output.
Keep evaluation and recovery records for each version
Intermediate checkpoints need not improve monotonically. Identify the evaluation set and metric used to select a best checkpoint and keep independent task checks. Minimum validation loss does not establish best factual accuracy or deployment latency.
A recovery check reads files, loads matching configuration, and confirms continuation at the expected step. This page explains the process without running a recovery experiment. Back up version identifiers and required configuration so identically named weights remain traceable to their training conditions.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.