What is pretraining? Learning broad capabilities before later adaptation
Understand pretraining through continuation learning, including its relationship to chat behavior, continued pretraining, fine-tuning, and data freshness.
Understand the concept
Pretraining is typically an initial training stage on a large dataset that learns reusable representations and capabilities. Language models often learn through objectives such as next-token prediction, though objectives vary. A pretrained model may need later adaptation for dialogue and does not guarantee complete, accurate, or continuously updated knowledge.
Training objectives differ from expected user behavior
Predicting text continuations teaches continuation patterns. Explaining a table for a user is not exactly the same format. Later training can improve instruction following, but all stages should not be described as a single chat-learning process.
Foundation-model research emphasizes adaptation to multiple tasks after broad training. This does not establish validation for every task or unbiased data. Check disclosed objectives and data scope; keep undisclosed details unknown rather than inferring them from a model name.
Distinguish continued pretraining from fine-tuning
Continued pretraining extends a similar objective on an existing model, for example with domain text. Supervised fine-tuning more often uses task inputs and target answers to adjust behavior. Terminology can overlap; inspect data formats and objectives rather than project titles.
Calling a pretrained model is inference; supplying a document does not automatically pretrain it. Conversation context and model parameters store information differently. Whether a service separately trains on conversations depends on its policy, not the behavior of one response.
Check data timing and task scope when choosing a model
The world continues changing after pretraining. Newly released features may require fresh evidence or retrieval. A specific answer does not establish current knowledge; verify dates and facts against original announcements.
Evaluate intended tasks and check factual errors separately from instruction following. Before training a model, assess whether adapting an existing one meets the need and account for data and compute. This page explains stages without reporting a pretraining experiment or a universal data threshold.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.