Training

Pretraining

Pretraining is the first phase of LLM training where models learn from enormous corpora of diverse text. The objective is typically next-token prediction (language modeling). During pretraining, the model learns grammar, facts, reasoning, and coding from data. Pretraining is computationally expensive (billions of dollars for state-of-the-art models). Quality and diversity of pretraining data heavily influence model quality. Most open-source models release pretrained versions that are further fine-tuned. Pretraining creates general-purpose models; specialization comes through fine-tuning.