Architecture

Cross-Entropy Loss

Cross-entropy loss is the standard objective for training LLMs. It measures how well the model's predicted probability distribution matches the true next token. Lower loss means better predictions. For language modeling, cross-entropy loss is applied at each token position. The model learns to maximize probability of correct continuations. Cross-entropy is differentiable, enabling backpropagation. Token-level cross-entropy is summed or averaged to get the batch loss. Understanding cross-entropy is fundamental to understanding LLM training.