Architecture

Perplexity

Perplexity is the exponential of average cross-entropy loss. It measures how 'surprised' the model is by test data. Lower perplexity indicates better predictions. Perplexity of 1 means perfect predictions (P=1 for correct tokens). Perplexity is intuitive: a perplexity of 50 means the model is effectively equally uncertain among 50 equally likely tokens. It's widely used as an evaluation metric for language models. Perplexity is comparable across datasets only if they use the same tokenizer. Recent models have dramatically lower perplexities than earlier ones.