Architecture

Vocabulary

The vocabulary is the complete set of tokens the tokenizer can produce. It's learned during tokenizer training on the pretraining corpus. Vocabulary size typically ranges from 30K to 50K tokens for LLMs. Larger vocabularies can represent rare words directly but increase embedding table size. Smaller vocabularies require more tokens to represent text but are more efficient. The vocabulary is fixed at inference—the model cannot generate tokens outside its vocabulary. Vocabulary misalignment (using wrong tokenizer) is a common source of errors.