← NLP Fundamentals
Free
Tokenization & Text Representations
From words to subwords — tradeoffs that affect every NLP system.
Tokenization & Text Representations
Tokenization decides the atomic units your model sees.
Options
- Whitespace / word tokens: simple, brittle on morphology
- Subwords (BPE, WordPiece, Unigram): handle rare words
- Bytes / characters: flexible, longer sequences
Embeddings
Map tokens to dense vectors. Static embeddings (word2vec/GloVe) vs contextual (transformers).