Tokenization
The conversion of input text or other content into discrete model units whose integer identifiers are processed by a neural network.
Related Concepts
- Byte Pair Encoding
- Context Window
- Embedding
- Language Model
The conversion of input text or other content into discrete model units whose integer identifiers are processed by a neural network.