Byte Pair Encoding
A subword tokenization method that repeatedly merges frequent adjacent symbol pairs to build a compact vocabulary from raw text units.
Related Concepts
- Tokenization
- Vocabulary
- Subword
- Context Window
A subword tokenization method that repeatedly merges frequent adjacent symbol pairs to build a compact vocabulary from raw text units.