SentencePiece Tokenization, Normalization and Vocabulary Boundaries

A single Turkish word can carry grammatical information that English may express with several words. If the tokenizer splits that structure poorly, the consequence is not only more tokens; the statistical units learned by the model change as well.

In speech and text-processing work I treat tokenization as part of the model contract, not disposable preprocessing.

SentencePiece model

SentencePiece learns subword units from raw text and supports methods such as BPE and unigram models. Vocabulary size is chosen before training, so the corpus must be represented through that finite symbol inventory.

Normalization comes first

The pipeline is:

raw text
→ normalization
→ subword segmentation
→ token ids

If training and inference use different normalization rules, visually similar text may produce different tokens. Normalization configuration therefore has to be versioned with the tokenizer.

Vocabulary size is a capacity decision

A larger vocabulary can reduce tokens per sentence but increases embedding/output vocabulary and creates rarer units. A smaller vocabulary reuses pieces more aggressively but lengthens sequences.

I compare vocabulary choices through token counts, long-tail words, domain terminology and downstream model cost rather than vocabulary size alone.

Turkish needs different measurements

Whitespace word count is a weak proxy for an agglutinative language. I prefer:

tokens / character
tokens / sentence
p95 token count

and inspect suffix-rich words, abbreviations, numbers and technical terminology.

Changing tokenizer changes the model contract

A token id has meaning only under the vocabulary used during model training. Replacing a .model file or normalization rule while keeping model weights fixed is therefore not a transparent deployment change.

Subword regularization

SentencePiece can sample alternative segmentations during training. This may improve robustness for some models, but it is a training choice that has to be evaluated against the downstream task.

End-to-end performance

Tokenizer throughput alone can mislead. I also measure produced token count, allocation, long-sentence tails and end-to-end inference latency. A fast tokenizer that emits longer sequences can increase total model cost.

The main point is that tokenization defines the discrete alphabet through which the model receives language.

References

• Kudo and Richardson, SentencePiece
• Google SentencePiece repository, normalization documentation and performance benchmark