# SentencePiece Tokenization, Normalization and Vocabulary Boundaries

> SentencePiece applies normalization before subword segmentation; in Turkish, vocabulary size and morphology directly affect token length and model cost.

- Author: Muhammet Ali Köker
- Language: en
- Canonical: https://alikoker.com.tr/en/sentencepiece-tokenization-normalization-and-vocabulary-boundaries
- Translation: https://alikoker.com.tr/sentencepiece-tokenization-normalizasyon-ve-vocabulary
- Published: 2018-11-01T12:00:00+03:00
- Modified: 2026-08-30T14:55:00+03:00
- Type: article

A single Turkish word can carry grammatical information that English may express with several words. If the tokenizer splits that structure poorly, the consequence is not only more tokens; the statistical units learned by the model change as well.

In speech and text-processing work I treat tokenization as part of the model contract, not disposable preprocessing.

## SentencePiece model

SentencePiece learns subword units from raw text and supports methods such as [BPE](/en/wiki/byte-pair-encoding) and unigram models. Vocabulary size is chosen before training, so the corpus must be represented through that finite symbol inventory.

## Normalization comes first

The pipeline is:

```text
raw text
→ normalization
→ subword segmentation
→ token ids
```

If training and inference use different normalization rules, visually similar text may produce different tokens. Normalization configuration therefore has to be versioned with the tokenizer.

## Vocabulary size is a capacity decision

A larger vocabulary can reduce tokens per sentence but increases embedding/output vocabulary and creates rarer units. A smaller vocabulary reuses pieces more aggressively but lengthens sequences.

I compare vocabulary choices through token counts, long-tail words, domain terminology and downstream model cost rather than vocabulary size alone.

## Turkish needs different measurements

Whitespace word count is a weak proxy for an agglutinative language. I prefer:

```text
tokens / character
tokens / sentence
p95 token count
```

and inspect suffix-rich words, abbreviations, numbers and technical terminology.

## Changing tokenizer changes the model contract

A token id has meaning only under the vocabulary used during model training. Replacing a `.model` file or normalization rule while keeping model weights fixed is therefore not a transparent deployment change.

## Subword regularization

SentencePiece can sample alternative segmentations during training. This may improve robustness for some models, but it is a training choice that has to be evaluated against the downstream task.

## End-to-end performance

Tokenizer throughput alone can mislead. I also measure produced token count, allocation, long-sentence tails and end-to-end inference latency. A fast tokenizer that emits longer sequences can increase total model cost.

The main point is that [tokenization](/en/wiki/tokenization) defines the discrete alphabet through which the model receives language.

## References

- Kudo and Richardson, *SentencePiece*
- Google SentencePiece repository, normalization documentation and performance benchmark

## Cite This Work

Köker, M. A. (2018). SentencePiece Tokenization, Normalization and Vocabulary Boundaries. alikoker.com.tr. https://alikoker.com.tr/en/sentencepiece-tokenization-normalization-and-vocabulary-boundaries

- BibTeX: https://alikoker.com.tr/en/sentencepiece-tokenization-normalization-and-vocabulary-boundaries.bib
- RIS: https://alikoker.com.tr/en/sentencepiece-tokenization-normalization-and-vocabulary-boundaries.ris
- CSL-JSON: https://alikoker.com.tr/en/sentencepiece-tokenization-normalization-and-vocabulary-boundaries.csl.json