Turkish Deasciification in C#

Restoring Turkish characters is not a mechanical conversion that replaces every occurrence of c with ç. In ASCII strings such as cocuk, cagri, saglik, and olcu, whether a character should be converted to its Turkish counterpart depends on the local context in which it appears. The same ASCII character remains unchanged in some words and is converted in others.

The C# implementation I developed was based on the decision-list approach used by Deniz Yüret in Emacs Turkish Mode. Yüret's work was itself influenced by the Turkish deasciification problem discussed in Gökhan Tür's 2000 doctoral dissertation. Tür treated restoration of the letters ü, ö, ç, ş, ğ, and ı in Turkish text written with ASCII characters as a statistical natural-language-processing task.

Yüret's implementation used decision lists learned from approximately one million words of Turkish news text. The model was produced with the Greedy Prepend Algorithm developed by Yüret and Michael de la Maza. The Emacs extension corrected the last word according to context when the user pressed space, tab, or newline.

When I transferred this algorithm to my own language-processing layer, I did more than translate Lisp syntax into C#. I rebuilt the character tables, model-loading format, context generation, and decision priority. My main goal was to adapt the runtime data structures to .NET while preserving the sequential decision behavior of the original algorithm.

A binary classification problem

The algorithm operates on six character families:

c <-> ç g <-> ğ i <-> ı o <-> ö s <-> ş u <-> ü

Uppercase characters are part of the same families. Every candidate position has two possible decisions:

0: retain the ASCII form 1: use the Turkish counterpart

The algorithm does not add or remove letters and does not change word boundaries. It performs no spell checking, stemming, or morphological analysis. It decides only whether the current character should be converted to the other form in the same family.

This distinction defines the boundary of the method. The input cocuk can become çocuk. A word with a missing letter, an incorrectly written suffix, or two words that should be joined is not a problem handled by this algorithm.

A separate pattern dictionary is kept for every character family:

c -> decision list g -> decision list i -> decision list o -> decision list s -> decision list u -> decision list

X in a pattern marks the position of the character being classified:

bu aXa na Xog birinX Xocuk

A pattern does not have to be a complete word. It can be a short segment taken from the left and right of the target. A fragment of the preceding word or a boundary character can also participate in the pattern.

A decision list is not a sum of probability scores. One older explanation in the shared source states that log-odds values of subpatterns are added, but the actual code does not operate this way. It selects the single highest-priority rule among the matching rules in the decision list.

This behavior is consistent with the classical definition of a decision list. Decisions form an ordered set of rules, and a new example receives the class of the first matching rule. The Greedy Prepend Algorithm was developed to produce lists of this kind.

In-memory representation of the model

In the C# port, patterns are stored in the following logical structure:

Dictionary<char, Dictionary<string, short>>

The outer dictionary identifies the character family, while the inner dictionary stores context patterns. The short value carries two pieces of information at once:

absolute value -> rule priority sign -> classification result

A rule with a smaller absolute value has higher priority. A positive value represents the Turkish form, while a negative value represents the ASCII form. Zero also belongs to the ASCII class and can be used as the highest-priority rule.

This encoding eliminates the need for a separate rule object. Pattern, order, and class information are sufficient for every dictionary entry. After model loading completes, the inference path consists only of dictionary reads.

The initial value is constructed with:

rank = pattern.Count << 1

The aim is to obtain a neutral value larger than all absolute rule ranks. In the historical model, the rule count for every character family remained below the short limit, so this representation was sufficient. If the model is retrained on a larger corpus, short overflow must be checked separately. When one family contains more than 16,383 rules, multiplying the count by two exceeds the signed 16-bit range.

Four helper tables are constructed for character operations:

A table that reduces a Turkish character to its ASCII counterpart

A table that converts a character to lowercase ASCII

A table that converts an already corrected character into a context marker

A table that switches between ASCII and Turkish forms

Building these tables once at startup avoids culture resolution or long conditional chains for every character. The model is also created during static initialization. As long as it is not modified later, the same dictionaries can be shared among all calls.

Asymmetric context

The most distinctive part of the algorithm is that it does not process the left and right context of the target character in the same way.

Text is scanned from left to right. Characters to the left of the target have already been classified. Their Turkish forms can be transferred as features to subsequent decisions. Characters on the right have not yet been processed and are therefore reduced only to lowercase ASCII.

Previously Turkishized characters in the left context are encoded with uppercase base letters:

ç -> C ğ -> G ı -> I ö -> O ş -> S ü -> U

Here, uppercase does not necessarily indicate the original case of the character. It is a feature marker that indicates that a Turkish character occurred at the previous position.

This structure makes the algorithm more than a collection of independent character classifiers. A previous prediction becomes an input to the next prediction:

y_i = f(x_(i-K), ..., y_(i-1), x_i, x_(i+1), ..., x_(i+K))

x_i is the original character, and y_i is the corrected character. Predicted y values are used on the left, while unprocessed x values are used on the right.

Processing direction is therefore part of the algorithmic contract. Scanning right to left, or processing all characters independently and combining them afterward, does not produce the same result. An incorrect previous decision can also be propagated into later contexts. The sequential model gains information from this mechanism while accepting the risk of error propagation.

Window and pattern search

The original Emacs approach uses ten context characters on each side of the target. In the C# port I examined, the constants are:

Current = 20 Size = 41

The target is kept at index 20. The theoretical window therefore consists of 20 characters on the left, X, and 20 characters on the right.

After the context is built, every contiguous substring containing X is tested. If the one-sided width is K, both the left and right length range from 0 to K:

candidate pattern count = (K + 1)^2

For this port:

K = 20 (K + 1)^2 = 441

At most 441 patterns are looked up in the dictionary for one candidate character. When several patterns match, the shortest or longest pattern is not selected. The rule with the smallest absolute rank wins.

Context collection also differs by direction. The right side is converted to lowercase ASCII, and scanning stops after the first unrecognized boundary character. The left side moves backward and represents previously corrected characters with markers. This asymmetry is consistent with the original usage model, which corrected the most recently typed word by using the context of the preceding word.

The Turkish i family

The characters i, ı, I, and İ cannot be left to general conversion logic:

i -> İ ı -> I

Lowercase ASCII i can represent either i or ı in Turkish text. Uppercase ASCII I can represent either I or İ. For lowercase, selecting the Turkish form means converting i -> ı. For uppercase, the same class decision requires preserving I.

The code handles this difference explicitly. When the model selects the dotless family:

i -> ı I -> I

When the model selects the dotted family:

i -> i I -> İ

The decision should therefore not be interpreted simply as "add a diacritic." In the i family, the positive class represents the dotless form. The current case of the character also determines the conversion direction.

Standard ToUpperInvariant and ToLowerInvariant calls do not establish this relationship correctly. Preserving Turkish-specific mappings in separate tables was necessary during the port for this reason.

Actual execution cost

Let text length be N, candidate-character count be A, and one-sided context width be K. The overall text scan costs Theta(N). Context generation costs Theta(K) for each candidate.

If only the number of hash lookups is considered, pattern search appears to cost:

Theta(A * K^2)

The current code, however, creates a new string for every lookup:

new string(temp, start, length)

Creating each pattern and calculating its hash are proportional to pattern length. Summed over all left and right lengths, the total number of generated characters reaches:

(K + 1)^3

For K = 20, one candidate position can create:

441 temporary strings 9,261 characters of total temporary content

The generalized runtime is therefore described more accurately as:

Theta(N + A * K^3)

When K is treated as a constant, total time remains Theta(N). In a high-traffic service, however, the allocation and hashing cost hidden by asymptotic notation becomes significant.

Persistent memory cost is approximately:

Theta(R * L)

for rule count R and average pattern length L. Because the input text is copied into a char[] on every call, additional working space is Theta(N). The context buffer has fixed size.

If I optimized the structure again, I would remove temporary string creation. A custom span key, collision-checked rolling hash, or pattern trie could be used. Every slice could then be represented by offset and length over the same 41-character buffer instead of allocating a new string.

Optimization must not change decision behavior. Rule priority is independent of pattern length. Selecting the first pattern found in a trie is insufficient. The smallest absolute rank among all matching rules must still be preserved.

Production boundaries

A strength of this implementation is that inference requires no external dictionary or morphological analyzer. The model resides in local memory. No network access is needed, and user text does not have to be sent to another system. Because the decision table is fixed, the same input produces the same output.

The following elements must remain fixed for decisions to be reproducible:

Character-processing direction

Unicode normalization

Context width

Pattern order

Separator behavior

Turkish upper-lowercase tables

Preservation of the first rule for duplicate patterns

Special handling of I, İ, i, and ı

Training the model on news text creates domain dependence. Proper names, technical terms, foreign words, URLs, email addresses, and source-code fragments can be converted incorrectly. These regions should be marked before the core algorithm or passed through a separate protection layer.

The shared review version deliberately omits the complete decision lists and direct character lookup table. It is therefore not, in its current form, a complete distribution package that can be compiled and reproduce historical accuracy values. The algorithmic core, data structures, and decision order can nevertheless be examined clearly.

The main result I obtained while developing this port was that accuracy in even a small natural-language-processing algorithm depends on more than the model table. The direction in which context is collected, the encoding of previous decisions, rule ordering, and handling of the four i characters are all parts of the model.

The method succeeds not because it uses a complex neural network, but because it retains the right features for the problem in a compact decision list. Training can be offline and expensive. Runtime remains small, local, and deterministic. In this respect, the algorithm is a useful example of how interpretable models and low-cost production code can be combined in the history of natural-language processing.