Table-Based ASCII Conversion for Turkish Text

Reducing Turkish text to the ASCII character set is a broader problem than replacing a few letters with their counterparts. Case behavior, separator handling, Unicode representation, memory allocation, and deterministic output for the same input must be addressed together. In the C# code I shared, the problem is solved with three constant conversion tables that use the character code directly as an index.

I developed this helper layer before AI-assisted code-generation tools became widespread. My objective was not to build a general and extensible transliteration framework. I wanted to produce a simple ASCII counterpart for Turkish text at low processing cost for use in search, naming, and natural-language-processing workflows. The code structure reflects that choice directly. It uses no regular expression, dictionary, chained Replace, culture-dependent case conversion, or runtime-created mapping object.

Direct-addressed conversion

The core of the algorithm contains three character tables:

An ASCII table that preserves case where possible

A table that reduces all supported letters to lowercase ASCII

A table that reduces all supported letters to uppercase ASCII

The index in each table corresponds to the input char value. The character stored at that index is the conversion result. The operation can be written mathematically as:

y = T[x]

Here, x is the numeric UTF-16 code unit of the input character, T is the conversion table, and y is the output character.

This structure does not use a lookup dictionary. It calculates no hash, compares no keys, and executes no long conditional chain that branches on character type. The char value is used directly as an array index. Time complexity for one character is therefore Theta(1).

At the processor level, table access is still subject to bounds checking and cache behavior. Even so, algorithmic cost is independent of the input character. A Turkish character, ASCII letter, digit, or punctuation mark is processed with the same number of table accesses.

Using three separate tables avoids leaving case conversion as a second step after ASCII reduction. Instead of first converting a Turkish character to its ASCII counterpart and then calling ToLower, the lowercase result is obtained directly from the lowercase table. No second conversion or culture resolution is required for each character.

This decision matters for the Turkish relationship among i, I, ı, and İ. Turkish case mapping is not identical to general culture-independent Latin mapping. The Unicode standard also defines dotted and dotless i mappings in Turkish as locale-dependent special cases. A constant table encodes the intended ASCII result deterministically and independently of culture settings.

If the conversion table is considered a function, the same input always produces the same output:

T(x) = y

The function is not one-to-one. Different characters such as c and ç, g and ğ, or i and ı can be reduced to the same ASCII value. A general inverse function therefore does not exist:

T(x1) = T(x2), x1 != x2

This is not a defect. It is a natural consequence of ASCII reduction. It must nevertheless be clear that the generated value cannot reconstruct the original text.

Cost of string conversion

The three primary string methods follow the same operation sequence. The input is copied into a char[]. The array is transformed in place in one pass. A new string is created from the array at the end.

For text of length n, operation count is linear:

T(n) = a.n + b T(n) = Theta(n)

Every character is read once and written once. There is no nested loop, backward scan, or search that restarts from the beginning of the text.

Auxiliary space complexity is also Theta(n), because of the working char[] and the new result string. Since .NET strings are immutable, a new result object is required whenever the content differs. ToCharArray() also copies the UTF-16 code units of the source string into a new character array.

An important property is that allocation count does not depend on text length. No separate object is created for every character. There is no StringBuilder capacity growth, intermediate string, or chained Replace result. One working array and one result string are sufficient on the normal path.

With chained character replacement, the text can be scanned again for every letter group to convert. For k separate replacement operations, cost is approximately:

T(n, k) = Theta(k.n)

For Turkish, k is small and constant, so the asymptotic result still appears linear. The same character sequence is nevertheless read several times, and each replacement can allocate a new string. The table-driven version scans the text once regardless of the size of the conversion set.

If a table covers the complete UTF-16 Basic Multilingual Plane, each table contains 65,536 char values. Because a char is 16 bits, the theoretical lower bound for the raw character data of three tables is:

3 x 65,536 x 2 = 393,216 bytes

The shared source explicitly states that the complete tables are not included. The actual coverage and memory size therefore cannot be verified from this version. The core tradeoff is still clear. A larger constant table removes runtime conditionals, dictionary objects, and multiple passes.

The snake_case state machine

The densest part of the code is the method that generates snake_case. It does more than convert characters to lowercase. In one pass, it transforms the text, merges invalid separators, skips leading separators, and removes a trailing underscore.

The algorithm uses the same char[] as both input and output buffer. It maintains two indices:

i = input position to read n = output position to write

i advances on every iteration. n advances only when a character is written. The following invariant is preserved throughout execution:

0 <= n <= i <= inputLength

The write index never advances beyond unread input. This property allows safe in-place compaction without a separate second buffer.

The character first passes through the lowercase ASCII table. It is then tested as a letter or digit. char.IsLetterOrDigit accepts not only ASCII characters but also Unicode letters and decimal-digit categories. The output is therefore guaranteed to be ASCII only if the conversion table reduces every supported input to an ASCII value.

A letter or digit is written directly. Other characters are treated as separator candidates. A state variable indicates whether the output is at the beginning or whether the last written character was a separator.

This one-bit state applies the following rules:

1. Leading separators are not written.

1. Only the first character in a separator sequence becomes an underscore.

1. Two underscores cannot be adjacent.

1. The first separator following a valid character is preserved.

1. A final underscore is removed when processing ends.

Consider a conceptual input with this structure:

[separator][word][separator][separator][word][separator]

The generated output is:

word_word

Correctness can be explained through a loop invariant. At the beginning of every step, the first n elements of the array are the valid snake_case result for the processed input prefix. That section has no leading underscore and no adjacent underscores. A valid new character is appended. An invalid character appends an underscore only if the previous output ended with a valid character. After the loop, the only possible violation is a trailing underscore. The final condition removes it.

The method has Theta(n) time complexity. Every input character is examined in both the best and worst case, so asymptotic cost does not change. For output length m:

0 <= m <= n

Space complexity is Theta(n) because of the working array. Apart from that array, in-place compaction requires no second temporary buffer of length n.

In the section that removes the trailing underscore, the read index is temporarily reused to calculate the final output position. The loop has already ended, so its previous input position is no longer needed. This is a small but deliberate choice that reduces length without introducing another variable.

Unicode boundaries

The direct indexing that provides speed requires the supported character domain to be defined explicitly. A C# char represents one UTF-16 code unit, not necessarily one complete Unicode character. Its value lies between U+0000 and U+FFFF.

A Unicode code point outside the Basic Multilingual Plane is represented by a surrogate pair consisting of two char values. Because the code processes each char independently, it cannot treat such a character as one semantic unit. Turkish Latin letters lie within the BMP, so the issue does not arise when the target data set is limited to Turkish text and common punctuation. A general Unicode text processor requires a design based on Rune.

Table length is another precondition:

table.Length > inputChar

If this condition is not satisfied, direct indexing causes an out-of-range access. Because the shared tables are shortened, the displayed source cannot be treated as a complete executable mapping. A production version should define every index up to the highest supported character code or provide a fallback for values outside the table.

Unicode normalization is a subtler issue. Visually identical text can be represented either by one precomposed code point or by a base letter followed by a combining mark. Unicode NFC and NFD forms address this canonical-equivalence problem.

A table-driven operation can convert a precomposed Turkish letter correctly while processing the decomposed form as a base letter and combining mark separately. If the combining-mark table entry is not defined appropriately, two canonically equivalent inputs can produce different outputs. Text received from external sources should be normalized to NFC, or to NFD according to the chosen policy, before conversion.

Its place in NLP and search systems

ASCII reduction is not a lossless normalization suitable for every purpose in Turkish natural-language processing. As the alphabet shrinks, some distinctions among words also disappear. Reducing ç, ş, ğ, ö, ü, and ı to basic Latin counterparts can map different words to the same string.

This information loss can be useful for tolerant search. When a user submits a query without a Turkish keyboard, matching can be performed through the ASCII key. The same property can be useful for URL segments, filenames, technical identifiers, and simplified text transfer between systems.

In machine learning, suitability depends on the objective. ASCII reduction can reduce vocabulary size and merge spelling differences in legacy data sets. It also removes distinctive Turkish letters and reduces linguistic information presented to the model. A safer data design retains both representations rather than deleting the original text:

original_text normalized_ascii_text

The ASCII form can serve as a search and matching key. The original text remains available for a language model, forensic record, legal document, or reproducible analysis.

A generated snake_case value is not a unique identifier by itself. Different inputs can produce the same result. If the value is used as a database key, file identity, or URL slug, collision checking, a numeric suffix, or a short content-derived identifier is required.

The approach I used while developing this code shows that a general abstraction is not always necessary in text processing. When the input domain is known, output policy is fixed, and latency matters, a direct-addressed table is highly effective. The three conversion tables combine character mapping and case policy in one access. Two-index compaction produces snake_case without a separate buffer or regular expression.

The strength of the algorithm does not come from appearing complex but from having explicit operating boundaries. Every character is processed once. Output length never exceeds input length. Separator rules are maintained with one bit of state. The same input produces the same result. When Unicode scope and irreversible conversion are defined correctly, the design provides a measured and predictable solution for high-traffic text-processing paths.