Natural Language Processing

Natural Language Processing (NLP) studies computational methods for analyzing, transforming, retrieving, classifying, generating, and grounding human language. It is broader than text generation and broader than large language models. Character encoding, normalization, morphology, syntax, semantics, information retrieval, speech recognition, language modeling, evaluation, and production-system behavior all belong to the field.

This course is deliberately Turkish-centered. Turkish exposes assumptions that remain hidden in many English-first pipelines: productive agglutinative morphology, vowel harmony, morphophonemic alternation, rich case marking, relatively flexible constituent order, and language-specific casing behavior. The goal is not to treat Turkish as an exception, but to use it to understand why NLP architectures must be evaluated against the linguistic properties of their target languages.

The course connects to Artificial Intelligence: Philosophy, Theory and Practice, Neural Networks and Learning Models, Statistical Learning and Machine Learning, Logic Programming and Prolog, and Large Language Models. Those courses explain broader learning and reasoning mechanisms; this course focuses on how natural language itself is represented and processed.

raw language data
       |
       v
encoding and normalization
       |
       v
segmentation / tokenization
       |
       v
morphology
       |
       v
syntax
       |
       v
semantics and discourse
       |
       v
task model
       |
       v
evaluation and production system

Modern end-to-end models can learn several of these stages jointly. The distinctions still matter when diagnosing errors, building datasets, designing interfaces, and operating real systems.

Unit 1: The NLP Problem Space

Natural language is not a formally designed programming notation. Surface strings are ambiguous, multiple forms can express similar meanings, and the same form can serve different grammatical or semantic roles.

In Turkish, constituent order may vary while case marking preserves much of the predicate–argument structure. In English, word order often carries a larger share of that information. The same preprocessing assumption therefore has different consequences in different languages.

Core NLP task families include normalization, segmentation, morphology, tagging, named-entity recognition, parsing, relation extraction, semantic similarity, retrieval, classification, sentiment analysis, question answering, summarization, machine translation, speech-language processing, dialogue, language modeling, and generation.

Rule-based, statistical, and neural methods should be viewed as engineering options rather than mutually exclusive historical camps.

explicit rule
   -> lexicon / grammar / finite-state model

statistical model
   -> n-gram / HMM / Naive Bayes / CRF

learned representation
   -> embedding / RNN / Transformer / LLM

A deterministic normalization rule may be preferable for a narrow text transformation, while contextual neural models may be appropriate for NER or semantic classification.

Unit 2: Language Typology and NLP Design

A Unicode-capable program can technically accept many languages without being linguistically language-independent.

Important cross-linguistic differences include word-boundary marking, inflectional complexity, derivation, constituent order, casing, orthography-to-pronunciation correspondence, compounding, clitics, and writing systems.

Turkish is agglutinative:

ev
evler
evlerimiz
evlerimizde
evlerimizdekiler
evlerimizdekilerden

English has much lighter inflection and relies more heavily on word order and separate function words.

German combines inflection with productive compounding.

Arabic involves root-and-pattern morphology together with attached elements and often omits short-vowel diacritics in ordinary writing.

Chinese does not normally mark all word boundaries with spaces, so word segmentation becomes a modeling decision.

Japanese combines multiple scripts and also requires explicit segmentation or morphological analysis.

Other Turkic languages share structural properties with Turkish but differ in writing systems, phonology, morphology, vocabulary, and resource availability. Relatedness helps transfer in some settings but is not a correctness guarantee.

Unit 3: Unicode, Orthography, and Normalization

The first NLP layer is often character handling. Byte decoding, Unicode normalization, invisible control characters, and language-specific case conversion can alter equality and search behavior.

For Turkish:

i <-> İ
ı <-> I

This differs from English-centric casing assumptions.

ASCII reduction may be useful as an auxiliary search representation:

ç -> c
ğ -> g
ı -> i
ö -> o
ş -> s
ü -> u

but it is lossy and should not replace canonical text. The design issues are explored in Table-Based ASCII Conversion for Turkish Text.

The reverse problem, Turkish deasciification, is context-sensitive. cagri can be restored as çağrı, while other strings remain ambiguous. Turkish Deasciification in C# is an example of a small deterministic/contextual NLP model whose scope is explicitly bounded.

Text normalization begins by defining which distinctions the application must preserve. Test Unicode equivalence, case folding, and punctuation transformations separately; some transformations irreversibly alter names, code, or evidentiary text. Keeping a mapping from raw to normalized text makes later matches auditable. Turkish-specific casing must not be replaced blindly with English-oriented rules.

3.1. Turkish Casing, Unicode, and Canonical Data

Text preprocessing is part of the model contract. Turkish has language-specific casing rules for i, İ, ı, and I; applying locale-insensitive lowercasing can silently change identifiers and words.

Unicode normalization should be explicit. Canonically equivalent sequences can compare differently at the code-unit level, while compatibility normalization can intentionally remove distinctions. A search index may keep several derived representations, but the original text should remain authoritative.

ASCII folding is useful as an auxiliary retrieval representation for texts written without Turkish characters. It is lossy and should not overwrite the canonical field.

3.2. Token Boundaries Are Engineering Decisions

Whitespace is a useful signal, not a universal token definition. Productive Turkish suffixes create many surface forms, and a word-level vocabulary can become sparse. BPE, Unigram, SentencePiece, character, and byte-level tokenizers make different trade-offs between vocabulary size, sequence length, morphology, and robustness.

Tokenizer choice therefore affects effective context length, memory, inference cost, and cross-lingual fairness as well as linguistic representation.

3.3. Unicode Equivalence and Turkish Text Keys

Two visually identical strings can be represented by different Unicode code-point sequences. A precomposed character and a base character followed by a combining mark may render identically without having identical bytes. A text-matching pipeline may adopt a normalization form such as NFC, but the source text should not be destroyed. Identity, search-key derivation, and user-facing display have different semantic responsibilities.

Turkish case mapping is language-sensitive, particularly for I, İ, ı, and i. Locale-independent lowercasing must not be assumed correct for Turkish search keys. Likewise, a phonetic or ASCII-folded representation is not a unique identifier: different words may collapse to the same key. Exact identifiers, approximate matching, and full-text search must not silently share one destructive normalization policy.

A validation corpus should separately cover composed and decomposed characters, Turkish letters, mixed scripts, whitespace variants, and invalid UTF-8 input. Each transform must declare whether it is reversible. When original spelling is evidentially significant, a normalized key should remain a derived index or temporary representation rather than replacing source text. Improved retrieval recall must be evaluated against the additional false matches that normalization may introduce.

3.4. Source Offset Mapping During Text Normalization

Unicode normalization, case conversion, and character cleanup may alter the text length or code-point sequence. Offsets found in normalized text therefore cannot safely be used as offsets into the original document. One visible grapheme may span multiple Unicode code points; JavaScript string indices count UTF-16 code units, while another layer may report UTF-8 byte offsets.

A robust pipeline carries a provenance map associating each transformed span with its source start and end positions. Many-to-one mappings should preserve source ranges or multiple correspondences rather than inventing a single character index. Synthetic characters without a source position are marked explicitly. When tokenization, search highlighting, named-entity recognition, and exports share the same offset contract, the evidence span remains consistent across layers.

Tests should include combining marks, emoji sequences, Turkish İ and ı, collapsed whitespace, and normalized line endings. Normalization may improve retrieval, but it must not overwrite the preserved source text. Highlighted spans must remain reconstructible from the original text and a recorded sequence of transformations.

3.5. Reversible and Information-Losing Normalization of Turkish Text

In Turkish text processing, İ/i, I/ı, diacritics, Unicode composition, and source encodings are not merely cosmetic details. Converting ş to s, or using an inappropriate locale for case conversion, may map distinct words to a shared representation. A normalization function should therefore be classified as one-to-one, many-to-one, or context-dependent.

Lossy normalization used for search and phonetic matching cannot replace the original forensic record or source document. Preserve source text, normalized representation, transformation order, and any required character-offset mapping separately. Composed transformations do not always commute: Unicode normalization, case folding, Turkish character conversion, and tokenization order can change results. Evaluation must report false mergers of distinct words as well as successful normalization.

Unit 4: Segmentation, Tokens, and Subwords

A token is a computational unit, not a universal linguistic fact.

text
 -> characters
 -> word candidates
 -> subwords
 -> model tokens

Whitespace is a strong cue in Turkish and English but not a complete solution. URLs, punctuation, abbreviations, multiword expressions, emojis, proper names, and malformed text all create edge cases.

Agglutinative morphology makes word-level vocabularies sparse in Turkish. Subword modeling can share statistical evidence across inflected and derived forms.

BPE-style methods merge frequent symbol sequences. Unigram methods score alternative segmentations from a learned subword inventory. SentencePiece can learn directly from raw text without requiring a pre-tokenized word sequence.

Model segmentation does not have to align with morphology:

linguistic:
ev + ler + imiz + den

possible model segmentation:
evl + erimiz + den

The distinction is important in SentencePiece Tokenization, Normalization and Vocabulary Boundaries.

Character- and byte-level models reduce vocabulary dependence but lengthen sequences. Tokenizer choice therefore affects memory, latency, effective context length, and multilingual fairness.

Segmentation begins by choosing whether task outputs require words, subwords, or morphemes. Compare boundary decisions for abbreviations, numbers, compounds, and inflected forms. Fewer tokens on average is not always better if distinctions needed downstream are lost. Evaluate labeled boundary examples alongside downstream task performance.

Unit 5: Turkish Morphology

Turkish morphology is productive. A root can take multiple inflectional and derivational suffixes.

yaz
yazıcı
yazıcılar
yazıcılarımız
yazıcılarımızdan

Suffixes also have context-dependent surface forms because of vowel harmony and consonant alternations.

Finite-state methods provide a natural formalism for much of this regular structure:

root lexicon
    |
    v
morphotactic transitions
    |
    v
surface alternation rules
    |
    v
word form

Running the relation in the opposite direction yields candidate analyses for a surface form.

Syllabification and morphological analysis are different problems. Finite-State Turkish Syllabification Algorithm and the open-source Turkish Finite-State Syllabifier demonstrate how a narrow phonotactic/orthographic problem can be solved with a small deterministic state machine without pretending to be a full morphological analyzer.

Unit 6: Morphological Ambiguity and Tagging

A morphological analyzer can produce multiple legal analyses. A disambiguator uses context to select among them.

surface form
    |
    v
morphological analyzer
    |
    +-> analysis A
    +-> analysis B
    +-> analysis C
    |
    v
contextual disambiguation
    |
    v
selected analysis

Part-of-speech tagging has been approached with rules, n-gram taggers, HMMs, transformation-based learning, CRFs, perceptrons, and contextual neural models.

For Turkish, morphology and syntax interact strongly. Case, agreement, possession, tense, aspect, and person information may be encoded inside the word rather than in separate tokens.

6.1. Morphological Ambiguity and Syntactic Evidence

A Turkish surface form can have multiple morphological analyses. Disambiguation uses surrounding words, syntactic relations, and sometimes semantic evidence to select the analysis that best fits the sentence.

This interaction is important because morphology and syntax are not cleanly separable pipeline stages. Case marking, agreement, derivation, and word order jointly influence dependency structure.

6.2. Dependency Parsing under Flexible Word Order

English relies strongly on position, whereas Turkish case marking allows more constituent movement. A parser that transfers well from English is therefore not guaranteed to preserve Turkish dependency relations.

Universal Dependencies provides a shared annotation framework, but language-specific constructions and annotation decisions still matter. Evaluation should inspect attachment errors by relation type, sentence length, morphology, and domain rather than reporting only one aggregate score.

Unit 7: Syntax and Dependency Parsing

Syntax models relationships between linguistic units. Dependency parsing represents head–dependent relationships directly.

          tested
          /    \
     engineer  model
      subject  object

Relatively flexible constituent order makes explicit dependency structure especially useful for Turkish.

Universal Dependencies provides a cross-linguistic annotation framework for parts of speech, morphological features, and dependency relations. It does not claim that languages are structurally identical; it provides shared annotation principles for comparing different grammatical systems.

The Turkish BOUN and IMST treebanks are important resources for Turkish dependency parsing.

Unit 8: Lexical Semantics, Entities, and Discourse

Meaning is contextual. A single surface form can have different senses, and a single concept can be expressed by multiple lexical forms.

Distributional proximity is therefore not equivalent to identity. Doctor and physician are near synonyms, whereas doctor and hospital are related concepts rather than interchangeable expressions. Polysemy makes the distinction sharper: bank may refer to a financial institution or the side of a river. The local predicate and surrounding arguments help disambiguate the intended sense.

Negation and modality also alter what a sentence asserts. Ali arrived, Ali did not arrive, and Ali may arrive contain similar lexical material but impose different truth conditions. A relation extracted from reported, hypothetical, or negated speech should not be stored as a verified external event.

Named-entity recognition identifies people, organizations, locations, dates, products, and other entity types. Turkish suffixes can attach directly to entity names, so normalization and boundary handling affect NER behavior.

NER produces at least a span and an entity type. With forms such as Ankara, Ankara'da, and Ankara'dan, tokenization and normalization must preserve an unambiguous mapping to the original character offsets. BIO labeling marks beginnings (B), continuation (I), and positions outside an entity (O); the treatment of attached morphology must remain consistent across training and evaluation.

Recognizing two mentions with the same spelling does not establish that they denote the same real-world entity. That is an entity-linking problem and may require independent identifiers or corroborating context. Name similarity alone must not merge distinct people or organizations.

Relation extraction goes beyond entity detection:

PERSON ---- works_at ----> ORGANIZATION
PERSON ---- born_in -----> LOCATION
EVENT  ---- occurred_on -> DATE

Coreference resolution and discourse analysis connect expressions across sentence boundaries.

Consider Ayşe completed the report. The engineer later sent it to her manager. Connecting the engineer to Ayşe and it to the report requires discourse context, and some texts permit more than one interpretation. Speech transcripts add uncertainty through imperfect punctuation, segmentation, and speaker changes.

Semantic roles should also be separated from surface mentions. In Ayşe said Mehmet would travel to Ankara, Ayşe is the speaker and Mehmet is the potential traveler. The travel is a reported proposition, not a confirmed journey. A structured extraction must retain its source sentence, offsets, reporting context, and modality if it is to support later verification.

Unit 9: Logic and Logic Programming

Natural language can be mapped into structured representations instead of being left as free-form text.

"Deniz sent the report."

        |
        v

sender = Deniz
object = report
action = send

A symbolic representation can then be expressed as a predicate:

sent(deniz, report).

Prolog and Definite Clause Grammars have a long relationship with natural-language parsing and formal semantics.

A modern hybrid system can combine neural extraction with symbolic validation:

text
  |
  v
neural NLP model
  |
  v
candidate entities / relations
  |
  v
logic rules and constraints
  |
  v
validated result

This is a direct bridge to Logic Programming and Prolog.

9.1. Leakage, near-duplicates, and Annotation Disagreement

Train/test separation must operate at the level at which information can leak. Randomly splitting sentences from the same document can place near-identical context on both sides of the split and produce an unrealistically high score.

Deduplication should therefore consider source identity and near-duplicate documents, not only byte-identical rows.

Annotator disagreement is also data about the task. Guidelines, adjudication, and inter-annotator agreement reveal whether a label is well-defined. Hiding disagreement can turn an ambiguous linguistic judgment into apparently precise ground truth.

Dataset versions should preserve source, license, normalization rules, annotation guideline version, split policy, and deduplication method.

Unit 10: Corpora, Annotation, and Data Lifecycle

A corpus is not merely a large folder of text. Source, time period, domain, language variety, licensing, deduplication, splits, and annotation rules all affect model behavior.

Document-level leakage is particularly important. Near-duplicate documents across train and test sets can create an unrealistically high score.

Annotation disagreement should be treated as information about the task rather than hidden. Guidelines, adjudication, and inter-annotator agreement are part of dataset quality.

Turkish NLP has developed morphological corpora, dependency treebanks, lexical resources, speech data, and task-specific datasets. Resource size must be considered together with annotation quality, domain, and license.

High percentage agreement between annotators may be misleading when one label dominates the corpus. Two annotators can agree on common cases while disagreeing systematically on rare, high-impact events. Observed agreement should be supplemented by an appropriate chance-adjusted statistic, label-wise disagreements and an analysis of their causes. Multi-label corpora need special care because several categories can legitimately co-occur in one passage. An annotation codebook improves reproducibility when it specifies, for each label, an inclusion rule, an exclusion rule and a boundary case rather than only a short name.

Unit 11: Classical Statistical NLP

N-gram language models estimate the next unit from a bounded history:

P(next word | recent context)

They are transparent and efficient but suffer from sparse observations and limited context. Morphologically rich languages can make word-level sparsity more severe.

Smoothing reallocates probability mass so unseen events are not automatically impossible.

Naive Bayes remains a strong baseline for many text-classification problems. My historical Spam Detection with Data Mining Techniques work used n-gram features and Naive Bayes implementations to compare practical classification behavior.

HMMs and CRFs historically played major roles in tagging and sequence labeling. Understanding them clarifies what modern contextual models changed and what evaluation problems remained unchanged.

Unit 12: Text Representation and Embeddings

Bag-of-words and TF-IDF discard much order information but remain valuable baselines and retrieval representations.

Word2Vec and GloVe learn dense distributional representations. FastText-style character n-gram information is particularly relevant to rare and morphologically complex word forms.

Contextual models assign different representations to the same surface word depending on its surrounding text. This is one of the key conceptual changes introduced by pre-trained contextual encoders such as BERT.

12.1. Jaccard, Dice, and Cosine: Choose the Representation Before the Similarity Measure

A similarity function should not be selected independently of the representation. The same two documents express different notions of "similar" when represented as sets, frequency vectors, or dense embeddings.

For token sets A and B, Jaccard measures intersection relative to union:

J(A,B) = |A intersection B| / |A union B|

Sørensen-Dice scales the overlap by the total set sizes:

D(A,B) = 2|A intersection B| / (|A| + |B|)

For set representations the two are monotonically related:

D = 2J / (1 + J)

Switching from Jaccard to Dice on the same set representation therefore does not necessarily add ranking information; much of the difference is score scaling.

Cosine similarity measures angular similarity between vectors:

cos(x,y) = dot(x,y) / (norm(x) * norm(y))

With sparse TF-IDF vectors it preserves term weights, whereas basic set-based Jaccard/Dice discard frequency. Cosine can also be applied to dense embedding vectors, but the score then represents geometric proximity in a learned space rather than token overlap. The same formula does not imply the same semantics.

A safer design order is:

analysis unit -> normalization/tokenization -> representation -> similarity -> threshold/ranking

The contract should also define empty-set and zero-vector behavior explicitly. Silently mapping an undefined denominator to 0 or 1 can change downstream ranking and threshold logic.

Unit 13: Neural Sequence Models

RNNs process sequences through recurrent hidden state. LSTM and GRU architectures introduced gates that make long-range dependencies easier to learn.

Encoder–decoder systems map one sequence to another, supporting tasks such as machine translation.

Attention allows the output process to weight different parts of the source sequence dynamically. This idea later became central to Transformers.

Unit 14: Transformers and Pre-training

Transformers replace recurrent sequence traversal with attention-based interactions between tokens. Their parallel training characteristics enabled much larger pre-training regimes.

Encoder-oriented models are commonly used for representation and understanding tasks, decoder-oriented models for autoregressive generation, and encoder–decoder models for sequence transformation.

Masked modeling and autoregressive next-token prediction produce different information flows.

Pre-training followed by task adaptation is valuable for lower-resource tasks, but multilingual training data is rarely balanced across languages.

Unit 15: Multilingual NLP and Cross-Lingual Transfer

A shared model does not guarantee equal treatment of languages. Tokenization efficiency, training volume, script, typological distance, and domain all affect transfer.

A Turkish sentence may consume more subword tokens than an English sentence with similar semantic content. That changes effective context length and inference cost.

Transfer between Turkish and related Turkic languages can benefit from shared morphology and vocabulary, but different alphabets, false cognates, and language-specific morphology still matter.

Machine translation must reconstruct grammar rather than replace words one by one. Turkish–English translation often requires reorganizing constituent order and expanding morphological information into separate English words.

Unit 16: Classification, Extraction, and Retrieval

Text classification includes spam filtering, topic classification, intent detection, routing, and moderation.

Evaluation should separate representation, model, decision threshold, and task metric.

Sentiment analysis is not simply positive/negative keyword counting. Negation, target, irony, intensity, domain, and morphology matter.

Information retrieval remains fundamental even in LLM systems. A hybrid design may combine lexical and vector search:

query
  |
  +-> lexical retrieval -----+
  |                          |
  +-> vector retrieval ------+-> reranker -> results

My Turkish Phonetic Matching with Numeric Soundex and open-source Turkish Numeric Soundex work treat phonetic coding as candidate generation rather than identity proof.

Unit 17: Speech and the NLP Boundary

Automatic speech recognition converts acoustic input into text. The resulting transcript often still requires NLP normalization.

audio
 |
 v
VAD / segmentation
 |
 v
ASR
 |
 v
raw transcript
 |
 v
punctuation / casing / text normalization
 |
 v
NLP task

Turkish ASR output can contain errors in suffixes, proper names, spacing, numbers, and written-language normalization.

Automatic Speech Recognition, Whisper Architecture in Speech Recognition Systems, and Capacity Engineering for Real-Time Speech Recognition cover the acoustic/model and production-system sides of this boundary.

WER is useful, but a single suffix error can turn a whole Turkish word into an error. CER can add diagnostic information, although neither metric directly measures semantic harm.

17.1. Sentence Boundaries and Speaker Turns in Noisy Transcripts

A line or segment boundary emitted by speech recognition is not necessarily a grammatical sentence boundary. A recognizer may end a segment during a short pause even when the speaker continues the same thought in the next segment. Conversely, one long segment may contain several sentences, questions, and topic changes. Three boundaries should therefore be represented separately: audio segment, speaker turn, and grammatical sentence. A merge decision can use word order, punctuation, pauses, and speaker changes, but uncertain boundaries must not be presented as facts established by the audio source.

Splitting a long unpunctuated segment at arbitrary character counts can destroy semantic continuity. A more reliable procedure first identifies intervals that are safe with respect to time and speaker identity, then proposes linguistic boundaries using cues such as conjunctions and predicate position. When the boundary remains uncertain, the larger unit is retained and the summarizer preserves the fact that its smaller spans share context. Every derived sentence retains source segment identifiers and original character spans; a sentence assembled from several segments must not receive a fabricated single timestamp.

The distinction also determines evaluation. Sentence-boundary quality, topic coverage in the summary, and accuracy of the mapping back to audio are separate metrics. A readable paragraph is not sufficient if it points to the wrong audio interval. Automatic paragraph formation is a reversible, traceable presentation layer rather than a correction written over the original transcript.

Word error rate is the sum of substitutions, deletions, and insertions divided by the number of reference words. Its value depends on normalization: “twenty five” and “25” may convey the same quantity but receive different costs in raw token alignment. Rules for number rendering, punctuation, named entities, and speaker changes should be fixed before evaluation. Low word error rate does not establish correct speaker attribution or guarantee that a safety-critical number was transcribed accurately; task-specific entities deserve separate accuracy checks.

17.2. Separating Segmentation Delay from Inference Cost

Voice activity detection (VAD) identifies intervals likely to contain speech; it does not decode the words being spoken. Its start and stop decisions depend on silence thresholds, minimum speech duration, and required context. Cutting a segment too early may truncate word endings, while waiting too long can increase the delay before text processing. An audio segment boundary is not the same event as a grammatical sentence boundary.

The computation required by a recognizer can be summarized by its real-time factor (RTF):

RTF = processing time / audio duration

Processing ten seconds of recorded audio in two seconds yields RTF 0.2. It does not imply that the user waited only two seconds: acquisition, endpointing, queueing, normalization, and delivery have distinct delays. NLP consumers must therefore preserve audio timestamps, speaker turns, and grammatical sentence spans as separate notions. Improving transcript readability must not silently rewrite the intervals to which the original words belong.

A low overall word error rate does not ensure that transcription errors are harmless for downstream analysis. Dropping a single negation between “acceptable” and “not acceptable”, for example, can reverse an intent or decision label. Transcription quality should therefore be assessed not only by word errors but also on task-relevant slices such as names, numerals, negation, domain terms and speaker transitions. Suspicious spans should not be silently rewritten during preprocessing; links to timestamps and original audio must remain available. Human review can focus on high-consequence or uncertain spans rather than uniformly inspecting every generated sentence.

Unit 18: Large Language Models Within NLP

LLMs can perform many NLP tasks but do not eliminate preprocessing, retrieval, data licensing, evaluation, privacy, multilingual imbalance, or production constraints.

Large Language Models covers Transformer scaling, pre-training, post-training, RAG, tool use, evaluation, and inference engineering. This course focuses on language data and NLP tasks.

Structured extraction from an LLM should still be validated:

text
 |
 v
LLM / NLP model
 |
 v
candidate structure
 |
 v
schema validation
 |
 v
domain rules
 |
 v
persistent result

RAG depends on retrieval quality. Turkish chunking, normalization, multilingual embeddings, reranking, and token budget all affect the final answer.

18.1. Evaluation under Turkish Variation and Calibration

A clean benchmark does not describe all production input. Turkish evaluation may need separate slices for spelling variation, ASCII Turkish, conversational transcripts, code-switching, proper names, dates, numbers, and domain shift.

Classification metrics should be paired with the decision threshold. Macro and micro averaging answer different questions under class imbalance, and a high score does not imply calibrated probabilities.

For ASR-derived text, WER and CER describe transcription error but do not directly measure semantic harm to a downstream NLP task. End-to-end evaluation should therefore measure the final task as well as the intermediate transcript.

Unit 19: Evaluation

There is no universal NLP score.

Classification may use precision, recall, F1, PR curves, and calibration.

Sequence labeling requires explicit boundary policy.

For example, a system that extracts only Ankara when the gold entity is Ankara Üniversitesi may receive partial token-level credit but fails exact entity-span matching. NER reporting should state whether entity-level precision, recall, and F1 require exact boundaries, and how nested entities are treated. The label ontology must also match the target application.

Dependency parsing evaluates heads and relations.

ASR commonly uses WER and CER.

Word Error Rate is WER = (S + D + I) / N, where substitutions, deletions, and insertions are divided by the number of reference words. In morphologically rich languages, an incorrect suffix may count as an entire word error. Conversely, a low WER does not guarantee that downstream entity, date, or negation extraction is correct. Evaluate the end-to-end NLP task on actual transcripts, not only the transcription error rate.

Machine translation uses automatic metrics as proxies; semantic adequacy and human judgment remain separate considerations.

Macro and micro averaging answer different questions under class imbalance.

Class-specific error counts and uncertainty are essential when rare entities or intents matter. A model's F1 score is not a calibrated probability: a confidence value of 0.90 cannot be read as a 90% chance of correctness without separate calibration evidence. Report the averaging policy, operating threshold, and the distribution of scores across the target classes.

Robust Turkish evaluation should include clean text together with spelling variation, ASCII Turkish, transcripts, domain shift, code-switching, proper names, dates, and numbers when these are relevant to the target application.

Regression corpora should deliberately include hard negatives: different people sharing a name, negated relations, invalid temporal expressions, missing discourse context, and speech-recognition substitutions that alter a key entity. Compare model or rule revisions on identical source records and label guidelines, and inspect error categories alongside aggregate metrics.

Unit 20: Production Engineering

An NLP service must satisfy accuracy, latency, throughput, memory, fault isolation, reproducibility, privacy, versioning, and observability constraints together.

UTF-8 text
   |
   v
normalization
   |
   v
tokenization
   |
   v
model inference
   |
   v
output validation
   |
   v
domain model

Preprocessing code is part of the model contract. Changing normalization without retraining or re-evaluating can create distribution shift.

Batch workloads optimize throughput; interactive systems care about p95/p99 latency. GPU batching can improve throughput while increasing queue delay.

Deterministic preprocessing can be cached safely with an explicit version key. Generative output may require a different caching contract.

Offline and privacy-sensitive NLP additionally requires explicit model licensing, local compute sizing, update procedures, and sensitive-log handling.

20.1. Preprocessing Versioning and Authorization Boundaries

Normalization, tokenization, chunking, embedding, model inference, and output validation should be versioned as one pipeline contract. Changing one stage can shift the input distribution seen by later stages even when the model file itself is unchanged.

Language input is untrusted input. Length limits, Unicode policy, timeouts, bounded queues, and output schemas belong to the service boundary.

LLM applications add a second concern: prompt injection can turn retrieved or user-controlled text into instructions. Authorization must remain outside the model. A model may propose an action, but the application must independently verify whether the caller is allowed to perform it.

Unit 21: Robustness and Security

Language input is untrusted input. Length limits, Unicode normalization policies, timeouts, quotas, and output schemas belong to service security.

LLM applications add prompt injection: natural language can become a control channel rather than passive data. Authorization must remain outside the model.

Sensitive information inside text can include names, identifiers, health data, or communication records. Removing filenames does not anonymize such content.

Unit 22: Turkish-Centered Engineering Patterns

A search system should often retain multiple representations rather than destructively normalize its canonical field:

canonical text
   |
   +-> exact index
   |
   +-> case-normalized index
   |
   +-> ASCII auxiliary index
   |
   +-> phonetic candidate index

The original representation remains authoritative.

Choosing Methods for Turkish Text Normalization and Phonetic Matching develops this design choice in more detail.

The open-source Turkish Finite-State Syllabifier shows that a deterministic state machine can be the right solution for a narrow linguistic task.

The open-source Turkish Numeric Soundex shows how a small phonetic key can reduce a candidate set without pretending to establish identity.

Natural Language Processing and Multimedia connects text with OCR, speech, and other media-derived language.

22.1. Candidate Generation and False-Positive Control in Phonetic Matching

Soundex-like phonetic codes are candidate-generation mechanisms, not complete linguistic analyzers. Turkish vowel harmony, agglutinative morphology, syllabic structure, and diacritic transformations can cause distinct words to share one code. Character normalization, token boundaries, phonetic coding, and contextual second-stage scoring must remain separate stages.

Strong exact matches should be preserved before broadening to phonetic candidates for ambiguous or unresolved queries. Validation reports precision, recall, candidate count, and query latency together. In searches involving identities, people, or technical terms, a false-positive match carries different consequences than an ordinary spelling suggestion; phonetic similarity is not identity equivalence.

Unit 23: Comparing the Same Task Across Languages

Tokenization differs by language:

Turkish:
whitespace is useful
+ productive suffixes
+ morphology/subwords matter

English:
whitespace is a strong baseline
+ contractions and multiword expressions

Chinese:
word spaces are not mandatory
+ segmentation is part of modeling

Japanese:
mixed scripts
+ explicit morphological segmentation

Morphology differs:

Turkish:
root + productive suffix chain

Arabic:
root + pattern + attached elements

English:
lighter inflection
+ more separate function words

German:
inflection
+ productive compounds

Syntax also differs. English relies heavily on word order, while Turkish case marking allows more flexibility. Cross-lingual transfer should therefore be evaluated, not assumed.

A shared multilingual model and shared tokenizer do not create linguistic equality by themselves.

Unit 24: Engineering Principles

A robust NLP system should keep the following principles visible:

1. Language data is more than a character string.
2. Document information loss introduced by normalization.
3. Do not equate model tokens with linguistic words.
4. Treat Turkish morphology as a central source of representation sparsity.
5. Keep classical methods as measurable baselines.
6. Do not use model size as a quality metric.
7. Prevent document-level and source-level data leakage.
8. Test domain and language variation explicitly.
9. Analyze error classes, not only average scores.
10. Version tokenizer, normalization, and model together.
11. Define sensitive-data lifecycle before model architecture.
12. Keep hard business constraints outside probabilistic inference when possible.
13. Avoid rigid rules where uncertainty genuinely requires learned models.
14. Do not assume multilingual models represent all languages equally.
15. Validate generated structure before turning it into persistent state.

Unit 25: Defining the NLU Problem Correctly

Natural Language Understanding (NLU) can be treated as the application-oriented part of NLP that converts text or speech-derived text into structures a software system can use: intents, entities, relations, events, semantic classes, or other constrained representations.

The first question in an NLU project is not which model to use. It is whether natural language is needed at all. If a user is selecting from a small, stable set of options, a form, menu, direct query, or deterministic parser may be more reliable. Natural language becomes valuable when people can express the same goal in many surface forms and forcing them into a rigid interface would create unnecessary friction.

business requirement
        |
        v
is natural language necessary?
        |
        +---- no ----> deterministic UI / rule / query
        |
        +---- yes
                |
                v
          what is the NLP task?
                |
        +-------+--------+
        |       |        |
      class   entities  open generation
        |       |        |
        +-------+--------+
                |
                v
         acceptance contract

25.1. Task Contracts

An NLP task is underspecified until the following are explicit:

• input unit,
• expected output,
• error classes,
• latency bound,
• behavior on failure,
• privacy boundary,
• policy for uncertain cases.

“Understand this text” is not testable. “Extract at most one primary intent and zero or more entities; route low-confidence cases for review” is a system contract.

25.2. Error Cost Comes Before Aggregate Accuracy

Two models with the same F1 score may have very different operational value. False positives and false negatives have task-dependent costs.

A search engine that misses one relevant result may still let the user reformulate a query. A system that attaches an identity or authorization attribute to the wrong person can have a much higher failure cost.

model error
   |
   v
application decision
   |
   v
business effect
   |
   v
risk / cost

Evaluation should therefore connect model errors to application consequences.

25.3. A Turkish Task Contract

Even the definition of a word boundary can be part of the contract in Turkish. For named-entity recognition:

Ankara
Ankara'da
Ankara'daki
Ankara'dakiler

the annotation policy must say which suffixes belong to the entity span.

ASCII Turkish, spelling variation, transcripts, proper-name suffixes, and compound/spacing variation should be part of the initial test design when they are expected in production.

Unit 26: Interactive NLU, Intents, Entities, and Dialogue State

Interactive language applications and offline document processing have different timing and failure contracts. A user-facing system must respond to short requests quickly; an offline classifier processing millions of documents may optimize throughput rather than single-request latency.

26.1. Spoken Interaction Layers

A spoken system can be decomposed conceptually:

audio
 |
 v
ASR
 |
 v
text
 |
 v
NLU
 | \
 |  +--> intent
 |  +--> entities
 |  +--> slots
 v
dialogue state / business rules
 |
 v
response content
 |
 v
NLG
 |
 v
TTS

ASR converts audio to text. NLU structures the text. Dialogue management determines the next system action. Natural-language generation forms the response, and text-to-speech renders it as audio.

An end-to-end model can merge some of these implementation stages, but keeping the responsibilities separate remains useful for debugging, authorization, and evaluation.

26.2. Intent

Intent represents what the user is trying to accomplish.

"What is the weather in Ankara tomorrow?"

may have the same underlying goal as several differently worded requests.

A useful intent taxonomy avoids creating classes that are almost indistinguishable while also avoiding one overly broad class that pushes all ambiguity into downstream business logic.

26.3. Entities and Slot Filling

Intent alone is often insufficient.

intent:   search_ticket
from:     Ankara
to:       Izmir
date:     ?

If a required date is missing, the correct action may be to ask for it instead of guessing.

user input
   |
   v
intent + extracted slots
   |
   v
required-field check
   |
   +--> complete --> business action
   |
   +--> missing  --> clarification

This makes missing information a deterministic system state rather than an invitation for a generative model to fabricate values.

26.4. Dialogue State

In a multi-turn interaction, the latest utterance is not always meaningful in isolation.

U: Find a ticket from Ankara to Izmir.
S: For which day?
U: Friday.

“Friday” completes a date slot established by the previous turn. A dialogue state can keep only the structured information needed by the task instead of treating the entire raw conversation as unbounded context.

26.5. Latency Budget

Interactive latency should be decomposed:

total latency
 = input capture
 + ASR
 + NLU
 + business service
 + NLG
 + TTS
 + network / queue

A fast NLU model does not imply a fast user experience. Endpoint detection, queues, remote calls, and sequential service dependencies can dominate model inference.

26.6. Discourse Focus, Elliptical Requests, and Corrections

Words in the most recent sentence are insufficient to determine a user's intention in a multi-turn conversation. In “show the third of those,” the referent may be the result list just displayed; the same expression can select a different object after a new list appears. Interpretation must therefore carry the visible result set, referent type, ordinal position, and context version alongside the latest intent. Discourse focus is more than the last-mentioned object: subject continuity, explicit selection, and context shifts provide different kinds of evidence.

A correction can be represented as a bounded change to one field of the previous query rather than an instruction to discard the entire task. “No, I meant the third record” changes the selected ordinal without silently altering the period or search criteria. “Search only the notes” changes the source scope, but is valid only when the preceding task supports that source and the new scope is authorized. If the referent cannot be established, the interpreter must not substitute a nearby record because it appears similar.

Preserving uncertainty does not require maintaining an unbounded list of alternatives for every sentence. Referents are limited by type and invalidated when the underlying result set changes. Explicit commands, contextual references, and phonetic suggestions carry different evidential strength; contextual or phonetic evidence alone must not authorize an effectful action. Effective dialogue management is measured both by preventing incorrect actions and by reducing unnecessary clarifications.

Unit 27: Data Engineering, Privacy, and Annotation

NLP behavior emerges from the data lifecycle as much as from the model. A large corpus with unknown provenance, duplicates, leakage, or an unrepresentative domain can be less valuable than a smaller well-defined dataset.

27.1. Provenance and Metadata

Useful metadata includes:

• source,
• collection time,
• language,
• domain,
• license or permission,
• transformations,
• annotation version,
• masking or pseudonymization steps.

raw data
  |
  v
provenance / license check
  |
  v
cleaning
  |
  v
annotation
  |
  v
splitting
  |
  v
modeling

A model can be reproducible at the weight level while the experiment remains irreproducible if the data transformations are unknown.

27.2. Training Data and Runtime Data Are Different Security Domains

Training data is often prepared as a relatively stable corpus. Runtime input changes continuously and is frequently user controlled.

Training concerns include usage rights, personal data, retention, leakage, and duplication.

Runtime input adds concerns such as malicious text, extreme length, malformed encodings, sensitive logging, prompt injection, and resource exhaustion.

27.3. Masking Is Not Automatically Anonymization

Replacing a person name with [PERSON] does not necessarily make a document anonymous. Organization, date, location, and event details may still identify the individual.

removing a direct identifier
        !=
making re-identification impossible

Privacy must therefore be evaluated against the data domain and the auxiliary information an attacker may have.

27.4. Class Imbalance

A rare production class can still be the most important class operationally. Matching production frequency exactly can make it difficult to learn; artificially balancing everything can distort priors.

The solution is not a single resampling recipe. Class-level evaluation should remain visible.

overall F1
    |
    +--> class A
    +--> class B
    +--> critical rare class

27.5. Annotation Guidelines Are Part of the Model

If annotators disagree on entity boundaries, irony, negation, or multi-intent behavior because the guidelines are unclear, the dataset itself imposes an upper bound on learnable consistency.

For Turkish, annotation policies should explicitly cover suffixes attached to proper names, multiword predicates, conversational fillers, and spelling variants when they matter to the task.

Unit 28: Weak Supervision, Distant Supervision, and Unlabeled Data

Labeled language data is expensive, while organizations often possess large amounts of unlabeled text. The key question is what learning signal can be extracted without pretending that unlabeled text is already ground truth.

28.1. Clustering and Topic Discovery

Unlabeled documents can be represented and clustered to inspect the data distribution.

unlabeled documents
       |
       v
representation
       |
       v
clustering
       |
       +--> candidate topic A
       +--> candidate topic B
       +--> outlier group

A cluster label is not automatically a true class. Clustering is nevertheless useful for discovering new intents, missing categories, domain shifts, and annotation gaps.

28.2. Weak Supervision

Weak supervision uses inexpensive but noisy labeling functions instead of requiring a human gold label for every example.

Signals can come from:

• dictionary matches,
• an existing database field,
• file or channel metadata,
• application source,
• another model.

Multiple labeling functions may disagree. The design should model the noise rather than silently treating every weak label as equally correct.

28.3. Distant Supervision

A structured knowledge source can project approximate labels onto text. This reduces manual work but does not guarantee that the relation in the structured source is expressed by the specific sentence.

Distant supervision therefore trades labeling cost for noise.

28.4. Active Learning

Active learning selects examples that are especially informative for human annotation. In a Turkish domain that requires experts, labeling boundary cases can be more efficient than labeling thousands of random examples.

Selecting only the most uncertain examples can distort the labeled distribution, so uncertainty should be combined with diversity and class coverage.

28.5. Synthetic Examples from Generative Models

LLMs can generate training examples, but synthetic text should not be assumed to match real user behavior.

Synthetic data can improve coverage of rare forms and help create stress tests. It can also reproduce model bias, over-clean natural spelling noise, distort frequency, and create artificial similarity between train and evaluation data.

Synthetic examples should therefore retain explicit provenance and be evaluated separately from real data.

Unit 29: Hybrid Architectures and Model Selection

A production NLP system is rarely described completely by naming its largest model. Rules, statistical models, neural networks, and LLMs can serve different responsibilities in one architecture.

29.1. The Most Complex Model Is Not the Default

A large language model is unnecessary for recognizing a well-formed email address. A deterministic parser or regular expression may be sufficient. Conversely, a contextual entity distinction may be too ambiguous for a simple pattern.

problem
  |
  +--> exact form? ------------> rule / parser
  |
  +--> learnable decision? ----> statistical / neural model
  |
  +--> open-ended generation? -> generative model

The choice should reflect error cost, latency, throughput, and operational complexity.

29.2. Cascaded Processing

A high-volume system can use cheap stages to reduce the number of requests that reach expensive models.

input
 |
 v
cheap validation / rule
 |
 +--> certain result ------> output
 |
 +--> uncertain
        |
        v
     small model
        |
        +--> sufficient confidence --> output
        |
        +--> uncertain
               |
               v
            large model

The early stage must avoid destructive false rejection. A powerful later model cannot recover an example that was discarded too early.

29.3. Turkish Morphology Plus Contextual Models

A Turkish pipeline may combine explicit and learned components:

raw text
  |
  v
Unicode / orthographic normalization
  |
  v
morphological candidates
  |
  v
contextual neural model
  |
  v
domain rule / validation

Explicit morphology is not mandatory for every task. It becomes especially useful when error analysis reveals systematic failures around suffixes, derivation, or case marking.

29.4. Model Capacity Is Not System Capacity

A larger model may increase task quality while reducing concurrency and increasing queue delay.

Model selection should consider:

task quality
compute cost
latency
operational complexity

A small F1 improvement that halves production capacity needs a business justification.

29.5. Explainability and Auditability

Rule-based behavior is often directly traceable. Neural behavior is less explicit but can cover language variation more effectively.

A hybrid system can keep the final high-impact boundary deterministic:

model -> candidate
rule  -> allow / deny / constrain

This is particularly important when authorization, policy, or legal constraints must not be delegated to probabilistic inference.

Unit 30: Moving Evaluation Beyond a Single Score

The most dangerous simplification in model development is treating one number as the whole system result. NLP evaluation spans data partitioning, statistical uncertainty, application behavior, and user outcomes.

30.1. Train, Validation, and Test

Training data updates parameters. Validation data supports model and threshold choices. Test data is reserved for final evaluation.

Repeatedly inspecting the test set while making design decisions turns it into an implicit validation set.

30.2. Grouped and Temporal Splitting

Random row splitting can leak information.

Sentences from the same document, turns from the same conversation, or near-duplicate messages from one user can appear on both sides of the split.

unsafe:
rows -> random train/test

safer:
document / user / conversation group
        |
        +--> train
        +--> validation
        +--> test

For domains that change over time, a temporal split can better approximate future production data.

30.3. Ablation

When a new component improves the final score, the gain may be caused by interactions elsewhere. Ablation removes one component at a time.

full system
   |
   +--> remove A -> measure
   +--> remove B -> measure
   +--> remove C -> measure

For Turkish NLP, ablations might remove morphological features, ASCII normalization, reranking, or domain lexicons independently.

30.4. Statistical Significance

A small metric difference is not necessarily a real improvement. Sample size and error correlation affect uncertainty.

Confidence intervals, resampling, or task-appropriate significance tests are useful when research claims depend on small differences.

30.5. User Testing

A high classifier score does not guarantee a successful user-facing application.

Interactive systems may also measure:

• task completion,
• reformulation count,
• handoff rate,
• abandonment turn,
• end-to-end latency.

User testing complements model metrics rather than replacing them.

30.6. Language and Domain Slices

Aggregate scores hide localized failures.

Useful Turkish slices may include:

standard orthography
ASCII Turkish
ASR transcripts
proper-name-heavy text
code-switching
long suffix chains
spelling errors

A model can improve overall while regressing on a critical production slice.

Unit 31: Post-Deployment Drift, Maintenance, and Feedback

Deployment is not the end of NLP development. Language use, product names, organizations, user behavior, and adversarial patterns change.

31.1. Three Kinds of Drift

At least three changes should be distinguished.

Input drift: Surface-form distribution changes.

Label-prior drift: Class frequencies change.

Concept drift: The relation between an input pattern and the target label changes.

yesterday:
pattern X -> class A

today:
pattern X -> class B

The response to each drift type can be different. Retraining is not automatically the correct reaction.

31.2. Production Observation Without Storing All Text

Sensitive text does not need to be logged in full just to observe the system.

Useful aggregate signals include:

• input-length distribution,
• language estimate,
• unknown-token rate,
• confidence histogram,
• class frequency,
• error code,
• latency,
• queue age.

31.3. The feedback-data Trap

Corrected failures are valuable, but collecting only failures creates a new dataset unlike normal traffic.

A healthier feedback pool combines:

failure examples
+
random production sample
+
critical-class examples

31.4. Regression Sets

Retraining on new data can reintroduce old failures. Previously fixed cases should become permanent regression tests.

new release
   |
   +--> current benchmark
   |
   +--> historical regressions
   |
   +--> critical security cases
   |
   +--> performance test

Passing only the average quality metric is not enough.

31.5. Canary and Shadow Evaluation

A new model can first receive a small traffic fraction, or it can run in shadow while the existing model remains authoritative.

Shadow execution lets the new model see real input distribution without affecting user decisions. It is useful for comparing latency, error classes, and output disagreement at lower risk.

31.6. Rollback

Model, tokenizer, normalization, lexicon, and thresholds should be versioned together.

NLP_RELEASE =
    normalization version
  + tokenizer version
  + model version
  + lexicon version
  + decision threshold

Rolling back only the weight file may not restore the old behavior. Rollback should restore the entire compatible pipeline contract.

Unit 32: The Trust Boundary Between NLU and Generative Models

LLMs can perform classical NLU tasks from natural-language instructions, but that convenience does not remove the need for a system contract.

32.1. Constrained Output

Instead of asking a model to “analyze this text,” the expected structure can be explicit:

intent
entities
confidence
needs_review

The generated structure still needs validation.

LLM
 |
 v
candidate JSON / structure
 |
 v
schema validation
 |
 v
domain validation
 |
 v
business decision

32.2. The Model Proposes; Authorization Decides

A tool-using model may propose an operation such as closing an account. Authorization must not be inferred from the language-model output.

natural language
     |
     v
model -> proposed action
     |
     v
identity + authorization + policy
     |
     +--> allow
     |
     +--> deny

This boundary directly reduces the effect of prompt injection and erroneous inference.

32.3. RAG Has Multiple Correctness Layers

A retrieval-augmented generation system has at least three separate correctness questions:

1. Was the right evidence retrieved?
2. Did the model interpret that evidence correctly?
3. Is the generated claim actually supported by the evidence?

A single answer-level score is therefore insufficient for root-cause analysis.

32.4. Structured Extraction in Turkish

Turkish suffixes can encode relations:

Ali'nin raporu
Ankara'daki toplantı
şirketten gelen yazı

A model can identify the entity stem while losing possession, location, or source relations carried by morphology. Structured extraction should validate relations, not only entity strings.

32.5. Deterministic, small-model, and large-model Layers

Production NLP is not a competition to use one model everywhere.

deterministic layer
       |
       +--> format / safety / contract

small task model
       |
       +--> high-volume routine classification

large language model
       |
       +--> open-ended, context-heavy cases

Explicit responsibility boundaries can make cost, latency, and failure impact easier to control.

Unit 33: End-to-End Engineering Review for an NLU System

Before production deployment, an NLP/NLU system should be reviewed as a lifecycle rather than only as an accuracy report.

problem
  |
  v
task contract
  |
  v
data provenance and labels
  |
  v
representation / model
  |
  v
offline evaluation
  |
  v
limited real traffic
  |
  v
production
  |
  v
drift + feedback
  |
  +--------------------+
          reevaluation

A practical review can ask:

1. Does this problem actually require natural language?
2. Are input and output contracts explicit?
3. Are critical error types identified separately?
4. Is training-data provenance and permission known?
5. Are Turkish-specific normalization and morphology decisions documented?
6. Are train, validation, and test splits leakage-resistant?
7. Was a simple rule or classical model measured as a baseline?
8. Was each new component justified with ablation?
9. Are language and domain slices reported separately?
10. Is end-to-end latency measured for interactive use?
11. Is uncertain behavior explicitly defined?
12. Is authorization outside probabilistic inference?
13. Is sensitive text unnecessarily retained in logs?
14. Are model, tokenizer, normalization, and thresholds versioned together?
15. Are historical failures retained as regression tests?
16. Is rollback defined?
17. Is production drift observed?
18. Are synthetic and weakly labeled samples distinguishable from real data?
19. Are retrieval and generation errors evaluated separately in RAG?
20. Is model quality connected to task completion and operational value?

The point of this review is not to make every project heavier. It is to identify which layers are genuinely necessary and avoid unnecessary models, data processing, and operational complexity.

33.1. Relationship to Artificial Intelligence

NLP is one of the longest-running problem areas in artificial intelligence, but NLP and AI are not synonyms.

Artificial Intelligence: Philosophy, Theory and Practice provides the broader framework for discussing understanding, intelligence, agency, and consciousness. Fluent language generation is behavioral evidence of linguistic capability; it does not by itself establish human-like intention or consciousness.

Neural Networks and Learning Models provides the learning machinery behind embeddings, sequence models, attention, Transformers, and modern NLP. NLP adds the linguistic representation and task structure.

Statistical Learning and Machine Learning provides probability, classification, generalization, evaluation, HMMs, and distribution-shift concepts that remain relevant across both classical and neural NLP.

Logic Programming and Prolog connects NLP with symbolic grammar, semantic representation, and rule-based inference. Neural extraction followed by Prolog or formal constraint checking is a practical neuro-symbolic pattern.

Fuzzy Logic models graded concepts through membership values. Token probabilities from a language model are not fuzzy membership degrees; they answer different modeling questions.

Genetic Algorithms and Applications are not NLP models by themselves but may be used for external search problems such as feature selection or hyperparameter optimization.

Large Language Models develops the generative-model layer in detail.

NLP
  = language data + representation + task + evaluation + system

LLM
  = a powerful model family that can perform many NLP tasks

LLM != all of NLP

The engineering objective is not to choose one historical paradigm. It is to decide which parts of a language problem require explicit rules, which require statistical learning, and which require formal validation.

Unit 34: Selecting Embedding Models by Task

An embedding model maps text into a fixed-dimensional vector, but embedding models are not optimized for one universal objective. A model can perform well on semantic similarity and poorly on retrieval, while another model may be stronger for clustering or classification.

Text
 |
 v
Embedding model
 |
 v
Vector
 |
 +--> similarity
 +--> retrieval
 +--> clustering
 +--> classification features

Benchmarks such as MTEB show why task coverage matters: no single embedding method dominates all embedding tasks. Production systems must still evaluate their own workload.

34.1. Bi-encoders and cross-encoders

A bi-encoder embeds the query and document separately.

Query ----> Encoder ----> q
Document -> Encoder ----> d

score = similarity(q,d)

Document vectors can be precomputed, making this architecture suitable for large-scale retrieval.

A cross-encoder jointly processes the query and candidate document.

[query ; document]
        |
        v
   Transformer
        |
        v
 relevance score

Cross-encoders model richer token interactions but are more expensive. A common production design retrieves with a bi-encoder or BM25 and reranks a much smaller candidate set with a cross-encoder.

This distinction forms the NLP foundation of the retrieval pipeline described in Retrieval-Augmented Generation.

Unit 35: BERT, GPT, and T5 as Different Transformer Configurations

The Transformer is not one model type. Different information-flow patterns and pre-training objectives produce different model families.

35.1. BERT-style Encoders

BERT-like models use bidirectional encoder representations. Their pre-trained representations can support text classification, token classification, named-entity recognition, semantic similarity, and feature extraction.

35.2. GPT-style Decoders

Decoder-only GPT models use causal masking and predict future tokens from previous tokens.

x1 -> x2 -> x3 -> ?

This naturally supports generation and completion.

35.3. T5-style encoder-decoders

T5 expresses many NLP tasks using a unified text-to-text interface.

input text
   |
 encoder
   |
 decoder
   |
output text

Even classification can be expressed by generating the class label as text.

Simple slogans such as “BERT understands, GPT generates, T5 translates” are too rigid. Architecture, masking, and training objective provide the more accurate distinction.

Unit 36: Tokenizer Training Is an NLP Modeling Decision

Tokenization directly affects sequence length, context cost, rare-word behavior, vocabulary size, and the fragmentation of Turkish morphology.

BPE-style methods merge frequent symbol sequences. Unigram models estimate likely segmentations from a candidate vocabulary. SentencePiece provides a framework for learning these subword models directly from raw Unicode text.

A Turkish tokenizer should be evaluated on:

tokens per word
tokens per character
suffix-chain fragmentation
proper-name behavior
numbers and codes
ASCII Turkish
English technical terms

A tokenizer optimized around English can create unnecessarily long sequences for Turkish.

Model-token boundaries are not linguistic morpheme boundaries.

linguistic:
gel + ecek + ler + den

possible tokenizer:
gele + cekler + den

That segmentation may be computationally useful without being a morphological analysis.

Unit 37: Multilingual Embeddings and Cross-Lingual Retrieval

Multilingual embedding models can place semantically related text from different languages near each other.

"bağlantı havuzu"
       |
multilingual embedding
       |
       +---- close ----> "connection pool"

This is useful when Turkish queries must retrieve English technical documents.

Multilingual support does not guarantee equal quality across languages, scripts, names, or domain terminology. Cross-language retrieval should therefore be evaluated in each direction:

TR query -> TR document
TR query -> EN document
EN query -> TR document
EN query -> EN document

Unit 38: Text Classification vs Token Classification

Modern Transformer libraries can expose these tasks through similar APIs, but the modeling problem is different.

38.1. Text Classification

One label is produced for an entire sentence or document.

"This message promises a fake reward."
             |
             v
            SPAM

38.2. Token Classification

Labels are assigned to individual tokens or spans.

"Ali traveled to Ankara."
 Ali    -> PERSON
 Ankara -> LOCATION

Named-entity recognition is a common token-classification task.

Subword tokenization complicates the mapping between model tokens and user-visible entity spans. In Turkish, suffixes attached to proper names make this especially important. Post-processing must define how subword labels are merged back into linguistic spans.

Unit 39: Benchmarking Embeddings on Your Own Corpus

A public leaderboard is a starting point, not a deployment decision.

A small gold retrieval set can contain:

query
expected_relevant_documents
hard_negatives
language
domain

Useful metrics include Recall@K, MRR, nDCG, language-specific slices, name/technical-term slices, latency, and embedding cost.

Hard negatives are superficially plausible but incorrect candidates and provide a stronger test of retrieval discrimination.

Turkish RAG research further demonstrates that retrieval and reranking choices need language-specific evaluation rather than direct transfer of English-centric assumptions.

Unit 40: Deterministic and Hybrid NLU Architecture

Natural-language understanding does not always require a large generative model. In bounded domains with a known action space, strict latency requirements, privacy constraints, and a need for reproducibility, a strong solution can combine deterministic rules, information retrieval, fuzzy matching, and dialogue state.

This is especially useful when the system requires offline operation, auditable routing, low latency, explicit intent boundaries, and authorization-controlled actions.

40.1. Do Not Reduce NLU to One Classifier

An input can be processed through progressively more flexible layers:

input
  ↓
normalization
  ↓
exact match
  ↓ otherwise
pattern match
  ↓ otherwise
conversation context
  ↓ otherwise
domain grammar
  ↓ otherwise
TF-IDF / cosine retrieval
  ↓ if still ambiguous
fuzzy matching
  ↓
fallback

The order matters. High-precision and highly explainable mechanisms run first; flexible similarity mechanisms are used only when deterministic paths fail.

40.2. Normalization Is an information-loss Decision

Turkish NLU may normalize case, Unicode representation, whitespace, punctuation, and selected spelling variation.

However, normalization can destroy information. The Turkish distinction among I, İ, ı, and i is meaningful.

A useful design therefore separates:

normalized form for matching
+
original form for display and evidence

40.3. Exact Matching for high-precision Intents

Some phrases are part of the product language itself:

"help"
"summarize"
"next step"
"show sources"

Exact matching is fast, deterministic, explainable, and easy to regression-test.

A visible UI action and the same phrase typed by the user should route to the same intent.

40.4. Pattern Matching

Regular expressions or explicit patterns can capture controlled variation.

Examples:

"show the last 7 days"
"inspect the last 30 days"
"return the first 5 results"

The number changes, but the underlying intent remains stable.

Pattern matching provides explicit ownership of surface forms, but the pattern set must be tested for overlap as it grows.

40.5. Short Contextual follow-ups

Users naturally say:

"why?"
"which one?"
"expand that"
"what are the limits?"

These phrases may be ambiguous without the previous turn.

A bounded system does not necessarily need open-ended coreference resolution. It can use explicit rules such as:

is there a meaningful previous intent?
is the new utterance short?
is a finding currently selected?

The limitation must remain explicit: this is controlled dialogue reference resolution, not general linguistic coreference.

40.6. Domain Grammar

Some applications contain a small command language:

[action] + [object] + [time]
[compare] + [period A] + [period B]
[show] + [top N] + [criterion]

A domain grammar can represent these structures directly.

Its advantages are explainability, low latency, direct regression testing, and strong control over false positives.

40.7. Intent Retrieval with TF-IDF and Cosine Similarity

If an input is close to known example utterances, a classical retrieval layer can be effective.

Treat each intent's example utterances as a document. TF-IDF weights terms according to their discriminative value:

tfidf(t, d) = tf(t, d) × log(N / df(t))

Then compare the query vector with intent vectors using cosine similarity:

cos(q, d) = (q · d) / (||q|| × ||d||)

This is less semantically flexible than a modern embedding model, but it can be highly useful for small intent sets, distinctive domain vocabulary, offline deployment, and deterministic behavior.

40.8. Do Not Trust top-1 Alone

The best score does not tell us how ambiguous the decision is.

Compare:

intent A: 0.61
intent B: 0.59

with:

intent A: 0.61
intent B: 0.18

The top score is identical, but the uncertainty is very different.

A router can therefore use both:

top-1 score
+
top-1 minus top-2 margin

Acceptance can require either a high absolute score or a moderate score with a strong margin.

40.9. Fuzzy Matching as a Late Safety Net

Fuzzy string matching helps with spelling errors and small surface-form differences.

It must not be confused with fuzzy logic:

fuzzy string matching:
  how similar are two strings?

fuzzy logic:
  graded membership and approximate reasoning

Because fuzzy matching is permissive, it belongs late in the routing cascade and should also use an ambiguity margin.

40.10. Match Provenance

The NLU layer can retain not only the selected intent but also how it was selected:

intent = X
provenance = EXACT
score = 1.0

or:

intent = Y
provenance = RETRIEVAL
score = 0.73
margin = 0.21

This supports debugging, regression analysis, threshold tuning, and explanation.

40.11. Separate Intent and Entity Extraction

For:

"compare the last 14 days"

there are two outputs:

intent:
  compare

entity:
  period = 14 days

They may be produced by one model, but treating them as separate responsibilities improves testability.

In a bounded domain, deterministic entity extraction can cover quantities, time units, ordinals, quoted text, explicit identifier forms, and controlled demonstrative references.

40.12. Dialogue State

Single-turn NLU is not the same as a conversational system.

Dialogue state may track:

previous intent
previous result
selected item
active period
active entity
response mode

A crucial rule is that a conversational turn that produces no new analytical result should not unnecessarily reset the user's current selection.

40.13. Separate Intent from Authority

Correctly understanding:

intent = open record

does not imply that the action is permitted.

A safer flow is:

NLU
 ↓
intent
 ↓
capability / authorization check
 ↓
tool invocation

The language layer therefore does not become a security boundary.

40.14. Tool Contracts

A conversational system becomes brittle if every tool returns a different arbitrary shape.

A small contract can separate:

reply    -> user-facing text
data     -> structured result
context  -> state to preserve for later turns

This keeps tool integration predictable.

40.15. Separate the Retrieval Corpus from Business Data

The corpus used for intent retrieval does not need to contain operational records.

A safer architecture separates:

NLU retrieval corpus:
  intent examples
  help text
  general knowledge

business data:
  separate authorization boundary
  separate query / analysis layer

This improves privacy, authorization clarity, and lifecycle management.

40.16. Deterministic and Generative Layers Can Coexist

A hybrid design can use:

high-precision deterministic layer
        ↓
small classifier / retrieval if needed
        ↓
generative model if needed
        ↓
constrained structured output
        ↓
authorization check

The generative model does not have to be the default route for every input.

For high-impact actions, deterministic routing can constrain the permitted intent set and prevent free-form model output from becoming an executable command.

40.17. The Regression Corpus Is Part of the NLU System

An NLU system is defined not only by training examples but also by its behavioral regression corpus.

At minimum, include:

positive cases:
  must route to the intended intent

negative cases:
  must not route to a particular intent

collision cases:
  two high-precision intents must not own the same utterance

Also test short contextual follow-ups, misspellings, neighboring intents, nonsense input, threshold boundaries, and top-1/top-2 margins.

40.18. Synthetic Example

Suppose the intent set is:

SUMMARIZE
ANALYZE_TIME
ANALYZE_NETWORK
SHOW_EVIDENCE
NEXT_STEP

The user says:

"inspect the bridges in the connection graph"

A pattern or domain grammar can route this to ANALYZE_NETWORK.

The next turn:

"why?"

is not necessarily a new independent request. Dialogue context can resolve it against the current analytical finding.

For:

"xyz qqq"

the system should prefer fallback over forcing the nearest intent.

A good NLU system must be able to remain uncertain.

40.19. Engineering Checklist

1. Does normalization preserve language-specific information?
2. Are high-precision commands protected by exact matching?
3. Can patterns collide?
4. Is context used only when a meaningful previous turn exists?
5. Is the order of grammar and retrieval explicit?
6. Does retrieval use both score and margin?
7. Can fuzzy matching force low-confidence input into an intent?
8. Can intent and entities be tested independently?
9. Is match provenance observable?
10. Can dialogue state mix stale and current results?
11. Can the NLU layer expand execution authority?
12. Is business data unnecessarily indexed inside the intent-retrieval corpus?
13. Do tools follow a stable contract?
14. Does the same input and state produce the same route?
15. Are negative and collision tests as strong as positive tests?

This architecture turns NLU from a model-selection exercise into a combined problem of language processing, state management, security, and software verification.

Unit 41: Span-Preserving and Compositional NLU

Normalization is useful in natural-language processing, but aggressive normalization can irreversibly change critical parts of user input. This matters when a request contains identifiers, numbers, dates, durations, links, file names, or domain terminology.

A safer token representation can preserve several views:

raw       = exact user text
canonical = normalized form for linguistic matching
folded    = tolerant search form
start/end = character span in the original input

Matching can therefore be flexible while critical parameter binding remains faithful to the original text.

41.1. Protected Literals

Not every token should be equally editable.

Protected literal categories can include numeric identifiers, dates and times, URLs, file paths, quoted strings, and code or version values.

Fuzzy correction or morphological rewriting of such values can silently convert one real identifier into another. Linguistic tolerance and data integrity require different rules.

41.2. Combine Morphology with Domain Knowledge

Agglutinative languages create many surface forms from one concept. A Turkish system can combine general tokenization, optional morphological analysis, domain vocabulary, and controlled suffix/stem variants.

A morphological analyzer provides linguistic evidence; its output should not become an execution authorization signal by itself.

41.3. Informal Language and ASCII Tolerance

Real input contains spelling variation and ASCII forms. Tolerant matching can support these variants while retaining the original text.

For Turkish, case folding is particularly sensitive because i, ı, I, and İ are not interchangeable in all contexts. Canonical and folded forms should therefore remain distinct.

41.4. Negation Is More Than an Intent Feature

Commands such as open/do not open, show/do not show, and play/do not play can be lexically similar while being operationally opposite.

For side-effecting actions, negation can produce an independent veto signal:

action detected = true
negation detected = true
=> invoke = false

This does not replace authorization. It adds a linguistic safety layer against false-positive routing.

41.5. Numbers and Ordinals

Conversational analytics often uses references such as first, second, last three, the first two, or evidence for the third.

Extracting ordinal = 2 is not enough. The system also needs the referenced result set:

ordinal = 2
referenceSet = CURRENT_FINDINGS

Reference resolution belongs to dialogue state rather than number parsing alone.

41.6. Semantic Segmentation

One request may contain several dependent operations:

"analyze the recent period, compare it with the previous period, and show evidence for the two strongest findings"

Compressing this into one intent name loses structure.

A bounded compositional parser can derive:

ANALYZE(period=current)
COMPARE(period=previous)
RANK(limit=2, metric=strength)
OUTPUT(evidence)

Simple one-action messages should remain on the simpler route.

41.7. Semantic Atoms

Segments can produce small domain-oriented semantic atoms:

ACTION=COMPARE
SUBJECT=RELATION
QUALIFIER=RECENT
METRIC=STRENGTH
LIMIT=2
OUTPUT=EVIDENCE

These atoms are not executable code. They are an intermediate representation for a planning layer.

41.8. Character n-grams and Word Bigrams

Word-level retrieval can be brittle under inflection and minor spelling errors.

Character 3–5-gram features can improve tolerance to inflectional variation, typos, and ASCII variants. Word bigrams preserve phrases such as “new relation,” “last week,” or “first two.”

Neither feature should be used to repair protected numeric identifiers.

41.9. Multi-feature Routing

An intent candidate can combine exact matching, patterns, domain grammar, TF-IDF cosine similarity, character n-grams, word bigrams, controlled fuzzy similarity, and conversation context.

Routing can then consider top1, top2, margin, and risk class and produce:

ACCEPT
CLARIFY
ABSTAIN / FALLBACK

Side-effecting intents should normally require a more conservative acceptance policy.

41.10. Clarification Is a Valid Result

If several entities share a label, the system should not silently choose the first.

one candidate  -> resolve
several        -> clarify
none           -> unavailable

41.11. Bound Compositional Plans

NLU output can feed a structured query or action plan, but the plan should bound the number of dependent steps, fan-out, and executable action types.

This is simultaneously a performance, safety, and verification concern.

Span-preserving compositional NLU keeps natural-language flexibility while treating critical literals, negation, context, and execution boundaries as separate contracts.

41.12. Decomposing Error Sources in an NLP Pipeline

A language system is a pipeline spanning tokenization, representation, retrieval, context selection, inference, and evaluation. In an NLP pipeline, retrieval components primarily expose recall and ranking failures, while generative components can introduce unsupported inferences or contradictions; fluency should therefore remain separate from factual correctness, coverage, calibration, and consistency with evidence. Agentic or tool-using systems add a new latency and failure boundary at every extra step.

• Evaluate tokenization, retrieval, and model-inference errors separately.
• Choose macro versus micro metrics according to the class distribution.
• Control domain shift and data leakage in linguistic evaluation sets.

Unit 42: Text Mining and Knowledge Discovery from Document Collections

Text mining is not a synonym for natural language processing. NLP provides methods for analyzing linguistic structure; information retrieval finds relevant documents or passages; machine learning learns decision functions from examples; text mining focuses on discovering collection-level patterns, relationships, and trends from large bodies of text. These areas can coexist in one system.

42.1. Documents, Collections, and Intermediate Representations

The analytical unit in text mining is often a document collection rather than an isolated sentence. A collection may be:

• static, with a stable analysis set,
• dynamic, with documents added, modified, or re-scoped over time.

"Unstructured text" is not literally structure-free. Headings, paragraphs, tables, punctuation, dates, authorship, section boundaries, and word order may carry implicit or semi-structured information. Production systems therefore usually transform raw text into an explicit intermediate representation before mining:

raw document
   ↓
parsing and normalization
   ↓
feature / term / concept representation
   ↓
collection-level analysis

42.2. Character, Word, Term, and Concept Levels

A document may be represented at different abstraction levels:

character → word → term → concept

Character n-grams can tolerate spelling variation. Words preserve surface forms. Terms represent meaningful single- or multiword units. Concepts can map several surface forms to a higher-level semantic identity.

Text collections create two important computational properties:

1. high dimensionality, because the possible feature space is large,
2. sparsity, because any one document contains only a small fraction of those features.

More features therefore do not automatically produce a better representation. Semantic coverage and computational cost must be balanced.

42.3. A Functional text-mining Architecture

A general system can be organized as:

document sources
      ↓
preprocessing
      ↓
canonical / intermediate representation
      ↓
core mining operations
      ↓
refinement and constraints
      ↓
presentation / browsing
      ↺
user query and revised constraints

The user is not merely a consumer of the final output. Discovery over large collections is commonly an iterative query → inspect → refine → query again process.

42.4. Collection-level Pattern Types

Text mining analyzes distributions across a corpus, not only the meaning of individual documents.

For a concept set X, simple support can be written as:

support(X) = |{d ∈ D : X ⊆ d}| / |D|

This supports several analytical views:

• distribution: how a concept occurs across collection subsets,
• frequent concept set: concepts co-occurring above a support threshold,
• near-frequent concept set: concept sets with strong overlap or similarity without exact co-occurrence,
• association: a directional or conditional co-occurrence tendency between concept sets.

Co-occurrence is evidence of textual association, not causation:

co-occurrence ≠ causality

42.5. Pattern Overabundance and Interestingness

High-dimensional corpora can produce thousands of technically valid patterns. The engineering problem is therefore not only discovery but also the creation of a manageable and meaningful result set.

Useful mechanisms include:

• minimum support,
• ranking,
• suppression,
• pruning,
• grouping,
• contextual constraints,
• result diversification.

A computable pattern is not automatically a report-worthy finding.

42.6. Dynamic Corpora and Temporal Patterns

With reliable timestamps, the same concept distribution can be compared across periods:

D(t1) → concept distribution
D(t2) → concept distribution
             ↓
           change

This supports emerging, decreasing, new, and disappearing concepts, as well as temporal deviation and transient-association candidates.

Recomputing an entire large corpus after every update may be unnecessarily expensive. Incremental methods aim to reuse valid prior state and process only the effect of changed data where correctness permits. This is not permission to trade correctness for speed; the validity conditions of reused state must remain explicit.

42.7. Background Knowledge and Controlled Concept Spaces

Lexicons, ontologies, taxonomies, and controlled vocabularies can support:

• synonym normalization,
• polysemy control,
• concept hierarchies,
• extraction validation,
• contextual query constraints.

Background knowledge does not replace source evidence. An ontology cannot make an unsupported claim true; it can make representation, matching, and query constraints more consistent.

42.8. Information Extraction: from Text to Structured Facts

A typical information-extraction pipeline is:

tokens / linguistic features
          ↓
entity extraction
          ↓
coreference resolution
          ↓
relation / event extraction
          ↓
normalized facts

Coreference is especially important. If multiple expressions referring to the same real-world entity are not reconciled, a downstream relation graph may fragment or duplicate evidence.

42.9. Rule-based, Probabilistic, and Hybrid Extraction

Information extraction has moved through hand-authored patterns; probabilistic approaches such as HMMs, maximum-entropy models, MEMMs, and CRFs; and later neural and Transformer-based models. The engineering lesson is not that one generation is universally superior, but that different data and error profiles justify different methods.

A hybrid system may combine:

high-precision rules
        +
statistical scoring
        +
domain constraints

Bootstrapping can expand a small set of high-confidence seeds, but erroneous seeds can amplify into semantic drift, so every iteration needs a quality gate.

42.10. A Rule DSL for Information Extraction

Rule-based extraction need not depend on a particular legacy language. A general DSL can expose:

CONCEPT
  attributes

RULE
  context
  pattern
  constraints
  actions

GUARD
  invariants

pattern generates a candidate, constraints decide whether a match is acceptable, and guard protects central invariants of the produced concept. This separates match logic from reusable validation.

42.11. Different Domains, Similar Pipeline

Finance, patents, biomedical literature, or security text require different vocabularies and relation types, yet their high-level architecture is often similar:

domain corpus
    ↓
domain lexicon / ontology
    ↓
entity and relation extraction
    ↓
pattern / trend analysis
    ↓
graph / visualization
    ↓
human validation

A robust text-mining system is therefore not only a model-selection problem. It combines representation, provenance, bounded search, refinement, presentation, and human review.

42.12. Coverage-aware Extractive Summarization of Long Documents

The first question in document summarization is not which sentence receives the highest score, but which parts of the information space must remain represented. A topic repeated near the beginning of a conversation can overshadow a consequential decision or time change mentioned only once near the end. Long text is therefore divided into sections by document structure, speaker turns, or reliable time intervals. Candidate sentences are selected within each section; a second selection across sections accounts for both relevance and coverage.

A sparse sentence-similarity graph need not compare every sentence with every other sentence. Lexical representations can first identify a bounded neighborhood, and centrality can be computed over those edges. Diversity selection downweights highly ranked sentences that repeat information already chosen. However, a penalty based only on surface similarity may collapse statements that differ in negation or temporal direction. Mood, polarity, speaker attribution, and time must remain available during sentence selection.

A summary cannot be more complete than the source material examined. If retrieval stops at a document count, character budget, or time interval, the result must be marked partial. For every chosen sentence, the exact source quotation, document identifier, character span, and available genuine audio times are retained. This ability to return to the source goes beyond a bibliography entry: each information unit in the summary can be audited against a specific passage.

Qualitative thematic analysis does not equate word frequency with the interpretation of a theme. The process first fixes a research question and unit of analysis, assigns initial codes to text spans, merges related codes under explicit criteria and challenges emergent themes with contrasting cases. The same phrase can have different meanings in different contexts, which limits automatic labeling based solely on embedding similarity. A model-proposed code is not a substitute for researcher judgment: the supporting span, code definition and rationale need to be retained. Annotation disagreements should inform revisions to the codebook rather than being hidden as noise.

42.13. Morphology, Negation, and Action Boundaries in Turkish Command Interpretation

Turkish surface forms can combine a stem with multiple suffixes. For analytic query interpretation, case conversion, Unicode normalization and morphological analysis are distinct stages. Language-neutral case folding that ignores the I/ı versus İ/i distinction can collapse different lexical forms into one key. Tolerance for spelling errors must not automatically grant permission to execute actions.

Detecting a named entity does not prove that a user authorized an operation. The Turkish utterances 'Kaydı açma' (do not open the record) and 'Kaydı aç' (open the record) share roots but imply opposite actions. 'Hangi kayıtlar açılmadı?' (which records were not opened?) is a query about state, not an instruction to open anything. Negation scope, interrogative intent and imperative mood must be resolved before action routing.

A test corpus should contain semantic paraphrases and minimal contrasting pairs: positive/negative, question/command, singular/plural, temporal context and references to a new entity independent of the current selection. Aggregate accuracy is insufficient; false-positive action rate, abstention and unnecessary refusal require separate reporting. Under ambiguity, a safe default is a source-grounded query or a request for clarification.

42.14. Evidence and Context Validity in Deterministic Routing

A deterministic analytic router need not execute a tool immediately after obtaining a linguistic similarity score. Entity resolution, time bounds, data coverage, current authorization and tool parameter contracts remain independent checks. A pronoun or elliptical utterance referring to previous dialogue does not prove that the same entity remains valid in the current context.

An interpretation must be invalidated when its supporting context version changes. Rule-based and statistical routers can be compared on the same corpus for wrong-tool selection, incorrect entity binding and ambiguity handling. Classifier confidence scores should not be confused with empirically calibrated probabilities.

References

• Alec Radford et al. “Robust Speech Recognition via Large-Scale Weak Supervision.” 2022. https://arxiv.org/abs/2212.04356

• Alexis Conneau et al. “Unsupervised Cross-lingual Representation Learning at Scale.” 2019. https://arxiv.org/abs/1911.02116

• Ashish Vaswani et al. “Attention Is All You Need.” NeurIPS, 2017. https://arxiv.org/abs/1706.03762

• Brendan Gregg. Systems Performance: Enterprise and the Cloud. Pearson Education, 2014.

• Chris Fregly. AI Systems Performance Engineering. O’Reilly Media, 2025.

• Daniel Jurafsky; James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. 3rd edition online manuscript, August 19, 2026. https://web.stanford.edu/~jurafsky/slp3/

• Deborah A. Dahl. Natural Language Understanding with Python. Packt Publishing, 2023. ISBN 978-1-80461-342-9.

• Dzmitry Bahdanau; Kyunghyun Cho; Yoshua Bengio. “Neural Machine Translation by Jointly Learning to Align and Translate.” ICLR, 2015. https://arxiv.org/abs/1409.0473

• Fernando C. N. Pereira; David H. D. Warren. “Definite Clause Grammars for Language Analysis.” Artificial Intelligence, 13(3), 1980, 231–278.

• Frederick Jelinek. Statistical Methods for Speech Recognition. MIT Press, 1997.

• Haşim Sak; Tunga Güngör; Murat Saraçlar. “Resources for Turkish Morphological Processing.” Language Resources and Evaluation, 45(2), 2011, 249–261. DOI: https://doi.org/10.1007/s10579-010-9128-6

• Ilya Sutskever; Oriol Vinyals; Quoc V. Le. “Sequence to Sequence Learning with Neural Networks.” NeurIPS, 2014.

• Jacob Devlin; Ming-Wei Chang; Kenton Lee; Kristina Toutanova. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” 2018. https://arxiv.org/abs/1810.04805

• Jeffrey Pennington; Richard Socher; Christopher D. Manning. “GloVe: Global Vectors for Word Representation.” EMNLP, 2014. https://aclanthology.org/D14-1162/

• John Lafferty; Andrew McCallum; Fernando Pereira. “Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.” ICML, 2001.

• Kemal Oflazer. “Two-level Description of Turkish Morphology.” Literary and Linguistic Computing, 9(2), 1994, 137–148. DOI: https://doi.org/10.1093/llc/9.2.137

• Kemal Oflazer; Murat Saraçlar (eds.). Turkish Natural Language Processing. Springer, 2018. DOI: https://doi.org/10.1007/978-3-319-90165-7

• Manning, Christopher D.; Raghavan, Prabhakar; Schütze, Hinrich. Introduction to Information Retrieval. Cambridge University Press, 2008. https://nlp.stanford.edu/IR-book/

• Marie-Catherine de Marneffe; Christopher D. Manning; Joakim Nivre; Daniel Zeman. “Universal Dependencies.” Computational Linguistics, 47(2), 2021, 255–308. DOI: https://doi.org/10.1162/coli_a_00402

• Marta R. Costa-jussà et al. “No Language Left Behind: Scaling Human-Centered Machine Translation.” 2022. https://arxiv.org/abs/2207.04672

• Muennighoff, N.; Tazi, N.; Magne, L.; Reimers, N. “MTEB: Massive Text Embedding Benchmark.” 2022. https://arxiv.org/abs/2210.07316

• Piotr Bojanowski et al. “Enriching Word Vectors with Subword Information.” Transactions of the Association for Computational Linguistics, 5, 2017, 135–146. https://aclanthology.org/Q17-1010/

• Raffel, C. et al. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.” Journal of Machine Learning Research, 21(140), 2020. https://jmlr.org/papers/v21/20-074.html

• Reimers, N.; Gurevych, I. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP-IJCNLP, 2019. https://arxiv.org/abs/1908.10084

• Rico Sennrich; Barry Haddow; Alexandra Birch. “Neural Machine Translation of Rare Words with Subword Units.” ACL, 2016. https://aclanthology.org/P16-1162/

• Ronen Feldman; James Sanger. The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Cambridge University Press, 2007.

• Stanley F. Chen; Joshua Goodman. “An Empirical Study of Smoothing Techniques for Language Modeling.” Harvard Computer Science Group Technical Report TR-10-98, 1998.

• Steven Bird; Ewan Klein; Edward Loper. Natural Language Processing with Python. O’Reilly Media, 2009. ISBN 978-0-596-51649-9.

• Taku Kudo; John Richardson. “SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing.” EMNLP System Demonstrations, 2018. DOI: https://doi.org/10.18653/v1/D18-2012

• Tomas Mikolov et al. “Efficient Estimation of Word Representations in Vector Space.” 2013. https://arxiv.org/abs/1301.3781

• Universal Dependencies. UD Turkish BOUN. https://universaldependencies.org/treebanks/tr_boun/

• Universal Dependencies. UD Turkish IMST. https://universaldependencies.org/treebanks/tr_imst/

• Utku Türk et al. “Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Tool.” Language Resources and Evaluation, 56, 2022, 259–307. DOI: https://doi.org/10.1007/s10579-021-09558-0