Statistical Learning and Machine Learning: Classification, Regression, and Clustering

Statistical learning estimates relationships, decision boundaries, probability distributions, or latent structure from observed examples. Instead of encoding every decision as a hand-written rule, it represents regularities in data through a model. Its mathematical foundation is developed in Probability and Statistics; the focus here is how that foundation becomes classification, regression, clustering, dimensionality reduction, and anomaly detection.

Unit 1: Foundations of Statistical Learning

Data, models, and learning

A learning problem can be described through observations, features, and targets. An observation x is the feature vector supplied to the model; y is the target or label when one exists.

data
↓
feature representation
↓
model family
↓
objective / loss
↓
parameter estimation or search
↓
prediction on unseen data

The goal is not to memorize training examples but to capture structure that remains useful on new observations. Training performance and real-world performance are therefore different quantities.

Statistical inference and prediction

Classical statistics often emphasizes inference about parameters, effects, and uncertainty. Machine learning often shifts emphasis toward predictive performance on unseen data. The boundary is not sharp: regression, Bayesian estimation, sampling, likelihood, and regularization are shared tools.

A model that explains variation well is not automatically the best predictor, and a strong predictor does not by itself establish a causal relation.

Parametric and non-parametric methods

A parametric method chooses a model form and estimates a finite parameter set. Linear and logistic regression are examples.

A non-parametric method makes a weaker fixed-form assumption rather than having no assumptions at all. k-Nearest Neighbors, kernel density methods, and several tree methods fall in this broad family. Distance, neighborhood, smoothness, or splitting criteria still provide inductive bias.

Generative and discriminative models

A generative model represents a joint distribution or the process that generates data conditional on a class:

p(x, y)
or
p(x | y) p(y)

Naive Bayes is a classic example.

A discriminative model focuses directly on the target conditional on the input or on a decision function:

p(y | x)
or
f(x) -> y

Logistic regression and SVMs are typical examples.

Learning paradigms

• Supervised learning: labeled x -> y examples; classification and regression are the main tasks.
• Unsupervised learning: no target label; clustering, density estimation, and some forms of dimensionality reduction search for structure in the inputs.
• Semi-supervised learning: a small labeled set is combined with a larger unlabeled set.
• Self-supervised learning: targets are derived from the structure of the data itself.
• Reinforcement learning: an agent learns a policy from rewards produced by interaction with an environment; it is a different sequential decision problem from the main focus of these notes.

Unit 2: Representation, Scale, and Distance

The data matrix

A data set is commonly represented with observations in rows and features in columns:

X ∈ R^(n×p)

where n is the number of observations and p the number of features. A target vector y is stored separately when present.

A categorical variable encoded with numbers does not thereby gain arithmetic meaning. Encoding 1=Ankara, 2=Istanbul does not imply that Istanbul is twice Ankara. Measurement level affects the valid notions of distance and the assumptions a model can make.

Scaling

Distance- and margin-based methods are sensitive to scale. If one feature is measured in kilometers and another lies in [0,1], the large-scale variable can dominate raw Euclidean distance.

Standardization is commonly written as:

z = (x - μ) / σ

Min-max scaling maps values into a selected range. Transformation parameters must be learned from the training data only and then applied unchanged to validation and test data. Fitting a scaler to the full data set before splitting leaks information.

Similarity and dissimilarity

For numeric data the Minkowski distance is:

d(x,y) = (Σ |x_i-y_i|^p)^(1/p)

with Euclidean distance at p=2 and Manhattan distance at p=1.

Sparse text and high-dimensional vectors often use cosine similarity:

cos(x,y) = (x·y) / (||x|| ||y||)

Nominal features may require matching-based distances; binary features may use Jaccard similarity. The distance measure is part of the model because it defines what “near” and “similar” mean.

Neighborhoods in high dimensions

As dimension grows, data become sparse and pairwise distances can become less distinctive. This family of effects is known as the curse of dimensionality. k-NN, density estimation, and clustering are especially exposed. Feature selection and dimensionality reduction can therefore change model geometry rather than merely reduce runtime.

Unit 3: Objectives, Likelihood, and Generalization

Empirical risk

For model f_θ and per-example loss L, a common training objective is:

R_emp(θ) = (1/n) Σ L(y_i, f_θ(x_i))

Minimizing training loss is only a proxy for the expected error on new data.

Maximum likelihood

For a probabilistic model p(x | θ) or p(y | x, θ), maximum likelihood chooses parameters that make the observed data most probable:

θ_MLE = argmax_θ Π p(z_i | θ)

Log-likelihood is normally used for numerical stability:

θ_MLE = argmax_θ Σ log p(z_i | θ)

MAP estimation

With a prior p(θ), maximum a posteriori estimation becomes:

θ_MAP = argmax_θ p(data | θ) p(θ)

Several forms of regularization can be interpreted through corresponding parameter priors.

Bias and variance

A model that is too rigid may have high bias and miss real structure. A model that is too flexible may have high variance and follow sampling noise. Model selection balances these sources of error under the available data and operational costs.

No Free Lunch

No single learning algorithm is uniformly best over every possible problem. Performance comes from matching the assumptions of a method to the structure of the data. Comparisons should therefore use the same data split, metrics, and computational budget.

Unit 4: Regression and Probabilistic Classification

Linear regression

A simple linear model is:

Y = β0 + β1 X + ε

and multiple regression extends it to:

Y = β0 + β1X1 + ... + βpXp + ε

Least squares minimizes:

min_β Σ (y_i - ŷ_i)^2

Coefficient interpretation, residual structure, multicollinearity, and influential observations should be read together with the inferential treatment in Probability and Statistics.

Polynomial regression

“Linear” in linear regression refers to linearity in the parameters, not necessarily in the raw input. For example:

Y = β0 + β1X + β2X^2 + ε

is still linear in its coefficients. Higher degree increases flexibility and also the risk of overfitting.

Logistic regression

For binary classification:

z = β0 + β^T x

can be mapped to a probability with the sigmoid:

p(y=1|x) = 1 / (1 + e^(-z))

A 0.5 threshold is not mandatory; it should reflect false-positive and false-negative costs. Despite its name, logistic regression is principally a classification method.

Ridge, Lasso, and Elastic Net

Ridge:

min_β SSE + λ Σ β_j^2

Lasso:

min_β SSE + λ Σ |β_j|

Elastic Net combines both penalties. Ridge shrinks coefficients; Lasso can drive some coefficients to zero and produce sparse solutions. Penalty strength should be selected by validation rather than chosen from training fit alone.

PCR and PLS

Principal Components Regression first maps predictors to PCA components and then performs regression. Since PCA does not use y, high-variance directions are not guaranteed to be the most predictive ones.

Partial Least Squares constructs components using covariance with the target as well as predictor structure. Both can be useful when predictors are numerous and strongly correlated.

Other regression families

• Support Vector Regression (SVR): extends margin ideas to continuous targets.
• Gaussian Process Regression (GPR): places a Gaussian-process prior over functions and can provide predictive uncertainty as well as a mean prediction; the kernel defines the covariance structure.
• Ordinal regression: for ordered categorical outcomes.
• Poisson regression: for count outcomes under a Poisson mean structure.
• Negative-binomial regression: can model overdispersed counts when Poisson assumptions are too restrictive.

The model family should match the nature of the target and the error process.

Unit 5: k-Nearest Neighbors

Core idea

k-Nearest Neighbors (k-NN) predicts a new observation from the nearest k training examples. Because it does not fit a complex global parametric model during training, it is often called instance-based or lazy learning.

Classification uses a local vote:

ŷ = majority class among the k nearest neighbors

Regression can average the target values of the neighbors.

Choosing k

k=1 creates a very local boundary and can be sensitive to noise. Large k values smooth the prediction but can erase genuine local structure. k is a hyperparameter and should be selected by validation.

Weighted voting

Closer neighbors can receive larger weight, for example:

w_i = 1 / (d_i + ε)

with explicit handling for zero distance.

Computational cost

A naive query compares the new point with all training examples. KD-trees and ball trees can help at low or moderate dimension, but high-dimensional indexing deteriorates. Approximate-nearest-neighbor methods provide a different speed–accuracy trade-off for large vector spaces.

Strengths and limitations

k-NN is simple, locally explainable, and can express nonlinear boundaries. It is also:

• scale sensitive,
• sensitive to irrelevant features,
• vulnerable to the curse of dimensionality,
• potentially expensive at query time,
• affected by class imbalance.

Preprocessing is therefore part of the algorithmic design.

Unit 6: Bayesian Classification and Naive Bayes

Bayes' theorem

For class C and observed features x:

P(C | x) = P(x | C) P(C) / P(x)

When comparing classes, the common denominator can be omitted:

P(C | x) ∝ P(x | C) P(C)

P(C) is the prior, P(x|C) the likelihood, and P(C|x) the posterior.

The naive assumption

Naive Bayes assumes conditional independence of the features given the class:

P(x1,...,xp | C) = Π P(x_j | C)

The assumption is rarely exact, yet the classifier can be a strong baseline, especially for sparse high-dimensional text data.

Gaussian, Multinomial, and Bernoulli variants

• Gaussian Naive Bayes: continuous features modeled with class-conditional Gaussian distributions.
• Multinomial Naive Bayes: count features such as token frequencies.
• Bernoulli Naive Bayes: binary feature presence/absence.

The representation determines which variant is appropriate.

Zero counts and Laplace smoothing

An unseen token can make a raw product probability zero. Additive smoothing uses:

P(w|C) = (count(w,C) + α) / (Σ count(.,C) + α|V|)

with α=1 for classical add-one smoothing.

Log-space computation

Products of many small probabilities can underflow numerically. The equivalent log-space score is:

log P(C|x) ∝ log P(C) + Σ log P(x_j|C)

Spam classification

Naive Bayes with token or n-gram count features is a classic spam-filtering baseline. Spam Detection with Data-Mining Techniques applies the idea together with text features and classification metrics.

Unit 7: Support Vector Machines

Maximum margin

A linear binary decision surface is:

w^T x + b = 0

An SVM seeks not just a separator but one with a large margin to the closest observations from the two classes. The observations that determine that geometry are the support vectors.

Hard margin

For perfectly separable data:

y_i (w^T x_i + b) >= 1

with objective:

min 1/2 ||w||^2

Real data often contain noise and overlap, making hard-margin assumptions too strict.

Soft margin and C

Slack variables ξ_i permit controlled violations:

min 1/2 ||w||^2 + C Σ ξ_i

Large C penalizes training violations more strongly; small C permits a wider margin with more violations. Neither direction is universally better.

Kernels

A nonlinear feature map φ(x) can be represented implicitly with:

K(x,z) = φ(x)^T φ(z)

Common kernels include linear, polynomial, RBF/Gaussian, and sigmoid kernels. With an RBF kernel, γ controls locality and interacts strongly with feature scaling and C.

Multiclass SVM

The basic SVM is binary. Multiclass classification is commonly implemented with one-vs-rest or one-vs-one decompositions; evaluation should account for the strategy used by the implementation.

SVR

Support Vector Regression extends the margin framework to continuous targets, often using an ε-insensitive region around the prediction function.

Unit 8: Decision Trees and Ensemble Learning

Decision trees

A decision tree repeatedly splits the data using feature tests. Internal nodes represent tests; leaves represent final predictions. A path from root to leaf can often be written as a human-readable rule.

Entropy and information gain

For class probabilities p_k:

H = -Σ p_k log2 p_k

A split can be scored by the reduction in entropy. ID3 is classically associated with information gain, while C4.5 uses gain ratio to reduce preference for high-cardinality attributes.

Gini impurity

CART commonly uses:

Gini = 1 - Σ p_k^2

for classification. Regression trees use continuous-error objectives such as squared error.

Overfitting and pruning

Deep trees can partition training data too finely and develop high variance. Maximum depth, minimum leaf size, minimum split gain, and pruning limit complexity.

Bagging and Random Forest

Bagging trains models on bootstrap samples and aggregates their outputs. Random Forest additionally considers a random subset of features at each split, reducing correlation among trees.

Classification commonly uses voting; regression commonly uses averaging.

Boosting

Boosting builds learners sequentially so later learners focus on earlier errors. AdaBoost and gradient boosting implement this general idea differently. Ensembles still require validation; they do not automatically dominate a well-matched simpler model.

Unit 9: Clustering and Mixture Models

The clustering problem

Clustering groups unlabeled examples according to a selected notion of similarity. A cluster has no intrinsic physical name: interpretation is supplied after the algorithm by domain knowledge.

K-Means

K-Means minimizes within-cluster squared distance:

J = Σ_k Σ_{x_i ∈ C_k} ||x_i - μ_k||^2

The Lloyd iteration is:

1. initialize K centers,
2. assign each point to its nearest center,
3. recompute each center as the mean of its assigned points,
4. repeat until assignments/centers stabilize.

The objective does not increase from one iteration to the next, but the final solution need not be the global optimum and depends on initialization.

K-Means++ and choosing K

K-Means++ improves initialization by spreading initial centers. The elbow method is heuristic; silhouette scores compare cohesion with separation. Some data sets do not contain one natural value of K.

K-Medoids, PAM, CLARA, and CLARANS

K-Medoids uses an observed data point as the representative of each cluster. This can be more robust to outliers and can work with dissimilarities where a mean is not meaningful.

PAM is the classical medoid algorithm. CLARA applies PAM to samples for scalability. CLARANS searches sampled neighborhoods of medoid solutions.

Fuzzy C-Means

Fuzzy C-Means gives each observation a degree of membership in multiple clusters instead of a single hard assignment. The mathematics of graded membership is developed in Fuzzy Logic.

Hierarchical clustering

Agglomerative clustering starts from individual points and merges them; divisive clustering starts from the full data set and splits it. Single, complete, average, and Ward linkage can produce materially different dendrograms.

AGNES and DIANA are classical hierarchical methods. BIRCH uses summary structures for scalability; CURE/CAMELEON-style approaches extend cluster representation and connectivity for more complex geometry.

DBSCAN

DBSCAN uses an ε neighborhood and minPts to identify dense regions and noise without requiring the number of clusters in advance. It can find non-spherical clusters but struggles when one global density threshold cannot represent all regions.

Gaussian mixtures and EM

A Gaussian Mixture Model represents the data as a mixture of Gaussian components:

p(x) = Σ_k π_k N(x | μ_k, Σ_k)

Each observation has a probabilistic responsibility for each component. Expectation-Maximization alternates:

E step: estimate component responsibilities
M step: update parameters from those responsibilities

GMMs therefore provide soft probabilistic membership in contrast to the hard assignments of standard K-Means.

Unit 10: Dimensionality Reduction and Feature Selection

Selection versus transformation

Feature selection keeps a subset of existing variables. Dimensionality reduction commonly creates new axes. They solve related but different problems.

PCA

Principal Component Analysis finds orthogonal linear directions that capture high variance. For centered data it can be computed from covariance eigenvectors or through SVD.

PCA does not use labels. High variance is therefore not necessarily the direction that best separates classes. Scaling can materially change PCA when input units differ.

SVD

A matrix can be decomposed as:

X = U Σ V^T

A truncated decomposition retains the largest singular values to produce a lower-rank approximation. SVD is mathematically closely related to common PCA computations.

LDA

Linear Discriminant Analysis uses class labels and seeks projections that increase between-class separation relative to within-class scatter. Conceptually:

J(w) = (w^T S_B w) / (w^T S_W w)

With C classes, the discriminant subspace has dimension at most C-1.

PCA and LDA therefore answer different questions: PCA preserves unlabeled variance; LDA uses class-separation information.

Feature-selection families

Filter methods use model-independent measures such as correlation, information gain, or statistical tests. Wrapper methods evaluate subsets through model performance. Embedded methods perform selection as part of model fitting, as with Lasso or some tree-based procedures.

Selection must be performed inside the training process; selecting features after examining the test set contaminates evaluation.

Unit 11: Anomaly and Outlier Detection

Context matters

An outlier is an observation that differs substantially from expected behavior, but unusual does not always mean erroneous. Useful distinctions include:

• point outlier: one observation departs from the overall distribution,
• contextual outlier: unusual only under a particular time, location, or condition,
• collective outlier: a group is abnormal even when individual members appear ordinary.

Noise may represent measurement uncertainty, whereas an anomaly may be the event of interest.

Statistical approaches

When a distributional model is credible, low-probability regions can define anomaly candidates. Gaussian z-scores, Grubbs-style tests, boxplots/IQR, histograms, and mixture models represent different levels of parametric commitment.

If the distributional model is wrong, a low-probability score can also be wrong. Outliers can themselves distort estimated parameters, making robust statistics important.

Regression residuals

Large residuals:

r_i = y_i - ŷ_i

can indicate abnormal behavior. A poor regression model, however, also produces large residuals; model error and anomaly must not be conflated.

Proximity and density

k-NN distances can score isolated points. Local Outlier Factor compares local density with the density of neighbors. Values near one indicate similar local density, whereas substantially larger values can indicate a locally sparse observation.

High dimension weakens distance and density notions, so representation quality matters.

Clustering-based detection

Candidates may include points far from cluster centers, members of very small/sparse clusters, or observations labeled as noise by DBSCAN. The approach fails when normal data do not form meaningful clusters.

Classification-based detection

With labeled normal/anomalous examples, ordinary binary classifiers can be used. When anomaly labels are scarce, one-class methods such as one-class SVM learn a boundary around normal observations.

Applications include intrusion detection, fraud, sensor faults, medical monitoring, image/audio analysis, and textual anomalies. The operational definition of anomaly must be specified for each domain.

Unit 12: n-grams and Statistical Language Models

Local sequence context

An n-gram is a contiguous sequence of n units. The unit may be a word, character, syllable, token, or domain-specific symbol.

1-gram = unigram
2-gram = bigram
3-gram = trigram
n-gram = contiguous sequence of length n

The word “gram” does not mean a weight in this terminology.

Chain rule and Markov approximation

A full sequence probability is:

P(w1,...,wT) = Π_t P(w_t | w1,...,w_{t-1})

Estimating full history is data intensive. An n-gram language model approximates history with the preceding n-1 units:

P(w_t | history) ≈ P(w_t | w_{t-n+1},...,w_{t-1})

For a bigram:

P(w_t | w_{t-1})

Count-based estimation

The maximum-likelihood bigram estimate is:

P(w_i | w_{i-1}) = count(w_{i-1}, w_i) / count(w_{i-1})

Unseen sequences receive zero probability under raw counts, motivating smoothing, backoff, and interpolation.

Word and character n-grams

Word n-grams capture local lexical and syntactic patterns. Character n-grams can be robust to spelling variation and morphology. In agglutinative languages such as Turkish, character/subword representations can reduce some sparsity created by many surface word forms.

Naive Bayes and text classification

n-gram counts can be features for Multinomial Naive Bayes. Unigrams ignore local order; bigrams can capture part of expressions such as “not good.” Larger n increases context but also expands the feature space and data sparsity.

Speech recognition

In classical automatic speech recognition, the acoustic model scores how well candidate sequences match the signal while an n-gram language model scores linguistic plausibility. The architecture is discussed in Automatic Speech Recognition.

Difference from modern language models

An n-gram model is a fixed-window count-based statistical model. Transformer language models use distributed vector representations and attention. Both can assign probabilities to sequences, but their representation and generalization mechanisms differ fundamentally.

Unit 13: Probabilistic Graphical Models and Hidden Markov Models

Bayesian networks

A Bayesian network represents conditional dependencies among random variables with a directed acyclic graph. Each node carries a distribution conditional on its parents, giving the factorization:

P(X1,...,Xn) = Π_i P(X_i | Parents(X_i))

The graph exposes conditional structure, but an observational Bayesian network is not automatically a causal graph.

Naive Bayes can be viewed as a highly constrained special case: the class is a common parent of the features, which are assumed conditionally independent given that class.

Markov chains

The Markov assumption states that, under the selected state representation, the next state depends on the current state rather than the full past:

P(S_t | S_1,...,S_{t-1}) = P(S_t | S_{t-1})

This connects the stochastic-process treatment in Probability and Statistics to sequential learning.

Hidden Markov Models

A Hidden Markov Model (HMM) combines an unobserved state sequence with observations emitted by those states:

hidden state S_t
   ↓ emission
 observation O_t

Its basic components are:

• initial-state probabilities,
• transition probabilities,
• observation/emission distributions.

Classical inference problems include:

• evaluation: probability of an observation sequence,
• decoding: most probable hidden-state sequence,
• learning: estimation of model parameters from data.

Forward–backward, Viterbi, and Baum–Welch/EM methods are classical solutions.

Speech and sequential data

In classical speech recognition, HMM states represent temporal structure, the acoustic model scores observations, and an n-gram model supplies linguistic sequence probability. The full architecture is described in Automatic Speech Recognition.

Bayesian networks and HMMs show why statistical learning is broader than drawing decision boundaries: many tasks are fundamentally about conditional dependence and latent state structure.

Unit 14: Model Evaluation

Training, validation, and test

• training: estimate model parameters,
• validation: choose models and hyperparameters,
• test: estimate final generalization after choices are complete.

Repeatedly tuning against the test set turns it into a validation set.

Confusion matrix

                 Actual +    Actual -
Predicted +         TP           FP
Predicted -         FN           TN

Accuracy

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Accuracy can be misleading under strong class imbalance. A classifier that predicts only the majority class may appear highly accurate while missing every rare event.

Precision, recall, specificity, and F1

Precision = TP / (TP + FP)
Recall    = TP / (TP + FN)
Specificity = TN / (TN + FP)
F1 = 2 * Precision * Recall / (Precision + Recall)

Precision matters when false positives are costly; recall matters when false negatives are costly. Metric choice should follow operational cost.

ROC and PR curves

Changing the threshold traces the TPR/FPR trade-off in ROC space. Precision–recall curves can be more informative when positives are rare. AUC does not reveal the cost at one particular deployment threshold.

Calibration

A model is well calibrated around a score of 0.8 if roughly 80% of examples receiving that score are positive. Ranking discrimination and probability calibration are separate properties.

Regression metrics

MAE  = mean(|y-ŷ|)
MSE  = mean((y-ŷ)^2)
RMSE = sqrt(MSE)
R²   = 1 - SSE/SST

Squared-error metrics emphasize large errors more strongly than MAE. A high R² can still coexist with operationally unacceptable error magnitude.

Clustering metrics

Inertia summarizes within-cluster dispersion. Silhouette combines cohesion and separation. When external labels exist, measures such as adjusted Rand index can compare structures, but a cluster is not necessarily a ground-truth class.

Unit 15: Model Selection, Regularization, and Leakage

Cross-validation

k-fold cross-validation partitions data into k folds and rotates the validation fold. Stratified folds preserve class proportions for classification. Time-dependent data require temporal splits so future observations do not leak into the past.

Hyperparameter selection

Neighbor count k, SVM C/γ, tree depth, K-Means K, and regularization strengths are hyperparameters rather than fitted model parameters. Grid search, random search, or adaptive search can be used, but the test set must remain outside the search loop.

Nested cross-validation can separate hyperparameter selection from performance estimation when unbiased comparison is important.

Data leakage

Leakage includes more than accidentally copying the target into the features. Examples include:

• fitting scalers on the entire data set,
• selecting features using test outcomes,
• placing records from the same subject into training and test,
• using future information for a past prediction,
• splitting near-duplicate augmented examples across folds.

Evaluation should reproduce the information boundary that will exist at prediction time.

Class imbalance

Class weighting, controlled resampling, threshold selection, PR metrics, and cost-sensitive learning can address rare classes. Synthetic methods such as SMOTE belong inside the training fold only.

Learning curves

Training and validation error as a function of data size can help distinguish underfitting, high variance, and potential benefit from more data. More data cannot fix a wrong target definition, systematic label error, or deployment distribution mismatch by itself.

Unit 16: Statistical Learning in Production

Training and inference must share one pipeline

The production system includes preprocessing, feature extraction, category dictionaries, missing-value policy, and decision thresholds, not just a serialized model.

raw data
→ validation
→ preprocessing
→ features
→ model
→ score
→ threshold / decision
→ monitoring

If production preprocessing differs from training, model behavior changes even when the model file is identical.

Distribution shift

Data drift changes the input distribution. Concept drift changes the relationship between inputs and targets. Performance can decay because the world changed even when the code did not.

Monitoring may include feature distributions, class proportions, calibration, error types, latency, and missing-value rates.

Reproducibility

At minimum record:

• data version,
• split strategy,
• random seed,
• preprocessing configuration,
• model/library version,
• hyperparameters,
• evaluation metrics.

The same source code can produce different results under different data or splits.

Interpretability

Linear coefficients, tree paths, and nearest examples can support direct forms of explanation, but interpretability is not causality. A threshold or coefficient estimated from observational data does not by itself prove the effect of an intervention.

Unit 17: Position Among AI Method Families

Statistical learning is one part of artificial intelligence rather than a synonym for the entire field:

Artificial intelligence
├─ symbolic search, logic, and planning
├─ statistical learning
│  ├─ regression / Bayesian models
│  ├─ k-NN / SVM / decision trees
│  ├─ clustering / dimensionality reduction
│  └─ n-grams / classical pattern recognition
├─ neural networks and deep learning
├─ fuzzy logic
├─ evolutionary computation and genetic algorithms
└─ reinforcement learning

Artificial Intelligence and Neural Networks and Artificial Neural Networks and Learning Models cover connectionist learning in detail.

Fuzzy Logic models graded membership rather than probability; Fuzzy C-Means is a useful intersection in which a clustering task uses graded membership.

Genetic Algorithms and Their Applications search through populations, selection, and variation rather than estimating a statistical model directly. Genetic search can still tune an SVM, select features, or optimize other learning systems; method families can be composed.

The broader historical and philosophical context is developed in Artificial Intelligence: Philosophy, Theory and Practice. The key engineering question is not which method is “more intelligent,” but which assumptions, data structure, error costs, and computational budget make a learning mechanism appropriate.

References

• Trevor Hastie; Robert Tibshirani; Jerome Friedman. The Elements of Statistical Learning. Springer, 2009.
• Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
• Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
• Ethem Alpaydın. Introduction to Machine Learning. MIT Press, 2010.
• Gareth James; Daniela Witten; Trevor Hastie; Robert Tibshirani. An Introduction to Statistical Learning. Springer, 2013.
• Thomas M. Cover; Peter E. Hart. “Nearest Neighbor Pattern Classification.” IEEE Transactions on Information Theory, 1967.
• Corinna Cortes; Vladimir Vapnik. “Support-Vector Networks.” Machine Learning, 1995.
• J. MacQueen. “Some Methods for Classification and Analysis of Multivariate Observations.” Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1967.
• Stuart Lloyd. “Least Squares Quantization in PCM.” IEEE Transactions on Information Theory, 1982.
• Leonard Kaufman; Peter J. Rousseeuw. Finding Groups in Data: An Introduction to Cluster Analysis. Wiley, 1990.
• Peter J. Rousseeuw. “Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis.” Journal of Computational and Applied Mathematics, 1987.
• Martin Ester; Hans-Peter Kriegel; Jörg Sander; Xiaowei Xu. “A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise.” KDD, 1996.
• Arthur P. Dempster; Nan M. Laird; Donald B. Rubin. “Maximum Likelihood from Incomplete Data via the EM Algorithm.” Journal of the Royal Statistical Society: Series B, 1977.
• Leo Breiman. “Random Forests.” Machine Learning, 2001.
• Markus M. Breunig; Hans-Peter Kriegel; Raymond T. Ng; Jörg Sander. “LOF: Identifying Density-Based Local Outliers.” SIGMOD, 2000.
• Christopher D. Manning; Prabhakar Raghavan; Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
• Daniel Jurafsky; James H. Martin. Speech and Language Processing. Prentice Hall, 2009.