Artificial Neural Networks and Learning Models

Neural-network model selection depends on more than architecture. The learning algorithm, loss function, regularization, data distribution, experimental design, and production behavior must be evaluated together. The progression from Perceptron and ADALINE extends through Multilayer Perceptrons (MLPs), backpropagation, Learning Vector Quantization (LVQ), Adaptive Resonance Theory (ART), recurrent networks, CNNs, LSTM/GRU, attention, Transformers, and generative models.

Unit 1: Early Neural Networks

Single-layer networks

A single-layer linear threshold classifier forms a decision boundary:

w^T x + b = 0

It can separate linearly separable classes but cannot represent arbitrary nonlinear classification boundaries.

Perceptron

For binary targets and a misclassified example, a classic perceptron update has the form:

w <- w + eta (target - output) x

under one common coding convention.

The perceptron convergence theorem guarantees convergence in finite updates when the training set is linearly separable and assumptions are met. It does not guarantee convergence for non-separable data.

XOR limitation

XOR cannot be separated by one linear decision boundary. This illustrates the limitation of a single-layer perceptron and motivates hidden layers/nonlinear feature transformations.

ADALINE

ADALINE uses a linear activation during learning and adjusts weights to reduce a mean-squared error criterion, historically through the Widrow-Hoff/LMS rule.

A basic gradient form is:

w <- w - eta * dL/dw

The update direction must follow the derivative of the defined loss; notation should not obscure the sign convention.

MADALINE

MADALINE combines multiple ADALINE units. It is historically important as an early multilayer adaptive architecture, although its training methods differ from contemporary generic backpropagation.

Unit 2: Supervised Learning and the Multilayer Perceptron

This unit treats the forward pass, loss computation, backpropagation, and weight update as one iterative supervised-learning optimization loop.

input -> hidden layers -> output -> loss
  ^                              |
  |                              v
  +---- weight update <- backpropagation

forward pass -> error -> gradient -> update -> next forward pass

Multilayer perceptron

An MLP composes affine transformations and nonlinear activations:

h = phi(W1 x + b1)
y_hat = psi(W2 h + b2)

With sufficient hidden capacity and suitable activations it can represent nonlinear decision functions that a single linear layer cannot.

Solving XOR

A hidden layer can construct intermediate regions/features that separate XOR. The important lesson is not a particular hand-coded weight set but that composition of nonlinear units changes the representable function class.

Forward propagation

A feed-forward pass computes layer outputs in order. For layer l:

z^(l) = W^(l) a^(l-1) + b^(l)
a^(l) = phi(z^(l))

The output and target define a loss.

Backpropagation

Backpropagation applies the chain rule to compute derivatives of the loss with respect to each parameter efficiently.

For output error signal delta^(L), a hidden-layer error is propagated conceptually as:

delta^(l) = (W^(l+1)^T delta^(l+1)) .* phi'(z^(l))

and weight gradient:

dL/dW^(l) = delta^(l) a^(l-1)^T

Gradient descent then updates parameters:

W <- W - eta * dL/dW
b <- b - eta * dL/db

The exact output delta depends on activation and loss. For example, softmax with cross-entropy leads to a particularly simple derivative under standard formulation.

Training procedure

A practical training loop is:

1. initialize parameters,
2. select a batch/example,
3. perform forward propagation,
4. compute loss,
5. backpropagate gradients,
6. update parameters,
7. evaluate validation behavior,
8. stop according to a defined criterion.

Randomizing/shuffling training examples can reduce systematic ordering effects in stochastic methods.

Error surfaces

Neural-network objectives are generally non-convex. Saddle points, flat regions and different local minima can occur. The goal of optimization is useful generalization, not necessarily discovery of one globally unique parameter vector.

Backpropagation as repeated chain rule

Backpropagation is an efficient way to apply the chain rule through a computational graph. For a layer

z = W a + b, a_next = φ(z),

the backward pass propagates the loss gradient from later layers to earlier activations and parameters.

The important engineering consequence is that gradient scale depends on activation derivatives, weight scale, depth, normalization, and the loss function. Vanishing and exploding gradients are consequences of repeatedly multiplying Jacobian factors.

Initialization and normalization should be chosen together with the activation. Saturating activations and poorly scaled weights can make learning slow even when the network has enough capacity.

Validation, calibration, and decision thresholds

Training loss measures optimization on the observed training set. It does not establish generalization.

A separate validation set supports model and hyperparameter selection; the final test set should remain isolated from that process. Under class imbalance, accuracy alone can hide failure on minority classes.

For probabilistic classifiers, calibration asks whether predicted confidence matches observed frequency. The decision threshold should follow application cost, recall/precision trade-offs, and operational capacity rather than being fixed automatically at 0.5.

Unit 3: MLP Design and Performance

This unit separates the roles of training, validation, and test data in model selection and estimation of generalization performance.

dataset
   |
   +--> training -----> update weights
   |
   +--> validation ---> model / threshold / stopping decisions
   |
   +--> test ---------> final generalization estimate only

Performance measurement

A loss used for optimization is not necessarily the only evaluation metric. Classification may require accuracy, precision, recall, F-score, ROC/PR analysis or calibrated error costs. Regression may use MAE, MSE/RMSE or domain-specific tolerances.

Metrics must be selected from the operational consequence of errors.

Overfitting

Overfitting occurs when the learned model captures training-specific noise/patterns that do not generalize. Symptoms include improving training error while validation error stops improving or worsens.

Controls include:

• more representative data,
• regularization,
• early stopping,
• appropriate capacity,
• augmentation where semantically valid,
• cross-validation for limited datasets.

Example selection

Training examples should cover the expected deployment distribution. Random train/test splitting can be misleading when samples from the same subject, session, machine or time period leak correlated information across splits.

Input and output representation

Input scale and encoding directly affect optimization. Categorical variables, bounded numerical measurements, images and time series require different representation choices.

Outputs should match the task: linear outputs for unconstrained regression, logistic-style outputs for binary probabilities, softmax-like normalized outputs for mutually exclusive classes, or structured outputs for more complex tasks.

Weight initialization

Initializing every hidden weight identically prevents symmetry breaking. Random initialization with variance scaled to layer fan-in/fan-out improves signal/gradient propagation. Later Xavier/Glorot and He-style initializations formalize this principle for common activation families.

Learning rate and momentum

A learning rate that is too large can destabilize training; one that is too small can make optimization impractically slow.

Classical momentum accumulates a velocity-like term so updates retain direction across iterations and can smooth oscillation:

v_t = beta v_(t-1) + gradient_term
w_t = w_(t-1) + v_t

with sign convention determined by the update definition.

Scaling

Features with very different numeric ranges can create poorly conditioned optimization. Standardization or bounded scaling should be fit using training data only and then applied consistently to validation/test/deployment data.

Stopping criteria

Training can stop after:

• a maximum epoch count,
• objective convergence,
• validation-based early stopping,
• reaching a domain performance target.

Stopping because training error is nearly zero is not automatically desirable.

Hidden layers and units

There is no universal formula for the correct number of hidden units. Capacity should be selected empirically against data scale, validation behavior, latency/memory constraints and regularization.

Constructive methods grow a network; pruning methods remove weak parameters/units. Modern architecture search and sparsification extend these ideas but do not eliminate the need for independent evaluation.

Unit 4: MLP for Industrial Prediction

Problem definition

An industrial prediction problem begins with a measurable target and a clear decision/use of the prediction. Sensor variables should be selected for causal/operational relevance, not simply because they are available.

Experimental design

If training data come from controlled experiments, the experiment must span the operating region in which the model will later be used. A network cannot be expected to interpolate reliably into regions that are absent or poorly sampled, and extrapolation is especially risky.

Inputs and outputs

Inputs may include process settings, environmental measurements and lagged state. Outputs may represent quality, energy, yield or another process variable.

Leakage must be prevented: a feature that becomes available only after the target event cannot be used for real-time prediction even if it produces excellent retrospective accuracy.

Optimization through a learned model

A trained model can be used as a surrogate inside an optimization loop, but an optimizer may exploit model errors and drive inputs toward unsupported regions. Input constraints and validation against the physical process are therefore necessary.

Unit 5: Learning Vector Quantization

Purpose

LVQ is a prototype-based supervised classification family associated with Kohonen. Each prototype/codebook vector represents a region and carries a class label.

Structure and distance

Given input x, the closest prototype under the chosen metric is found:

c = argmin_j ||x - w_j||

The winning prototype determines the class.

LVQ1 update

A common LVQ1 rule moves the winning prototype toward a correctly classified sample:

w_c <- w_c + alpha (x - w_c)

and away from a sample of another class:

w_c <- w_c - alpha (x - w_c)

The learning rate is commonly reduced over training.

Decision regions

Nearest-prototype classification partitions feature space into Voronoi-like regions. The quality therefore depends strongly on feature scaling, distance metric and prototype placement.

LVQ2 and boundary refinement

LVQ2-style variants focus updates on samples near class boundaries and adjust prototypes from competing classes. The exact window conditions and variant definitions should be stated when implementing them; "LVQ" is a family rather than one universal update rule.

Penalized and extended variants

Historical penalized or modified LVQ forms aim to reduce prototype domination, improve boundary representation or handle different data structures. Their names are less standardized than core LVQ1/LVQ2, so implementations should document their exact rule.

Strengths and weaknesses

LVQ is interpretable in terms of prototypes and computationally light for low-dimensional engineered features. It is sensitive to feature scaling and metric choice and does not learn hierarchical representations from raw high-dimensional signals in the manner of deep networks.

Unit 6: LVQ for Pattern Recognition

Statistical quality-control example

A process can be represented by feature vectors derived from measured signals or summary statistics. LVQ prototypes then represent known normal/fault classes.

The engineering sequence is:

1. define measurable states/classes,
2. collect representative examples,
3. normalize features,
4. initialize class prototypes,
5. train on labeled examples,
6. evaluate on independent data.

Evaluation should use a confusion matrix and class-sensitive metrics, particularly when faults are rare.

Later perspective

For low-dimensional engineered features, LVQ remains a useful simple baseline. For high-dimensional image, audio or long sensor sequences, later CNN, recurrent or Transformer-based models can learn representations directly, but they introduce greater data and compute requirements.

Unit 7: Adaptive Resonance Theory Networks

Stability-plasticity problem

An adaptive system should learn new patterns without catastrophically destroying previously learned categories. ART architectures were designed around this stability-plasticity tension.

Basic ART mechanism

ART uses competitive category selection followed by a similarity/vigilance test. If the candidate category sufficiently matches the input, resonance allows learning. Otherwise the category is reset and another candidate is tried or a new category is created.

F1 and F2 layers

In the traditional description, F1 represents the input/comparison field and F2 represents category units. Bidirectional weights participate in category choice and match checking.

Vigilance

The vigilance parameter controls category granularity. Higher vigilance demands a closer match and tends to create more categories; lower vigilance allows broader categories.

ART1 and ART2

ART1 is designed for binary input patterns. ART2 extends the idea to continuous-valued inputs. These are historical model families; later clustering methods use different objectives and optimization formulations.

Unit 8: ART1 and Group Technology

Group technology

Group technology in manufacturing groups parts with similar design/manufacturing characteristics so processes can be organized into families/cells.

Binary feature vectors can describe properties such as machine requirements or part characteristics. ART1 can cluster such vectors adaptively.

Iterative operation

Each input selects a candidate category, is tested against vigilance, then either updates that category or causes reset/search. Results depend on vigilance, input order and preprocessing; the clustering is therefore not a unique ground truth.

Engineering interpretation

ART1 is valuable here as an example of incremental unsupervised categorization with an explicit similarity threshold. Modern clustering should still be compared against simpler and more robust baselines for the actual data distribution.

Sequence memory, truncation, and recurrent-state stability

A recurrent network carries state across sequence positions, so training couples parameters through time. Backpropagation through time unfolds the recurrence and accumulates gradients over that chain.

Long sequences increase memory cost and can worsen vanishing or exploding gradients. Truncated BPTT limits the number of unfolded steps and trades long-range credit assignment for bounded computation.

LSTM and GRU architectures introduce gates that make state retention and update more controllable, but they do not remove the need for sequence masking, state-reset policy, gradient clipping, and evaluation under distribution shift.

Unit 9: Recurrent Networks and the Elman Network

This unit explains how hidden state is carried through time and how shared parameters model sequential dependencies.

x(t-1) -> h(t-1) -> y(t-1)
            |
            v
x(t)   ->  h(t)   -> y(t)
            |
            v
x(t+1) -> h(t+1) -> y(t+1)

time unrolling with shared parameters

Recurrent-network concept

A recurrent network feeds information from previous states back into later computation. A simple state equation is:

h_t = phi(W_x x_t + W_h h_(t-1) + b)

The hidden state creates temporal memory.

Elman network

An Elman network uses a hidden layer whose previous activation is copied to context units and provided as additional input at the next time step.

Conceptually:

x_t + context(h_(t-1)) -> hidden -> output

This allows the network to represent sequence dependencies beyond a memoryless feed-forward model.

Backpropagation through time

A recurrent network can be unfolded through time and differentiated as a deeper computational graph. Backpropagation through time accumulates gradients across temporal steps.

Long sequences can produce vanishing or exploding gradients, making long-range dependency learning difficult for simple recurrent units.

LSTM, GRU, Attention, and Transformers

LSTM/GRU, attention, and Transformer-based sequence modeling are developed further in Unit 11.

Unit 10: Model Selection and Evaluation Across Application Domains

Model-evaluation criteria vary by application domain. Accuracy should be considered together with latency, error cost, calibration, distribution shift, resource budget, and reliability. For general application areas, advantages, and limitations, see Artificial Intelligence and Neural Networks.

Application Selection

Before using an Artificial Neural Network, determine whether the problem genuinely benefits from a learning-based solution. Nonlinear relationships, high-dimensional data, many examples, hard-to-write rules, and complex patterns can justify ANN use; a simple problem with a clear and verified algorithmic solution may not.

Industrial Systems

In industrial systems, accuracy is only one requirement. Latency, deterministic behavior, distribution drift, and safe behavior under failure can be equally important.

Financial Systems

Historical performance does not guarantee future performance. Regime change, data leakage, and selection bias are critical evaluation risks.

Military and Security Systems

False-positive and false-negative costs are usually asymmetric. Evaluation should reflect operational conditions, distribution shift, and adversarial effects.

Healthcare Systems

Clinical use requires more than accuracy. Data-representation fairness, calibration, external validation, and human oversight must be considered together.

Data Type and Model Choice

Image, audio, text, and time-series data have different structural properties. A model family should not be chosen mechanically by modality; representation structure, latency, memory, compute budget, error cost, and validation requirements should be evaluated together.

The best engineering model is not the largest model. It is the model that meets the requirement with sufficient reliability at the lowest total cost.

Unit 11: Other Models and Later Developments

Model families differ in memory mechanisms, connectivity, learning objectives, and representation capacity.

Hopfield network

A Hopfield network is a recurrent associative-memory model with an energy function under the standard symmetric-weight formulation. Updating units reduces or preserves energy until an attractor/local minimum is reached.

Stored patterns are not retrieved as an exact database lookup; capacity and spurious attractors limit behavior.

Counterpropagation

Counterpropagation combines competitive representation with supervised association, historically linking Kohonen-style clustering and Grossberg-style output learning.

Cognitron and Neocognitron

Fukushima's neocognitron introduced hierarchical local receptive fields and shift-tolerant feature extraction, making it an important conceptual predecessor of convolutional neural networks.

Self-Organizing Map

A SOM maps high-dimensional samples to a usually low-dimensional grid. The winner and its neighborhood are moved toward an input, preserving topology approximately:

w_i <- w_i + alpha h_ci (x - w_i)

It is useful for visualization and exploratory clustering, not as a probabilistic guarantee of class structure.

Radial-basis-function networks

RBF networks use localized basis functions, commonly Gaussian-like responses around centers:

phi_j(x) = exp(-||x-c_j||^2 / (2 sigma_j^2))

A linear output combines these features. Center and width selection determine behavior.

Probabilistic neural networks

PNN-style models estimate class likelihood/density from examples with kernel-like functions. They can train quickly but inference/storage grow with retained examples.

Boltzmann machines

Boltzmann machines are stochastic energy-based networks. Restricted Boltzmann Machines simplify connectivity to enable more practical learning and were important in the history of deep generative pretraining.

Convolutional neural networks

CNNs exploit local connectivity and shared kernels:

feature_map = convolution(input, kernel)

This inductive bias is efficient for spatially local patterns. Modern CNNs use many architectural refinements, but convolution, nonlinear activation and hierarchical features remain the core idea.

LSTM and GRU

Gated recurrent networks control state updates with learned gates, improving optimization of longer sequences compared with simple RNNs. They were major sequence-modeling tools before Transformers and remain relevant in streaming and resource-constrained scenarios.

Attention and Transformer

Attention computes content-dependent weighted combinations. A standard scaled dot-product attention is:

Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V

A Transformer stacks attention and feed-forward blocks with residual paths and normalization. Positional information is required because pure self-attention does not inherently encode sequence order.

The original Transformer architecture appeared in 2017 and established self-attention as the central sequence-mixing mechanism of the model.

Autoencoders

An autoencoder learns:

x -> encoder -> z -> decoder -> x_hat

with a reconstruction objective. Bottleneck, sparsity, denoising or variational constraints determine what representation is learned.

Generative adversarial networks

GANs were introduced in 2014. A generator and discriminator play an adversarial optimization game: the generator produces candidate samples while the discriminator provides a learned signal for distinguishing generated samples from training data. Training instability, mode collapse and evaluation difficulty are practical concerns.

Diffusion models

Diffusion generative models learn to reverse a progressive noising process and have become a major family for high-dimensional generation. Training spans multiple noise levels, while generation applies a sequence of denoising steps that progressively maps a noisy state toward the learned data distribution.

Self-supervised representation learning

Self-supervised methods derive prediction targets from the data itself, allowing large unlabeled corpora to train representations that can later be adapted to downstream tasks.

Regularization

Regularization includes weight penalties, dropout, data augmentation, early stopping and architecture constraints. The purpose is not simply to make weights small but to improve generalization under the expected data distribution.

Normalization

Batch normalization uses mini-batch statistics during training and running statistics for inference under its standard form. Layer normalization normalizes within each sample representation and is common in Transformer architectures. The correct choice depends on architecture and batch/sequence behavior.

Modern optimization

Stochastic gradient descent with momentum remains a strong baseline. Adaptive optimizers such as Adam use moving estimates of first and second gradient moments. An optimizer does not remove the need to select learning-rate schedules, regularization and validation protocol.

From neural-network simulators to learning frameworks

Older neural-network study often used specialized simulators. Modern frameworks represent differentiable computational graphs, automatic differentiation, accelerator execution and distributed training.

The abstraction changed, but the same engineering questions remain: define the data contract, loss, model, optimization, evaluation and deployment constraints explicitly.

Current engineering principles

For production neural systems:

• separate train/validation/test correctly,
• prevent data leakage,
• version preprocessing together with the model,
• measure latency and memory on target hardware,
• monitor distribution shift,
• calibrate or threshold outputs according to risk,
• retain reproducible model/data metadata,
• treat external validation as distinct from training success.

Historical neural models are valuable not because every one remains state of the art, but because they expose the core ideas of linear separability, error-driven learning, prototypes, competition, recurrence, and representation that later architectures build upon.

Reliability boundaries also include adversarial inputs, data poisoning, privacy constraints, and the role of human oversight in high-risk use. Progress is therefore better evaluated as an algorithm + data + hardware + validation + operations problem than as a catalogue of newly named architectures.

Experiment tracking, ablation, and production observation

Recording only the final metric does not record an experiment. Reproducible comparison requires the baseline, data version, preprocessing, model configuration, hyperparameters, evaluation set, and metric definition to remain traceable together.

baseline
→ one controlled change
→ same evaluation protocol
→ result
→ interpretation

This produces stronger evidence than changing many components at once and then being unable to identify which change produced the gain.

Hyperparameter search and evaluation discipline

Grid search, random search, and probabilistic optimization generate different candidate configurations. A sophisticated search algorithm does not repair a weak evaluation protocol. Hyperparameter selection should use validation data; the final test set should not become part of the repeated selection loop.

When many experiments are run, simply choosing the highest score can also be misleading. Runs should use comparable splits, preprocessing, and metrics, and sources of variation should be inspected where they matter.

Ablation studies

An ablation study removes a model component or feature/data group and compares the result under a controlled protocol.

Model A
Model A - component X
Model A - feature group Y
Model A - augmentation Z

Ablation is not a replacement for ordinary software testing. A test can ask whether required behavior holds; ablation asks how much a component contributes to observed performance. The two forms of evidence answer different questions.

Error slices and explainability

A single average score can hide where a model fails. Error rates can be inspected across meaningful slices such as class, time, source, device, or signal quality. Residual plots and systematic residual patterns serve a similar purpose in regression, while confusion matrices and class-specific measures help in classification.

Explainability methods such as SHAP can help inspect feature contributions to a model decision, but feature attribution is not causal proof. The explanation method itself is another analytical layer with assumptions and approximations.

Production observation and drift

Success on a validation set does not guarantee unchanged production behavior. Input distributions, class proportions, sensors, or client behavior can drift. System behavior such as latency, memory use, and failure rate also remains part of model quality.

offline evaluation
→ controlled rollout
→ data and model observation
→ drift / error-slice analysis
→ retrain or rollback decision

Online or incremental learning may adapt more quickly, but it introduces risks from bad labels, abrupt distribution change, irreversible updates, backward-compatibility failures, and catastrophic forgetting. Update criteria, data-quality checks, model versioning, and rollback must therefore be explicit.

Configuration and release management are developed further in Software Engineering, while evidence and validation scope are covered in Software Test Engineering.

Related Topics

For artificial neurons, activation functions, layer structure, and learning paradigms, see Artificial Intelligence and Neural Networks. For the philosophical and theoretical framework of Artificial Intelligence (AI), see Artificial Intelligence: Philosophy, Theory and Practice.

Calibration, thresholds, and error slices

Ranking quality and probability quality are different. A classifier can have good AUC while being poorly calibrated; a score of 0.9 may not correspond to an event rate near 90 percent. Calibration curves and Brier score help expose the difference.

Decision thresholds should follow business costs and operating constraints rather than defaulting to 0.5.

Aggregate metrics can hide systematic failures. Errors should be sliced by meaningful factors such as device, source, noise level, region, or time, with uncertainty reported for small samples.

Building a reproducible learning experiment

Architecture alone does not define an experiment. Dataset version, normalization, augmentation, random seed, optimizer, learning rate, batch size, early stopping, and model-selection criteria should be recorded with the result.

Train/validation/test roles should be fixed before model selection. Repeatedly tuning against the test set removes its independence. On small datasets, repeated cross-validation or confidence intervals can be more informative than one split.

Failure cases are also results. Error slices by class, source, noise, device, lighting, or another relevant dimension can reveal weaknesses hidden by aggregate metrics.

Separate the learning result from the model

In a neural-network problem, architecture, training objective, and evaluation metric belong to different layers. An activation function introduces the layer's nonlinearity, a loss function defines what training attempts to minimise, and a metric reports performance. A classifier can, for example, be trained with cross-entropy while accuracy is reported separately.

Backpropagation is not a prediction method; it is a way to compute gradients by repeated application of the chain rule. The optimiser determines how those gradients change the parameters. A learning rate that is too large can create unstable updates, while one that is too small can make progress impractically slow.

Training, validation, and test sets have different roles. Training data changes model parameters. Validation data supports model and hyperparameter selection. A test set estimates generalisation only to the extent that it remains outside that selection loop. Repeatedly changing a design after inspecting test results effectively turns the test set into another validation set.

Overfitting is not merely low training error; it is the gap between training behaviour and performance on unseen data. Regularisation, data augmentation, early stopping, and appropriate model capacity can reduce that gap. High training error can instead indicate under-capacity, optimisation failure, poor data, or unsuitable feature scaling.

A high classifier score is not automatically a calibrated probability. If cases reported near 0.9 are correct only half the time, the model may rank examples well and still be poorly calibrated. Accuracy, calibration, and the decision threshold should therefore be evaluated separately, especially with imbalanced classes.

Unit 12: Generative Neural Network Families

Generative AI includes several neural model families rather than only large language models.

Variational autoencoders

A VAE maps observations into a probabilistic latent space and samples from that space.

x
 |
Encoder
 |
q(z|x)
 |
 z
 |
Decoder
 |
x'

Its objective combines reconstruction with regularization of the latent distribution.

Generative adversarial networks

GANs train a generator against a discriminator.

noise -> Generator -> fake sample
                       |
real sample ------------+--> Discriminator

This competitive objective can produce high-quality samples but can also suffer from instability and mode collapse.

Diffusion models

Diffusion models define a forward noise process and learn a reverse denoising process.

data -> noise -> noise -> ...
                    |
                    v
              learned reverse
                    |
                  sample

Generation often requires multiple denoising steps.

Autoregressive Transformers

Language generation proceeds sequentially:

x1 -> x2 -> x3 -> x4 ...

These families share the broad label “generative models” while differing in objectives, sampling cost, modalities, and failure modes.

Unit 13: Transformers Are Neural Networks

Transformers are not separate from artificial neural networks. Attention, linear projections, normalization, and feed-forward blocks form a trainable neural architecture.

Large language models are therefore large-scale Transformer-based neural networks rather than a replacement for neural networks as a paradigm.

Unit 14: Pruning, Distillation, and Quantization

Production constraints include memory, latency, and energy as well as predictive quality.

Pruning removes low-value parameters or structures.

Distillation transfers useful behavior from a larger teacher to a smaller student.

Quantization reduces numeric precision.

Model
 |
 +--> pruning
 +--> distillation
 +--> quantization
 |
 v
deployment trade-off

The techniques can be combined but are conceptually different.

Very low-bit research such as 1-bit/1.58-bit architectures should not be confused with ordinary post-training quantization of arbitrary models.

Compression should be measured across quality, latency, throughput, memory, energy, and hardware support.

References

• David E. Rumelhart; Geoffrey E. Hinton; Ronald J. Williams. Learning Representations by Back-propagating Errors. Nature, 1986. DOI

• Frank Rosenblatt. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review, 1958. DOI

• Hinton, G.; Vinyals, O.; Dean, J. “Distilling the Knowledge in a Neural Network.” 2015. https://arxiv.org/abs/1503.02531

• Ho, J.; Jain, A.; Abbeel, P. “Denoising Diffusion Probabilistic Models.” NeurIPS, 2020. https://arxiv.org/abs/2006.11239

• Kingma, D. P.; Welling, M. “Auto-Encoding Variational Bayes.” 2013. https://arxiv.org/abs/1312.6114

• Sepp Hochreiter; Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 1997. DOI

• Warren S. McCulloch; Walter Pitts. A Logical Calculus of the Ideas Immanent in Nervous Activity. Bulletin of Mathematical Biophysics, 1943. DOI