Probability and Statistics: Distributions, Sampling and Regression

Unit 1: Introduction to Statistics

A statistical investigation first defines the outcome variable, observational unit, and sampling design. Selecting summaries such as means or correlations without considering measurement scale can be misleading. Report central tendency together with dispersion; investigate whether extreme values are data errors or genuine observations. Do not infer population properties from a merely descriptive sample without addressing selection and uncertainty.

1.1. Core Concepts

Statistics is the set of methods used to collect, organize, analyze, and interpret data.

A population is the complete set of units about which information is sought.

A sample is a subset selected from the population.

A census examines every unit in the population.

Sampling selects a sample from the population according to a defined procedure.

A unit is a single object of observation.

A variable is a measured or classified property of a unit.

A numerical quantity describing a population is a parameter; its sample-based counterpart is a statistic. Statistical inference uses the sample to draw conclusions about the population.

1.2. Levels of Measurement

• Nominal: identifies categories without an inherent order.
• Ordinal: has an order, but distances between levels are not assumed equal.
• Interval: equal differences are meaningful, but there is no true zero.
• Ratio: has equal intervals and a true zero, so ratios are meaningful.

Encoding a category as 1, 2, or 3 does not make it quantitative. Coding and level of measurement are different concepts.

Interpretation must also respect the level of aggregation. Inferring individual behavior from relationships observed at a regional, organizational, or time-aggregated level is an ecological fallacy.

A storage type does not determine a variable's measurement level. Encoding three product categories as 1, 2 and 3 does not create a quantitative distance or a twofold relation between categories. Nor does summing several ordinal responses automatically justify treating the result as an interval-scale measure; the construction of the scale and the assumptions of the intended model must be examined. The observational unit is equally important: when each row denotes a district rather than a person, an association between district means need not hold for individuals. A defensible sequence is to specify the observational unit, establish what each variable measures, separate encoding from measurement, and only then select an inferential operation.

Reverse-coding an item in a multi-item scale is a substantive scoring operation, not merely a display adjustment. On a five-point scale, the transformation new_x = 6 − old_x maps 1 to 5 and 2 to 4 while leaving the midpoint 3 unchanged. Aggregation without aligning item directions may create an incoherent composite score. An internal-consistency coefficient alone cannot prove that all items validly measure the intended construct. Reliability concerns consistency of measurement; validity concerns support for the intended interpretation. A scale can be consistent while measuring the wrong thing. Item polarity, missing-response rules, scoring and validity evidence should be reviewed before using the composite in a model.

1.3. Descriptive and Inferential Statistics

Descriptive statistics summarizes the observed data.

Inferential statistics uses a sample to make uncertainty-aware statements about a population.

The distinction is fundamental: a mean and standard deviation describe data; a standard error and confidence interval quantify sampling uncertainty.

1.4. From Frequency Tables to Grouped Distributions

A frequency table records how often each value occurs. For n valid observations and absolute frequency f_i, relative frequency is r_i = f_i / n; when missing entries are excluded, the denominator must count only valid observations. In 1, 1, 2, 2, 2, 3, 4, 4, the value 2 has frequency 3, relative frequency 3/8, and cumulative frequency 5. Cumulative counts are meaningful only after values or classes are ordered.

For continuous measurements, observations may be assigned to intervals. Half-open boundaries such as [a,b) prevent one observation from entering adjacent classes simultaneously; treatment of the final upper boundary must be explicit. Sturges' heuristic gives a starting class count k = ceil(1 + log2(n)). At n=100 it suggests 8 classes, not a compulsory histogram specification. Skewness, outliers, and repeated values may call for different boundaries. If class widths differ, histogram bar heights should represent frequency density rather than raw count so that bar areas remain proportional to observation shares.

1.5. Measures of Central Tendency

1.5.1. Arithmetic Mean

x̄ = Σxᵢ / n
x̄ = Σ(fᵢ·xᵢ) / Σfᵢ

For grouped data, xᵢ is the class midpoint.

The arithmetic mean uses every observation and is sensitive to outliers.

1.5.2. Geometric Mean

G = ⁿ√(x₁·x₂·...·xₙ)
log G = Σ(log xᵢ) / n

It is appropriate for positive multiplicative changes, growth rates, and compound returns.

1.5.3. Harmonic Mean

H = n / Σ(1/xᵢ)

It is useful for averaging rates when the denominator structure requires it.

1.5.4. Root Mean Square

K = √(Σxᵢ² / n)

It measures magnitude independently of sign.

For positive values:

H ≤ G ≤ x̄ ≤ K

1.5.5. Median

The median is the middle value of an ordered data set. It is more robust to outliers than the mean.

For grouped data:

Md = L + [ (n/2 − F) / f ] · c

L is the lower boundary of the median class, F the cumulative frequency before it, f its frequency, and c the class width.

1.5.6. Mode

The mode is the most frequent value. A distribution may have no mode or multiple modes.

For grouped data:

Mo = L + [ Δ₁ / (Δ₁ + Δ₂) ] · c

For a symmetric unimodal distribution:

x̄ = Md = Mo

1.6. Measures of Dispersion

Range:

R = x_max − x_min

It depends only on the minimum and maximum and is therefore sensitive to extremes.

Interquartile range:

IQR = Q₃ − Q₁

It measures the spread of the middle 50% and is more robust to outliers.

Population variance:

σ² = Σ(xᵢ − μ)² / N

Sample variance:

s² = Σ(xᵢ − x̄)² / (n − 1)

Standard deviation:

s = √s²

Standard deviation describes the spread of observations. Standard error describes the sampling variability of an estimator.

For independent observations, the estimated standard error of the sample mean is:

SE(x̄) = s / √n

Coefficient of variation:

CV = (s / x̄) · 100

It can compare relative variability of positive ratio-scale quantities. It becomes unstable when the mean is near zero.

1.7. Percentiles and Quantiles

The median is the 50th percentile. Q₁, Q₂, and Q₃ correspond to the 25th, 50th, and 75th percentiles.

P90, P95, and P99 summarize upper-tail behavior. P95, for example, is a threshold at or below which approximately 95% of observations fall.

Percentile algorithms can differ in rank definition and interpolation, especially in small samples. Comparisons should use the same method.

1.8. Robust Statistics

When outliers distort the mean and standard deviation, the median and median absolute deviation can provide robust summaries:

MAD = median(|xᵢ − median(x)|)

Modified z-score:

Mᵢ = 0.6745 · (xᵢ − median(x)) / MAD

It is an anomaly indicator, not proof. The expression cannot be applied directly when MAD = 0.

Classical z-score:

zᵢ = (xᵢ − x̄) / s

1.9. Match the Statistical Tool to the Question

Describing the observed sample and making an inference about a population are different tasks. Mean, median, variance, and quantiles summarize observed data. Confidence intervals, hypothesis tests, and regression make statements under uncertainty beyond the sample. A formula can be computed correctly while the interpretation remains wrong if this distinction is missed.

Conditional probability is directional:

P(A|B) = P(A ∩ B) / P(B)

P(A|B) and P(B|A) are generally different. Bayes' rule reverses the conditioning direction by combining prior probability with likelihood. In low-base-rate problems, test accuracy alone is insufficient; the prior prevalence can dominate the posterior probability.

Expected value is a probability-weighted long-run average, whereas variance measures dispersion around that centre. Two distributions can have the same mean and very different risk profiles. Standard deviation is easier to interpret because it has the same unit as the variable. For skewed samples or data with strong outliers, the median and interquartile range can be more robust than the mean and standard deviation.

A confidence interval must also be distinguished from a prediction interval for a future observation. A confidence interval represents uncertainty about a parameter; a prediction interval additionally contains the natural variability of a new observation and is therefore usually wider.

Correlation describes association, not causation. Likewise, a small p value does not imply a large or practically important effect. Statistical results should be read together with sampling design, effect size, uncertainty, and the context in which the data were produced.

Unit 2: Set Theory

2.1. Sets and Subsets

A set is a collection of objects; its objects are elements.

If every element of A belongs to B:

A ⊆ B

A set with n elements has:

2ⁿ

subsets.

2.2. Intersection, Union, and Complement

A ∩ B = {x : x ∈ A and x ∈ B}
A ∪ B = {x : x ∈ A or x ∈ B}
A' = {x ∈ E : x ∉ A}
A \ B = A ∩ B'

De Morgan's laws:

(A ∪ B)' = A' ∩ B'
(A ∩ B)' = A' ∪ B'

Inclusion-exclusion:

|A ∪ B| = |A| + |B| − |A ∩ B|

Set theory provides the language of probability: the sample space is a set and an event is a subset.

Unit 3: Permutations and Combinations

3.1. Counting Principles

Counts are added for mutually exclusive alternatives and multiplied for successive stages.

3.2. Permutations

Order matters:

P(n,r) = n! / (n−r)!
P(n,n) = n!

Sampling with replacement:

nʳ

With identical elements:

n! / (n₁!·n₂!·...·nₖ!)

3.3. Combinations

Order does not matter:

C(n,r) = n! / [r!·(n−r)!]

Combinations with repetition:

C(n+r−1,r)

Properties:

C(n,r) = C(n,n−r)
C(n,r) = C(n−1,r−1) + C(n−1,r)
Σ C(n,r) = 2ⁿ

If changing order changes the outcome, use a permutation; otherwise use a combination.

Unit 4: Introduction to Probability

Probability calculations begin with an explicit sample space and event definitions. Determine whether events are disjoint and whether independence is genuinely justified. In conditional probability, compute numerator and denominator under the same conditioning event. After applying total probability or Bayes' rule, check that probabilities lie between zero and one and that sums over the appropriate partition remain consistent.

4.1. Basic Concepts

A random experiment has an outcome that cannot be known with certainty in advance. The set of all possible outcomes is the sample space. A subset of the sample space is an event.

4.2. Definitions of Probability

For equally likely outcomes:

P(A) = favorable outcomes / possible outcomes

In the frequentist view, probability is the limiting relative frequency under repeated trials.

Kolmogorov axioms:

0 ≤ P(A) ≤ 1
P(S) = 1
If A and B are disjoint:
P(A ∪ B) = P(A) + P(B)

Basic results:

P(∅) = 0
P(A') = 1 − P(A)
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

For three events:

P(A∪B∪C)
= P(A)+P(B)+P(C)
− P(A∩B)−P(A∩C)−P(B∩C)
+ P(A∩B∩C)

4.3. Conditional Probability

P(A|B) = P(A ∩ B) / P(B),   P(B) > 0

Multiplication rule:

P(A ∩ B) = P(B)·P(A|B)
         = P(A)·P(B|A)

4.4. Independence

P(A|B) = P(A)

equivalently:

P(A ∩ B) = P(A)·P(B)

Disjointness is not independence. Two disjoint events with nonzero probabilities cannot be independent.

4.5. Law of Total Probability

If B₁, ..., Bₙ partition the sample space:

P(A) = ΣP(Bᵢ)·P(A|Bᵢ)

4.6. Bayes' Theorem

P(Bₖ|A)
= P(Bₖ)·P(A|Bₖ) / Σ[P(Bᵢ)·P(A|Bᵢ)]

P(Bₖ) is the prior probability and P(Bₖ|A) the posterior probability. New evidence updates prior information.

Combining Bayesian updating with a decision threshold. Suppose the true fault rate in a device population is 2%, a fault triggers an alarm with probability 95%, and a healthy device raises a false alarm with probability 4%. Treating an alarm as 95% certainty of failure ignores the base rate. Bayes' rule gives P(fault | alarm) = (0.02·0.95) / (0.02·0.95 + 0.98·0.04) = 0.019/0.0582 ≈ 0.3265. Thus, among alarmed devices, the probability of a true fault is about 32.65%. A sound inspection decision also needs the cost of false alarms, the cost of missed failures, and the accuracy of a confirmatory test. This calculation presupposes that sensitivity and false-alarm estimates apply to the same device population.

Unit 5: Random Variables

5.1. Discrete and Continuous Variables

A discrete random variable takes countable values.

A continuous random variable can take values across an interval.

5.2. Discrete Distributions

p(xᵢ) = P(X = xᵢ)
p(xᵢ) ≥ 0
Σp(xᵢ) = 1

Cumulative distribution:

F(x) = P(X ≤ x)

Expected value:

E(X) = μ = Σxᵢ·p(xᵢ)

Variance:

Var(X) = E[(X−μ)²]
       = E(X²) − [E(X)]²

Properties:

E(aX+b) = aE(X)+b
E(X+Y) = E(X)+E(Y)
Var(aX+b) = a²Var(X)

In general:

Var(X+Y)
= Var(X)+Var(Y)+2Cov(X,Y)

5.3. Continuous Distributions

Probability density function:

f(x) ≥ 0
∫f(x)dx = 1
P(a ≤ X ≤ b) = ∫[a,b] f(x)dx

For a continuous distribution, a single point has probability zero.

F(x) = ∫(−∞,x] f(t)dt
E(X) = ∫x·f(x)dx
Var(X) = ∫(x−μ)²f(x)dx

5.4. Joint, Marginal, and Conditional Distributions

When two random variables are studied together, their individual distributions do not preserve all dependence information. A joint distribution describes how values of X and Y occur together. Summing or integrating the joint distribution over one variable yields a marginal distribution.

A conditional distribution describes how knowledge of one variable changes the distribution of another. In the discrete case,

P(X=x | Y=y) = P(X=x, Y=y) / P(Y=y)

for a positive denominator. Independence can also be read as the condition that conditioning does not change the relevant distribution.

This distinction matters in data science: two columns can have identical marginal histograms while exhibiting very different joint behavior. Dependence cannot generally be recovered from marginals alone.

5.5. Covariance, Conditional Expectation, and the Laws of Total Expectation and Variance

Covariance measures linear co-variation:

Cov(X,Y) = E[(X-E[X])(Y-E[Y])].

Correlation removes units by scaling covariance, but it is not equivalent to independence. Independence implies zero covariance when the moments exist, whereas zero covariance does not imply independence in general.

Conditional expectation E[X|Y] describes the average behavior of X after information about Y is available. The law of total expectation is

E[X] = E[E[X|Y]],

while the law of total variance decomposes uncertainty as

Var(X) = E[Var(X|Y)] + Var(E[X|Y]).

The first term describes within-condition uncertainty and the second variation across conditional means. This decomposition is useful in hierarchical data, error budgets, and model interpretation.

Unit 6: Discrete Probability Distributions

6.1. Bernoulli Distribution

A single trial with two outcomes:

P(X=1) = p
P(X=0) = 1−p = q
E(X) = p
Var(X) = p·q

6.2. Binomial Distribution

Number of successes in n independent Bernoulli trials:

P(X=k) = C(n,k)·pᵏ·q^(n−k)
E(X) = n·p
Var(X) = n·p·q

The model requires a fixed number of trials, two outcomes per trial, constant success probability, and independent trials.

6.3. Multinomial Distribution

An extension of the binomial model to more than two outcomes:

P(X₁=n₁,...,Xₖ=nₖ)
= [n!/(n₁!·...·nₖ!)] · p₁^n₁ · ... · pₖ^nₖ

6.4. Poisson Distribution

Models the number of events in a fixed time or spatial interval:

P(X=k) = e^(−λ)·λᵏ/k!
E(X) = λ
Var(X) = λ

Under the theoretical Poisson model, mean and variance are equal. Strong overdispersion or underdispersion in observed data calls model adequacy into question.

A Poisson approximation to the binomial is appropriate when n grows, p becomes small, and λ=n·p remains bounded. Fixed numerical cutoffs are only rules of thumb.

6.5. Geometric Distribution

Number of trials until the first success:

P(X=k) = q^(k−1)·p
E(X) = 1/p

It is memoryless.

6.6. Negative Binomial Distribution

Models the number of trials until the rth success. The geometric distribution is the special case r=1.

Unit 7: Continuous Probability Distributions

7.1. Normal Distribution

f(x)
= [1/(σ√(2π))]
  · e^(−(x−μ)²/(2σ²))

μ determines location and σ dispersion.

Properties:

• symmetric about μ;
• mean, median, and mode coincide;
• total area under the curve is 1;
• inflection points occur at μ ± σ.

Approximate empirical rule:

μ ± σ   → 68%
μ ± 2σ  → 95%
μ ± 3σ  → 99.7%

7.2. Standard Normal Distribution

Z = (X−μ)/σ

Z expresses how many standard deviations an observation lies from the mean.

7.3. Central Limit Theorem

Under suitable independence and regularity conditions, the standardized distribution of the sample mean approaches normality as sample size grows:

X̄ ≈ N(μ, σ²/n)

This does not imply that the raw observations become normally distributed. Convergence depends on skewness, tail behavior, and dependence.

The central limit theorem must not be interpreted as a claim that the observed data themselves become normally distributed. Under suitable conditions it concerns the sampling distribution of standardized sums or means. Independent, identically distributed observations with finite variance provide the classical setting; strong dependence, heavy tails or infinite variance do not inherit this conclusion automatically. Sample size also cannot repair selection bias: a very large nonrepresentative sample remains nonrepresentative. The analysis must establish the collection mechanism and dependence structure before selecting a sampling distribution, estimating uncertainty or claiming a normal approximation.

7.4. Exponential Distribution

Models waiting time between events:

f(x) = λe^(−λx),   x ≥ 0
F(x) = 1−e^(−λx)
E(X) = 1/λ
Var(X) = 1/λ²

Memorylessness:

P(X>s+t | X>s) = P(X>t)

7.5. Gamma Distribution

For waiting time until the rth event:

E(X) = r/λ
Var(X) = r/λ²

The exponential distribution is the case r=1.

7.6. Beta Distribution

A flexible distribution on [0,1], useful for modeling proportions and probabilities.

7.7. Chi-square Distribution

χ² = Z₁² + Z₂² + ... + Zₖ²

k is the degrees of freedom. The distribution appears in variance inference and tests of independence.

7.8. F Distribution

F = (χ₁²/k₁) / (χ₂²/k₂)

It is used in variance comparisons and analysis of variance.

7.9. Student's T Distribution

t = Z / √(χ²/k)

It has heavier tails than the normal distribution and approaches the standard normal as the degrees of freedom increase.

It is used for inference about a mean when the population standard deviation is unknown and the model assumptions are appropriate.

7.10. Law of Large Numbers versus the Central Limit Theorem

The Law of Large Numbers (LLN) states, under suitable conditions, that a sample mean approaches the population expectation as the sample grows. The Central Limit Theorem (CLT) instead describes the approximate distribution of a standardized sum or mean.

They answer different questions:

• LLN concerns convergence toward the target.
• CLT concerns the scale and approximate shape of sampling fluctuations.

For the familiar independent identically distributed setting, the standard error of the sample mean scales as sigma/sqrt(n). More data reduces sampling uncertainty, but it does not automatically correct selection bias or a changing data-generating process.

Unit 8: Sampling and Estimation

8.1. Sampling Frame

A sampling frame is the list of population units eligible for selection. An incomplete frame can create coverage bias.

A sampling design defines the population, frame, sampling method, sample size, data collection process, and estimators.

8.2. Point and Interval Estimation

A point estimate provides a single value for a parameter.

A confidence interval expresses sampling uncertainty.

If the population standard deviation is known:

x̄ ± z(α/2)·σ/√n

If it is unknown and the normal model is appropriate:

x̄ ± t(α/2,n−1)·s/√n

In frequentist terms, a 95% confidence procedure produces intervals that cover the fixed but unknown parameter in about 95% of repeated samples.

8.3. Bias, Consistency, and Efficiency of an Estimator

An estimator is a rule that maps observed data to an estimate of an unknown parameter. It should be evaluated through repeated-sampling behavior, not by one observed data set alone.

• Bias is E[theta_hat] - theta.
• Consistency means theta_hat approaches the true parameter as sample size grows.
• Variance measures sample-to-sample fluctuation.
• Mean squared error combines variance and squared bias.
• Efficiency concerns obtaining the same target with lower uncertainty.

An unbiased estimator is not automatically the best estimator. A controlled increase in bias can reduce variance enough to lower total error, which connects naturally to regularization and the bias-variance trade-off in statistical learning.

Sufficiency concerns whether reducing the sample to a statistic T(X) preserves all sample information about a parameter. Conditional on a sufficient statistic, the distribution of the original sample does not depend on the unknown θ. In models satisfying the relevant factorization conditions, the Fisher–Neyman criterion checks whether the likelihood can be separated through T(X). Sufficiency does not imply unbiasedness or minimum variance.

Asymptotic efficiency must be interpreted under the assumed model and regularity conditions. Fisher information can be written I(θ) = E[(∂ log L(θ;X)/∂θ)²]; under appropriate conditions it also equals the negative expected second derivative of log likelihood. For an unbiased estimator in a regular one-parameter model, the Cramér–Rao bound gives Var(θ̂) ≥ 1/I(θ). This is not an unconditional variance bound for biased estimators, irregular families, or parameters on the boundary.

8.4. Finite Population Correction

For sampling without replacement when the sampling fraction is substantial:

√[(N−n)/(N−1)]

8.5. Sample Size

For a simple mean problem with target margin of error d:

n = [z(α/2)·σ/d]²

Standard error decreases approximately with 1/√n; halving the margin of error requires roughly four times the sample size.

8.6. Simple Random Sampling

Each unit has the same probability of selection. The design is theoretically simple but does not guarantee representation of small subgroups.

8.7. Systematic Sampling

k = N/n

Choose a random start among the first k units, then take every kth unit. Periodicity in the ordering can introduce bias.

8.8. Stratified Sampling

The population is divided into internally homogeneous strata and a sample is drawn from each:

N = N₁ + ... + Nₖ
n = n₁ + ... + nₖ

Efficiency improves when within-stratum variability is low and between-stratum differences are large.

Allocation may be equal, proportional, or based on Neyman allocation.

8.9. Cluster Sampling

The population is divided into natural clusters and a subset of clusters is sampled. This can reduce field cost, but similarity within clusters can reduce effective sample size.

Observations sampled from the same cluster are often correlated, so nominal sample size does not fully describe the amount of independent information. A common approximation for equal-sized clusters is the design effect DEFF ≈ 1 + (m−1)ρ, where m is average cluster size and ρ the intracluster correlation. For ten observations per cluster and ρ=0.2, this yields DEFF ≈ 2.8. The approximation indicates why a variance calculation assuming simple random sampling may be misleading; it cannot replace design-aware variance estimation with stratification, unequal weights or complex selection. Sampling units and observation units need separate definitions.

8.10. Weighting

With unequal selection probabilities:

x̄_w = Σ(wᵢxᵢ) / Σwᵢ

A complex design affects standard errors as well as point estimates.

Weights can represent inverse selection probabilities, nonresponse adjustment or calibration to known population totals; these interpretations are not interchangeable. For observations 10, 20, 30 and 40 with weights 1, 1, 2 and 2, the weighted mean is (10 + 20 + 2×30 + 2×40) / 6 = 28.33, whereas the unweighted mean is 25. Obtaining a weighted point estimate does not complete uncertainty estimation. Ignoring strata, clusters and finite-population information can underestimate standard errors. An analysis protocol therefore needs to identify the target population, explain the purpose of the weights and specify an appropriate variance estimator.

8.11. Missing Data

Common missingness mechanisms:

• MCAR: missingness is independent of observed and unobserved values.
• MAR: missingness can be explained by observed variables.
• MNAR: missingness also depends on the unobserved value itself.

Deleting incomplete records or replacing them with a simple mean is not generally unbiased.

Why a value is missing matters before deciding how to replace it. Missing completely at random (MCAR), missing at random conditional on observed variables (MAR), and missing not at random (MNAR) require different assumptions. If expensive items disproportionately omit their cost and that omission is unexplained by observed features, replacing missing costs with the overall mean can bias the estimate downward. A blank for a question that does not apply to the respondent is not a failed measurement and should be distinguished from nonresponse. Deletion, multiple imputation and sensitivity analysis must be chosen in relation to the estimand, the missingness mechanism and uncertainty propagation.

A missing-data mechanism is an assumption about why a value is unobserved. Under MCAR, missingness is independent of observed and unobserved values; under MAR, it no longer depends on the missing value after conditioning on observed covariates; under MNAR, dependence may persist even after conditioning. These mechanisms cannot be identified conclusively from missing-cell counts alone.

Multiple imputation does not substitute one supposedly true value. It samples multiple completed data sets from an imputation model, then combines their estimates while accounting for within-imputation and between-imputation uncertainty. Model incompatibility, an incorrect missingness assumption, or nonidentifiability can still undermine inference. Measurement error and censoring require observation models distinct from ordinary missing-value imputation.

8.12. Heavy Tails, Robust Estimation, and Performance Data

Robust statistics asks how much an estimator degrades under small departures from an idealized model. In robust statistics, the median and MAD are practical summaries, whereas the influence function describes the marginal effect of small contamination and the breakdown point describes how much contamination an estimator can tolerate before it ceases to provide meaningful information. In heavy-tailed or contaminated data, decisions based only on the mean and standard deviation can be misleading.

• Report mean latency together with median and high percentiles.
• Determine from context whether an outlier is an error or a legitimate rare event.
• Connect influence function and breakdown point to estimator selection.

8.13. Frequency, Binning, and Sampling Are Different Questions

Statistical sampling and signal sampling share a word but solve different problems. A Frequency Table summarizes observations by value or class counts, while Sturges' Rule gives a rough starting estimate for the number of classes. Sampling Rate, by contrast, describes how many signal samples are represented per second.

The distinction matters: sample size in statistics is a count of observations, whereas sampling rate in digital signal processing is a measurement rate over time. The Sampling Theorem completes the second context by relating the rate to a band-limited signal.

8.14. How Sampling Design Shapes Confidence Intervals

A confidence interval is produced by an estimation procedure applied to a sample; in the classical frequentist interpretation it should not be treated unconditionally as the posterior probability that a fixed parameter lies inside one realized interval. Its confidence level is the long-run coverage of the procedure over repeated samples under the stated assumptions. Random sampling, a suitable distributional model, and a defensible standard error are all material to that interpretation.

For independent observations, the standard error of the sample mean may be estimated by \(s/\sqrt{n}\). Treating a thousand events from one person as a thousand independent observational units can severely understate uncertainty. Within-cluster dependence, temporal autocorrelation, and selection bias can make confidence intervals misleadingly narrow even for very large datasets. Cluster-aware or block-based resampling may be considered when their assumptions match the data-generating process.

A sound report states the point estimate, confidence level, standard-error method, independent sampling unit, and missing-data policy. Statistical significance is not the same as practical effect size. A narrow interval does not repair systematic undercoverage in the original sampling frame.

8.15. Frequency Bins, Information Loss, and Distribution Representation

Converting continuous observations into a frequency table requires a bin count and interval width, both of which affect histogram appearance. Sturges' rule k ≈ 1 + log2(n) provides a starting point, but is not universally appropriate for large, heavy-tailed, or multimodal datasets. Equal-width bins can obscure outliers, while equal-frequency bins produce unequal resolution on the measurement scale.

Means, variances, and percentiles calculated from grouped data are often approximations to values computed from raw observations. Rare tail events can matter in critical systems, so p95/p99 and exceedance probabilities should be estimated from source measurements whenever possible. Histogram presentation choices must not be confused with the statistical estimation method.

Unit 9: Regression, Correlation, and Hypothesis Testing

9.1. Scatterplot

A scatterplot is the first diagnostic for two quantitative variables. Nonlinearity, outliers, and changing spread are often visible before formal modeling.

9.2. Linear Regression

Y = a + bX + ε

Least-squares estimates:

b = Σ[(xᵢ−x̄)(yᵢ−ȳ)] / Σ(xᵢ−x̄)²
a = ȳ − b·x̄

b is the expected linear change in Y for a one-unit increase in X.

Ordinary linear regression can be unsuitable for count data because it may predict negative values and ignores the changing variance associated with different means. The Poisson model equates conditional mean and variance; systematically larger observed variance calls for investigating overdispersion. A shift with a mean count of 4 and sample variance of 19 challenges the Poisson assumption, but those summaries alone do not establish that a negative-binomial model is correct. Excess zeros, unequal observation periods, and shared sources of variation also matter. Model choice should follow the data-generating process, not just a fit statistic.

9.3. Deriving the least-squares Criterion

Linear-regression coefficients need not be treated as memorized formulas; they follow from an explicit optimization problem. For observation pairs (xᵢ,yᵢ), let

ŷᵢ = a + b xᵢ

and define the residual

eᵢ = yᵢ − ŷᵢ.

Ordinary least squares minimizes

S(a,b) = Σ eᵢ²
       = Σ [yᵢ − (a + b xᵢ)]².

Minimizing the raw sum of residuals would be unsuitable because positive and negative residuals can cancel. Squaring removes the sign, penalizes large residuals more strongly, and produces a differentiable convex objective. This does not make squared error the unique valid loss for every data-generating process; it defines the classical least-squares criterion.

At an interior minimum the partial derivatives vanish:

∂S/∂a = −2 Σ [yᵢ − a − b xᵢ] = 0
∂S/∂b = −2 Σ xᵢ[yᵢ − a − b xᵢ] = 0.

These are the normal equations:

n a + b Σxᵢ       = Σyᵢ
a Σxᵢ + b Σxᵢ²    = Σxᵢyᵢ.

When the xᵢ values have nonzero variation, solving the system yields

b = Σ[(xᵢ−x̄)(yᵢ−ȳ)] / Σ(xᵢ−x̄)²
a = ȳ − b x̄.

For ordinary least squares with an intercept, the first-order conditions also imply

Σeᵢ = 0
Σxᵢeᵢ = 0.

The residuals therefore sum to zero, and the residual vector is orthogonal to both the intercept column and the predictor column. The fitted line also passes through (x̄,ȳ). These are useful algebraic checks on an implementation.

In matrix form,

y = Xβ + ε
S(β) = ||y − Xβ||²,

and setting the gradient to zero gives

XᵀX β = Xᵀy.

If X has full column rank, the solution is unique. Although the symbolic expression β=(XᵀX)⁻¹Xᵀy is valid under the corresponding invertibility condition, explicitly forming and inverting XᵀX can worsen numerical conditioning. QR- or SVD-based solutions are generally more reliable for large or ill-conditioned problems; this connects directly to Linear Algebra and Numerical Analysis.

9.4. Optimization Criterion versus Statistical Model

Computing least-squares coefficients and making statistical inferences about those coefficients are different problems. Minimizing S(β) does not require normally distributed errors. Standard errors, confidence intervals, and classical small-sample t/F results, however, depend on assumptions of the chosen probabilistic model.

A classical linear model commonly separates conditions such as

E[ε | X] = 0

from assumptions about independence or an appropriate dependence structure and about the variance of the errors. In the special case of conditionally Gaussian errors with constant variance, least-squares coefficient estimates also coincide with maximum-likelihood estimates under that Gaussian model.

The following statements are therefore not equivalent:

• "A least-squares line has been computed."
• "The statistical assumptions of the model have been validated."

The second requires separate examination of residual structure, heteroscedasticity, dependence, influential observations, the sampling process, and model specification.

9.5. Correlation

Pearson correlation:

r = Σ[(xᵢ−x̄)(yᵢ−ȳ)]
    / √[Σ(xᵢ−x̄)² · Σ(yᵢ−ȳ)²]

−1 ≤ r ≤ 1.

r≈0 indicates weak linear association, not necessarily absence of dependence.

For ordinal data, monotonic nonlinear relationships, or situations in which rank robustness matters, Spearman rank correlation may be more appropriate.

Correlation does not imply causation.

Correlation summarizes the direction and strength of a linear association, not an intervention effect or causal mechanism. Production intensity, for example, may increase both electricity use and recorded equipment failures. A positive correlation between these outcomes does not demonstrate that higher electricity consumption directly causes failure; intensity may be their common cause. Adjusting for observed covariates in multiple regression cannot automatically eliminate unmeasured confounders. A scientific report must state the hypothesized data-generating process, plausible alternative explanations and the boundary between observational association and an intervention claim.

9.6. Coefficient of Determination

For simple linear regression:

R² = 1 − SSE/SST

It summarizes the proportion of observed variability explained by the model. A high R² does not by itself establish correctness, causality, or out-of-sample validity.

9.7. Residuals

eᵢ = yᵢ − ŷᵢ

Systematic residual patterns can reveal model problems.

• curvature may indicate nonlinearity;
• changing spread may indicate heteroscedasticity;
• serial structure may indicate autocorrelation.

An outlier is not the same as an influential observation. Leverage and Cook's distance help assess observations that strongly affect fitted coefficients.

9.8. Multiple Regression

Y = β₀ + β₁X₁ + ... + βₖXₖ + ε

A coefficient represents the conditional linear association with its predictor while the other included predictors are held fixed.

Strong linear dependence among predictors creates multicollinearity. The variance inflation factor is one diagnostic; it is not an automatic variable-deletion rule.

9.9. Hypothesis Testing

The null hypothesis H₀ states the model or restriction under test; H₁ represents the alternative.

A p-value is the probability, assuming H₀ and the statistical model are correct, of obtaining a test statistic at least as extreme as the observed one.

It is not:

P(H₀ | data)

A small p-value does not imply a large, important, or causal effect. A large p-value does not prove equality.

A p-value is based on the probability, under the null hypothesis and its model assumptions, of observing a test statistic at least as extreme as the one obtained; it is not the probability that the null hypothesis is true. With a sufficiently large sample a negligible difference may be statistically significant without having practical importance. A report should therefore combine the test statistic with effect size, confidence interval, sampling design and a prespecified decision threshold. The frequentist interpretation of a 95% confidence procedure is that approximately 95% of its intervals would cover the fixed parameter over repeated samples; it does not assign a 95% probability to the parameter being inside one already computed interval.

A Wald test standardizes a parameter estimate by its standard error; a score test evaluates the likelihood slope under the null; and a likelihood-ratio test compares maximized log likelihoods of restricted and unrestricted models. Under suitable regularity conditions these procedures can agree asymptotically, but they need not agree for small samples, boundary parameters, or misspecified models.

The Neyman–Pearson construction gives a most-powerful test at a chosen size for specified simple hypotheses, not a universal test for every composite problem. Power also depends on sample size, effect, variability, and the rejection rule. A Bayes factor compares marginal likelihoods under the hypotheses; it is not another expression of a p-value. Causal interpretation additionally requires assumptions about intervention, confounding, and selection.

9.10. Effect Size

Statistical significance and effect magnitude are different.

A basic standardized mean difference is:

d = (x̄₁ − x̄₂) / s_pooled

Raw effect, standardized effect size, and confidence interval are more informative when reported together.

9.11. t-tests and Analysis of Variance

A t-test compares two group means. Welch's t-test is useful when equal variances cannot be assumed.

For three or more groups, one-way analysis of variance uses:

F = between-group variability / within-group variability

A significant F indicates incompatibility with the model of equal group means; it does not identify which groups differ.

Independent groups and repeated measurements on the same individuals arise from different data-generating processes. A paired t-test evaluates the within-pair differences rather than treating two measurements from each participant as independent. For independent groups with unequal variances, Welch's method can be preferable; a Levene test is not a universal switch determining the entire analysis. Chi-square independence tests additionally require attention to expected cell counts and observation independence, while Pearson correlation concerns linear association and can be strongly affected by outliers. The test cannot be selected from a file extension or from variables merely appearing numeric in a software interface.

9.12. Chi-square Test of Independence

Expected count:

Eᵢⱼ = (row totalᵢ · column totalⱼ) / grand total

Pearson chi-square:

χ² = Σ (Oᵢⱼ − Eᵢⱼ)² / Eᵢⱼ

Phi or Cramér's V can summarize association strength.

Tetrachoric correlation is appropriate only when binary observations can reasonably be modeled as thresholded versions of latent continuous variables that are jointly normal.

9.13. Multiple Testing

Testing many hypotheses on the same data increases the probability of false positives. Bonferroni procedures control family-wise error; Benjamini-Hochberg procedures control the false discovery rate.

9.14. Type I Error, Type II Error, and Nonparametric Tests

A Type I error rejects a true null hypothesis. A Type II error fails to reject a false null hypothesis. The significance level alpha controls the first type, while statistical power concerns the probability of detecting a real effect.

A p value is not the probability that the null hypothesis is true. It concerns the probability, under the null model and test assumptions, of observing a test statistic at least as extreme as the one obtained.

When parametric assumptions are unsuitable, rank- or empirical-distribution-based procedures can be useful. Mann-Whitney addresses two independent samples, Wilcoxon signed-rank paired measurements, Kruskal-Wallis more than two groups, and Kolmogorov-Smirnov distributional differences. Method selection should follow the dependence structure, measurement scale, and hypothesis rather than the vague rule "the data are not normal."

9.15. Resampling and Simulation

When an analytical sampling distribution is unknown or a closed-form calculation is difficult, resampling the observed data or running random simulation can make uncertainty measurable.

9.15.1. Bootstrap

The bootstrap repeatedly draws n observations with replacement from an observed sample of size n and recomputes the statistic of interest.

observed sample
   ↓ sample with replacement
bootstrap sample 1 -> θ1
bootstrap sample 2 -> θ2
...
bootstrap sample B -> θB

The empirical distribution of θ can be used to estimate standard error or construct uncertainty intervals. Bootstrap does not remove systematic bias in the data-generating process; an unrepresentative sample remains unrepresentative when resampled.

A parametric bootstrap resamples from a fitted probability model, whereas a nonparametric bootstrap usually resamples observed units with replacement. When observations are not independent and identically distributed, the resampling unit must match the dependency structure. Block bootstrap for time series and cluster-based resampling address different uncertainty structures from naive independent observation resampling.

Bootstrap replicates can estimate standard error, bias, and confidence intervals. Percentile, studentized, and bias-corrected intervals rely on different conditions and may fail for boundary estimators or very small samples. If the original analysis selects a model from the data, resampling only the final selected model can omit uncertainty introduced by selection.

9.15.2. Permutation Tests

Under a null hypothesis in which group labels are exchangeable, the labels can be shuffled repeatedly to construct a null distribution for a test statistic.

This can reduce reliance on a parametric distributional assumption, but it is not valid when the exchangeability condition is violated.

9.15.3. Monte Carlo Simulation

When an output depends on several random inputs, repeated simulation can approximate its distribution:

sample inputs from their distributions
        ↓
run model
        ↓
record output
        ↓
repeat N times

The resulting distribution can estimate quantities such as expected value, failure probability, or tail behavior.

Monte Carlo sampling error decreases with more simulation, but model error does not. Running a wrong model more times only estimates the wrong model more precisely.

9.15.4. Statistical Power

A non-significant result is not equivalent to evidence that an effect is absent. Sample size, effect magnitude, variance, and significance threshold determine the probability of detecting a real effect.

A study design therefore links:

effect size
+ noise / variance
+ sample size
+ alpha
-> power

Resampling and simulation do not automatically replace classical formulas. They are especially useful when assumptions are fragile or when the sampling distribution of a derived statistic is difficult to obtain analytically.

Unit 10: Stochastic Processes and Markov Chains

10.1. From a Random Variable to a Stochastic Process

A single random variable models one numerical outcome of an experiment. When uncertain behavior evolves over time, space, or another index, one variable is not enough. A stochastic process is a family of random variables indexed by that dimension.

X0, X1, X2, ...

When the index is time, Xt can represent the state or measurement of a system at time t. The index may be discrete or continuous, and the state space may independently be discrete or continuous.

Engineering applications include queue length, failure state, network load, user-session state, sensor measurements, and demand: all are quantities whose evolution can contain uncertainty.

10.2. The Markov Property

A process has the Markov property when the conditional distribution of the next state, given the full past, depends only on the current state.

future | present, past = future | present

This is a modeling assumption, not a universal law of physical systems. If the current state does not preserve enough information from the past, a Markov model loses relevant dependence. The remedy may be to enlarge the state or to use a richer temporal model.

10.3. Transition Probabilities and the Transition Matrix

In a finite-state discrete-time Markov chain, transition probabilities can be collected in a matrix. A three-state example is:

             next state
             A     B     C
current A    0.80  0.15  0.05
        B    0.20  0.60  0.20
        C    0.10  0.25  0.65

Each row is the probability distribution of the next state conditional on the current state, so each row sums to 1.

If the initial distribution is represented as a row vector, the distribution one step later is:

pi_next = pi_current * P

Multiple-step transitions follow by repeated application of the same matrix:

two steps  -> P * P
four steps -> P * P * P * P

This does not predict one certain future state. It propagates a distribution over possible states.

10.4. Stationary Distribution

A distribution is stationary if a transition leaves it unchanged:

pi = pi * P

A stationary distribution can summarize long-run behavior, but it is unsafe to assume that every chain converges to one unique stationary distribution. Reachability, reducibility, and periodicity affect convergence.

For finite chains, the idea that every state can reach every other state captures irreducibility. The absence of a forced common return period captures aperiodicity. These structural properties matter when interpreting long-run behavior.

10.5. Absorbing States

A state that cannot be left once entered is an absorbing state:

P(A -> A) = 1

Failure, completion, cancellation, or another irreversible outcome can sometimes be modeled this way. Questions can then concern not only the final state but also the probability of absorption and the expected number of transitions before it occurs.

10.6. Modeling Boundary

A Markov chain is not created simply by counting adjacent labels in sequential data. State meaning, observation interval, and whether the current state actually captures relevant history must be stated first.

Different state definitions can produce different chains from the same records. A disciplined workflow is:

define states
→ define observation and time scale
→ count transitions
→ estimate transition probabilities
→ inspect structural properties
→ analyze long-run or absorbing behavior
→ interpret with domain knowledge

A Markov model can be descriptive or predictive; by itself it does not establish causality.

10.7. Data Analysis: Distribution, Time, and Concentration

10.7.1. Frequency and Empirical Distribution

Relative frequency:

pᵢ = fᵢ / Σfᵢ

Empirical cumulative distribution function:

Fₙ(x) = (1/n)·ΣI(Xᵢ ≤ x)

A histogram summarizes distributional shape through bins; its appearance depends on bin width.

10.7.2. Cumulative Share and Concentration

Category share:

pᵢ = xᵢ / Σxⱼ

Cumulative share of the top k categories:

Cₖ = Σ(i=1..k)pᵢ

Pareto analysis is useful for concentration, but the 80/20 ratio is not a universal law.

Herfindahl-Hirschman Index:

HHI = Σpᵢ²

Shannon entropy:

H = −Σpᵢln(pᵢ)

Effective diversity:

Dₑ = eᴴ

The Gini coefficient measures inequality. HHI, Gini, and entropy capture different properties and are not interchangeable.

10.7.3. Time-series Summaries

Absolute change:

Δₜ = xₜ − xₜ₋₁

Relative change:

rₜ = (xₜ − xₜ₋₁) / xₜ₋₁

Relative change is undefined when the denominator is zero.

Moving average:

MAₜ = (1/m)·Σ(i=0..m−1)xₜ₋ᵢ

A larger window smooths more strongly but reacts more slowly to change.

A simple short-term volatility measure is the standard deviation of successive differences:

jitter = SD(xₜ − xₜ₋₁)

A linear time trend can be written as:

xₜ = a + b·t + εₜ

The slope b summarizes linear direction and rate of change; it does not imply causality.

Daily, hourly, and weekday profiles help distinguish calendar structure from unusual behavior.

10.7.4. Point Deviations, Local Level, and Persistent Change

A single high observation, a rising local level, and a persistent regime shift are different time-series problems. Compressing them into one "anomaly score" may look convenient, but it hides which behavior is actually being measured.

A robust baseline can use the median for location and median absolute deviation for spread:

m = median(x)
MAD = median(|x_i - m|)

When a scale comparable to standard deviation is useful:

sigma_r = MAD / 0.6745
z_r = (x_t - m) / sigma_r

This is valuable when a few extreme observations would pull the mean and standard deviation. MAD = 0 remains a real boundary case; a system should use a documented fallback, a longer history, or another spread measure rather than divide blindly.

An exponentially weighted moving average (EWMA) describes the recent level by assigning the newest observation the greatest weight and letting older weights decay geometrically:

E_t = lambda * x_t + (1 - lambda) * E_(t-1)
0 < lambda <= 1

A larger lambda reacts faster, while a smaller value smooths more aggressively. A value such as lambda = 0.30 can be used as a transparent engineering choice when moderate smoothing is desired. EWMA need not be presented as forecasting; it is often more useful as a local baseline. A classical reference is S. W. Roberts, Control Chart Tests Based on Geometric Moving Averages (1959): https://doi.org/10.1080/00401706.1959.10489860

CUSUM answers a different question: are modest deviations accumulating consistently in one direction? With robustly standardized values,

z_t = (x_t - m) / sigma_r

a two-sided cumulative scheme can be written as:

S+_t = max(0, S+_(t-1) + z_t - k)
S-_t = min(0, S-_(t-1) + z_t + k)

k controls how much small variation is ignored, while a decision threshold h determines when accumulated evidence becomes a signal. An isolated spike can therefore disappear quickly, whereas a sequence of moderate deviations in the same direction can become visible. The classical starting point is E. S. Page, Continuous Inspection Schemes (1954): https://doi.org/10.1093/biomet/41.1-2.100

The methods answer distinct questions:

Robust MAD   -> How unusual is the current observation?
EWMA         -> Where is the recent local level?
CUSUM        -> Are small deviations accumulating persistently?
Regression   -> How strong is the longer-term linear direction?

Keeping these signals separate is usually more interpretable than producing one composite score. A current observation may be individually ordinary while EWMA has moved upward and CUSUM indicates a persistent shift. Conversely, a one-day spike may have a large robust deviation but leave no lasting cumulative signal.

Time-axis construction is part of the model. If the observed date interval is known, days with genuinely zero events should usually be represented by zero rather than removed. Computing a baseline only from active days can inflate the apparent normal level. A missing day is different, however: zero observation and missing observation are not equivalent. Data acquisition gaps must remain distinguishable from observed inactivity.

10.7.5. Correct Aggregation of Period Shares

A simple mean of daily percentages does not equal the overall period share when daily totals differ.

The period share is:

P = Σ(category value) / Σ(overall total)

Ratios should not be aggregated without their denominators.

10.7.6. Data Quality and Denominators

Keep distinct:

physical record count
logical observation count
valid analysis count

Duplicate physical records can inflate denominators; incorrect deduplication can remove real events.

0, missing data, absence of a record, and unpublished data are different states. Treating a missing time point as zero can create artificial drops and rebounds.

10.7.7. Description, Anomaly, and Evidence

Descriptive statistics summarizes observed data. Exploratory analysis exposes patterns. An anomaly score prioritizes observations for review. None of these alone establishes causality or proof.

A disciplined sequence is:

observation definition
→ data quality
→ correct denominator
→ descriptive summary
→ distribution and temporal pattern
→ association or anomaly indicator
→ assumption checks
→ domain knowledge
→ inference when justified

10.8. Effect Size, Multiple Testing, and Reproducibility

Statistical significance is not the same as practical importance. With very large samples, tiny effects can produce small p-values, so effect sizes and confidence intervals should accompany hypothesis tests.

Testing many hypotheses on the same data increases false positives. Family-wise corrections or false-discovery-rate methods can control that risk depending on the goal.

Changing the analysis plan after inspecting the data can make results optimistic. Predefined outcomes, exclusions, and independent validation improve reproducibility.

10.9. Reporting Statistical Results in Context

A p-value does not describe effect size or practical importance. Effect size, confidence intervals, sample size, and model assumptions make the result more interpretable.

Assumptions should be checked with residual plots and appropriate diagnostics. In regression, linearity, independence, variance structure, and influential observations may each matter. Violations do not automatically invalidate a model, but they change the limits of interpretation.

Multiple hypothesis tests increase false-positive risk. The correction strategy should follow the analysis goal and preferably be decided before inspecting final results.

10.10. Hitting Time and first-passage Time

A stochastic process raises not only the question "which state comes next?" but also when a target state is reached for the first time. For a target set A, the hitting time is:

T_A = inf { t >= 0 : X_t in A }

If the process starts inside A, this definition permits T_A = 0. When the question is return to a state after departure, one usually imposes t >= 1 and studies a first return time. That small change affects the interpretation of the expectation.

For a finite Markov chain, the expected hitting time h_i can, under the usual finiteness conditions, be obtained from a linear system:

h_i = 0                                  if i in A
h_i = 1 + sum_j P[i,j] * h_j            if i not in A

This can provide an analytical answer without simulation. Two quantities must still be kept separate:

P(T_A < infinity)    -> probability of eventually reaching A
E[T_A]               -> expected time required to reach A

In infinite-state chains, reaching the set with probability one does not by itself guarantee a finite expected hitting time. The distinction matters in reliability and queueing analysis.

Hitting times model questions such as the first time a queue reaches capacity, a system enters a failure state, inventory crosses a lower threshold, or a random walk reaches a target region.

10.11. Multiple Testing and Rare-Event Alert Rates

Testing many candidate outliers at a fixed significance level can generate substantial numbers of false alerts. If all m null hypotheses hold, the expected false-rejection count is mα; independence is not needed for that expectation. Benjamini–Hochberg false-discovery-rate control serves a different purpose from family-wise error control. When event prevalence is low, even a highly specific test may have limited positive predictive value under Bayes’ rule. Alert loss and red-flag verification are developed in Anomaly Detection.

Unit 11: From Statistics to Statistical Learning

11.1. From Inference to Prediction

Sampling, probability distributions, Bayes' theorem, and regression form the basis of statistical machine learning as well as classical statistical inference. The main difference is often the center of the question rather than the mathematics itself. Inferential statistics emphasizes uncertainty about parameters or effects; statistical learning often emphasizes prediction error on unseen observations.

sample
↓
probabilistic / geometric model
↓
parameter or decision function
↓
unseen data
↓
prediction + uncertainty / error measurement

Strong predictive accuracy does not establish causality, and statistical significance of a coefficient does not guarantee the best out-of-sample predictor.

11.2. Likelihood and Parameter Estimation

For observations z_1,...,z_n and parameter θ, the likelihood is:

L(θ) = Π p(z_i | θ)

Maximum likelihood estimation chooses the parameter maximizing this quantity. Computation normally uses the log-likelihood:

log L(θ) = Σ log p(z_i | θ)

A Bayesian treatment adds a prior over parameters. Maximum a posteriori estimation can be written as:

θ_MAP = argmax p(data | θ) p(θ)

Bayes' theorem therefore supports parameter estimation as well as event-probability calculations.

11.3. Loss Functions

Prediction error is represented by a loss function. Squared or absolute error can be used for regression; log loss/cross-entropy is common for probabilistic classification. Different losses can define different optimal models on the same data.

Operational cost may differ from mathematical loss. When false negatives are much more costly than false positives, the threshold with the highest accuracy may not be the right decision threshold.

11.4. Regularization and Generalization

Low training error does not guarantee low population error. A flexible model can fit sample noise. Regularization constrains parameters or model capacity to reduce that risk.

Ridge and Lasso are explicit examples in linear models. The same generalization problem appears through tree depth, neighbor count k, SVM C/γ, or the initialization and cluster count of K-Means.

11.5. Training, Validation, and Test

The roles of data should be separated:

• training data estimate parameters,
• validation data support model and hyperparameter selection,
• test data estimate generalization after choices are complete.

Cross-validation improves evaluation efficiency on limited data. Scaling, feature selection, imputation, or other preprocessing must remain inside the training process; fitting them on the complete data leaks test information.

11.6. Statistical-learning Methods

Different assumptions lead to different method families:

• probabilistic: Naive Bayes, Gaussian mixtures, Bayesian networks, and HMMs,
• distance based: k-Nearest Neighbors,
• margin based: Support Vector Machines,
• partition based: decision trees and ensembles,
• clustering: K-Means, K-Medoids, DBSCAN, and Fuzzy C-Means,
• dimensionality reduction: PCA, SVD, and LDA,
• sequence statistics: n-gram language models.

Their algorithms, evaluation, and engineering limits are developed in Statistical Learning and Machine Learning.

11.7. Calibration, Uncertainty, and Decision Thresholds

A classifier score and a real-world probability are not the same concept. If examples receiving p̂=0.8 are positive about 80% of the time, the model is calibrated in that region. Accuracy can be high while probability estimates remain poorly calibrated.

The decision threshold is also a separate business choice:

score → threshold → decision

Changing the threshold changes the false-positive/false-negative tradeoff without changing model parameters.

11.8. Distribution Shift and Sampling Limits

Statistical learning usually relies on training data being sufficiently representative of future data. Changes in population, sensors, user behavior, or collection policy can break that relationship. A large dataset is not automatically representative; millions of observations collected through a biased mechanism remain biased.

11.9. A Continuum Between Statistics and Learning

Regression, likelihood, regularization, and Bayesian estimation belong to both statistical inference and machine learning. A more useful view is a continuum:

parameter / effect uncertainty ← statistics → prediction / generalization

Reliable systems often need both: good prediction and defensible evaluation of sampling and uncertainty.

Unit 12: Multivariate, Bayesian, and Temporal Statistics

12.1. Multivariate Normal Distribution and the Covariance Matrix

With multiple continuous variables, location is represented by a mean vector and joint variability by a covariance matrix. In a multivariate normal model, the covariance matrix contains both marginal variances and linear dependence information.

Euclidean distance treats axes as equally scaled and geometrically independent. Mahalanobis distance adapts to covariance structure:

d_M(x,mu)^2 = (x-mu)^T Sigma^(-1) (x-mu).

This gives a data-aware geometry for correlated or differently scaled features. When Sigma is ill-conditioned or rank deficient, a numerically appropriate factorization or pseudoinverse is preferable to forming an explicit inverse.

A joint multivariate distribution specifies both marginal behavior and dependence. With continuous marginal distribution functions F₁,…,Fₚ, a copula representation writes the joint distribution as F(x₁,…,xₚ) = C(F₁(x₁),…,Fₚ(xₚ)). This separates marginal distribution choices from the dependence model. Zero Pearson correlation does not generally imply independence or lack of tail dependence.

Extreme-value analysis can use block maxima or exceedances above a high threshold; these are distinct sampling formulations. Generalized extreme-value and generalized Pareto families arise under particular limit conditions. Rare-event probabilities remain sensitive to stationarity, threshold or block choice, model fit, and parameter uncertainty.

12.2. Bayesian Inference and the Posterior Predictive Distribution

Bayesian inference treats parameter uncertainty explicitly:

posterior ∝ likelihood * prior.

The prior represents information before the observed data, the likelihood measures compatibility of the data with parameter values, and the posterior combines them.

A single MAP estimate does not preserve the full posterior. The posterior predictive distribution integrates parameter uncertainty when predicting a future observation. A frequentist confidence interval and a Bayesian credible interval are therefore not the same object and carry different probability interpretations.

For Bernoulli observations, a Beta(α, β) prior over the unknown success probability θ is conjugate to the likelihood. After h successes and t failures, the posterior is Beta(α+h, β+t). With a Beta(2,2) prior and three successes followed by one failure, the result is Beta(5,3) and the posterior mean is 5/8 = 0.625. The update reshapes the entire uncertainty distribution instead of merely changing a point estimate.

For a future Bernoulli trial, the posterior predictive probability integrates over the posterior parameter distribution. Here P(X_new=1 | data) = E[θ | data] = 5/8. This distinguishes a posterior density over parameters from a predictive distribution over future observations.

Prior selection can materially affect the posterior, particularly with limited data or weak identifiability. For suitable regular models, a Jeffreys prior can be written π(θ) ∝ √I(θ) in terms of Fisher information. Its parameter-transformation property does not make it proper or suitable in every problem. Empirical Bayes estimates some prior parameters from data, whereas hierarchical Bayes models between-group variability through another probability level.

When posterior integrals are intractable, a Laplace approximation uses a local second-order expansion around a mode, while variational inference optimizes over a restricted family of distributions. Monte Carlo approaches approximate integrals using random draws. Their convergence behavior, uncertainty representation, and sensitivity to multimodal posteriors differ.

12.3. Why MCMC Is Needed

When a posterior normalizing constant or expectation cannot be computed analytically, Markov Chain Monte Carlo can generate dependent samples whose stationary distribution is the target distribution. The objective is not to pretend the samples are independent, but to construct a chain with the correct long-run distribution.

Metropolis-Hastings provides a general proposal-and-accept framework, while Gibbs sampling cycles through conditional distributions when they are available. MCMC output requires diagnostics: burn-in choices, autocorrelation, effective sample size, and agreement across multiple chains all matter.

12.4. Stationarity, Autocorrelation, and Change Points

Time-ordered data contain information in their order. Stationarity formalizes which distributional properties remain unchanged over time; under weak stationarity the mean is constant and covariance depends only on lag.

Autocorrelation measures linear dependence between a series and lagged versions of itself. When autocorrelation is strong, random train-test splitting or methods assuming independent observations can be misleading.

A change point is a candidate persistent structural change in a mean, variance, relationship, or other data-generating parameter. A single outlier is not automatically a change point. Temporal statistics therefore forms a natural bridge among statistical inference, signal processing, and production monitoring.

A change point in a time series is not equivalent to one unusually large observation. A shift in mean, variance, or serial dependence represents a different hypothesis. If a sensor's first 50 readings fluctuate near 20 and its next 50 near 24, changes in clocking, calibration, and operating regime should be investigated before treating the shift as a statistical finding. Selecting the break position from the observations and then applying an unadjusted test to the same data inflates false-positive risk. A held-out validation period or a method that accounts for selection is needed.

12.5. Numerical Reliability of MCMC Output

Samples generated by a Markov chain are not independent draws from the same distribution. Having a stationary distribution does not by itself establish the reliability of an estimate computed from a finite run. For the posterior expectation of a function h(θ), write μ̂_N = (1/N) Σ(t=1..N) h(θ_t). Under suitable regularity conditions, its asymptotic variance depends on lagged covariances as well as marginal variance:

Var(μ̂_N) ≈ [γ₀ + 2 Σ(k≥1) γ_k] / N
γ_k = Cov(h(θ_t), h(θ_(t+k)))

Using an independent-sample standard error without estimating the long-run variance in brackets may understate uncertainty for positively autocorrelated chains. The Monte Carlo standard error (MCSE) quantifies numerical uncertainty from a finite algorithm run, not the dispersion of the posterior distribution itself. Effective sample size expresses that uncertainty on an independent-sample scale; it is not interchangeable with the number of iterations.

Running multiple chains from dispersed initial states and comparing within-chain and between-chain variation helps diagnose convergence. No single diagnostic, however, guarantees adequate exploration of every mode of a difficult multimodal distribution. Removing warm-up draws cannot by itself repair poor mixing. A stopping decision should consider MCSE, effective sample size and the behavior of separate chains for the particular parameters or functionals of interest.

12.6. Weight Concentration in Importance Sampling

When direct sampling from a target density p(x) is difficult, samples can be drawn from a proposal q(x) that is positive wherever the relevant target has support. With unnormalized importance weights w_i = p(x_i) / q(x_i), the self-normalized estimator is

Î = Σ(i=1..N) w_i f(x_i) / Σ(i=1..N) w_i

If a handful of observations dominate the weights, the estimator can remain unstable despite a large nominal sample. The quantity N_eff,w = (Σ_i w_i)² / Σ_i w_i² is a useful weight-concentration diagnostic, not the same effective sample size used for autocorrelated MCMC output. Increasing the sample count without examining the proposal's support and the tails of the weight distribution is not a sufficient remedy.

References

• Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.

• Douglas C. Montgomery, George C. Runger. Applied Statistics and Probability for Engineers, 7th Edition. Wiley, 2018.

• George Casella, R. L. B. Statistical Inference, 2nd Edition. Duxbury, 2002.

• Göran Kauermann; Helmut Küchenhoff; Christian Heumann. Statistical Foundations, Reasoning and Inference: For Science and Data Science. Springer, 2021. https://doi.org/10.1007/978-3-030-69827-0
• Kevin P. Murphy. Probabilistic Machine Learning: An Introduction. MIT Press, 2022.

• Mor Harchol-Balter. Performance Modeling and Design of Computer Systems: Queueing Theory in Action. Cambridge University Press, 2013.

• Morris H. DeGroot, Mark J. Schervish. Probability and Statistics, 4th Edition. Pearson, 2012.

• NIST/SEMATECH. e-Handbook of Statistical Methods. National Institute of Standards and Technology.

• Peter J. Huber; Elvezio M. Ronchetti. Robust Statistics. 2nd ed. Wiley, 2009.

• Sheldon M. Ross. A First Course in Probability, 10th Edition. Pearson, 2018.

• Walter W. Piegorsch, Richard A. Levine, Hao Helen Zhang, Thomas C. M. Lee (eds.). Computational Statistics in Data Science. Wiley, 2022. ISBN 9781119561071.

• William G. Cochran. Sampling Techniques, 3rd Edition. Wiley, 1977.
• Charu C. Aggarwal. Outlier Analysis, 2nd ed.. Springer, 2017. https://doi.org/10.1007/978-3-319-47578-3