Probability and Statistics: Distributions, Sampling and Regression

I prepared my probability and statistics notes during undergraduate Computer Engineering courses in the 2013-2014 period. This revision preserves the original sequence from descriptive statistics and probability through random variables, distributions, sampling, regression, and correlation, while reviewing terminology and conditions with later sources.

Unit 1: Introduction to Statistics

Fundamental concepts

Statistics studies how data are collected, summarized, modeled, and used for inference. A population is the complete set of units under study; a sample is a subset observed to learn about that population. A parameter describes a population, while a statistic is calculated from a sample.

Levels of measurement

Data can be classified as nominal, ordinal, interval, or ratio scale. The measurement level determines which comparisons and numerical operations are meaningful.

• Nominal data identify categories.
• Ordinal data add an order relation.
• Interval scales have meaningful differences but no absolute zero.
• Ratio scales have meaningful differences and ratios because zero has an absolute interpretation.

Margin of error and decision making

A sample does not normally reproduce the population exactly. Sampling variability therefore needs to be quantified. Point estimates, standard errors, confidence intervals, and hypothesis tests are different tools for describing uncertainty rather than guarantees of exact population values.

Relationship between statistics and probability

Probability starts with a model and studies possible outcomes. Statistical inference works in the reverse direction: observed data are used to learn about an unknown process or population. Probability distributions provide the mathematical language for sampling uncertainty.

Statistical series

Raw observations can be organized into simple, frequency, or grouped series. Grouping improves readability for large data sets but can discard some individual detail.

Arithmetic mean

For observations x1,...,xn,

x_bar = (1/n) Σ x_i.

The mean uses every observation and is sensitive to extreme values.

Geometric mean

For positive observations,

G = (Π x_i)^(1/n).

It is useful for multiplicative rates, growth factors, and proportional changes.

Root-mean-square

The quadratic or root-mean-square value is

RMS = sqrt[(1/n) Σ x_i^2].

It emphasizes larger magnitudes and is important in signal and engineering measurements.

Median

The median is the central ordered observation, or the mean of the two central observations when the sample size is even. It is more resistant to extreme values than the arithmetic mean.

Mode

The mode is the most frequent value or class. A data set can have one mode, several modes, or no uniquely dominant mode.

Relationship among mean, median, and mode

For a symmetric unimodal distribution, the three measures can coincide. In skewed distributions they usually differ, and their relative locations provide qualitative information about the direction of skewness.

Measures of dispersion

Range, variance, standard deviation, and related measures describe spread. The sample variance is commonly written as

s^2 = Σ(x_i - x_bar)^2 / (n-1)

when estimating population variance from an independent sample under the usual framework.

Properties of variance

For a constant a, Var(a)=0. For constants a,b,

Var(aX+b) = a^2 Var(X).

For two random variables,

Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y).

Independence implies zero covariance when the required moments exist, but zero covariance does not in general imply independence.

Unit 2: Set Theory

Sets and subsets

A set is a collection of distinct elements. A ⊆ B means every element of A belongs to B. Set notation provides the language for events in probability.

Intersection, union, and complement

A ∩ B
A ∪ B
A^c

represent intersection, union, and complement. These operations correspond to logical AND, OR, and NOT in event algebra.

Properties of operations

Union and intersection are commutative and associative, distribute over each other, and satisfy De Morgan's laws:

(A ∪ B)^c = A^c ∩ B^c
(A ∩ B)^c = A^c ∪ B^c.

Unit 3: Permutations and Combinations

Counting principles

The addition principle applies to mutually exclusive alternatives; the multiplication principle applies to sequential choices. Counting should be organized before formulas are selected.

Permutation

When order matters, the number of arrangements of r distinct objects selected from n is

P(n,r) = n!/(n-r)!.

Combination

When order does not matter,

C(n,r) = n!/[r!(n-r)!].

Selection criterion

The central distinction is simple: if changing the order creates a different outcome, use permutation logic; if it does not, use combination logic.

Unit 4: Introduction to Probability

Fundamental concepts

A random experiment has an outcome that is not known with certainty in advance. The set of all possible outcomes is the sample space S; an event is a subset of S.

Definition of probability

For equally likely finite outcomes,

P(A) = |A|/|S|.

More generally, probability is a measure satisfying nonnegativity, normalization, and countable additivity.

Basic theorems

0 ≤ P(A) ≤ 1
P(S) = 1
P(A^c) = 1 - P(A)

and

P(A ∪ B) = P(A) + P(B) - P(A ∩ B).

Conditional probability

For P(B)>0,

P(A|B) = P(A ∩ B)/P(B).

Conditional probability updates the event space after information that B occurred is known.

Independence

Events A and B are independent when

P(A ∩ B) = P(A)P(B).

Equivalently, when the relevant probabilities are nonzero, learning one event does not change the probability of the other.

Law of total probability

If B1,...,Bk form a partition of the sample space with positive probabilities, then

P(A) = Σ_i P(A|B_i)P(B_i).

Bayes' theorem

Bayes' theorem reverses conditioning:

P(B_j|A) = P(A|B_j)P(B_j) / Σ_i P(A|B_i)P(B_i).

It combines prior probabilities with the likelihood of the observed evidence.

Unit 5: Random Variables and Their Distributions

Random-variable concept

A random variable maps outcomes in the sample space to real numbers. It can be discrete, taking countable values, or continuous, described through a density over intervals.

Discrete distribution

A probability mass function p(x) satisfies

p(x) ≥ 0
Σ_x p(x) = 1.

Expected value of a discrete variable

E(X) = Σ_x x p(x).

Expected value is a probability-weighted average, not necessarily an attainable outcome.

Variance of a discrete variable

Var(X) = E[(X-E(X))^2]
       = E(X^2) - [E(X)]^2.

Continuous distribution

A continuous variable has a density f(x) satisfying

f(x) ≥ 0
∫ f(x) dx = 1.

Probabilities are areas:

P(a ≤ X ≤ b) = ∫_a^b f(x) dx.

For a truly continuous variable, the probability of one exact point is zero.

Expected value and variance of a continuous variable

E(X) = ∫ x f(x) dx
Var(X) = ∫ (x-μ)^2 f(x) dx.

Unit 6: Discrete Probability Distributions

Discrete uniform distribution

If n possible values are equally likely, each has probability 1/n. Its exact mean and variance depend on the chosen support.

Bernoulli distribution

A Bernoulli variable models one trial with outcomes conventionally coded 1 and 0:

P(X=1)=p
P(X=0)=1-p.

Then

E(X)=p
Var(X)=p(1-p).

Binomial distribution

The binomial distribution counts successes in n independent Bernoulli trials with common success probability p:

P(X=k) = C(n,k)p^k(1-p)^(n-k).

Its mean and variance are

E(X)=np
Var(X)=np(1-p).

Multinomial distribution

The multinomial model generalizes binomial trials to more than two mutually exclusive categories. The category counts sum to the fixed number of trials.

Poisson distribution

For k=0,1,2,...,

P(X=k) = e^-λ λ^k/k!.

The theoretical Poisson distribution satisfies

E(X)=Var(X)=λ.

Sample estimates of mean and variance are not expected to be exactly equal. Strong overdispersion or underdispersion can indicate that a Poisson model is inadequate for the observed data.

Geometric and negative-binomial distributions

The geometric distribution models waiting until a first success under a specified counting convention. The negative-binomial distribution extends the idea to waiting for a specified number of successes.

Unit 7: Continuous Probability Distributions

Normal distribution

The normal density with mean μ and standard deviation σ is

f(x) = [1/(σ sqrt(2π))] exp[-(x-μ)^2/(2σ^2)].

It is symmetric around μ and has variance σ^2.

Standard normal distribution

Standardization uses

Z = (X-μ)/σ.

The standard normal has mean 0 and variance 1.

Importance of the normal distribution

Normal models arise in measurement error and many aggregated phenomena, and the central limit theorem explains why suitably standardized sums or averages often approach normality under broad conditions. This is an approximation principle, not a claim that every data set is normal.

Exponential distribution

For x≥0,

f(x)=λe^-λx.

It is associated with waiting times in a Poisson-process model and has the memoryless property.

Gamma distribution

The gamma family generalizes the exponential distribution and models positive continuous quantities such as waiting time until multiple events under a Poisson-process interpretation.

Beta distribution

The beta distribution is supported on [0,1] and is flexible for modeling proportions and probabilities.

Chi-square distribution

A chi-square variable can be formed as a sum of squares of independent standard-normal variables. It plays a central role in variance inference and contingency-table methods.

F distribution

The F distribution arises from a ratio of scaled independent chi-square variables. It is used in comparing variances and in analysis-of-variance and regression settings.

t distribution

Student's t distribution arises when a standardized mean uses an estimated standard deviation under the standard normal-sampling model. It has heavier tails than the normal distribution and approaches the normal as the degrees of freedom increase.

Relationships among distributions

Bernoulli trials aggregate to binomial counts. Under appropriate limiting conditions, binomial distributions can be approximated by Poisson or normal models. Exponential waiting times and Poisson counts are connected through the Poisson-process model. Chi-square, t, and F distributions are built from normal and chi-square quantities in classical inference.

Unit 8: Sampling

Fundamental concepts

A census observes the entire population; sampling observes part of it. A sampling frame is the operational list or mechanism from which units are selected. Bias and random sampling error are different problems.

Sampling plan

A sampling plan defines the target population, sampling unit, frame, selection method, sample size, and operational procedure before data collection.

Estimation

A point estimator returns one value; an interval estimator gives a range together with a confidence level under the model and sampling design. Unbiasedness, variance, and consistency are different estimator properties.

Simple random sampling

In simple random sampling, samples of a fixed size have equal selection probability under the standard design. Random selection protects against systematic preference in choosing units.

Systematic sampling

After a random start, systematic sampling selects units at a regular interval. It is operationally simple but can interact badly with hidden periodicity in the ordered frame.

Stratified sampling

The population is divided into internally meaningful strata, and sampling occurs within each stratum. Proper stratification can improve precision and guarantee representation of important groups.

Cluster sampling

The population is divided into clusters, and selected clusters provide the observed units according to the design. It can reduce field cost but often changes the variance structure because units within a cluster can be similar.

Unit 9: Regression and Correlation Analysis

Fundamental concepts

Regression models a systematic relationship between a response variable and one or more explanatory variables. Correlation summarizes the strength and direction of association; it does not by itself establish a causal mechanism.

Model forms

Relationships can be linear or nonlinear. A model is selected from the scientific problem and observed pattern rather than from correlation alone.

Linear regression

For simple linear regression,

Y = β0 + β1 X + ε.

Least squares estimates the coefficients by minimizing the sum of squared residuals. Interpretation depends on the model assumptions and study design.

Two regression equations

Classical bivariate descriptions distinguish regression of Y on X from regression of X on Y. They solve different prediction problems and should not be treated as the same line unless special conditions hold.

Correlation coefficient

Pearson correlation is

r = Cov(X,Y)/(s_X s_Y)

in sample notation, with values between -1 and 1. It measures linear association and is sensitive to outliers.

Coefficient of determination

In simple linear regression with an intercept,

R^2 = r^2.

More generally, R^2 measures the proportion of response variation represented by the fitted regression relative to the selected baseline model.

Correlation and causation

Correlation can arise from causal influence, common causes, selection effects, time trends, measurement structure, or chance. Causal inference requires design and assumptions beyond a correlation coefficient.

Contingency coefficient

For categorical data, association can be summarized through quantities derived from a contingency table and the chi-square statistic. These measures are not interchangeable with Pearson correlation for quantitative variables.

General Conceptual Framework

The course connects data description with probability models and inference:

Observed data
    ↓
Descriptive statistics
    ↓
Probability model
    ↓
Random variables and distributions
    ↓
Sampling design
    ↓
Estimation / regression / association
    ↓
Decision with quantified uncertainty

The main discipline is to keep population, sample, model, and observed statistic distinct. Probability supplies uncertainty models; statistics uses data to estimate or test unknown features of those models.

Conceptual Distinctions

Population ≠ sample. A sample is observed to learn about a larger target population.

Parameter ≠ statistic. A parameter describes the population; a statistic is calculated from sample data.

Mean ≠ median ≠ mode. They summarize different aspects of location and respond differently to skewness and outliers.

Variance of a sum is not generally the sum of variances.

Var(X+Y)=Var(X)+Var(Y)+2Cov(X,Y).

Var(X+Y)=Var(X)+Var(Y) holds when Cov(X,Y)=0; independence is sufficient but not necessary for that equality.

Conditional probability ≠ independence. Independence is the special case in which conditioning does not change the relevant probability.

Probability mass ≠ density value. For a continuous variable, probability is an integral over an interval; the probability of an exact point is zero.

Poisson theoretical equality ≠ exact sample equality. The model has E(X)=Var(X)=λ, while sample mean and variance fluctuate. Strong dispersion mismatch can question model adequacy.

t ≠ Z. When the population standard deviation is unknown and the standard t-model assumptions hold, Student's t distribution is used in the classical standardized-mean framework, especially for small samples.

Sampling error ≠ sampling bias. Larger samples can reduce random error but do not automatically remove systematic bias.

Correlation ≠ causation. Association alone does not identify a causal mechanism.

Interpolation of a statistical trend ≠ causal explanation. A fitted line describes the model relationship in the observed data; interpretation requires context and assumptions.

References

• Meral Yay. İstatistik ve Olasılık (TBIL209 ders materyalleri). Ahmet Yesevi Üniversitesi, Bilgisayar Mühendisliği Bölümü.
• Sheldon M. Ross. A First Course in Probability, 10th Edition. Pearson, 2018.
• Douglas C. Montgomery, George C. Runger. Applied Statistics and Probability for Engineers, 7th Edition. Wiley, 2018.
• Morris H. DeGroot, Mark J. Schervish. Probability and Statistics, 4th Edition. Pearson, 2012.
• George Casella, Roger L. Berger. Statistical Inference, 2nd Edition. Duxbury, 2002.
• William G. Cochran. Sampling Techniques, 3rd Edition. Wiley, 1977.