Linear Algebra: Matrices, Linear Systems and Eigenvalues

Linear algebra provides a representation language that appears repeatedly across computer engineering. Linear systems, vector spaces, determinants, eigenvalues and eigenvectors are presented with their assumptions and structural interpretations kept explicit.

Mechanical matrix arithmetic can easily be delegated to software. The harder task is understanding what structural question each operation answers. Rank, null space, determinant and eigenvalues expose different properties of the same matrix and should not be reduced to interchangeable recipes.

Unit 1: Linear Equations and Matrices

Linear equation

A linear equation in variables x1, x2, ..., xn has the form

a1 x1 + a2 x2 + ... + an xn = b.

The coefficients and constant term are scalars. Variables appear only to the first power and are not multiplied together.

System of linear equations

A linear system is a collection of linear equations sharing the same unknowns. Depending on consistency and rank, a system can have one solution, no solution, or infinitely many solutions.

Geometry of a two-variable system

In two dimensions, each nondegenerate linear equation represents a line. Two distinct nonparallel lines intersect at one point; parallel distinct lines have no common solution; coincident lines represent infinitely many common solutions.

Elementary operations and elimination

The solution set is preserved by exchanging equations, multiplying an equation by a nonzero scalar, and adding a multiple of one equation to another. These are the algebraic basis of elimination.

Matrix concept

A matrix organizes coefficients in rows and columns. An m x n matrix has m rows and n columns. Matrices provide a compact representation of systems and linear transformations.

Addition, subtraction, and scalar multiplication

Matrices of the same dimensions are added or subtracted element by element. Scalar multiplication multiplies every entry by the same scalar.

Matrix multiplication

If A is m x n and B is n x p, then

(AB)_ij = Σ_k a_ik b_kj.

The product exists only when the inner dimensions match. In general,

AB ≠ BA.

Special matrices

Frequently used forms include the zero matrix, identity matrix, diagonal matrix, scalar matrix, upper and lower triangular matrices, symmetric matrix, and transpose. Structural properties can simplify later computations.

Unit 2: Solving Linear Systems with Matrices

Matrix representation

A linear system can be written as

Ax = b.

Here A is the coefficient matrix, x is the unknown vector, and b is the right-hand-side vector.

Augmented matrix

The augmented matrix

[A | b]

stores the coefficients and right-hand side in one array and is convenient for row reduction.

Gaussian elimination

Gaussian elimination uses elementary row operations to transform the augmented matrix to row-echelon or upper-triangular form. Back substitution then recovers the unknowns.

Gauss-Jordan method

Gauss-Jordan elimination continues row reduction until pivot columns are reduced further, ideally producing reduced row-echelon form. This makes free variables and consistency conditions directly visible.

Reading the solution set

A contradictory row such as

0 0 ... 0 | c,   c ≠ 0

shows inconsistency. If the system is consistent and every variable column contains a pivot, the solution is unique. Free variables imply infinitely many solutions.

Inverse matrix

For a square matrix A, an inverse A^-1 satisfies

AA^-1 = A^-1A = I.

An inverse exists exactly when A is nonsingular.

Solving with the inverse matrix

If A is invertible,

Ax = b
x = A^-1 b.

This identity is useful conceptually. In practical numerical computation, explicit inversion is usually not the preferred way to solve a single linear system; factorization or elimination is more efficient.

Unit 3: Determinants

Definition of the determinant

The determinant maps a square matrix to a scalar. It captures information about invertibility, orientation, and volume scaling.

Second- and third-order determinants

For

A = [a b; c d],

det(A) = ad - bc.

For 3 x 3 matrices, cofactor expansion or Sarrus' rule can be used. Sarrus' rule applies only to 3 x 3 determinants and does not generalize to higher dimensions.

Properties of determinants

Important properties include:

• interchanging two rows changes the sign;
• multiplying a row by k multiplies the determinant by k;
• adding a multiple of one row to another leaves the determinant unchanged;
• a triangular matrix has determinant equal to the product of its diagonal entries;
• det(AB) = det(A)det(B);
• det(A^T) = det(A).

Determinants using row operations

Row reduction can transform the matrix to triangular form while tracking row exchanges and scalings. This avoids the combinatorial cost of a large cofactor expansion.

Minor and cofactor

The minor M_ij is the determinant obtained after deleting row i and column j. The cofactor is

C_ij = (-1)^(i+j) M_ij.

Cofactor expansion

A determinant can be expanded along any row or column:

det(A) = Σ_j a_ij C_ij.

The value is independent of the chosen expansion row or column.

Unit 4: Invertibility, Adjugate Matrix, and Cramer's Rule

Invertibility criterion using the determinant

For a square matrix,

det(A) ≠ 0

is equivalent to invertibility. A zero determinant means the matrix is singular.

Adjugate matrix

The adjugate is the transpose of the cofactor matrix. For a nonsingular matrix,

A^-1 = adj(A) / det(A).

This is a closed general formula and is useful for theory and small matrices. For large numerical problems, cofactor-based inversion is normally replaced by factorization and linear-system solvers.

Cramer's rule

For a nonsingular square system,

x_i = det(A_i)/det(A),

where A_i is obtained by replacing column i of A with b. The rule is mathematically elegant but not computationally efficient for large systems.

Unit 5: Vectors

Scalar and vector quantities

A scalar has magnitude only. A vector has magnitude and direction and can be represented by ordered components in a chosen coordinate system.

Vector operations

Vectors can be added, subtracted, and multiplied by scalars componentwise. These operations satisfy the vector-space laws.

Components and magnitude

For

v = (v1, v2, ..., vn),

the Euclidean magnitude is

||v|| = sqrt(v1^2 + v2^2 + ... + vn^2).

Dot product

The dot product is

u·v = Σ_i u_i v_i.

In Euclidean space,

u·v = ||u|| ||v|| cos θ.

For nonzero vectors, a zero dot product means orthogonality.

Cross product

In three dimensions,

u x v

is perpendicular to both u and v and has magnitude

||u x v|| = ||u|| ||v|| sin θ.

If two nonzero vectors have zero cross product, they are parallel.

Scalar triple product

The scalar triple product

u · (v x w)

represents the signed volume of the parallelepiped generated by the three vectors. A zero value indicates coplanarity.

Linear combination

A vector v is a linear combination of v1,...,vk if

v = c1 v1 + ... + ck vk.

The set of all such combinations is their span.

Linear dependence and independence

Vectors are linearly independent if

c1 v1 + ... + ck vk = 0

implies all coefficients are zero. Otherwise they are linearly dependent.

Fundamental subspaces and orthogonality

For an m x n matrix A, four related spaces organize much of linear algebra: the column space, null space, row space, and left null space.

Rank is the dimension of the column space and also of the row space. Rank-nullity gives

rank(A) + nullity(A) = n.

Orthogonal complements connect these spaces. The null space of A is orthogonal to the row space, while the left null space is orthogonal to the column space. These relationships explain why inconsistent systems leave a residual component outside the column space.

Orthogonal projection and least squares

When Ax=b has no exact solution, least squares minimizes ||Ax-b||². The optimum residual is orthogonal to the column space:

A^T(Ax-b)=0.

This produces the normal equations, but forming A^T A can square the condition number. QR factorization is usually numerically safer, and SVD is especially useful when the matrix is ill-conditioned or rank-deficient.

Unit 6: Rank and the Structure of Solutions

Rank of a matrix

The rank is the dimension of the row space, equivalently the column space. Computationally it equals the number of pivots in a row-echelon form.

Rank and linear independence

Rank measures how many independent row or column directions the matrix contains. A set of columns is independent exactly when those columns contribute separate pivot directions.

Linear systems and rank

For

Ax = b,

the system is consistent exactly when

rank(A) = rank([A|b]).

If this common rank equals the number of unknowns, the solution is unique. If it is smaller, consistent systems have free variables and infinitely many solutions.

Homogeneous systems

For

Ax = 0,

the zero vector is always a solution. A nontrivial solution exists when the null space has positive dimension; for a square matrix this occurs when det(A)=0.

Comparing solution methods

Gaussian elimination is the general direct framework. Gauss-Jordan exposes reduced form and free variables. Inverse and Cramer formulas are useful conceptually and for small problems, but they are not preferred for large numerical systems.

Singular values, numerical rank, and PCA

The singular-value decomposition

A = U Σ V^T

exists for any real matrix, including rectangular and rank-deficient matrices. Singular values measure how strongly the matrix stretches orthogonal directions.

A very small singular value indicates a direction that is nearly lost by the transformation. This makes SVD useful for numerical rank, pseudoinverses, low-rank approximation, and diagnosing ill-conditioning.

Principal component analysis applies the same geometry to centered data. Principal directions can be obtained from covariance eigenvectors or directly from the SVD of the centered data matrix. PCA is therefore an application of orthogonal projection and low-rank approximation rather than a separate mathematical trick.

Unit 7: Eigenvalues and Eigenvectors

Basic concept

A nonzero vector v is an eigenvector of A if

Av = λv.

The scalar λ is the corresponding eigenvalue. The transformation changes the magnitude and possibly the sign/direction along that eigenvector without changing its one-dimensional span.

Characteristic equation

Rearranging gives

(A - λI)v = 0.

A nonzero solution exists only if

det(A - λI) = 0.

This is the characteristic equation.

Computation procedure

First solve the characteristic equation for eigenvalues. For each eigenvalue, solve

(A - λI)v = 0

to obtain the corresponding eigenspace.

Properties of eigenvalues

The sum of eigenvalues counted with algebraic multiplicity equals the trace, and their product equals the determinant. For triangular matrices, the eigenvalues are the diagonal entries.

Basis and dimension

A basis is a linearly independent spanning set. Every basis of a finite-dimensional vector space contains the same number of vectors; that number is the dimension.

Diagonalization

A matrix is diagonalizable if there is an invertible matrix P such that

A = P D P^-1,

with diagonal D. The columns of P are independent eigenvectors. Having n linearly independent eigenvectors is the key requirement for an n x n matrix.

Orthogonality and orthonormality

Vectors are orthogonal when their dot product is zero. They are orthonormal when they are also unit vectors. Orthonormal bases simplify coordinates, projections, and numerical computation.

Gram-Schmidt method

Gram-Schmidt converts an independent set into an orthogonal or orthonormal set spanning the same subspace. Each new vector has its projections onto the previously constructed directions removed.

Diagonalization of symmetric matrices

A real symmetric matrix has real eigenvalues and can be diagonalized by an orthogonal matrix:

A = Q D Q^T.

This spectral structure is particularly important in numerical analysis, optimization, and quadratic forms.

General Conceptual Framework

The course connects systems, matrices, vector spaces, and transformations:

Linear equations
      ↓
Matrix representation
      ↓
Row reduction / rank
      ↓
Vector spaces and independence
      ↓
Linear transformations
      ↓
Eigenvalues and invariant directions

Determinants provide compact structural tests, but row reduction and factorization are the practical tools for systems. Rank explains consistency and degrees of freedom. Eigenvectors describe directions preserved by a linear transformation and lead to diagonal representations when enough independent eigenvectors exist.

Conceptual Distinctions

Matrix multiplication ≠ elementwise multiplication. The row-by-column rule defines the ordinary product, and multiplication is generally not commutative.

Determinant ≠ rank. The determinant is a scalar defined for square matrices; rank is defined for rectangular matrices and measures independent directions.

Sarrus' rule ≠ a general determinant algorithm. It applies only to 3 x 3 matrices.

Inverse formula ≠ preferred large-scale solver. The adjugate formula is mathematically valid, but direct factorization or elimination is usually preferable numerically.

Dot product ≠ cross product. A zero dot product indicates orthogonality. For two nonzero vectors in three dimensions, a zero cross product indicates parallelism.

Linear dependence ≠ geometric equality. Dependence means that at least one vector can be expressed from the others through a nontrivial linear relation.

**Rank(A) ≠ rank([A|b]) in an inconsistent system.** Equality is the consistency criterion.

Eigenvalue ≠ eigenvector. The eigenvalue is a scalar; the eigenvector is a nonzero direction satisfying Av=λv.

Algebraic multiplicity ≠ geometric multiplicity. Repetition in the characteristic polynomial does not automatically provide the same number of independent eigenvectors.

Diagonalizable ≠ invertible. A matrix may be diagonalizable and singular, or invertible without being diagonalizable.

The engineering power of linear algebra is its ability to place many numerical problems under a common representation. Equation solving, least squares, transformations, and eigenvalue problems look different but share vector-space structure; choosing the right method usually begins by reading that structure correctly.

Conditioning, numerical rank, and scaling

A linear system can have a unique solution and still be numerically unreliable. A large condition number means small input perturbations can create large solution changes.

Rank is also not always a sharp concept with measured data. Very small singular values can make a system effectively low-rank, and the threshold depends on scale and noise.

Scaling variables and avoiding transformations that unnecessarily square the condition number are basic numerical-engineering practices.

Numerical trust in linear algebra

An algebraically solvable linear system may still be numerically sensitive in floating-point arithmetic. A large condition number can amplify small input perturbations; this is a property of the problem, not necessarily an algorithm defect.

Residual and solution error should be distinguished. Ax-b can be small even when the solution is inaccurate for an ill-conditioned system. Scaling, suitable factorizations, and singular-value inspection help explain the difference.

Eigenvalue computations also depend on matrix structure. Symmetric/Hermitian problems should use structure-aware algorithms; tiny imaginary artifacts from a general solver may have no physical meaning.

The Role of Linear Algebra in Artificial Intelligence

Modern machine learning represents much of its data and parameters as vectors, matrices, or higher-order tensors. Linear algebra is therefore not an auxiliary calculation layer added to AI; it is the shared language of representation and computation.

Vector representation and feature space

An observation may be written as:

x = [x1, x2, ..., xd]^T

Distances and inner products describe geometry in this feature space. Cosine similarity:

cos(θ) = (x^T y) / (||x|| ||y||)

is widely used with embeddings. Similarity in embedding space, however, is not logical identity; it depends on the representation learned by the embedding model.

A weight matrix is a transformation

A basic neural-network layer has the form:

z = W x + b

W maps the input space to another vector space. Once a nonlinear activation is added, the full layer is no longer linear, yet matrix multiplication remains a dominant computational primitive. For a batch of samples:

Z = X W^T + b

which also explains why modern accelerators emphasize matrix operations.

Rank and learned representation

Rank describes the number of independent directions represented by a matrix. High-dimensional data may lie near a lower-dimensional subspace. A low-rank approximation:

W ≈ U_r Σ_r V_r^T

can reduce storage or computation at the cost of approximation error. Rank selection is therefore a tradeoff, not a lossless simplification.

Eigenvectors, covariance, and PCA

For centered data, eigenvectors of the covariance structure identify directions of variance. PCA projects onto dominant directions. It is unsupervised: the direction with highest variance is not guaranteed to be the most discriminative for a downstream label.

SVD and numerical reliability

Singular Value Decomposition:

A = U Σ V^T

supports rank analysis, pseudoinverses, least squares, and low-rank approximations. Solving least squares through normal equations may worsen conditioning; QR or SVD can be numerically safer. Large AI computations do not escape conditioning problems.

Attention is also built on linear algebra

Transformer attention contains operations such as:

Q = X W_Q
K = X W_K
V = X W_V
A = softmax(Q K^T / √d_k)
Y = A V

Softmax is nonlinear, but projections, similarity scores, and output aggregation are dense matrix operations.

Norms, scale, and optimization

Norms define distance and parameter magnitude. L2 regularization adds a term such as:

λ ||w||²_2

while L1 has a different geometry and can promote sparsity. The choice of norm changes the optimization problem rather than merely changing notation.

Linear algebra does not explain every aspect of learning. Generalization and uncertainty require probability and statistics; parameter optimization requires multivariable calculus. Its distinctive strength is that it describes data representation, subspaces, transformations, and the algebraic core of large model computations within one framework.

Read the structural meaning of the operation

In linear algebra, it helps to ask what structural question an operation answers before carrying it out. For a square matrix, a zero determinant is not just a numerical result: it is connected to non-invertibility, loss of full rank, and linear dependence of the columns. For a rectangular matrix, concepts such as rank, null space, and least squares are usually more meaningful than searching for an inverse.

For a system

A x = b

existence and uniqueness depend on the relationship between A and b. Rank conditions distinguish inconsistent systems, unique solutions, and infinitely many solutions. Gaussian elimination is therefore not merely an arithmetic procedure; it exposes this structure through row operations.

In an eigenvalue problem, A v = λv says that the direction represented by v is preserved up to scaling by the transformation. Not every matrix has a complete real eigenvector basis, so diagonalisation is not automatic. Real symmetric matrices are an important special case because their eigenvectors can be chosen orthogonally.

Least squares does not force an inconsistent system to become exactly solvable. It chooses the vector x that minimises the residual norm between Ax and b. Geometrically, b is projected onto the column space of A. The same structure appears in regression, signal processing, and model fitting.

Dimension checking is a simple but powerful verification step. If A is m×n and x is n×1, the product must be m×1. Many mistakes can be detected before any arithmetic by checking dimensions and the spaces in which the objects live.

References

• Ahmet Yesevi Üniversitesi Bilgisayar Mühendisliği Bölümü. Lineer Cebir (TBIL203) ders materyalleri.

• David C. Lay, Steven R. Lay, Judi J. McDonald. Linear Algebra and Its Applications, 6th Edition. Pearson, 2021.

• Gene H. Golub, Charles F. Van Loan. Matrix Computations, 4th Edition. Johns Hopkins University Press, 2013.

• Gilbert Strang. Introduction to Linear Algebra, 5th Edition. Wellesley-Cambridge Press, 2016.

• Gilbert Strang. Linear Algebra and Learning from Data. Wellesley-Cambridge Press, 2019.

• Howard Anton, C. R. Elementary Linear Algebra, 12th Edition. Wiley, 2019.

• Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. MIT Press, 2016.

• Lloyd N. Trefethen, David Bau III. Numerical Linear Algebra. SIAM, 1997.

• Roger A. Horn, Charles R. Johnson. Matrix Analysis, 2nd Edition. Cambridge University Press, 2012.