Linear Algebra: Matrices, Linear Systems and Eigenvalues

Linear algebra provides a representation language that appears repeatedly across computer engineering. Linear systems, vector spaces, determinants, eigenvalues and eigenvectors are presented with their assumptions and structural interpretations kept explicit.

Mechanical matrix arithmetic can easily be delegated to software. The harder task is understanding what structural question each operation answers. Rank, null space, determinant and eigenvalues expose different properties of the same matrix and should not be reduced to interchangeable recipes.

Unit 1: Linear Equations and Matrices

1.1. Linear Equation

A linear equation in variables x1, x2, ..., xn has the form

a1 x1 + a2 x2 + ... + an xn = b.

The coefficients and constant term are scalars. Variables appear only to the first power and are not multiplied together.

1.2. System of Linear Equations

A linear system is a collection of linear equations sharing the same unknowns. Depending on consistency and rank, a system can have one solution, no solution, or infinitely many solutions.

1.3. Geometry of a two-variable System

In two dimensions, each nondegenerate linear equation represents a line. Two distinct nonparallel lines intersect at one point; parallel distinct lines have no common solution; coincident lines represent infinitely many common solutions.

1.4. Elementary Operations and Elimination

The solution set is preserved by exchanging equations, multiplying an equation by a nonzero scalar, and adding a multiple of one equation to another. These are the algebraic basis of elimination.

1.5. Matrix Concept

A matrix organizes coefficients in rows and columns. An m x n matrix has m rows and n columns. Matrices provide a compact representation of systems and linear transformations.

1.6. Addition, Subtraction, and Scalar Multiplication

Matrices of the same dimensions are added or subtracted element by element. Scalar multiplication multiplies every entry by the same scalar.

1.7. Matrix Multiplication

If A is m x n and B is n x p, then

(AB)_ij = Σ_k a_ik b_kj.

The product exists only when the inner dimensions match. In general,

AB ≠ BA.

The geometric meaning of the product is the composition of linear maps. Applying A to a vector v and then applying B gives B(Av) = (BA)v. For the standard basis vectors e₁ and e₂, the columns of BA are B(Ae₁) and B(Ae₂). Tracking a unit square through the two maps reveals what the row-by-column computation actually does in space.

For example, shear and rotation generally give different results when performed in opposite orders. Both are linear maps that preserve the origin. A translation, in contrast, is not an ordinary linear transformation unless one moves to homogeneous coordinates.

1.8. Special Matrices

Frequently used forms include the zero matrix, identity matrix, diagonal matrix, scalar matrix, upper and lower triangular matrices, symmetric matrix, and transpose. Structural properties can simplify later computations.

1.9. Matrix Transformations and Homogeneous Coordinates

A matrix is not merely an array of numbers; it can be read as a transformation from one vector space to another:

y = A x.

The columns of A show where the standard basis vectors are sent. Scaling, reflection, rotation, and other linear transformations can be expressed in this common form.

A two-dimensional rotation is

R(θ) =
[ cosθ  −sinθ ]
[ sinθ   cosθ ].

It is an orthogonal transformation and preserves lengths and angles.

Translation is not linear because it does not keep the origin fixed. Computer graphics handles this by introducing homogeneous coordinates. Representing (x,y) as (x,y,1) allows translation, scaling, and rotation to be represented and composed through matrix multiplication.

Order matters. ABx applies B first and A second. The noncommutativity of matrix multiplication therefore has a direct geometric interpretation.

1.10. Reading Linear Systems, Transformations, and Eigenstructure Together

The course connects systems, matrices, vector spaces, and transformations:

Linear equations
      ↓
Matrix representation
      ↓
Row reduction / rank
      ↓
Vector spaces and independence
      ↓
Linear transformations
      ↓
Eigenvalues and invariant directions

Determinants provide compact structural tests, but row reduction and factorization are the practical tools for systems. Rank explains consistency and degrees of freedom. Eigenvectors describe directions preserved by a linear transformation and lead to diagonal representations when enough independent eigenvectors exist.

1.11. Distinguishing Rank, Determinant, Invertibility, and Eigenvalues

Matrix multiplication ≠ elementwise multiplication. The row-by-column rule defines the ordinary product, and multiplication is generally not commutative.

Determinant ≠ rank. The determinant is a scalar defined for square matrices; rank is defined for rectangular matrices and measures independent directions.

Sarrus' rule ≠ a general determinant algorithm. It applies only to 3 x 3 matrices.

Inverse formula ≠ preferred large-scale solver. The adjugate formula is mathematically valid, but direct factorization or elimination is usually preferable numerically.

Dot product ≠ cross product. A zero dot product indicates orthogonality. For two nonzero vectors in three dimensions, a zero cross product indicates parallelism.

Linear dependence ≠ geometric equality. Dependence means that at least one vector can be expressed from the others through a nontrivial linear relation.

**Rank(A) ≠ rank([A|b]) in an inconsistent system.** Equality is the consistency criterion.

Eigenvalue ≠ eigenvector. The eigenvalue is a scalar; the eigenvector is a nonzero direction satisfying Av=λv.

Algebraic multiplicity ≠ geometric multiplicity. Repetition in the characteristic polynomial does not automatically provide the same number of independent eigenvectors.

Diagonalizable ≠ invertible. A matrix may be diagonalizable and singular, or invertible without being diagonalizable.

The engineering power of linear algebra is its ability to place many numerical problems under a common representation. Equation solving, least squares, transformations, and eigenvalue problems look different but share vector-space structure; choosing the right method usually begins by reading that structure correctly.

1.12. Computation with Sparse and Structured Matrices

Many engineering matrices contain mostly zeros. A sparse matrix is not merely a storage optimization; sparsity changes the data structures and algorithms that should be used. Storing and multiplying explicit zero entries can dominate cost at large scale.

Banded, block, Toeplitz, symmetric, and positive-definite structure can likewise guide algorithm selection. A generic dense-matrix routine is not always the correct computational model.

For a large linear system, the relevant questions extend beyond existence of a solution: rank, conditioning, sparsity pattern, fill-in, memory locality, parallelism, and the required error tolerance all matter. The engineering value of linear algebra lies in converting mathematical structure into computational structure.

1.13. Read the Structural Meaning of the Operation

In linear algebra, it helps to ask what structural question an operation answers before carrying it out. For a square matrix, a zero determinant is not just a numerical result: it is connected to non-invertibility, loss of full rank, and linear dependence of the columns. For a rectangular matrix, concepts such as rank, null space, and least squares are usually more meaningful than searching for an inverse.

For a system

A x = b

existence and uniqueness depend on the relationship between A and b. Rank conditions distinguish inconsistent systems, unique solutions, and infinitely many solutions. Gaussian elimination is therefore not merely an arithmetic procedure; it exposes this structure through row operations.

In an eigenvalue problem, A v = λv says that the direction represented by v is preserved up to scaling by the transformation. Not every matrix has a complete real eigenvector basis, so diagonalisation is not automatic. Real symmetric matrices are an important special case because their eigenvectors can be chosen orthogonally.

Least squares does not force an inconsistent system to become exactly solvable. It chooses the vector x that minimises the residual norm between Ax and b. Geometrically, b is projected onto the column space of A. The same structure appears in regression, signal processing, and model fitting.

Dimension checking is a simple but powerful verification step. If A is m×n and x is n×1, the product must be m×1. Many mistakes can be detected before any arithmetic by checking dimensions and the spaces in which the objects live.

Unit 2: Solving Linear Systems with Matrices

Before solving a linear system, verify the dimensions of the coefficient matrix and right-hand side. During elimination, consider pivoting for zero or small pivots and assess possible growth of rounding error. Substitute the computed solution into the original equations to obtain the residual vector. A small residual does not necessarily imply a small solution error, so matrix conditioning also matters.

2.1. Matrix Representation

A linear system can be written as

Ax = b.

Here A is the coefficient matrix, x is the unknown vector, and b is the right-hand-side vector.

2.2. Augmented Matrix

The augmented matrix

[A | b]

stores the coefficients and right-hand side in one array and is convenient for row reduction.

2.3. Gaussian Elimination

Gaussian elimination uses elementary row operations to transform the augmented matrix to row-echelon or upper-triangular form. Back substitution then recovers the unknowns.

2.4. Gauss-Jordan Method

Gauss-Jordan elimination continues row reduction until pivot columns are reduced further, ideally producing reduced row-echelon form. This makes free variables and consistency conditions directly visible.

2.5. Reading the Solution Set

A contradictory row such as

0 0 ... 0 | c,   c ≠ 0

shows inconsistency. If the system is consistent and every variable column contains a pivot, the solution is unique. Free variables imply infinitely many solutions.

2.6. Inverse Matrix

For a square matrix A, an inverse A^-1 satisfies

AA^-1 = A^-1A = I.

An inverse exists exactly when A is nonsingular.

2.7. Solving with the Inverse Matrix

If A is invertible,

Ax = b
x = A^-1 b.

This identity is useful conceptually. In practical numerical computation, explicit inversion is usually not the preferred way to solve a single linear system; factorization or elimination is more efficient.

2.8. LU Factorization and Repeated right-hand Sides

The elimination process can, under suitable conditions, factor the coefficient matrix as

A = L U,

where L is lower triangular and U is upper triangular.

Then

Ax = b
LUx = b

is solved in two stages:

Ly = b
Ux = y.

The first system uses forward substitution and the second uses backward substitution. Once the factorization has been computed, solving the same matrix A against many different right-hand sides becomes much cheaper.

Pivoting is usually required in numerical work. With row exchanges represented by a permutation matrix,

P A = L U.

The main benefit is not the construction of an inverse. It is the reuse of the elimination structure. Explicitly forming A^-1 and multiplying by b is usually more expensive and can be less numerically reliable.

2.9. Cholesky Factorization and positive-definite Systems

For a real, symmetric, positive-definite matrix A, a more specialized factorization is available:

A = L L^T,

where L is lower triangular. Cholesky factorization exploits symmetry and can require roughly half the storage and less arithmetic than a general LU treatment.

Positive definiteness can be expressed through the quadratic form

x^T A x > 0    for every x != 0.

This structure appears naturally in least squares, covariance-like matrices, and second-order optimization models. A matrix that merely looks symmetric is not automatically positive definite, so the assumption must be verified before selecting the method.

The numerical consequence of pivot selection. For 10⁻¹² x + y = 1 and x + y = 2, eliminating directly from the first row uses a pivot of 10⁻¹² and produces a multiplier of roughly 10¹². Under finite precision, such intermediate values can amplify loss of significant digits. Partial pivoting selects the entry of magnitude 1 in the first column and swaps the rows. Elimination then gives x ≈ 1.000000000001 and y ≈ 0.999999999999. A basic check substitutes the result back into the original system and evaluates the residual r = b − Ax; a small residual alone, however, does not guarantee a small forward error when the matrix is ill-conditioned. Conditioning must then be assessed separately. Algebraically equivalent row operations need not be equally reliable in finite-precision arithmetic.

Unit 3: Determinants

3.1. Definition of the Determinant

The determinant maps a square matrix to a scalar. It captures information about invertibility, orientation, and volume scaling.

3.2. Second- and third-order Determinants

For

A = [a b; c d],

det(A) = ad - bc.

For 3 x 3 matrices, cofactor expansion or Sarrus' rule can be used. Sarrus' rule applies only to 3 x 3 determinants and does not generalize to higher dimensions.

3.3. Properties of Determinants

Important properties include:

• interchanging two rows changes the sign;
• multiplying a row by k multiplies the determinant by k;
• adding a multiple of one row to another leaves the determinant unchanged;
• a triangular matrix has determinant equal to the product of its diagonal entries;
• det(AB) = det(A)det(B);
• det(A^T) = det(A).

3.4. Determinants Using Row Operations

Row reduction can transform the matrix to triangular form while tracking row exchanges and scalings. This avoids the combinatorial cost of a large cofactor expansion.

3.5. Minor and Cofactor

The minor M_ij is the determinant obtained after deleting row i and column j. The cofactor is

C_ij = (-1)^(i+j) M_ij.

3.6. Cofactor Expansion

A determinant can be expanded along any row or column:

det(A) = Σ_j a_ij C_ij.

The value is independent of the chosen expansion row or column.

Unit 4: Invertibility, Adjugate Matrix, and Cramer's Rule

The choice to compute an inverse or apply Cramer's rule should not be based simply on formula availability. First verify that the matrix is square and nonsingular; a near-zero determinant may warn of numerical sensitivity, depending on scaling. For a large system with one right-hand side, factorization is often preferable to explicitly forming an inverse. Assess correctness through the original system's residual and error sensitivity, not by reapplying the same formula.

4.1. Invertibility Criterion Using the Determinant

For a square matrix,

det(A) ≠ 0

is equivalent to invertibility. A zero determinant means the matrix is singular.

4.2. Adjugate Matrix

The adjugate is the transpose of the cofactor matrix. For a nonsingular matrix,

A^-1 = adj(A) / det(A).

This is a closed general formula and is useful for theory and small matrices. For large numerical problems, cofactor-based inversion is normally replaced by factorization and linear-system solvers.

4.3. Cramer's Rule

For a nonsingular square system,

x_i = det(A_i)/det(A),

where A_i is obtained by replacing column i of A with b. The rule is mathematically elegant but not computationally efficient for large systems.

Unit 5: Vectors

5.1. Scalar and Vector Quantities

A scalar has magnitude only. A vector has magnitude and direction and can be represented by ordered components in a chosen coordinate system.

5.2. Vector Operations

Vectors can be added, subtracted, and multiplied by scalars componentwise. These operations satisfy the vector-space laws.

5.3. Components and Magnitude

For

v = (v1, v2, ..., vn),

the Euclidean magnitude is

||v|| = sqrt(v1^2 + v2^2 + ... + vn^2).

5.4. Dot Product

The dot product is

u·v = Σ_i u_i v_i.

In Euclidean space,

u·v = ||u|| ||v|| cos θ.

For nonzero vectors, a zero dot product means orthogonality.

5.5. Cross Product

In three dimensions,

u x v

is perpendicular to both u and v and has magnitude

||u x v|| = ||u|| ||v|| sin θ.

If two nonzero vectors have zero cross product, they are parallel.

5.6. Scalar Triple Product

The scalar triple product

u · (v x w)

represents the signed volume of the parallelepiped generated by the three vectors. A zero value indicates coplanarity.

5.7. Linear Combination

A vector v is a linear combination of v1,...,vk if

v = c1 v1 + ... + ck vk.

The set of all such combinations is their span.

5.8. Linear Dependence and Independence

Vectors are linearly independent if

c1 v1 + ... + ck vk = 0

implies all coefficients are zero. Otherwise they are linearly dependent.

5.9. Vector Spaces and Subspaces

Vectors are not restricted to R² or R³. Any set whose addition and scalar multiplication satisfy the vector-space axioms can form a vector space. Polynomials, matrices, and functions provide important examples under suitable operations.

A nonempty subset W of a vector space is a subspace when it is closed under vector addition and scalar multiplication. A compact equivalent test is

u, v ∈ W and α, β scalars
⇒ αu + βv ∈ W.

The span of a set of vectors is always a subspace and is the smallest subspace containing those vectors. This turns linear combination from a coefficient-manipulation exercise into a geometric construction of a space.

5.10. Inner-product Spaces and Orthogonal Complements

The ordinary dot product defines Euclidean geometry in R^n. An inner product generalizes this structure to other vector spaces while preserving the properties required for notions of angle and length.

It induces a norm and distance:

||v|| = sqrt(<v,v>)
d(u,v) = ||u−v||.

For a subspace W, the set of vectors orthogonal to every vector in W is the orthogonal complement W⊥.

In a finite-dimensional Euclidean space,

dim(W) + dim(W⊥) = n.

A vector can be decomposed into a component in W and an orthogonal component in W⊥. This decomposition is the geometric basis of least-squares approximation.

5.11. Fundamental Subspaces and Orthogonality

For an m x n matrix A, four related spaces organize much of linear algebra: the column space, null space, row space, and left null space.

Rank is the dimension of the column space and also of the row space. Rank-nullity gives

rank(A) + nullity(A) = n.

Orthogonal complements connect these spaces. The null space of A is orthogonal to the row space, while the left null space is orthogonal to the column space. These relationships explain why inconsistent systems leave a residual component outside the column space.

5.12. Orthogonal Projection and Least Squares

When Ax=b has no exact solution, least squares minimizes ||Ax-b||². The optimum residual is orthogonal to the column space:

A^T(Ax-b)=0.

This produces the normal equations, but forming A^T A can square the condition number. QR factorization is usually numerically safer, and SVD is especially useful when the matrix is ill-conditioned or rank-deficient.

Unit 6: Affine Geometry with Vectors — Lines and Planes

Every linear subspace contains the origin, while geometric lines and planes generally do not. The natural bridge between analytic geometry and linear algebra is therefore an affine set: a translated linear subspace.

affine set = p + U

Here U is a linear subspace and p is a reference point. When p=0, the affine set reduces to the linear subspace itself.

This distinction prevents statements such as “every line is a vector space” or “every plane is a subspace.”

6.1. Vector and Parametric Equations of a Line

A line through point p with nonzero direction vector v is

L = { p + t v : t ∈ R }.

In three dimensions, if

p = (x0,y0,z0)
v = (a,b,c),

the component equations are

x = x0 + at
y = y0 + bt
z = z0 + ct.

Any nonzero scalar multiple of v defines the same direction. The direction vector is therefore not unique; what matters is the one-dimensional subspace that it spans.

Given two distinct points p and q, a suitable direction is

v = q-p.

6.2. Point-normal Equation of a Plane

A plane through p and perpendicular to a nonzero normal vector n is the set of points satisfying

n · (x-p) = 0.

With n=(A,B,C) and p=(x0,y0,z0),

A(x-x0) + B(y-y0) + C(z-z0) = 0,

or equivalently,

Ax + By + Cz = D.

The direction space of the plane is the two-dimensional subspace orthogonal to n:

U = {v ∈ R^3 : n·v = 0}.

Thus the affine structure of the plane is explicitly p+U.

6.3. A Plane Through Three Points

Three noncollinear points p, q, and r determine a plane. Form two direction vectors,

u = q-p
v = r-p,

and obtain a normal from

n = u × v.

If u × v = 0, the three points do not determine a unique plane because their direction vectors are linearly dependent. This condition should be checked before using the formula mechanically.

6.4. Parallelism and Perpendicularity

Two lines are parallel when their direction vectors are linearly dependent.

For a line with direction v and a plane with normal n,

v · n = 0

means the line direction is parallel to the plane. If one point of the line also satisfies the plane equation, the entire line lies in the plane.

Two planes are parallel when their normals are linearly dependent. If their normals are orthogonal, the planes are perpendicular.

These vector criteria generalize slope-based rules from two-dimensional coordinate geometry.

6.5. Line-plane Intersection

Let the line be

x = p + t v

and the plane

n · (x-q) = 0.

Substitution gives

n · (p + tv - q) = 0.

When n·v ≠ 0, the unique parameter is

t =
- n·(p-q) / (n·v).

When n·v = 0, distinguish two cases:

• if p satisfies the plane equation, the line lies in the plane and there are infinitely many intersection points;
• otherwise the line is parallel to the plane and there is no intersection.

This is the geometric form of the unique / infinitely-many / inconsistent classification of a linear system.

6.6. Intersection of Two Planes

Two nonparallel planes intersect in a line. If their normals are n1 and n2, a direction for the intersection line is

v = n1 × n2.

A point on the line can be found by solving the two plane equations as a linear system. This connects the geometry directly to Gaussian elimination.

The common intersection of three planes is similarly a three-variable linear system; rank can distinguish a point, a line, a plane, or an empty solution set.

6.7. Perpendicular Distance from a Point to a Plane

For point x and plane

n · y = D,

the perpendicular distance is the magnitude of the projection onto the normal direction:

d =
|n·x - D|
/
||n||.

For n=(A,B,C) this becomes

d =
|Ax0 + By0 + Cz0 - D|
/
√(A^2+B^2+C^2).

Dividing by ||n|| removes the arbitrary scaling of the plane equation.

6.8. Projection of a Point Onto a Line

Let L = p + span(v) and let x be a point. The orthogonal projection of x-p onto v is

proj_v(x-p) =
[(x-p)·v / (v·v)] v.

The nearest point on the line is

x_near =
p + proj_v(x-p),

and the distance is the norm of the residual:

d(x,L) =
|| (x-p) - proj_v(x-p) ||.

This is not merely a coordinate-geometry formula; it is the one-dimensional geometric prototype of orthogonal projection and least squares.

6.9. Relative Position of Two Lines in Space

In three dimensions, two lines need not be only intersecting or parallel. Nonparallel lines that do not lie in the same plane are skew lines.

For

L1 = p + s u
L2 = q + t v

with u × v ≠ 0, coplanarity requires

(q-p) · (u × v) = 0.

If this scalar triple product is nonzero, the lines are skew. Their shortest distance is the projection onto the common-normal direction:

d =
| (q-p) · (u × v) |
/
||u × v||.

The formula is also a direct consequence of the volume interpretation of the scalar triple product.

6.10. Common Language of Affine Geometry and Linear Systems

Line and plane equations may appear to be a separate geometry topic, but they are fundamentally linear systems:

line / plane intersection
          ↓
linear system
          ↓
rank and consistency
          ↓
dimension of the solution set

Whenever a system Ax=b is consistent, its full solution set has the affine form

x = xp + Null(A),

where xp is one particular solution and Null(A) is the solution space of the homogeneous system.

Geometric lines and planes are low-dimensional examples of this general statement. This viewpoint integrates analytic geometry with rank, null spaces, orthogonal projections, and linear transformations rather than treating it as an unrelated collection of formulas.

Unit 7: Rank and the Structure of Solutions

7.1. Rank of a Matrix

The rank is the dimension of the row space, equivalently the column space. Computationally it equals the number of pivots in a row-echelon form.

7.2. Rank and Linear Independence

Rank measures how many independent row or column directions the matrix contains. A set of columns is independent exactly when those columns contribute separate pivot directions.

7.3. Linear Systems and Rank

For

Ax = b,

the system is consistent exactly when

rank(A) = rank([A|b]).

If this common rank equals the number of unknowns, the solution is unique. If it is smaller, consistent systems have free variables and infinitely many solutions.

7.4. Homogeneous Systems

For

Ax = 0,

the zero vector is always a solution. A nontrivial solution exists when the null space has positive dimension; for a square matrix this occurs when det(A)=0.

7.5. Comparing Solution Methods

Gaussian elimination is the general direct framework. Gauss-Jordan exposes reduced form and free variables. Inverse and Cramer formulas are useful conceptually and for small problems, but they are not preferred for large numerical systems.

7.6. Linear Transformations, Kernel, and Image

A function T: V → W between vector spaces is a linear transformation when

T(u + v) = T(u) + T(v)
T(αu) = αT(u).

Vectors mapped to zero form the kernel,

ker(T) = {v ∈ V : T(v)=0},

while all attainable outputs form the image,

im(T) = {T(v) : v ∈ V}.

After bases are chosen, a finite-dimensional linear transformation is represented by a matrix. For T(x)=Ax, the kernel is Null(A) and the image is Col(A). Matrix language and abstract transformation language therefore describe the same structure.

The transformation form of rank-nullity is

dim(V) = dim(ker T) + dim(im T).

7.7. Change of Coordinates and Similarity

The same linear transformation can have different matrices in different bases. If P is the change-of-basis matrix, the two representations satisfy

B = P^-1 A P,

and A and B are similar matrices.

Similarity changes coordinates rather than the underlying transformation. Structural quantities such as determinant, trace, and characteristic polynomial are therefore preserved.

Diagonalization is a search for a basis in which the transformation matrix becomes especially simple. If an eigenvector basis exists, P^-1 A P is diagonal.

7.8. Singular Values, Numerical Rank, and PCA

The singular-value decomposition

A = U Σ V^T

exists for any real matrix, including rectangular and rank-deficient matrices. Singular values measure how strongly the matrix stretches orthogonal directions.

A very small singular value indicates a direction that is nearly lost by the transformation. This makes SVD useful for numerical rank, pseudoinverses, low-rank approximation, and diagnosing ill-conditioning.

Principal component analysis applies the same geometry to centered data. Principal directions can be obtained from covariance eigenvectors or directly from the SVD of the centered data matrix. PCA is therefore an application of orthogonal projection and low-rank approximation rather than a separate mathematical trick.

7.9. Pseudoinverse and Best low-rank Approximation

A rectangular or rank-deficient matrix may have no ordinary inverse. From the SVD

A = U Sigma V^T,

the Moore-Penrose pseudoinverse A^+ is constructed by inverting the nonzero singular values. A^+ b gives a least-squares solution for inconsistent systems and, in the appropriate underdetermined setting, a minimum-norm solution.

The SVD also gives a precise low-rank approximation. Keeping only the first k singular components,

A_k = U_k Sigma_k V_k^T,

produces the best rank-k approximation in the Frobenius and spectral norms. This makes the information-versus-dimension trade-off explicit in image compression, denoising, and latent representations.

7.10. Conditioning, Numerical Rank, and Scaling

A linear system can have a unique solution and still be numerically unreliable. A large condition number means small input perturbations can create large solution changes.

Rank is also not always a sharp concept with measured data. Very small singular values can make a system effectively low-rank, and the threshold depends on scale and noise.

Scaling variables and avoiding transformations that unnecessarily square the condition number are basic numerical-engineering practices.

7.11. Numerical Trust in Linear Algebra

An algebraically solvable linear system may still be numerically sensitive in floating-point arithmetic. A large condition number can amplify small input perturbations; this is a property of the problem, not necessarily an algorithm defect.

Residual and solution error should be distinguished. Ax-b can be small even when the solution is inaccurate for an ill-conditioned system. Scaling, suitable factorizations, and singular-value inspection help explain the difference.

Eigenvalue computations also depend on matrix structure. Symmetric/Hermitian problems should use structure-aware algorithms; tiny imaginary artifacts from a general solver may have no physical meaning.

7.12. Condition Number, Rank, and Stable Linear Solutions

A linear system having a unique mathematical solution does not guarantee a numerically reliable answer. For \(Ax=b\), amplification of small data errors is related to the condition number \(\kappa(A)=\lVert A\rVert\lVert A^{-1}\rVert\) in a chosen norm. Perturbations remain comparatively controlled for well-conditioned matrices, whereas small measurement errors may be strongly amplified by an ill-conditioned system.

For example, \(A=\begin{bmatrix}1&1\\1&1+\varepsilon\end{bmatrix}\) is invertible for every \(\varepsilon\neq0\), yet its rows approach dependence and its condition number grows as \(\varepsilon\to0\). A nonzero determinant is therefore insufficient as a numerical-reliability test. Scaling, pivoting, and QR- or SVD-based solution methods should be selected with reference to data precision and problem structure.

Rank-deficient or overdetermined systems lead naturally to the least-squares objective \(\min_x\lVert Ax-b\rVert_2\). The SVD exposes small singular values and supports an assessment of effective numerical rank, but deciding when a singular value counts as zero depends on a tolerance and a model of data noise. Residual \(\lVert Ax-b\rVert\), condition number, and expected measurement uncertainty are distinct quantities in the solution report.

Unit 8: Eigenvalues and Eigenvectors

8.1. Basic Concept

A nonzero vector v is an eigenvector of A if

Av = λv.

The scalar λ is the corresponding eigenvalue. The transformation changes the magnitude and possibly the sign/direction along that eigenvector without changing its one-dimensional span.

8.2. Characteristic Equation

Rearranging gives

(A - λI)v = 0.

A nonzero solution exists only if

det(A - λI) = 0.

This is the characteristic equation.

8.3. Computation Procedure

First solve the characteristic equation for eigenvalues. For each eigenvalue, solve

(A - λI)v = 0

to obtain the corresponding eigenspace.

8.4. Properties of Eigenvalues

The sum of eigenvalues counted with algebraic multiplicity equals the trace, and their product equals the determinant. For triangular matrices, the eigenvalues are the diagonal entries.

8.5. Basis and Dimension

A basis is a linearly independent spanning set. Every basis of a finite-dimensional vector space contains the same number of vectors; that number is the dimension.

8.6. Diagonalization

A matrix is diagonalizable if there is an invertible matrix P such that

A = P D P^-1,

with diagonal D. The columns of P are independent eigenvectors. Having n linearly independent eigenvectors is the key requirement for an n x n matrix.

8.7. Orthogonality and Orthonormality

Vectors are orthogonal when their dot product is zero. They are orthonormal when they are also unit vectors. Orthonormal bases simplify coordinates, projections, and numerical computation.

8.8. Gram-Schmidt Method

Gram-Schmidt converts an independent set into an orthogonal or orthonormal set spanning the same subspace. Each new vector has its projections onto the previously constructed directions removed.

8.9. QR Factorization: Gram-Schmidt, Householder, and Givens

Under suitable conditions an m x n matrix can be written as

A = Q R,

where Q has orthonormal columns and R is upper triangular. Classical Gram-Schmidt is pedagogically transparent; Modified Gram-Schmidt is usually more numerically reliable. Householder reflections are efficient for eliminating many entries below a pivot, while Givens rotations are useful for targeting selected entries.

For least squares, QR avoids forming the normal equations A^T A, which can unnecessarily square the condition number. If A=QR, orthonormality reduces

min ||Ax-b||

to a triangular problem.

Forming the normal equations for an overdetermined least-squares problem approximately squares the condition number through AᵀA. Nearly dependent columns can then amplify floating-point error. QR factorization uses orthogonal transformations instead of explicitly constructing that matrix, reducing avoidable numerical degradation. If two columns are almost linearly dependent, sensitivity to small perturbations in the data is intrinsic: QR does not remove that uncertainty, but helps avoid adding instability from the solution procedure. Singular-value decomposition is especially informative when rank deficiency is suspected.

8.10. Diagonalization of Symmetric Matrices

A real symmetric matrix has real eigenvalues and can be diagonalized by an orthogonal matrix:

A = Q D Q^T.

This spectral structure is particularly important in numerical analysis, optimization, and quadratic forms.

8.11. Discrete Dynamical Systems Through Eigenvalues

If a system state is updated by the same linear transformation at every step,

x_(k+1) = A x_k,

then

x_k = A^k x_0.

When A is diagonalizable, A^k = P D^k P^-1 exposes long-term behavior directly through the eigenvalues. Directions associated with eigenvalues of larger magnitude tend to dominate, while components with magnitude below one decay under repeated application.

For stochastic transition matrices, the eigenvalue λ=1 is especially important. Under suitable conditions, an associated eigenvector describes a stationary distribution. Convergence is not guaranteed merely by the existence of λ=1; the remaining eigenvalue magnitudes and structural properties of the chain also matter.

This viewpoint turns eigenvalues from roots of a characteristic polynomial into dynamic quantities that explain which directions grow, decay, or persist under iteration.

8.12. The Role of Linear Algebra in Artificial Intelligence

Modern machine learning represents much of its data and parameters as vectors, matrices, or higher-order tensors. Linear algebra is therefore not an auxiliary calculation layer added to AI; it is the shared language of representation and computation.

8.13. Vector Representation and Feature Space

An observation may be written as:

x = [x1, x2, ..., xd]^T

Distances and inner products describe geometry in this feature space. Cosine similarity:

cos(θ) = (x^T y) / (||x|| ||y||)

is widely used with embeddings. Similarity in embedding space, however, is not logical identity; it depends on the representation learned by the embedding model.

8.14. A Weight Matrix Is a Transformation

A basic neural-network layer has the form:

z = W x + b

W maps the input space to another vector space. Once a nonlinear activation is added, the full layer is no longer linear, yet matrix multiplication remains a dominant computational primitive. For a batch of samples:

Z = X W^T + b

which also explains why modern accelerators emphasize matrix operations.

8.15. Rank and Learned Representation

Rank describes the number of independent directions represented by a matrix. High-dimensional data may lie near a lower-dimensional subspace. A low-rank approximation:

W ≈ U_r Σ_r V_r^T

can reduce storage or computation at the cost of approximation error. Rank selection is therefore a tradeoff, not a lossless simplification.

8.16. Eigenvectors, Covariance, and PCA

For centered data, eigenvectors of the covariance structure identify directions of variance. PCA projects onto dominant directions. It is unsupervised: the direction with highest variance is not guaranteed to be the most discriminative for a downstream label.

8.17. SVD and Numerical Reliability

Singular Value Decomposition:

A = U Σ V^T

supports rank analysis, pseudoinverses, least squares, and low-rank approximations. Solving least squares through normal equations may worsen conditioning; QR or SVD can be numerically safer. Large AI computations do not escape conditioning problems.

8.18. Attention Is Also Built on Linear Algebra

Transformer attention contains operations such as:

Q = X W_Q
K = X W_K
V = X W_V
A = softmax(Q K^T / √d_k)
Y = A V

Softmax is nonlinear, but projections, similarity scores, and output aggregation are dense matrix operations.

8.19. Norms, Scale, and Optimization

Norms define distance and parameter magnitude. L2 regularization adds a term such as:

λ ||w||²_2

while L1 has a different geometry and can promote sparsity. The choice of norm changes the optimization problem rather than merely changing notation.

Linear algebra does not explain every aspect of learning. Generalization and uncertainty require probability and statistics; parameter optimization requires multivariable calculus. Its distinctive strength is that it describes data representation, subspaces, transformations, and the algebraic core of large model computations within one framework.

8.20. From Eigenvectors to Network Centrality

Network science studies how connectivity patterns produce system-level behavior rather than reducing analysis to isolated node attributes. Degree distributions, centralities, communities, bridges, cores, and robustness answer different questions; no single metric universally measures 'importance'. Metric selection must follow the directed/weighted nature of the data and the actual analytical question.

• Connect eigenvector centrality and PageRank-like methods to their linear-algebra formulation.
• Relate spectral properties to components and community structure.
• State how directed versus symmetric matrices change eigenvalue interpretation.

8.21. From Matrix Rank to Tensor Rank

The singular-value decomposition A = UΣVᵀ constructs a rank-r approximation using the leading r singular values and their vectors. The truncated matrix A_r = U_r Σ_r V_rᵀ is optimal among matrices of at most that rank in both the Frobenius and spectral norms. The squared Frobenius approximation error equals the sum of squares of the discarded singular values; approximation loss is therefore directly measurable.

For tensors with three or more modes, there is no single notion of rank identical to matrix rank. A mode-n unfolding treats one tensor axis as rows and combines the remaining axes into columns, bringing matrix rank into the analysis. Tucker decomposition uses lower-dimensional subspaces for these unfoldings; CP decomposition represents a tensor through sums of rank-one outer products. The matrix theorem for best rank-r approximation does not transfer unchanged to every tensor factorization.

Preserving tensor structure makes it explicit which axis is compressed in data organized as time × sensor × measurement type. Low rank, however, is a modeling assumption rather than a physical law. Increasing the selected ranks raises storage and computation; making them too small may suppress important local structure. Rank selection should therefore consider explained variation, the residual-error pattern and the application's accepted tolerance.

For a centered observation matrix X, the eigenvectors of the covariance matrix XᵀX/(n−1) establish the principal directions used by PCA. Right singular vectors determine the same directions, while squared singular values recover the eigenvalues after the relevant normalization. If one measurement is recorded in meters and another in millimeters, however, raw covariance may be dominated by the arbitrary units. Standardization can mitigate this but also reweights physically meaningful variation. Units and the analytical objective must therefore be specified before PCA, and explained variance must not be mistaken for predictive accuracy.

References

• Ahmet Yesevi Üniversitesi Bilgisayar Mühendisliği Bölümü. Lineer Cebir (TBIL203) ders materyalleri.

• Albert-László Barabási. Network Science.

• David C. Lay, Steven R. Lay, Judi J. McDonald. Linear Algebra and Its Applications, 6th Edition. Pearson, 2021.

• Gene H. Golub, Charles F. Van Loan. Matrix Computations, 4th Edition. Johns Hopkins University Press, 2013.

• Gilbert Strang. Introduction to Linear Algebra, 5th Edition. Wellesley-Cambridge Press, 2016.

• Gilbert Strang. Linear Algebra and Learning from Data. Wellesley-Cambridge Press, 2019.

• Howard Anton, C. R. Elementary Linear Algebra, 12th Edition. Wiley, 2019.

• Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. MIT Press, 2016.

• Jure Leskovec; Anand Rajaraman; Jeffrey D. Ullman. Mining of Massive Datasets.

• Lloyd N. Trefethen, David Bau III. Numerical Linear Algebra. SIAM, 1997.

• Roger A. Horn, Charles R. Johnson. Matrix Analysis, 2nd Edition. Cambridge University Press, 2012.

• Walter W. Piegorsch, Richard A. Levine, Hao Helen Zhang, Thomas C. M. Lee (eds.). Computational Statistics in Data Science. Wiley, 2022. ISBN 9781119561071.