9.18 min read

Motivation for SVD

The spectral theorem gives a beautiful decomposition $A = U\Lambda U^\top$ for symmetric matrices. But what about a general $m \times n$ matrix $A$ that might not even be square? We need a generalization — something that reveals the geometric structure of any linear transformation, not just symmetric ones.

The key insight: even if $A$ itself is not symmetric, the products $A^\top A$ (an $n \times n$ matrix) and $AA^\top$ (an $m \times m$ matrix) are always symmetric and PSD. We can apply the spectral theorem to each of them. The Singular Value Decomposition (SVD) packages this information into a single clean factorization $A = U \Sigma V^\top$ that works for any matrix.

Geometrically, the SVD says every linear map $\mathbf{x} \mapsto A\mathbf{x}$ is a composition of: (1) a rotation/reflection $V^\top$ in the domain, (2) a scaling $\Sigma$ along coordinate axes, and (3) a rotation/reflection $U$ in the codomain. This is the most fundamental geometric description of any linear transformation.

Formal View

Remark 9.1 — Need for SVD

The spectral theorem applies only to symmetric matrices. For a general

m \times n

matrix

A

, the products

A^\top A

(size

n \times n

) and

AA^\top

(size

m \times m

) are both symmetric and PSD. SVD packages both spectral decompositions into one unified factorization

A = U\Sigma V^\top

Why This Matters

SVD is the Swiss army knife of linear algebra — it solves least squares, compresses data, finds low-rank structure, and enables PCA.

Image compression: keep only the top singular values and vectors to get a compact representation.
Natural language processing: latent semantic analysis uses SVD to find topics in text.
Recommendation systems: matrix factorization via SVD identifies latent factors in user-item matrices.

Learning Resources

Singular value decomposition (SVD)

3Blue1Brown

Visual and geometric explanation of what the SVD means for any linear map.

15 min

SVD: motivation and computation

Steve Brunton

Data scientist's perspective on why SVD matters and how to use it.

20 min

Quiz

Question 1

The spectral theorem applies to all matrices, not just symmetric ones.

Question 2

For any $m \times n$ matrix $A$ , the product $A^\top A$ is:

Common Mistakes

Thinking SVD is just for square matrices — it works for any $m \times n$ matrix, even non-square.
Confusing SVD with eigendecomposition — eigendecomposition applies to square matrices, SVD applies universally.