9.1310 min read

The SVD Solution

The solution to the linear reduction problem is: compute $A = U\Sigma V^\top$ and take $V_k = U_k = [\mathbf{u}_1, \ldots, \mathbf{u}_k]$ — the top $k$ left singular vectors. The minimum reconstruction error is $\|A - U_k U_k^\top A\|_F^2 = \sigma_{k+1}^2 + \cdots + \sigma_r^2$ .

The fraction of variance retained is $\sum_{i=1}^k \sigma_i^2 / \|A\|_F^2$ . As $k$ increases, more variance is retained and error decreases. When $k = r$ (rank), the error is zero.

Formal View

Theorem 9.5 — SVD Solves Linear Reduction

The optimal

V_k = U_k = [\mathbf{u}_1, \ldots, \mathbf{u}_k]

(first

k

left singular vectors of

A

). Minimum reconstruction error:

\|A - U_k U_k^\top A\|_F^2 = \sum_{i=k+1}^{\min(m,N)} \sigma_i^2.

Explained variance: $\sum_{i=1}^k \sigma_i^2$ out of total $\|A\|_F^2 = \sum_i \sigma_i^2$ .

Interactive Visualization

SVD as Three Transformations

Why This Matters

The SVD gives the provably optimal way to compress data while minimizing reconstruction error.

PCA: project onto top- $k$ singular vectors to capture maximum variance.
Image compression: only largest singular values and vectors needed.
Collaborative filtering: SVD finds latent user/item factors.

Learning Resources

SVD and optimal approximation

Steve Brunton

SVD as the solution to low-rank approximation with image compression examples.

20 min

PCA and SVD connection

StatQuest

How the SVD solution gives principal components.

22 min

Quiz

Question 1

The optimal $k$ -dimensional subspace for dimensionality reduction is spanned by:

Question 2

Keeping only the top- $k$ singular vectors minimizes the Frobenius-norm reconstruction error.

Common Mistakes

Using right singular vectors $V$ instead of left singular vectors $U$ for the reduction subspace.
Confusing "top- $k$ " (largest) with "bottom- $k$ " — always keep largest singular values.