9.1710 min read

Classical PCA

The full Classical PCA pipeline: (1) Center: $\mathbf{a}_i = \mathbf{q}_i - \mathbf{m}$ ; (2) form $A = [\mathbf{a}_1 \cdots \mathbf{a}_N]$ ; (3) compute SVD $A = U\Sigma V^\top$ ; (4) choose $k$ ; (5) reconstruct: $\tilde{\mathbf{q}} = \mathbf{m} + U_k U_k^\top (\mathbf{q} - \mathbf{m})$ .

The formula projects the centered point onto the top- $k$ subspace, then adds the mean back. This is the best rank- $k$ affine approximation of the original data.

Formal View

Definition 9.6 — Classical PCA Reconstruction

For centered data

A = U\Sigma V^\top

and data mean

\mathbf{m}

, the Classical PCA rank-

k

reconstruction of

\mathbf{q}

\tilde{\mathbf{q}} = \mathbf{m} + U_k U_k^\top (\mathbf{q} - \mathbf{m}).

Why This Matters

Classical PCA is the standard dimensionality reduction method throughout science and engineering.

Face recognition: project onto eigenfaces.
Signal compression: project onto dominant PCA modes.
Visualizing high-dimensional data in 2D.

Learning Resources

PCA step-by-step

StatQuest

Complete walkthrough of classical PCA.

22 min

PCA and SVD in practice

Steve Brunton

Practical PCA implementation and connection to SVD.

16 min

Quiz

Question 1

The classical PCA reconstruction of $\mathbf{q}$ is:

Question 2

If $\mathbf{q} - \mathbf{m}$ lies in $\operatorname{col}(U_k)$ , then $\tilde{\mathbf{q}} = \mathbf{q}$ .

Common Mistakes

Forgetting to add the mean back after projection.
Confusing the low-dimensional score $U_k^\top(\mathbf{q}-\mathbf{m}) \in \mathbb{R}^k$ with the reconstruction $\tilde{\mathbf{q}} \in \mathbb{R}^m$ .