9.1610 min read

Principal Components

The principal components of centered data $A$ are the left singular vectors $\mathbf{u}_1, \mathbf{u}_2, \ldots$ of $A$ , ordered by decreasing $\sigma_i$ . The first principal component $\mathbf{u}_1$ is the direction of greatest variance; $\mathbf{u}_2$ is perpendicular to $\mathbf{u}_1$ and captures the next most variance.

The principal coordinates (scores) are the projections: $C = U_k^\top A \in \mathbb{R}^{k \times N}$ . Each column is the position of one data point in the reduced-dimension space. The fraction of variance explained by the top $k$ components is $\sum_{i=1}^k \sigma_i^2 / \sum_i \sigma_i^2$ .

Formal View

Definition 9.5 — Principal Components and Scores

For centered data matrix

A = U\Sigma V^\top

: - Principal components: columns

\mathbf{u}_1, \ldots, \mathbf{u}_k

U

(sorted by descending

\sigma_i

). - Principal coordinates (scores):

C = U_k^\top A \in \mathbb{R}^{k \times N}

Fraction of variance explained by top $k$ : $\sum_{i=1}^k \sigma_i^2 / \sum_i \sigma_i^2$ .

Interactive Visualization

Orthogonal Projection

Why This Matters

Principal components reveal structure in high-dimensional data, enabling visualization and compression.

Genome data: top 2 PCA components separate ethnic groups in GWAS.
Finance: PCA of stock returns identifies market, sector, and idiosyncratic risk factors.
Image recognition: eigenfaces are principal components of a face dataset.

Learning Resources

PCA Step by Step

StatQuest

Complete walkthrough including principal components and scores.

22 min

PCA and SVD explained

3Blue1Brown

Geometric interpretation of principal components as directions of variance.

15 min

Quiz

Question 1

Principal components are:

Question 2

The fraction of total variance explained by the top- $k$ components is $\sum_{i=1}^k \sigma_i^2 / \sum_i \sigma_i^2$ .

Common Mistakes

Confusing principal components (directions, columns of $U$ ) with principal coordinates (scores, $U_k^\top A$ ).
Forgetting to center before computing principal components.