Orthogonal Projection

Every vector $\mathbf{b} \in \mathbb{R}^m$ can be split uniquely into two pieces: one piece inside a subspace $V$, and one piece in $V^\perp$. This decomposition $\mathbf{b} = \mathbf{v} + \mathbf{z}$ (with $\mathbf{v} \in V$, $\mathbf{z} \in V^\perp$) is guaranteed to exist and be unique — this is the Orthogonal Projection Theorem.

Given an orthonormal basis $U$ for $V$, the projection is computed as $\mathbf{v} = UU^t\mathbf{b}$ (apply weight finding to all of $\mathbf{b}$, not just vectors in $V$). The perpendicular piece is $\mathbf{z} = \mathbf{b} - \mathbf{v}$. The key fact is that $\mathbf{z} \in V^\perp$: you can verify $U^t \mathbf{z} = U^t(\mathbf{b} - UU^t\mathbf{b}) = U^t\mathbf{b} - U^t\mathbf{b} = \mathbf{0}$.

The component $\mathbf{v} = UU^t\mathbf{b}$ is called the orthogonal projection of $\mathbf{b}$ onto $V$, written $\text{Proj}_V(\mathbf{b})$. The matrix $P = UU^t$ is called the projection matrix for $V$. Importantly, $P$ does not depend on the choice of orthonormal basis — any orthonormal basis for $V$ gives the same $P$.