12.148 min read

Quadratic Case: Critical Points

For a quadratic function $f(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} + \mathbf{b}^T \mathbf{x} + c$ (symmetric $A$ ), the gradient is $\nabla f(\mathbf{x}) = 2A\mathbf{x} + \mathbf{b}$ .

Setting the gradient to zero: $2A\mathbf{x} + \mathbf{b} = \mathbf{0}$ , giving $A\mathbf{x} = -\mathbf{b}/2$ . If $A$ is invertible, the unique critical point is $\mathbf{x}^* = -A^{-1}\mathbf{b}/2$ .

If $A$ is singular (not invertible): the system either has no solution (if $\mathbf{b}/2 \notin \text{col}(A)$ , meaning no critical points) or infinitely many solutions (if $\mathbf{b}/2 \in \text{col}(A)$ , meaning an entire subspace of critical points).

Formal View

Theorem 12.8 — Critical Point of a Quadratic

For

f(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} + \mathbf{b}^T \mathbf{x} + c

with symmetric

A

: the critical points satisfy

2A\mathbf{x} = -\mathbf{b}

. If

A

is invertible, the unique critical point is

\mathbf{x}^* = -\frac{1}{2}A^{-1}\mathbf{b}

Interactive Visualization

Eigenvector Explorer

Why This Matters

Quadratic optimization is the backbone of all second-order methods and regularized regression.

Least squares has a quadratic objective whose unique critical point is the normal equation solution
Ridge regression adds $\lambda\|\mathbf{x}\|^2$ making the Hessian $A^T A + \lambda I$ positive definite and invertible
All quadratic programming (QP) solvers exploit this structure

Learning Resources

Optimization of Quadratic Forms

MIT OpenCourseWare

Quadratic optimization and the role of symmetric matrices.

48 min

Matrix Calculus and Optimization

Steve Brunton

Minimizing quadratic objectives using matrix calculus.

18 min

Quiz

Question 1

For $f(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} + \mathbf{b}^T \mathbf{x}$ with invertible symmetric $A$ , the unique critical point is:

Question 2

If $A$ is singular and $\mathbf{b} \notin \text{col}(A)$ , then $f(\mathbf{x}) = \mathbf{x}^T A \mathbf{x} + \mathbf{b}^T \mathbf{x}$ has no critical points.

Common Mistakes

Forgetting the factor of 2 in $\nabla(\mathbf{x}^T A \mathbf{x}) = 2A\mathbf{x}$ .
Assuming a unique critical point exists without checking invertibility of $A$ .
Confusing the critical point with the Hessian — the critical point is a vector, the Hessian is a matrix.