Linear Algebra
16.58 min read

Quadratic Approximation

You know the linear (first-order) approximation to ff at x0\mathbf{x}_0: f(x)≈f(x0)+Df(x0)(x−x0)f(\mathbf{x}) \approx f(\mathbf{x}_0) + Df(\mathbf{x}_0)(\mathbf{x} - \mathbf{x}_0). This is good "to first order."

The quadratic (second-order) approximation adds a curvature term using the Hessian: f~(x)=f(x0)+Df(x0)u+12ut[Hf(x0)]u,u=x−x0\tilde{f}(\mathbf{x}) = f(\mathbf{x}_0) + Df(\mathbf{x}_0)\mathbf{u} + \frac{1}{2}\mathbf{u}^t [Hf(\mathbf{x}_0)] \mathbf{u}, \quad \mathbf{u} = \mathbf{x} - \mathbf{x}_0

This is the multivariable Taylor expansion up to second order. The 12\frac{1}{2} factor comes directly from the 12\frac{1}{2} in f(x)=12xtMxf(\mathbf{x}) = \frac{1}{2}\mathbf{x}^t M \mathbf{x}.

The approximation is "really good" near x0\mathbf{x}_0: the error f(x)−f~(x)f(\mathbf{x}) - \tilde{f}(\mathbf{x}) goes to zero faster than ∥x−x0∥2\|\mathbf{x} - \mathbf{x}_0\|^2. This is what it means to be a local quadratic approximation.

Formal View

Definition 16.4 — Local Quadratic Approximation
A quadratic function gg is a local quadratic approximation of ff at x0\mathbf{x}_0 if
lim⁡x→x0f(x)−g(x)∥x−x0∥2=0\lim_{\mathbf{x} \to \mathbf{x}_0} \frac{f(\mathbf{x}) - g(\mathbf{x})}{\|\mathbf{x} - \mathbf{x}_0\|^2} = 0
Theorem 16.3 — Existence and Uniqueness
A function ff has a local quadratic approximation at x0\mathbf{x}_0 if and only if f∈D2f \in D^2 at x0\mathbf{x}_0. In this case the unique approximation is:
f~(x)=f(x0)+Df(x0)u+12utHf(x0)u,u=x−x0\tilde{f}(\mathbf{x}) = f(\mathbf{x}_0) + Df(\mathbf{x}_0)\mathbf{u} + \frac{1}{2}\mathbf{u}^t Hf(\mathbf{x}_0)\mathbf{u}, \quad \mathbf{u} = \mathbf{x} - \mathbf{x}_0

Why This Matters

The quadratic approximation is the foundation of second-order optimization: Newton's method, trust region methods, and more.

  • Newton's method minimizes the local quadratic approximation at each step
  • Fisher information matrix in statistics is a Hessian of the log-likelihood
  • Curvature-aware learning rates in deep learning (AdaHessian, K-FAC)
  • Structural analysis: the potential energy of a deformed structure is approximated quadratically

Quiz

Question 1

The quadratic approximation to ff at x0\mathbf{x}_0 requires knowing:

Question 2

If x0\mathbf{x}_0 is a critical point, the linear term Df(x0)uDf(\mathbf{x}_0)\mathbf{u} vanishes from the quadratic approximation.

Question 3

The error in the quadratic approximation near x0\mathbf{x}_0 behaves as:

Common Mistakes

  • Forgetting the 12\frac{1}{2} factor in the Hessian term of the quadratic approximation.
  • Confusing the quadratic approximation (a local model) with the actual function values far from x0\mathbf{x}_0.
  • Applying the quadratic approximation at a non-differentiable point.