17.37 min read

Constrained Optimization — The Setup

We want to minimize $f(\mathbf{x})$ subject to the constraint $g(\mathbf{x}) = k$ for some function $g$ . The feasible set is the isosurface $S = \{\mathbf{x} : g(\mathbf{x}) = k\}$ — a hypersurface in $\mathbb{R}^n$ .

We cannot just run unconstrained gradient descent — it might step off the surface. We need to understand which directions of movement are feasible (stay on $S$ ) and which make $f$ decrease.

Feasible directions at $\mathbf{x}_0$ are exactly the tangent space $T_{\mathbf{x}_0} S = \text{Null}(Dg(\mathbf{x}_0))$ . A direction $\mathbf{u}$ decreases $f$ if $Df(\mathbf{x}_0)\mathbf{u} < 0$ .

So if there exists a $\mathbf{u} \in T_{\mathbf{x}_0} S$ with $Df(\mathbf{x}_0)\mathbf{u} < 0$ , we can move along $\mathbf{u}$ to decrease $f$ while staying on $S$ — $\mathbf{x}_0$ is not optimal. For $\mathbf{x}_0$ to be a constrained minimum, no such direction can exist: $Df(\mathbf{x}_0)\mathbf{u} = 0$ for all $\mathbf{u} \in T_{\mathbf{x}_0} S$ .

Formal View

Definition 17.3 — Constrained Optimization

The constrained optimization problem:

\min_{\mathbf{x}} f(\mathbf{x}) \quad \text{s.t.} \quad g(\mathbf{x}) = k

with

f, g \in C^1

. The feasible set is

S = \{\mathbf{x} : g(\mathbf{x}) = k\}

. We assume

Dg(\mathbf{x}) \neq \mathbf{0}

S

(so

S

is a well-behaved hypersurface).

Lemma 17.1 — Necessary Condition for a Constrained Minimum

\mathbf{x}_0 \in S

is a constrained minimum of

f

S

, then:

\text{Null}(Dg(\mathbf{x}_0)) \subseteq \text{Null}(Df(\mathbf{x}_0))

Every direction tangent to

S

\mathbf{x}_0

must be an isovector of

f

Why This Matters

Equality-constrained optimization is everywhere: engineering design with fixed resources, physics with conservation laws, economics with budget constraints.

Budget-constrained utility maximization in economics
Robot arm reaching target position (kinematic constraint)
Circuit design: minimize power subject to performance constraints
Machine learning: training with constraints (e.g., orthogonality)

Learning Resources

Lagrange multipliers introduction

3Blue1Brown

Visual explanation of constrained optimization via level curves.

13 min

Constrained optimization — Lagrange multipliers

Khan Academy

Sets up the constrained optimization problem and introduces the key condition.

12 min

Quiz

Question 1

For constrained optimization $\min f$ s.t. $g(\mathbf{x}) = k$ , the feasible directions at $\mathbf{x}_0$ are:

Question 2

At a constrained minimum $\mathbf{x}_0$ , what must be true about all tangent directions $\mathbf{u} \in T_{\mathbf{x}_0} S$ ?

Question 3

A constrained minimum of $f$ on $S$ must also be an unconstrained critical point of $f$ .

Question 4

The necessary condition for a constrained minimum is $\text{Null}(Dg) \subseteq \text{Null}(Df)$ . This means:

Common Mistakes

Confusing the feasible set (the surface $S$ ) with the constraint function $g$ .
Forgetting that unconstrained critical points of $f$ need not be constrained critical points, and vice versa.
Applying unconstrained gradient descent on the full space when the feasible set is a lower-dimensional surface.