10.27 min read

Why We Need Derivatives

To minimize $f$ , we need to know which direction is "downhill" at any point. This is exactly what the derivative tells us: the rate at which $f$ changes as we perturb the input. Without derivatives, we would need to evaluate $f$ at every possible input — which is impossible for continuous functions.

For a one-variable function $f(x)$ , the derivative $f'(x_0)$ gives the slope of the tangent line at $x_0$ . If $f'(x_0) > 0$ , moving right increases $f$ ; if $f'(x_0) < 0$ , moving right decreases $f$ . At a minimum, $f'(x^*) = 0$ — flat tangent, no immediate improvement from either direction.

In multiple variables, the gradient $\nabla f(\mathbf{x})$ generalizes the derivative — it points in the direction of steepest ascent. The negative gradient $-\nabla f$ points downhill. This is the direction we follow in gradient descent.

Formal View

Remark 10.1 — Why Derivatives Matter for Optimization

At a local minimum

x^*

of a differentiable function

f

, the derivative must satisfy

f'(x^*) = 0

(in 1D) or

\nabla f(\mathbf{x}^*) = \mathbf{0}

(in multiple dimensions). This necessary condition dramatically restricts where minima can occur.

Interactive Visualization

Interactive Line Explorer

Why This Matters

Derivatives make optimization tractable — they tell you where to step without evaluating $f$ everywhere.

Backpropagation in neural networks: compute derivatives of loss w.r.t. every parameter via the chain rule.
Newton's method: use both first and second derivatives to find roots or minima faster than gradient descent.
Sensitivity analysis: $f'(x_0)$ tells how much the optimum changes if a constraint changes slightly.

Learning Resources

Essence of calculus: derivatives

3Blue1Brown

Geometric intuition for the derivative as a rate of change.

17 min

Introduction to derivatives

Khan Academy

Basics of derivatives and why they measure rates of change.

9 min

Quiz

Question 1

If $f'(x^*) = 0$ , then $x^*$ must be a minimum.

Question 2

The negative gradient $-\nabla f(\mathbf{x})$ points in the direction of:

Common Mistakes

Thinking $f'(x^*) = 0$ is sufficient for a minimum — it is only necessary. Check second-order conditions or function values to confirm.
Confusing the derivative (a number, the slope) with the gradient (a vector, pointing uphill).