Gradient, Critical Points, and Optimization
Finding Minima Using First- and Second-Order Information
Building on partial derivatives and the Jacobian, this chapter introduces the gradient vector, directional derivatives, gradient descent, critical point theory, and the practical methods for finding global minima of functions on subsets of Rⁿ.
Sections
Directional (U-) Derivative
10 min · 2 quiz questions
U-Derivative of a Linear Function
6 min · 1 quiz question
U-Derivative and the LLA
8 min · 2 quiz questions
U-Derivative from the Jacobian
7 min · 1 quiz question · Interactive diagram
Sign of the U-Derivative
7 min · 2 quiz questions
The Gradient Vector
10 min · 2 quiz questions
Steepest Descent Direction
7 min · 2 quiz questions
Gradient Descent Algorithm
10 min · 2 quiz questions · Interactive diagram
Critical Points
8 min · 2 quiz questions
Local Minima
8 min · 2 quiz questions
Fermat's Theorem
7 min · 2 quiz questions
Saddle Points
8 min · 2 quiz questions
Calculating Critical Points
10 min · 2 quiz questions
Quadratic Case: Critical Points
8 min · 2 quiz questions · Interactive diagram
Quadratic Case: Classifying Extrema
10 min · 2 quiz questions · Interactive diagram
Least Squares as Optimization
10 min · 2 quiz questions · Interactive diagram
Global Minima
8 min · 2 quiz questions
Strategies for Finding Global Minima
8 min · 2 quiz questions
Compact Domains
8 min · 2 quiz questions
Extreme Value Theorem
7 min · 2 quiz questions · Interactive diagram
Practical Minimization Method
10 min · 2 quiz questions
Non-Quadratic Example
10 min · 2 quiz questions · Interactive diagram
Computational Tools for Optimization
7 min · 2 quiz questions