Linear Algebra
12.237 min read

Computational Tools for Optimization

For most practical optimization problems, closed-form solutions are unavailable. Computational tools are essential. The key tool in scientific computing environments (MATLAB, Python/NumPy, Julia) for optimization is a numerical solver that iteratively improves a solution.

In MATLAB, fminunc minimizes an unconstrained differentiable function; you provide the function handle and optionally its gradient. fmincon handles constrained optimization. In Python, scipy.optimize.minimize provides similar functionality.

The solver needs: (1) an objective function handle, (2) an initial guess x0\mathbf{x}_0, (3) optionally a gradient function, and (4) solver options (tolerances, maximum iterations). Providing the gradient dramatically speeds up convergence and improves accuracy.

Formal View

Example 12.4 — Numerical Optimization
To minimize f(x,y)=(x−1)2+(y−2)2+xyf(x,y) = (x-1)^2 + (y-2)^2 + xy numerically: MATLAB: `[x,fval] = fminunc(@(v) (v(1)-1)^2 + (v(2)-2)^2 + v(1)*v(2), [0;0])` Python: `from scipy.optimize import minimize; res = minimize(lambda v: (v[0]-1)2 + (v[1]-2)2 + v[0]*v[1], [0,0])` Both require a starting point and return an approximate minimizer.

Why This Matters

Numerical optimization solvers are used in virtually every area of science and engineering.

  • Training machine learning models with optimizers like Adam, SGD, L-BFGS
  • Parameter estimation in statistical models
  • Design optimization in engineering (shape, topology, material selection)

Quiz

Question 1

What does providing the gradient function (in addition to the objective) to a numerical optimizer typically achieve?

Question 2

Numerical optimization solvers for unconstrained problems only need an objective function and a starting point.

Common Mistakes

  • Providing the wrong gradient (off by a factor of 2, wrong sign) — this causes the optimizer to move in wrong directions.
  • Using an initial point far from the solution, leading to slow convergence or convergence to a bad local minimum.
  • Forgetting to check convergence diagnostics — solvers can fail silently, returning a suboptimal or unconverged solution.