Numerical methods (how you actually solve)

Parent: optimization

Closed-form ∇=0 rarely solves; you iterate xₖ₊₁ = xₖ + step. The families:

First-order (use the gradient)

Gradient descent x ← x − η∇f (= forward Euler on the gradient-flow ODE, dynamics-to-a-fixed-point);
Subgradient (non-smooth), proximal-gradient (smooth + simple non-smooth), mirror descent (non-Euclidean geometry);
Accelerated: heavy-ball (Polyak), Nesterov (= a damped 2nd-order ODE); optimal O(1/k²) for convex;
Coordinate descent (one block at a time).

Second-order (use curvature)

Newton x ← x − (∇²f)⁻¹∇f (quadratic convergence, but Hessian expensive);
Quasi-Newton (BFGS / L-BFGS — build curvature from gradients);
Gauss–Newton / Levenberg–Marquardt (least squares).

Constrained solvers

Projected gradient (project onto the feasible set each step);
Interior-point / barrier (push away from the boundary; the workhorse for convex cone programs);
Augmented Lagrangian, ADMM (operator splitting — decompose + a consensus step; = decompose-and-reassemble);
Active-set, SQP (sequential quadratic programming).

Globalization — the "gate" of optimization

A proposed step must be accepted:

Line search (Armijo / Wolfe sufficient-decrease);
Trust region (accept iff the model's predicted decrease is realized, ratio ρ good; else shrink & retry). → This accept-or-reject test is exactly the gate of stages-gates-as-hard-optimization.

The unifying lens

Most of these are fixed-point iterations of a contraction / a discretized flow to an equilibrium (dynamics-to-a-fixed-point); operator-splitting/monotone-operator theory (e.g. ADMM, proximal) unifies many under one fixed-point framework.