Optimizers in Practice

Gradient descent minimizes loss by stepping opposite the gradient.

Variants

• SGD + momentum: strong default for many vision workloads
• Adam / AdamW: adaptive per-parameter rates; AdamW decouples weight decay
• Schedules: cosine decay, warmup, reduce-on-plateau

Debugging checklist

• Loss NaNs → LR too high, bad init, mixed precision issues
• Loss flat → LR too low, buggy labels, saturated activations
• Val diverges → regularization / early stop / data mismatch