Training is a mathematical game. We want to find the bottom of a deep valley (the minimum of the loss function). The loss measures how far the current prediction is from the correct answer. The optimizer decides how we step down into that valley. Steps that are too large overshoot the goal; steps that are too small make training drag on forever.