How does a model decide which way to change millions of parameters?

examplepredictionlossgradientchanged parameters

01 · Smallest useful mechanism

The gradient describes the local uphill direction of the loss surface.

A model with two parameters can be pictured as a point above a loss landscape. Partial derivatives measure how loss changes when each parameter moves slightly. Together they form a gradient pointing toward steepest local increase. Subtracting it produces a downhill step. Repeating the calculation traces a path shaped by the data, surface, and learning rate.

Gradient descent subtracts a learning-rate-scaled gradient, trading step speed against the risk of oscillation or divergence.

02 · Experiment

Test the prediction

Exact regression objective · analytic and numerical gradients

A two-parameter model exposes its complete loss surface, update direction, and finite-difference verification.

Learning rate

Model space

Three samples and one line

y = 2x - 1

Parameter space

Loss over w and b

Ready to descend

Exact batch calculation

Prediction → residual → loss → gradient

MSE loss

17.167

Parameters

w -1.500 · b 2.000

x	target	prediction	residual	residual²	2·residual·x	2·residual
-1	-3	3.500	6.500	42.250	-13.000	13.000
0	-1	2.000	3.000	9.000	0.000	6.000
1	1	0.500	-0.500	0.250	-1.000	-1.000

Analytic gradient

[dw -4.667, db 6.000]

Finite-difference check

[dw -4.667, db 6.000]

Update rule

θnew = θ − 0.20∇L

Important limitation

This convex two-parameter surface is deliberately inspectable. A large model has millions or billions of parameter directions; a two-dimensional picture can be a useful slice, but never the full geometry.

Loading the learning experiment…

03 · Reason from evidence

Loss becomes NaN a few steps after the learning rate is increased tenfold.

Your brief: Design a minimal investigation that can distinguish a derivative bug from an unstable update and a numerically unsafe loss calculation.

Evidence your answer should use

An analytic-versus-finite-difference gradient check at a finite point
Loss, gradient norm, parameter norm, and update norm by step
The first operation that produces NaN or infinity
A rerun at the previous learning rate with the same batch and seed

There is no reveal button here. A strong response names the claim, the evidence that would test it, and what the evidence cannot establish.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

The weight gradient is negative. What does gradient descent do to that weight?

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: A shallow model has an easy derivative. Deep networks need an efficient way to reuse derivatives across many layers.