Recap: Neural Networks I — How Learning Works
DS701 Session 19 — Mon Nov 9, 2026
A neuron, a layer, a network — one composed function
\[\text{out} = f\Big(\sum_i w_i x_i + b\Big)\]
- With \(f = \sigma\) this is logistic regression — weights, bias, squash.
- A layer: many neurons in parallel on the same inputs → \(\mathbf{h} = f(\boldsymbol\beta + \boldsymbol\Omega\mathbf{x})\).
- A network: layers in sequence, each layer’s outputs the next one’s inputs: \[\hat{\mathbf y} = \boldsymbol\beta_K + \boldsymbol\Omega_K\, f(\cdots f(\boldsymbol\beta_0 + \boldsymbol\Omega_0\mathbf{x})\cdots)\]
- So an MLP is a function of its parameters \(\theta = \{\boldsymbol\Omega_k, \boldsymbol\beta_k\}\) — and the activation \(f\) is what stops the stack collapsing to one linear map.
Training is a loop: loss → gradient → update
- Loss — a number that says how wrong the network is on the data: MSE \(\frac1N\sum(\hat y_i - y_i)^2\), or cross-entropy for classes.
- Gradient — \(\nabla_\theta L\): how the loss changes with every parameter.
- Update — step downhill: \[\theta \leftarrow \theta - \eta\,\nabla_\theta L\]
- Repeat until the loss stops falling.
Non-convex surface: no closed form — you walk.
The learning rate \(\eta\) decides whether you walk down or bounce out.
Where do the gradients come from?
\[ L = \ell\big(\,\boldsymbol\beta_K + \boldsymbol\Omega_K\, f(\boldsymbol\beta_{K-1} + \boldsymbol\Omega_{K-1}\, f(\cdots f(\boldsymbol\beta_0 + \boldsymbol\Omega_0 \mathbf{x})\cdots)),\ y\big) \]
- By hand, for millions of parameters — hopeless.
- Numerically, wiggling each parameter — one forward pass per parameter.
- We want all the partials for about the cost of one forward pass.
Backpropagation = the chain rule, organised on a computation graph. Record every operation as a node; store data (forward) and grad (backward) on each.
\(d = a\cdot b,\ e = d + c,\ L = e\cdot f\) — after the forward pass, every grad is still 0.
The chain rule, node by node
\[ \frac{\partial L}{\partial c} = \frac{\partial L}{\partial e}\cdot\frac{\partial e}{\partial c} \qquad\text{— gradient from above} \times \text{local derivative, both at the forward-pass values.} \]
L.grad = 1. Then \(L = e\cdot f\): \(\ \partial L/\partial e = f = 2,\ \ \partial L/\partial f = e = -4\) — a product hands each operand the other’s value.- \(e = d + c\): local derivative 1 — a sum just routes the gradient:
d.grad = c.grad = 2. - \(d = a\cdot b\):
b.grad = a.data * d.grad = 8,a.grad = b.data * d.grad = -6.
Check: wiggle \(b\) by 1 → \(L\) changes by 8. Every step was local — that is why it costs about one forward pass, however many parameters.
The recipe on a real neuron
\(o = \mathrm{ReLU}(x_1w_1 + x_2w_2 + b)\), with \(x_1 = 2,\ x_2 = 0\).
o.grad = 1- ReLU: input was \(> 0\) →
n.grad = 1 - two
+nodes route it →x1w1.grad = x2w2.grad = b.grad = 1 *nodes:w1.grad = x1 · 1 = 2,w2.grad = x2 · 1 = 0
Then one step for each parameter leaf: \(w \leftarrow w - \eta\,\partial L/\partial w\).
The recipe, once more: forward pass; L.grad = 1; walk back multiplying by local derivatives — + routes, * swaps, ReLU gates. That is what you will implement, check against finite differences, and be asked to do on paper.
Why stochastic gradient descent
Backprop gives \(\nabla_\mathbf{w}\ell_i\) for one sample. How many before a step?
| per step | gradient | noise | |
|---|---|---|---|
| full batch | all \(N\) | exact | none |
| SGD (\(B=1\)) | 1 | very noisy | lots |
| mini-batch | \(B \approx 32\)–\(256\) | good enough | some |
Cost is the obvious reason. The subtle one: on a non-convex surface the noise is a feature — it can shake you out of a poor local minimum. Mini-batch is the compromise everyone uses.
(illustration, not a real training curve)
Activity: backprop by hand, then by machine — and what the gradients tell you
Goal: train a single neuron with a hand-derived gradient; build a minimal Value autograd class (you write backward for * and ReLU) and check it against finite differences; use it to train a two-layer MLP on your section’s toy dataset. Then the stretch: send gradients back through a deeper net with sigmoid / tanh / ReLU and watch which ones vanish, and see with your own eyes why all-zeros and too-large initialisations fail.
- Work in groups of 2–3. Open your section’s notebook.
- Parts marked (autograded) are submitted to Gradescope; the rest is participation.
- Staff will circulate — be ready to explain any part of your work.
- Dependencies:
numpy,matplotlib,scikit-learn(datasets only). No PyTorch.
Section A1 
Section B1
The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.