Recap: Generalization — Why Training Error Lies
DS701 Session 8 — Wed Sep 30, 2026
Training error lies
\(N = 10\) points, \(y = \sin(2\pi x) + \varepsilon\). Least squares finds the \(\mathbf{w}^*\) that minimises training error \(E(\mathbf{w}) = \sum_n [f(x_n, \mathbf{w}) - y_n]^2\). Training error: \(1.91 \to 1.45 \to 0.64 \to\) 0.00 at \(k = 9\) (10 coefficients, 10 points — it interpolates). Which one would you bet on for the next point?
The money plot: train vs test
- Score every fit on fresh points from the same generator.
- Training error falls monotonically — a bigger family can always reproduce a smaller one — and reaches zero.
- Test error falls, bottoms out at \(k = 3\), then rises.
- Training ↓ while test ↑ is overfitting: fitting the training data “too well” — noise included.
Why it happens: too many parameters for the data; a model more complex than the phenomenon. What helps: more data, less complexity (hyperparameters), regularization.
Two ways to be wrong
Bias — the model is too simple to capture the pattern; it underfits (\(k = 0, 1\)). Refit it on new data and it is wrong the same way every time.
Variance — the model is so flexible it fits the noise; it overfits (\(k = 9\)). Refit it on a different sample from the same distribution and you get a very different answer.
Both are defined over repeated draws of the training set — which is exactly what you will do in Part 3 today.
The trade-off is the U-shape
- More flexibility: bias ↓, variance ↑.
- Left of the minimum, bias dominates; right of it, variance.
- The test-error minimum (\(k = 3\)) is the sweet spot — model selection is the search for it.
- \(k\) is a hyperparameter: fixed before the parameters \(\mathbf{w}\) are learned; it controls complexity.
Same trade-off, different knob, every week from now on: tree depth, \(k\) in \(k\)-NN, the regularization strength, early stopping in neural nets.
Choosing \(k\) honestly: hold out, then grid search
- We may not choose \(k\) by training error, and we may not choose it by the test set either.
- So hold out data:
train_test_splitputs a random half aside. - Grid search: for each candidate \(k\), split randomly 5×, fit on train, score on held-out, average — the error bars are \(\sigma/\sqrt{5}\).
- Read off the minimum: \(k = 3\).
Random, not first-half/second-half — a systematic split breaks the i.i.d. contract between the two halves.
\(K\)-fold cross-validation
- Partition once into \(K\) folds; each fold is the validation set exactly once; average the \(K\) scores.
- Every point is tested once and trained on \(K-1\) times — nothing wasted, no lucky split.
- Larger \(K\): more training data per fit, more fits to run; \(K = n\) is leave-one-out.
The full protocol: lock away a test set → choose the hyperparameter by CV on the rest → refit on all training data → score the test set once. That last number is the only one you may report.
Activity: train/test discipline, then learning curves
Goal: on your section’s synthetic regression, reproduce the train-vs-test curve for degrees 0–12 and pick the degree; pick it again with 5-fold CV without touching the test set; measure bias and variance for \(k = 1\) vs \(k = 9\) by refitting on many samples; then draw learning curves for an under-fit and an over-fit model and diagnose a mystery model from its curve.
- Work in groups of 2–3. Open your section’s notebook.
- Parts marked (autograded) are submitted to Gradescope; the rest is participation.
- Staff will circulate — be ready to explain any part of your work.
- Dependencies:
numpy,matplotlib,scikit-learn.
Section A1 
Section B1
The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.