Recap: Neural Networks II — Making Training Work

DS701 Session 20 — Wed Nov 11, 2026

Today’s plan

  • 5 min — knowledge-check review
  • 20 min — highlights and Q&A (answer or pass — answering always earns credit)
  • 60 min — in-class activity (small groups)
  • wrap-up and cold-call check-ins

Knowledge-Check Review

KC 1: The knobs

The lecture maps the ideas from Neural Networks I onto MLPClassifier arguments. Name the scikit-learn argument that sets each of the following, and say in one phrase what a larger value does: (a) the number of layers and neurons per layer, (b) the learning rate, (c) the mini-batch size, (d) the maximum number of epochs, (e) the strength of L2 regularization. Which solver does the lecture recommend as the default?

KC 2: What the loss curve does — and does not — tell you

After training the MNIST model, the lecture plots mlp.loss_curve_ and remarks that “the loss decreases smoothly — our model is learning.” What exactly is being plotted (which data, which quantity), and why does a smoothly decreasing curve of that kind not by itself guarantee good performance on the test set? Which two MLPRegressor arguments in the California-housing example address that concern, and how?

KC 3: “Because neural networks”

A colleague has 8,000 rows of tabular customer data with 25 numeric features and wants to build the classifier in PyTorch “because neural networks.” Using the lecture’s scikit-learn-vs-PyTorch/TensorFlow criteria, what would you recommend and why? Name two things about the problem that would change your recommendation, and one preprocessing step you would insist on either way.

Highlights

Every idea from last time is now an argument

Concept (NN I) scikit-learn
layers and neurons hidden_layer_sizes
activation \(f\) activation
learning rate \(\eta\) learning_rate_init
mini-batch size batch_size
optimizer solver — adam / sgd / lbfgs
epochs / stopping max_iter, early_stopping, n_iter_no_change
regularization alpha (L2)
  • Start simple: one hidden layer, sizes between the input and output dimension; add depth only when the data can pay for it.
  • Input and output sizes come from the data — you never set them.
  • max_iter is a ceiling, not a target: adam stops when the training loss stalls for n_iter_no_change epochs.
  • adam forgives a mediocre learning rate; sgd does not.

Standardize inputs. Always.

Gradient descent takes one step size for every direction. If one feature spans 0–4000 and another 0–0.2, that feature dominates every weighted sum and every gradient, the loss surface is a long narrow valley, and no single learning rate suits both.

  • MNIST: pixels / 255 → \([0, 1]\).
  • Housing: StandardScaler → zero mean, unit variance per feature.
  • Fit the scaler on the training split only; transform the test split with it. Fitting on everything leaks test statistics into training.
  • Best done inside a Pipeline([('scaler', StandardScaler()), ('mlp', MLPClassifier(...))]) so cross-validation and grid search re-fit it correctly per fold.

When it barely matters: features already on one common scale (pixels).

When it rescues you: real measurements with wildly different units — the difference between a model that works and one that flails.

Today’s activity has one dataset of each kind.

Read the loss curve before you read the accuracy

One point = training loss after one epoch. Healthy: fast initial drop, smooth flattening. That is what “the model is learning” looks like.

Too low: barely moves. Too high: starts absurdly high, bounces, stalls, and stops “because it stopped improving”. The right-hand model happened to score well — would you ship it?

Overfitting: what the loss curve cannot show you

  • A (512, 512) network has ~300K weights; 50 training digits cannot pin them down. Training loss → 0 — the samples are memorized.
  • Loss on data the model has never seen bottoms out around epoch 20 and drifts back up. Everything after the dashed line is fitting noise.
  • You only see this if you hold data out and measure it — loss_curve_ alone would have said “learning nicely”.

The three repairs — and what each one changes

repair scikit-learn what it changes
L2 regularization alpha=0.01 (default 1e-4) the objective: penalizes large weights
early stopping early_stopping=True, validation_fraction=0.1, n_iter_no_change=10 the training loop: stop on validation evidence, not training loss
cross-validation cross_val_score(mlp, X, y, cv=5) the evaluation: an honest estimate, so you notice — a measurement, not a repair

Not on the lecture’s list, but the lever that usually matters most: more data. Regularizers trade variance for bias; they cannot create information.

Early stopping is not free either — it takes validation_fraction of your training data away. Ask what that costs when the training set is tiny.

In the activity you will overfit on purpose, then pull each lever and measure it — including the one the lecture only gestured at.

Tuning honestly

  • GridSearchCV(mlp, param_grid, cv=3) — every configuration × every fold is a full training run: \(3 \text{ architectures} \times 2\ \alpha \times 3 \text{ folds} = 18\) fits before the refit. Budget accordingly (the lecture used 1,500 samples for the demo).
  • Tune on the folds, never on the test set; the test set is spent once, at the end.
  • Read cv_results_ — the spread (std_test_score) tells you whether the “best” setting is really better or just lucky.
  • Solvers on the same budget (2,000 digits, 50 epochs): adam 0.90, lbfgs 0.88, sgd 0.83 — the one that lost had simply not converged yet.

When the toy tool is enough

scikit-learn MLPClassifier / MLPRegressor

  • small–medium tabular data (< ~100K rows)
  • standard feed-forward architectures
  • CPU training is fine
  • lives inside Pipeline, cross_val_score, GridSearchCV
  • fastest way to a tuned baseline — and to a fair comparison with logistic regression, trees, kNN

PyTorch / TensorFlow

  • large data, GPUs, data that does not fit in memory
  • CNNs, RNNs, transformers, custom losses
  • production and research

Depth on those architectures lives in the optional modules M4 (CNNs) and M3 (RNNs), and in a dedicated course — DS542, Deep Learning for Data Science. In DS701 the point is to train a network correctly and know when you have.

In-Class Activity

Activity: train it properly, then break it and fix it

Goal: run the lecture’s recipe on a real dataset — standardize, fit, read the loss curve, evaluate, compare solvers and learning rates on a fixed budget — then overfit a huge network on purpose, watch training and validation loss diverge, repair it three ways (alpha, early_stopping, more data) and measure which lever actually helped, and finally diagnose three mystery loss curves.

  • Work in groups of 2–3. Open your section’s notebook.
  • Parts marked (autograded) are submitted to Gradescope; the rest is participation.
  • Staff will circulate — be ready to explain any part of your work.
  • Dependencies: numpy, matplotlib, scikit-learn. Every fit takes well under a second.

Section A1

Section B1

The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.

Going deeper

Full lecture notes: Neural Networks II: Making Training Work

Back to top