import warnings
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
from sklearn.exceptions import ConvergenceWarning
from slideUtilities import OKABE_ITO
%matplotlib inlineThis module picks up where Neural Networks II: Making Training Work leaves off. That lecture covers MLPClassifier/MLPRegressor, scaling, loss curves, grid search, overfitting and solver choice; it is not repeated here.
Here we cover four things the lecture does not:
solver='sgd'warm_start: continue training a fitted modelpartial_fit: train one batch (or one epoch) at a timeAll examples use sklearn’s 8×8 digits (load_digits: 1,797 images, 64 features), standardized, so every fit takes about a second.
digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(
digits.data, digits.target, test_size=0.3,
stratify=digits.target, random_state=701)
scaler = StandardScaler().fit(X_train)
X_train_s = scaler.transform(X_train)
X_test_s = scaler.transform(X_test)
classes = np.unique(digits.target)
print(f"train {X_train_s.shape}, test {X_test_s.shape}")train (1257, 64), test (540, 64)learning_rate sets how sgd’s step size changes during training (adam ignores it):
'constant': always learning_rate_init'invscaling': decays as learning_rate_init / t**power_t'adaptive': kept constant while the training loss improves; divided by 5 each time it fails to improve for n_iter_no_change epochsschedules = {}
for schedule in ['constant', 'invscaling', 'adaptive']:
with warnings.catch_warnings():
warnings.simplefilter('ignore', ConvergenceWarning) # fixed 100-epoch budget
m = MLPClassifier(hidden_layer_sizes=(64,), solver='sgd',
learning_rate=schedule, learning_rate_init=0.1,
max_iter=100, random_state=701)
m.fit(X_train_s, y_train)
schedules[schedule] = m
print(f"{schedule:10s}: epochs {m.n_iter_:3d} final loss {m.loss_:.4f} "
f"test acc {m.score(X_test_s, y_test):.3f}")constant : epochs 57 final loss 0.0028 test acc 0.978
invscaling: epochs 100 final loss 0.1951 test acc 0.920
adaptive : epochs 100 final loss 0.0026 test acc 0.980fig, ax = plt.subplots(figsize=(10, 4))
styles = {'constant': (OKABE_ITO[5], '-', 4), 'invscaling': (OKABE_ITO[0], '-', 2),
'adaptive': (OKABE_ITO[3], '--', 2)}
for name, m in schedules.items():
c, ls, lw = styles[name]
ax.plot(np.arange(1, m.n_iter_ + 1), m.loss_curve_, color=c, ls=ls, lw=lw, label=name)
ax.set_yscale('log')
ax.set_xlabel('epoch')
ax.set_ylabel('training loss (log scale)')
ax.set_title("sgd on sklearn's 8×8 digits: three learning-rate schedules")
ax.legend()
ax.grid(alpha=0.3, which='both')
plt.show()
adaptive follows constant exactly until the training loss stops improving (epoch 57). There constant stops “because it stopped improving”, while adaptive divides the step by 5 and keeps going. invscaling shrinks the step from the first epoch, so from the same starting rate it slows to a crawl and ends with a loss about 75 times higher.
With warm_start=True, calling fit again continues from the current weights instead of re-initializing:
mlp_warm = MLPClassifier(hidden_layer_sizes=(64,), max_iter=5,
warm_start=True, random_state=701)
for round_ in range(1, 5):
with warnings.catch_warnings():
warnings.simplefilter('ignore', ConvergenceWarning) # 5 epochs per call, on purpose
mlp_warm.fit(X_train_s, y_train)
print(f"after {5 * round_:2d} epochs: test acc {mlp_warm.score(X_test_s, y_test):.3f} "
f"loss {mlp_warm.loss_:.4f}")after 5 epochs: test acc 0.674 loss 1.3157
after 10 epochs: test acc 0.874 loss 0.5857
after 15 epochs: test acc 0.920 loss 0.2907
after 20 epochs: test acc 0.948 loss 0.1633Useful for growing the epoch budget without starting over, or for sweeping a hyperparameter (e.g. alpha) while reusing the previous solution as the starting point.
partial_fit runs one pass over whatever data you hand it. Feed it chunks when the data does not fit in memory or arrives over time:
mlp_online = MLPClassifier(hidden_layer_sizes=(64,), random_state=701)
chunks = np.array_split(np.arange(len(y_train)), 5)
for epoch in range(3):
for idx in chunks:
# the first call must list every class, since one chunk may not contain them all
mlp_online.partial_fit(X_train_s[idx], y_train[idx], classes=classes)
print(f"after epoch {epoch + 1} ({len(chunks)} chunks): "
f"test acc {mlp_online.score(X_test_s, y_test):.3f}")after epoch 1 (5 chunks): test acc 0.280
after epoch 2 (5 chunks): test acc 0.472
after epoch 3 (5 chunks): test acc 0.583Because each call is exactly one pass, partial_fit is also how you record validation loss after every epoch – loss_curve_ holds training loss only. The lecture’s overfitting figure is built this way.
The same (128, 64) ReLU network for 64 input features and 10 classes:
from sklearn.neural_network import MLPClassifier
# Define and train
mlp = MLPClassifier(
hidden_layer_sizes=(128, 64),
activation='relu',
max_iter=100
)
mlp.fit(X_train, y_train)
# Predict
predictions = mlp.predict(X_test)import torch
import torch.nn as nn
class MLP(nn.Module):
def __init__(self):
super().__init__()
self.layers = nn.Sequential(
nn.Linear(64, 128),
nn.ReLU(),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 10)
)
def forward(self, x):
return self.layers(x)
# Training loop required...The scikit-learn fit hides the whole training loop. In PyTorch you write it yourself:
model = MLP()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
for epoch in range(100):
for xb, yb in train_loader: # mini-batches
optimizer.zero_grad()
loss = loss_fn(model(xb), yb) # forward pass
loss.backward() # backpropagation
optimizer.step() # gradient stepThat is more code, and also the point: every line is a place to add dropout, batch normalization, a custom loss, a learning-rate warm-up, or a GPU (model.to('cuda')) – none of which MLPClassifier offers. See Module M4: CNNs and Module M3: RNNs for networks built this way.