Recap: Logistic Regression and Regularization

DS701 Session 13 — Mon Oct 19, 2026

Today’s plan

  • 5 min — knowledge-check review
  • 20 min — highlights and Q&A (answer or pass — answering always earns credit)
  • 60 min — in-class activity (small groups)
  • wrap-up and cold-call check-ins

Knowledge-Check Review

KC 1: The model, in one line

Write down the logistic regression model: what linear function does it fit, what quantity is that linear function equal to, and how do you get from it to a probability \(p(x)\)? Then say how the coefficients are chosen (which principle) and why, unlike linear regression, they cannot be found in closed form.

KC 2: Why the coefficients blow up, and two ways to stop them

When two columns of the design matrix are nearly linearly dependent (multicollinearity), the least-squares coefficients tend to become very large in magnitude. Explain in a sentence or two why that happens, how ridge regression’s penalty \(c\Vert\boldsymbol{\beta}\Vert_2^2\) addresses it, and what LASSO’s \(c\Vert\boldsymbol{\beta}\Vert_1\) penalty does that ridge cannot.

KC 3: 0.5 is not a law of nature

To turn logistic regression into a classifier the lecture used the rule “say yes if \(p(x) > t\)” and found on the admissions data that F1 peaked near \(t \approx 0.3\), not \(0.5\). Explain what lowering the threshold does to precision and to recall, why \(0.5\) is not automatically the right choice, and give one real setting where you would deliberately pick a low threshold and one where you would pick a high one.

Highlights

Logistic regression = a linear model + a sigmoid

Last week: a continuous \(y\) as a linear function of \(x\). This week: a 0/1 \(y\), and we want a probability.

  • A line takes values in \((-\infty,\infty)\); a probability lives in \([0,1]\). So fit the line to something unbounded: the log-odds \(\log\frac{p}{1-p}\) (the logit).
  • Invert: \(p(x) = \sigma(\beta_0 + \beta_1 x)\), the sigmoid.
  • \(\beta_1\) sets the steepness (and direction); \(\beta_0\) shifts the curve; \(p = 0.5\) at \(x = -\beta_0/\beta_1\); \(\beta_0\) alone is the base rate.
  • Read a coefficient exactly as in linear regression — but on the log-odds scale.

Fit by maximum likelihood — same principle as Lecture 05

One observation: \(\;\text{P}(y_i \mid x_i) = p_i^{\,y_i}\,(1-p_i)^{1-y_i}, \qquad p_i = \sigma(\boldsymbol{\beta}^\top\mathbf{x}_i).\)

  • Data are independent, so the likelihood is the product; the log-likelihood is the sum \(\;\ell(\boldsymbol{\beta}) = \sum_i \big[y_i\log p_i + (1-y_i)\log(1-p_i)\big].\)
  • In Lecture 05 you maximized likelihoods for a Poisson and a Gaussian and got a formula. Linear regression is the lucky case: Gaussian noise ⇒ MLE = least squares ⇒ closed form.
  • Here there is no closed form. Gradient ascent: \(\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} + \alpha\,\nabla\ell(\boldsymbol{\beta}^{(t)})\), with the tidy gradient \(\nabla\ell = X^\top(\mathbf{y} - \mathbf{p})\) — residual times feature, just like the normal equations, with a sigmoid inside.
  • statsmodels.Logit / sklearn.LogisticRegression do the climb for you and give a coefficient per feature (plus the column of ones for \(\beta_0\)).

Today’s activity, Part 1: you write \(\sigma\), \(\ell\), and one gradient step by hand — then check that sklearn lands in the same place.

The boundary is still a line

  • “Assign to class 1 if \(p(x) > 0.5\)” \(\iff\) log-odds \(> 0\) \(\iff\) \(\boldsymbol{\beta}^\top\mathbf{x} > 0\).
  • The sigmoid is monotone, so the decision depends only on the sign of the linear score — the boundary is a hyperplane. The nonlinearity lives in the confidence, not the boundary.
  • Any other threshold \(t\) just slides the plane parallel to itself, to \(\boldsymbol{\beta}^\top\mathbf{x} = \log\frac{t}{1-t}\).
  • Once you classify, you owe an evaluation. Four outcomes: TP, FP, FN, TN, and two rates that pull in opposite directions: \[\text{Precision} = \frac{TP}{TP+FP}, \qquad \text{Recall} = \frac{TP}{TP+FN}, \qquad F_1 = \frac{2PR}{P+R}.\]

Evaluation beyond accuracy: the threshold is a decision

Admissions: F1 peaks near \(t \approx 0.3\) — the model rarely says “admit” at 0.5 because admits are the minority.

ROC: TPR vs FPR, one point per threshold; diagonal = random; AUC summarizes ranking quality without choosing a threshold (0.69 here).

AUC tells you how good the scores are. It does not tell you where to cut — that depends on the costs of FN vs FP and on the base rate. Today’s stretch: a dataset where 95% accuracy is worthless.

Regularization: a penalty that trades bias for variance

Lecture 07: flexible fits chase noise → high variance. Regularization is the knob:

\[\min_{\boldsymbol{\beta}} \Vert X\boldsymbol{\beta} - \mathbf{y}\Vert_2^2 + c\,R(\boldsymbol{\beta})\]

  • Ridge: \(R = \Vert\boldsymbol{\beta}\Vert_2^2\), closed form \((X^\top X + cI)^{-1}X^\top\mathbf{y}\) — always exists, always unique, even with multicollinearity.
  • LASSO: \(R = \Vert\boldsymbol{\beta}\Vert_1\).
  • \(c = 0\): least squares. \(c \to \infty\): \(\boldsymbol{\beta} \to 0\). In between: fit vs. size — bias bought, variance sold.
  • Multicollinearity is the classic reason: nearly dependent columns ⇒ huge \(\kappa(X) = \sigma_{\max}/\sigma_{\min}\) ⇒ wildly unstable \(\hat{\boldsymbol{\beta}}\) (Longley: \(\kappa \approx 10^8\)).

Longley, ridge: held-out \(R^2\) rises with \(c\), then collapses — the sweet spot is found by cross-validation.

\(\ell_2\) shrinks, \(\ell_1\) selects

Same contours, different penalty balls. The \(\ell_1\) diamond has corners on the axes — the contours usually hit a corner first, and a corner means a coefficient is exactly zero.

Longley, LASSO: as \(c\) grows, ARMED, UNEMP, POP go to zero; GNP and GNPDEFL remain — the near-dependent variables are the ones dropped.

Ridge never zeroes anything (until \(c \to \infty\), all at once). LASSO gives you model selection for free — instead of searching \(2^n - 1\) subsets — at similar prediction error, with a much more interpretable answer.

Two rules you must not skip

  1. Standardize first. The penalty is on the size of \(\boldsymbol{\beta}\). A useful feature in tiny units needs a huge coefficient and gets punished for it; a noise feature in huge units is nearly free. Plain least squares did not care about units — the coefficients absorbed them; a penalized fit does. Standardize \(X\), centre \(\mathbf{y}\), don’t penalize the intercept.
  2. Choose \(c\) on held-out data. Training error is always lowest at \(c = 0\) — the penalty can only make the training fit worse. Cross-validate. (And know what CV optimizes: prediction error, not sparsity.)

Today’s activity, Part 2: coefficient paths for both penalties on data where you know which features matter, CV to pick \(\alpha\), and one fit where you “forget” to standardize and watch the lasso throw away a real feature.

Where this goes next

  • \(\kappa(X) = \sigma_{\max}/\sigma_{\min}\) used the singular values of \(X\) without ever saying what they are. Next lecture — the SVD — is the tool underneath: it explains why \(X^\top X\) squares the conditioning problem, why ridge is “shrink each singular direction by \(\sigma_j^2/(\sigma_j^2 + c)\)”, and how the same decomposition gives low-rank compression and, later, PCA.
  • Regularization itself will come back unchanged for logistic regression (sklearn’s C is \(1/c\)) and for neural networks (weight decay).
  • And the threshold question comes back every time a model outputs a probability.

In-Class Activity

Activity: from the likelihood to a threshold you would ship

Goal: build logistic regression from its likelihood and check it against sklearn; trace ridge and lasso paths on data with correlated and irrelevant features, pick \(\alpha\) by CV, and see what forgetting to standardize costs; then take a 90/10 (A1) or 95/5 (B1) imbalanced problem, watch accuracy lie, sweep the threshold, plot ROC and precision–recall, choose a threshold for a stated cost, and try class_weight="balanced".

  • Work in groups of 2–3. Open your section’s notebook.
  • Parts marked (autograded) are submitted to Gradescope; the rest is participation.
  • Staff will circulate — be ready to explain any part of your work.
  • Dependencies: numpy, pandas, matplotlib, scikit-learn.

Section A1

Section B1

The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.

Going deeper

Full lecture notes: Logistic Regression and Regularization