Recap: Decision Trees and Random Forests

DS701 Session 9 — Mon Oct 5, 2026

Today’s plan

  • 5 min — knowledge-check review
  • 20 min — highlights and Q&A (answer or pass — answering always earns credit)
  • 60 min — in-class activity: growing and breaking decision trees (small groups)
  • wrap-up and cold-call check-ins

Knowledge-Check Review

KC 1: What does a decision tree actually learn?

A tree is fit to a table of loans. What is the learned object, and how is a new loan classified?

KC 2: Why not use training error as the splitting criterion?

We split on Home Owner, then Marital Status, then Income, and error dropped to 0%. Why do we bother with Gini or entropy instead of just minimizing error?

KC 3: Zero training error — is the tree good?

The full Titanic tree drove training error to ~0 but did worse on the validation set than the one-rule model predict survived = (Sex == female). What happened?

Highlights

1. Growing a tree: greedy recursive partitioning

  • Start with all \(N\) records in the root; predict the majority class.
  • For every attribute, for every candidate test condition, score the split.
  • Take the single best-scoring split, create the children, and recurse independently on each child.
  • Stop when a node is pure, or a stopping criterion fires.

The search is greedy and local — the best split now is never revisited, so the tree is not the globally optimal tree. Finding that one is NP-hard.

2. Where can a split go? Depends on the attribute type

  • Binary — one obvious split.
  • Nominal — multi-way (a branch per value) or binary (a partition of the values into two groups); \(\mathcal{O}(2^k)\) groupings to search.
  • Ordinal — binary splits are fine if the ordering is preserved: {Small, Medium} | {Large, X-Large} is legal, {Small, Large} | {Medium, X-Large} is not.
  • Continuous — threshold \(x \le \tau\): sort, then sweep the midpoints between consecutive values, \(\mathcal{O}(n)\) after the sort.

Pick the threshold with the lowest weighted impurity.

3. Impurity measures and how a split is scored

\[ \begin{aligned} \textnormal{Gini} &= 1 - \sum_{i=0}^{c-1} p_i(t)^2 \\ \textnormal{Entropy} &= -\sum_{i=0}^{c-1} p_i(t) \log_2 p_i(t) \\ \textnormal{Error} &= 1 - \max_i p_i(t) \end{aligned} \]

\(p_i(t)\) = relative frequency of class \(i\) at node \(t\); \(0 \log_2 0 = 0\).

Score a split by the weighted average over its children and take the largest gain:

\[ I(\textnormal{ch}) = \sum_{j=1}^{k} \frac{N(v_j)}{N} I(v_j), \quad \Delta = I(\textnormal{parent}) - I(\textnormal{ch}) \]

  • All three are 0 for a pure node, maximal for a uniform mix.
  • Entropy \(\ge\) Gini \(\ge\) Error, always.
  • Gini and entropy almost always pick the same split; Gini is cheaper (no logs) and is scikit-learn’s default.
  • Largest gain = smallest weighted child impurity, since the parent term is fixed.

4. The pathology: raw gain rewards many-way splits

  • Splitting the loans table on the customer ID gives 10 pure leaves and collective entropy exactly 0 — the maximum possible gain — and is completely worthless. It generalizes to nothing.
  • Fix 1: binary trees only (CART) — sidesteps varying fan-out entirely.
  • Fix 2: gain ratio (C4.5) — divide the gain by the split information \(-\sum_i \frac{N(v_i)}{N}\log_2\frac{N(v_i)}{N}\), penalizing wide splits.
  • On the loans data, Marital Status has slightly higher gain (0.195 vs 0.192) but much larger split info (1.486 vs 0.881), so Home Owner wins on gain ratio (0.218 vs 0.131).
  • Remember this slide when you read feature importances later.

5. Overfitting: bias, variance, and how to stop it

  • Bias — error from an over-simple model; a depth-1 stump underfits.
  • Variance — error from an over-complex model; a fully grown tree fits the training set exactly and its structure changes wildly under resampling.
  • A fully grown tree is the canonical low-bias / high-variance learner — exactly the raw material ensembles want.

  • Pre-prune (stopping criteria): max_depth, min_samples_split / min_samples_leaf, minimum impurity decrease.
  • Post-prune: grow fully, then collapse subtrees that don’t pay for themselves on held-out data (ccp_alpha).
  • On Titanic, both min_samples_split=20 and max_depth=3 beat the unrestricted tree on validation error — less tree, better model.

6. Bagging: average away the variance

  • Draw \(B\) bootstrap samples (sample \(N\) records with replacement).
  • Fit a fully grown, unpruned tree on each — keep bias low on purpose.
  • Predict by majority vote across the \(B\) trees.
  • Averaging \(B\) noisy, roughly independent predictors cuts the variance term without inflating bias.
  • Free bonus: each tree omits ~37% of the data, so the out-of-bag records give a validation estimate with no separate split.

Catch: bagged trees are correlated. If one feature dominates, every tree splits on it first and the trees look nearly identical — so averaging buys much less than it should.

7. Random forests = bagging + feature subsampling

  • Everything bagging does, plus: at each node, consider only a random subset of \(m\) features (max_features, commonly \(m \approx \sqrt{p}\)).
  • This forces different trees to use different features, decorrelating them, which is what makes the average actually pay off.
  • Each base tree gets slightly worse; the ensemble gets substantially better.
  • The lecture’s forest never names this knob — in today’s activity you sweep max_features from 1 to all 30 and measure the trees decorrelating (their pairwise prediction correlation drops) while the OOB score barely moves.
  • Almost no tuning needed: more trees never hurts accuracy, it just costs time.

On Titanic, a 100-tree RandomForestClassifier beat every single tree we fit. A forest is a strong, low-effort baseline for tabular data — no feature scaling, mixed types welcome, automatic feature selection. Try one before anything fancier.

8. Feature importance — read it with suspicion

Trees hand you an importance score for free. Four ways it will mislead you:

  • Cardinality bias — impurity-based importance inflates high-cardinality and continuous features (the customer-ID effect, again).
  • Correlated features split the credit — two near-duplicate strong features each look half as important as either would alone.
  • Computed on training data — a feature can score high purely by memorizing noise. Prefer permutation importance on a held-out set.
  • It is not causal, and it is not signed — importance says a feature was used, not which direction it pushes or why.

In-Class Activity

Activity: growing and breaking decision trees

Goal: compute impurity by hand, watch a tree overfit as depth grows, fit a random forest and sweep max_features, then deliberately break the feature importances — noise, duplicates, cardinality — and compare against permutation importance.

  • Work in groups of 2–3. Open the notebook for your section — the data split differs, so the other section’s numbers are wrong for you.
  • Parts 1–4 are autograded and submitted to Gradescope; part 5 is open-ended and graded for participation.
  • Staff will circulate — be ready to explain any part of your work.

Section A1:

Section B1:

The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.

Going deeper

Full lecture notes: Decision Trees and Random Forests

Reference: Tan, Steinbach, Karpatne & Kumar, Introduction to Data Mining, Ch. 3–4.