| Home Owner | Marital Status | Annual Income | Defaulted Borrower | |
|---|---|---|---|---|
| ID | ||||
| 1 | Yes | Single | 125000 | No |
| 2 | No | Married | 100000 | No |
| 3 | No | Single | 70000 | No |
| 4 | Yes | Married | 120000 | No |
| 5 | No | Divorced | 95000 | Yes |
| 6 | No | Married | 60000 | No |
| 7 | Yes | Divorced | 220000 | No |
| 8 | No | Single | 85000 | Yes |
| 9 | No | Married | 75000 | No |
| 10 | No | Single | 90000 | Yes |
Decision Trees and Random Forests
Loan Default Example
We’ll use an example from (Tan et al. 2018).
You are a loan officer at Terrier Savings and Loan.
You have a dataset on loans that you have made in the past.
You want to build a model to predict whether a loan will default.
Simple Model
Looking at the table, let’s just start with the simplest model possible and just predict that no one will default.
So the output of our model is just to always predict “No”.
We see a 30% error rate since 3 out of 10 loans defaulted.
Let’s split the data based on the “Home Owner” field. (values = [# No, # Yes]).
We see that the left node (Home Owner == Yes) has a 0% error rate since all the samples are Defaulted == No. We don’t split this node since all the samples are of the same class. We call this node a leaf node and we’ll color it orange.
The right node (Home Owner == No) has a 43% error rate since 3 out of 7 loans defaulted.
Let’s split this node into two nodes based on the Marital Status field.
Let’s split on the “Marital Status” field.
We see that the 3 defaulted loans are all for single or divorced people. Since the node is all one class, we don’t split this node and we call it a leaf node.
Let’s try to split on the “Annual Income” field.
We see that the person with income of 70K doesn’t default, so we split the node into two nodes based on the “Income” field.
We arbitrarily pick a threshold of $75K.
For a Nominal (Categorical) attribute:
- In a Multi-way split we can use as many partitions as there are distinct values of the attribute:
For a Nominal (Categorical) attribute:
In a Binary split we divide the values into two groups.
In this case, we need to find an optimal partitioning of values into groups, which we discuss shortly.
For an Ordinal attribute, we can use a multi-way split with as many partitions as there are distinct values.
Or we can use a binary split as long we preserve the ordering of the values.
Warning
Be careful not to violate the ordering of values such as {Small, Large} and {Medium, X-Large}.
A Continuous attribute can be handled two ways:
Note that finding good partitions for \(k\) nominal attributes can be expensive, \(\mathcal{O}(2^k)\), possibly involving combinatorial searching of groupings.
However for ordinal or continuous attributes, sweeping through a range of \(n\) threshold values can be more efficient if \(n \approx k\). \(\mathcal{O}(n)\) for a sorted list.
We can plot the three impurity indices to get a sense of how they behave for binary classification problems.
They all maintain the same ordering for every relative frequency, i.e., Entropy > Gini > Misclassification error.
Impurity Example
Let’s compute collective impurity on our loans dataset to see which feature to split on.
Tip
Try calculating the collective Entropy for (a) and (b) and see if you get the same values.
Important
The collective entropy for (c) is 0. Why would we not want to use this node?
Identifying the Best Attribute Test Condition
Here’s an example of how to identify the best attribute test condition using the Collective Impurity of the Gini Impurity index.
Code
def gini_impurity(yes_count, no_count):
"""Calculate Gini impurity for a node"""
total = yes_count + no_count
if total == 0:
return 0
p_yes = yes_count / total
p_no = no_count / total
gini = 1 - (p_yes**2 + p_no**2)
return gini
def weighted_gini(n1_yes, n1_no, n2_yes, n2_no):
"""Calculate weighted average Gini impurity after split"""
n1_total = n1_yes + n1_no
n2_total = n2_yes + n2_no
total = n1_total + n2_total
gini_n1 = gini_impurity(n1_yes, n1_no)
gini_n2 = gini_impurity(n2_yes, n2_no)
weighted_gini = (n1_total/total) * gini_n1 + (n2_total/total) * gini_n2
return weighted_gini
# Parent node
parent_yes = 3
parent_no = 7
parent_gini = gini_impurity(parent_yes, parent_no)
print("=" * 60)
print("PARENT NODE")
print("=" * 60)
print(f"Yes: {parent_yes}, No: {parent_no}")
print(f"Gini Impurity: {parent_gini:.3f}")
print()
# Define all four splits
splits = {
"Home Owner": {
"N1": {"yes": 0, "no": 3}, # Yes (owns home)
"N2": {"yes": 3, "no": 4} # No (doesn't own home)
},
"Marital Status (Split 1: Single vs Married+Divorced)": {
"N1": {"yes": 2, "no": 3}, # Single
"N2": {"yes": 1, "no": 4} # Married, Divorced
},
"Marital Status (Split 2: Single+Married vs Divorced)": {
"N1": {"yes": 2, "no": 6}, # Single, Married
"N2": {"yes": 1, "no": 1} # Divorced
},
"Marital Status (Split 3: Single+Divorced vs Married)": {
"N1": {"yes": 3, "no": 4}, # Single, Divorced
"N2": {"yes": 0, "no": 3} # Married
}
}
# Calculate for each split
results = []
for split_name, nodes in splits.items():
n1_yes = nodes["N1"]["yes"]
n1_no = nodes["N1"]["no"]
n2_yes = nodes["N2"]["yes"]
n2_no = nodes["N2"]["no"]
gini_n1 = gini_impurity(n1_yes, n1_no)
gini_n2 = gini_impurity(n2_yes, n2_no)
weighted_gini_value = weighted_gini(n1_yes, n1_no, n2_yes, n2_no)
gini_gain = parent_gini - weighted_gini_value
results.append({
'name': split_name,
'weighted_gini': weighted_gini_value,
'gini_gain': gini_gain,
'gini_n1': gini_n1,
'gini_n2': gini_n2,
'n1_yes': n1_yes,
'n1_no': n1_no,
'n2_yes': n2_yes,
'n2_no': n2_no
})
print("=" * 60)
print(f"SPLIT: {split_name}")
print("=" * 60)
print(f"Node N1: Yes={n1_yes}, No={n1_no}, Total={n1_yes+n1_no}")
print(f" Gini(N1) = {gini_n1:.3f}")
print(f"Node N2: Yes={n2_yes}, No={n2_no}, Total={n2_yes+n2_no}")
print(f" Gini(N2) = {gini_n2:.3f}")
print(f"\nWeighted Gini (after split): {weighted_gini_value:.3f}")
print(f"Gini Gain: {gini_gain:.3f}")
print()
# Summary: Find best split
print("=" * 60)
print("SUMMARY - BEST SPLIT")
print("=" * 60)
results_sorted = sorted(results, key=lambda x: x['gini_gain'], reverse=True)
for i, result in enumerate(results_sorted, 1):
print(f"{i}. {result['name']}")
print(f" Weighted Gini: {result['weighted_gini']:.3f}, Gini Gain: {result['gini_gain']:.3f}")
print()
print(f"Best split: {results_sorted[0]['name']}")
print(f"Maximum Gini Gain: {results_sorted[0]['gini_gain']:.3f}")============================================================
PARENT NODE
============================================================
Yes: 3, No: 7
Gini Impurity: 0.420
============================================================
SPLIT: Home Owner
============================================================
Node N1: Yes=0, No=3, Total=3
Gini(N1) = 0.000
Node N2: Yes=3, No=4, Total=7
Gini(N2) = 0.490
Weighted Gini (after split): 0.343
Gini Gain: 0.077
============================================================
SPLIT: Marital Status (Split 1: Single vs Married+Divorced)
============================================================
Node N1: Yes=2, No=3, Total=5
Gini(N1) = 0.480
Node N2: Yes=1, No=4, Total=5
Gini(N2) = 0.320
Weighted Gini (after split): 0.400
Gini Gain: 0.020
============================================================
SPLIT: Marital Status (Split 2: Single+Married vs Divorced)
============================================================
Node N1: Yes=2, No=6, Total=8
Gini(N1) = 0.375
Node N2: Yes=1, No=1, Total=2
Gini(N2) = 0.500
Weighted Gini (after split): 0.400
Gini Gain: 0.020
============================================================
SPLIT: Marital Status (Split 3: Single+Divorced vs Married)
============================================================
Node N1: Yes=3, No=4, Total=7
Gini(N1) = 0.490
Node N2: Yes=0, No=3, Total=3
Gini(N2) = 0.000
Weighted Gini (after split): 0.343
Gini Gain: 0.077
============================================================
SUMMARY - BEST SPLIT
============================================================
1. Home Owner
Weighted Gini: 0.343, Gini Gain: 0.077
2. Marital Status (Split 3: Single+Divorced vs Married)
Weighted Gini: 0.343, Gini Gain: 0.077
3. Marital Status (Split 1: Single vs Married+Divorced)
Weighted Gini: 0.400, Gini Gain: 0.020
4. Marital Status (Split 2: Single+Married vs Divorced)
Weighted Gini: 0.400, Gini Gain: 0.020
Best split: Home Owner
Maximum Gini Gain: 0.077Splitting Continuous Attributes
For quantitative attributes like Annual Income, we need to find some threshold \(\tau\) that minimizes the impurity index.
The following table illustrates the process.
Procedure:
- Sort all the training instances by Annual Income in increasing order.
- Pick thresholds half way between consecutive values.
- Compute the Gini impurity index for each threshold.
- Select the threshold that minimizes the Gini impurity index.
Code
def gini_impurity(yes_count, no_count):
"""Calculate Gini impurity for a node"""
total = yes_count + no_count
if total == 0:
return 0
p_yes = yes_count / total
p_no = no_count / total
gini = 1 - (p_yes**2 + p_no**2)
return gini
def weighted_gini_split(left_yes, left_no, right_yes, right_no):
"""Calculate weighted Gini impurity for a binary split"""
left_total = left_yes + left_no
right_total = right_yes + right_no
total = left_total + right_total
if total == 0:
return 0
gini_left = gini_impurity(left_yes, left_no)
gini_right = gini_impurity(right_yes, right_no)
weighted_gini = (left_total/total) * gini_left + (right_total/total) * gini_right
return weighted_gini
# Data from the table
sorted_values = [60, 70, 75, 85, 90, 95, 100, 120, 125, 220]
classes = ['No', 'No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'No', 'No', 'No']
# Calculate split positions (midpoints between consecutive values)
split_positions = []
for i in range(len(sorted_values) - 1):
split_pos = (sorted_values[i] + sorted_values[i+1]) / 2
split_positions.append(split_pos)
# Parent node statistics
parent_yes = classes.count('Yes')
parent_no = classes.count('No')
parent_gini = gini_impurity(parent_yes, parent_no)
print("=" * 80)
print("CONTINUOUS ATTRIBUTE: Annual Income")
print("=" * 80)
print(f"Parent Node: Yes={parent_yes}, No={parent_no}")
print(f"Parent Gini Impurity: {parent_gini:.3f}")
print()
print("=" * 80)
print("EVALUATING ALL POSSIBLE SPLITS")
print("=" * 80)
best_split = None
best_gini = float('inf')
best_gain = -float('inf')
results = []
for i, split_pos in enumerate(split_positions):
# Count classes on left (<= split) and right (> split)
left_yes = 0
left_no = 0
right_yes = 0
right_no = 0
for j, value in enumerate(sorted_values):
if value <= split_pos:
if classes[j] == 'Yes':
left_yes += 1
else:
left_no += 1
else:
if classes[j] == 'Yes':
right_yes += 1
else:
right_no += 1
# Calculate weighted Gini
weighted_gini = weighted_gini_split(left_yes, left_no, right_yes, right_no)
gini_gain = parent_gini - weighted_gini
results.append({
'split_position': split_pos,
'left_yes': left_yes,
'left_no': left_no,
'right_yes': right_yes,
'right_no': right_no,
'weighted_gini': weighted_gini,
'gini_gain': gini_gain
})
print(f"Split at {split_pos:6.1f}:")
print(f" Left (<=): Yes={left_yes}, No={left_no}, Total={left_yes+left_no}")
print(f" Right ( >): Yes={right_yes}, No={right_no}, Total={right_yes+right_no}")
print(f" Weighted Gini: {weighted_gini:.3f}")
print(f" Gini Gain: {gini_gain:.3f}")
if weighted_gini < best_gini:
best_gini = weighted_gini
best_split = split_pos
best_gain = gini_gain
print()
print("=" * 80)
print("BEST SPLIT")
print("=" * 80)
print(f"Best Split Position: {best_split}")
print(f"Minimum Weighted Gini: {best_gini:.3f}")
print(f"Maximum Gini Gain: {best_gain:.3f}")
# Show the split details
best_result = [r for r in results if r['split_position'] == best_split][0]
print(f"\nSplit Details (Annual Income <= {best_split}):")
print(f" Left (<=): Yes={best_result['left_yes']}, No={best_result['left_no']}")
print(f" Right ( >): Yes={best_result['right_yes']}, No={best_result['right_no']}")================================================================================
CONTINUOUS ATTRIBUTE: Annual Income
================================================================================
Parent Node: Yes=3, No=7
Parent Gini Impurity: 0.420
================================================================================
EVALUATING ALL POSSIBLE SPLITS
================================================================================
Split at 65.0:
Left (<=): Yes=0, No=1, Total=1
Right ( >): Yes=3, No=6, Total=9
Weighted Gini: 0.400
Gini Gain: 0.020
Split at 72.5:
Left (<=): Yes=0, No=2, Total=2
Right ( >): Yes=3, No=5, Total=8
Weighted Gini: 0.375
Gini Gain: 0.045
Split at 80.0:
Left (<=): Yes=0, No=3, Total=3
Right ( >): Yes=3, No=4, Total=7
Weighted Gini: 0.343
Gini Gain: 0.077
Split at 87.5:
Left (<=): Yes=1, No=3, Total=4
Right ( >): Yes=2, No=4, Total=6
Weighted Gini: 0.417
Gini Gain: 0.003
Split at 92.5:
Left (<=): Yes=2, No=3, Total=5
Right ( >): Yes=1, No=4, Total=5
Weighted Gini: 0.400
Gini Gain: 0.020
Split at 97.5:
Left (<=): Yes=3, No=3, Total=6
Right ( >): Yes=0, No=4, Total=4
Weighted Gini: 0.300
Gini Gain: 0.120
Split at 110.0:
Left (<=): Yes=3, No=4, Total=7
Right ( >): Yes=0, No=3, Total=3
Weighted Gini: 0.343
Gini Gain: 0.077
Split at 122.5:
Left (<=): Yes=3, No=5, Total=8
Right ( >): Yes=0, No=2, Total=2
Weighted Gini: 0.375
Gini Gain: 0.045
Split at 172.5:
Left (<=): Yes=3, No=6, Total=9
Right ( >): Yes=0, No=1, Total=1
Weighted Gini: 0.400
Gini Gain: 0.020
================================================================================
BEST SPLIT
================================================================================
Best Split Position: 97.5
Minimum Weighted Gini: 0.300
Maximum Gini Gain: 0.120
Split Details (Annual Income <= 97.5):
Left (<=): Yes=3, No=3
Right ( >): Yes=0, No=4Let’s fit a decision tree to the data.
Let’s plot the tree.
Interestingly, the tree was built using only the Income field.
That’s arguably an advantage of Decision Trees: they automatically perform feature selection.
Let’s explore some fields starting with survival rate by gender.
Code
import seaborn as sns
import matplotlib.pyplot as plt
fig,axs = plt.subplots(1,2, figsize=(11,5))
sns.barplot(data=df_train, y='Survived', x="Sex", ax=axs[0], hue="Sex", palette=["#3374a1","#e1812d"]).set(title="Survival rate")
sns.countplot(data=df_train, x="Sex", ax=axs[1], hue="Sex", palette=["#3374a1","#e1812d"]).set(title="Histogram");
Indeed, “women and children first” was enforced on the Titanic.
Alternatively, we could try splitting on a continuous column. We have to use a somewhat different chart to see how this might work – here’s an example of how we could look at LogFare:
Code
The boxenplot above shows quantiles of LogFare for each group of Survived==0 and Survived==1.
It shows that the average LogFare for passengers that didn’t survive is around 2.5, and for those that did it’s around 3.2.
So it seems that people that paid more for their tickets were more likely to get put on a lifeboat.
Full Decision Tree
Ok. Let’s build a decision tree model using all the features.
Let’s draw the tree.
Stopping Criteria – Minimum Samples Split
Let’s train the decision tree again but with stopping criteria based on the number of samples in a node.
Let’s draw the tree.
Decision Tree – Maximum Depth
Let’s train the decision tree again but with a maximum depth of 3.
Let’s draw the tree.
