Recap: \(k\)-Nearest Neighbors, the Curse of Dimensionality, and Naive Bayes

DS701 Session 11 — Tue Oct 13, 2026 (Monday schedule)

Today’s plan

• 5 min — knowledge-check review
• 20 min — highlights and Q&A (answer or pass — answering always earns credit)
• 60 min — in-class activity (small groups)
• wrap-up and cold-call check-ins

Knowledge-Check Review

KC 1: What does \(k\)-NN actually do?

Describe what a \(k\)-nearest-neighbors classifier does at training time and at test time, and name the one hyperparameter that controls how complex the resulting model is. Is \(k\)-NN parametric or nonparametric, and why?

KC 2: What the curse does to “nearest”

The lecture showed that in \(d\) dimensions the fraction of a unit ball’s volume lying in an outer shell of thickness \(\epsilon\) is \(f_d = 1-(1-\epsilon)^d\), and that the ratio of the minimum to the average pairwise distance among random points climbs toward 1 as \(d\) grows. Explain in your own words what these two facts do to the word “nearest” in \(k\)-nearest neighbors, and why that hurts the classifier.

KC 3: Naive Bayes meets the curse

State the “naive” assumption in Naive Bayes and say why it is called naive. Then connect it to the curse of dimensionality: a spam filter uses \(d = 1{,}000\) binary word features. Roughly how many probabilities per class must Naive Bayes estimate with the assumption versus without it, and why might Naive Bayes cope with those 1,000 features better than \(k\)-NN would?

Highlights

The data is the model

walks like a duck, swims like a duck, quacks like a duck…

• Train: store the labeled points. That is all.
• Predict for a query \(x\): distance to every stored point → the \(k\) closest → majority vote (hard) or vote fractions \(p(c \mid x, k)\) (soft).
• Nonparametric: the “parameters” are the training set itself; they grow with the data. Contrast: linear/logistic regression, Naive Bayes — fixed parameter counts.
• The bill comes at test time: every prediction is \(O(n)\) distances. Training is free, prediction is not.
• Built on Euclidean distance, so scale the features — an unscaled dollars column picks the neighbours by itself (same lesson as \(k\)-means).

\(k\) is the bias–variance knob

• \(k=1\): islands around single points; training accuracy 100%; low bias, high variance — move one point and the boundary moves.
• \(k=25\): smooth, broad regions; low variance, more bias — at \(k=n\) it predicts the majority class everywhere.
• Every \(k\) is a different model. From lecture 07: complexity is chosen on held-out data, never on training error.

Choosing \(k\) — and judging the result

Iris, \(k = 2 \dots 19\), each point the mean of 50 random train/test splits.

• The peak is at \(k = 13\) — but \(\pm 1\sigma\) bars (std\(/\sqrt{50}\)) overlap \(k = 9\) and \(11\). Honest answer: around 9–13.
• One split is one noisy sample of the accuracy. Average many splits — or use \(K\)-fold cross-validation (lecture 07): same idea, no point wasted.

Accuracy is not the whole story. Two-circles, 5-NN: 72% accuracy — but every Negative right and only half the Positives. Recall \(= TP/(TP+FN) = 50\%\); precision \(= TP/(TP+FP) = 100\%\). With 90/10 imbalance, “always predict the majority” scores 90% and is useless — read the confusion matrix, not the single number.

The curse, part 1: space explodes

\(10^2\) bins

\(10^3\) bins

• \(k\)-NN needs the training data to be dense — a genuinely close point near every query.
• Quantise each feature into 10 bins: \(d = 40\) gives \(10^{40}\) bins. With \(10^4\) points that is one point per \(10^{36}\) bins. Almost every bin is empty.
• To be close in \(\mathbb{R}^d\) two points must be close in each of the \(d\) coordinates — harder and harder as \(d\) grows.
• Data needed to keep a given density grows exponentially in \(d\).

The curse, part 2: “nearest” stops meaning anything

Fraction of a unit ball in its outer \(\epsilon\)-shell: \(f_d = 1-(1-\epsilon)^d \to 1\).

1,000 uniform points: min pairwise distance / average pairwise distance \(\to 1\).

Both say: in high \(d\) every candidate neighbour sits at about the same distance. The \(k\) “nearest” are a random handful; classifying from them is like classifying from points at the average distance. What helps: a different dissimilarity (cosine did a little better), or reduce the dimension — feature selection, PCA/SVD (lectures 12–13). Today’s activity measures all of this.

Naive Bayes: classify with Bayes’ rule

\[ P(C_i \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid C_i)\,P(C_i)}{P(\mathbf{x})} \qquad\Longrightarrow\qquad \hat{C} = \arg\max_{C_i} \; P(\mathbf{x} \mid C_i)\,P(C_i) \]

• Meningitis: \(P(S \mid M) = 0.75\), \(P(S) = 0.05\), \(P(M) = 10^{-4}\) → \(P(M \mid S) = 0.75/0.05 \times 10^{-4} = 15 \times 10^{-4}\). Prior \(\to\) posterior: the evidence multiplied the odds by 15.
• \(P(\mathbf{x})\) is the same for every class — drop it; the MAP class wins.
• The obstacle: \(P(x_1,\dots,x_d \mid C_i)\) is a joint histogram — \(10^{20}\) bins for 20 features of 10 values. Not estimable.

The naive assumption — and why it still works

\[ P(x_1,\dots,x_d \mid C_i) \;=\; \prod_{j=1}^{d} P(x_j \mid C_i) \]

• Conditional independence given the class: \(10^{20}\) bins become \(20 \times 10\). Training is counting (or one Gaussian per feature and class).
• Naive: features are rarely independent. Works anyway: only the arg max must be right, and each factor is estimated from all the data — no distances, no density requirement, so high \(d\) is not the same enemy it is for \(k\)-NN.
• Refund = No, Married: Evade=No \(\approx 0.7 \cdot \tfrac47 \cdot \tfrac47 \approx 0.23\); Evade=Yes \(= 0.3 \cdot 1 \cdot \mathbf{0}\). One empty bin kills the product → Laplace smoothing.

In-Class Activity

Activity: a classifier bake-off, then measure the curse

Goal: on your section’s slice of a real diagnostic dataset, sweep \(k\) under two metrics and choose it by cross-validation, then pit \(k\)-NN against Gaussian Naive Bayes and a decision tree — accuracy, confusion matrix, precision, recall — and hand-compute Naive Bayes on a toy table. Then measure the curse: nearest/farthest distance ratios and shell volumes as \(d\) grows, watch \(k\)-NN collapse as noise dimensions are added, and rescue it — and find out whether Naive Bayes suffers the same way.

• Work in groups of 2–3. Open your section’s notebook.
• Parts marked (autograded) are submitted to Gradescope; the rest is participation.
• Staff will circulate — be ready to explain any part of your work.
• Dependencies: numpy, pandas, matplotlib, scipy, scikit-learn.

Section A1

Section B1

The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.

Going deeper

Full lecture notes: \(k\)-Nearest Neighbors, Curse of Dimensionality, and Naive Bayes

• Parametric vs. Nonparametric Models
• \(k\)-Nearest Neighbors — the 1-, 2-, 3-NN pictures; Varying \(k\) — decision regions for \(k = 1, 5, 25\)
• Challenges for \(k\)-NN — cost, scaling, and the curse
• Points are far apart in high D · Fraction of points in outer shell · Curse of Dimensionality Example · Implications of the Curse
• Precision and Recall and the Confusion Matrix on the two-circles data
• Train-Test Splitting and Hyperparameters: k-NN — many splits, error bars, choosing \(k\) on iris
• Naive Bayes — Bayes Rule, The “Naive” Assumption, Worked Example
• Where \(k\) came from: Generalization — bias–variance, hold-out and \(K\)-fold cross-validation