Recap: \(k\)-Means Clustering

DS701 Session 4 — Wed Sep 16, 2026

Today’s plan

  • 5 min — knowledge-check review
  • 20 min — highlights and Q&A (answer or pass — answering always earns credit)
  • 60 min — in-class activity (small groups)
  • wrap-up and cold-call check-ins

Knowledge-Check Review

KC 1: What does \(k\)-means minimize?

Name the quantity and write it as a formula.

KC 2: One iteration by hand

Points \(1.2,\ 1.8,\ 2.1,\ 7.3,\ 7.9,\ 8.2,\ 9.1,\ 10.5\), initial centers \(A = 4.0\), \(B = 11.0\).

KC 3: Why scale the features?

Cluster customers on age (18–80) and income ($20k–$200k), unscaled. What determines the clusters?

Highlights

What are we actually asking for?

A clustering is a grouping of data objects such that objects within a group are similar and objects in different groups are dissimilar.

So we want to:

  • minimize intra-cluster distances
  • maximize inter-cluster distances

And it is unsupervised — no labels go in, and any labels we put on the clusters we invent afterwards.

Partitional setup for \(k\)-means: data are points in \(\mathbb{R}^d\), every object belongs to one and only one cluster (mutually exclusive), the clusters cover everything (exhaustive), and \(K\) is given in advance.

Hard problem, easy algorithm

  • Minimizing WCSS exactly is NP-hard for \(d \ge 2\) — no efficient exact algorithm is expected.
  • \(k=1\) and \(k=n\) are the easy corner cases (global mean; every point its own center, WCSS \(=0\)).
  • Nonetheless: Lloyd’s algorithm (1957) is a simple heuristic that works very well in practice.
  • Voted a top-10 data mining algorithm, and independently rediscovered several times.
  • Careful with the wording: the \(k\)-means problem \(\neq\) the \(k\)-means algorithm.

Lloyd’s algorithm

  1. Pick \(K\) cluster centers \(\{c_1, \dots, c_K\}\)
  2. Assign: define \(C_j\) as the points closest to center \(c_j\)
  3. Update: set \(c_j\) to the center of mass of \(C_j\)
  4. Repeat step 2 until convergence
    • centers move below a threshold, or inertia changes below a threshold, or max iterations

Neither step 2 nor step 3 can increase WCSS \(\Rightarrow\) it always converges — but only to a local minimum.

Initialization matters

  • Same data, same \(K\), different random start \(\Rightarrow\) different answer.
  • A bad start gets stuck: the algorithm can never climb back out, because it never increases WCSS.
  • Strategy: pick initial centers that are far apart — this is \(k\)-means++.
  • It is scikit-learn’s default, and n_init restarts the whole thing several times and keeps the best WCSS.

Choosing \(K\)

\(K\) is our first hyperparameter: a parameter that must be set before the model parameters (the centers) can be learned.

WCSS alone can’t pick it — WCSS decreases monotonically in \(K\), all the way to \(0\) at \(K = n\).

So: sweep \(K = 1, 2, 3, \dots\), fit each, and score it.

  • Elbow method — plot WCSS vs. \(K\) and look for the bend where the marginal gain flattens.
  • Silhouette score — how much closer is a point to its own cluster than to the next-nearest one? Higher is better, and it has a maximum.
  • Both are heuristics; the elbow is often ambiguous — and they can disagree. You will build the silhouette from its definition and confront exactly that in today’s activity.

Where \(k\)-means breaks

Assignment is “nearest center”, so cluster regions are straight-sided Voronoi cells and clusters are implicitly assumed to be spheres around their center.

  • Non-spherical — two half-moons get sliced by a straight boundary.
  • Anisotropic — stretched/rotated blobs, same failure after a linear transformation.
  • Unequal variance — squared distances make the wide cluster expensive, so it gets split.
  • Unequal sizes — the objective prefers roughly balanced, equal-radius cells (500/100/10 will not survive).
  • Bad init — a local minimum of WCSS.
  • Wrong \(K\) — ask for 2 and you get 2, whatever the data says.

Feature scaling

Because \(k\)-means looks for spherical clusters in Euclidean distance, the units of each feature are part of the model.

\[ \begin{bmatrix}\text{age } 27\\\text{income } 75000\\\text{gender } 0\end{bmatrix}, \qquad \begin{bmatrix}\text{age } 45\\\text{income } 42000\\\text{gender } 1\end{bmatrix} \]

Standard remedy, per feature and independently:

  1. subtract the mean (centers the feature at zero), then
  2. divide by the standard deviation (puts every feature on a comparable range).

Whenever you build or pick a distance metric, think about feature scale first.

Where it works well

\(k\)-means is a workhorse when the assumptions roughly hold:

  • Customer segmentation — spending / order value / order count \(\rightarrow\) personas (scale first: dollars vs. counts).
  • Iris — recovers the three species almost exactly; setosa is cleanly separated, versicolor/virginica overlap a little.
  • Image compression — cluster pixels in \((R,G,B)\) space, replace each pixel with its centroid. \(k=16\) means a 4-bit palette index instead of 24 bits: \(24/4 = 6\times\).

Original, 24-bit color

\(k=16\) centroids, \(6\times\) smaller

In-Class Activity

Activity: \(k\)-Means from scratch and in scikit-learn

Goal: implement one Lloyd’s iteration yourself, check it against sklearn.cluster.KMeans, then the stretch — build the silhouette coefficient from its definition, sweep \(k\), and settle a case where the elbow and the silhouette disagree — and finally break \(k\)-means on purpose and diagnose why.

  • Work in groups of 2–3. Open the notebook for your section — A1 and B1 get different data.
  • Parts marked (autograded) are submitted to Gradescope; the rest is participation.
  • Staff will circulate — be ready to explain any part of your work.
  • Dependencies: numpy, matplotlib, scikit-learn only.

Section A1:    Section B1:   

The activity notebook goes live on the day of the lecture. Colab is optional — you can also open it on GitHub and run it locally.

Going deeper

Full lecture notes: \(k\)-Means Clustering