Recall that the GMM generates each point by:
If we know that point \(x_i\) was generated by cluster \(z_i\), the joint probability is:
\[ p(x_i, z_i \mid \theta) = p(z_i \mid \theta) \cdot p(x_i \mid z_i, \theta) = \pi_{z_i} \cdot \mathcal{N}(x_i \mid \mu_{z_i}, \Sigma_{z_i}) \]
This is just applying the chain rule: probability of choosing cluster \(z_i\) times probability of generating \(x_i\) from that cluster.
Summing this joint probability over the \(K\) possible values of \(z_i\) gives back the mixture density from the lecture:
\[ p(x_i \mid \theta) = \sum_{k=1}^{K} p(x_i, z_i = k \mid \theta) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(x_i \mid \mu_k, \Sigma_k). \]
So the observed-data likelihood is the complete-data likelihood with the unknown \(z_i\) averaged out. That averaging is where the sum inside the log comes from.
For all \(n\) independent data points:
\[ p(X, Z \mid \theta) = \prod_{i=1}^{n} \pi_{z_i} \cdot \mathcal{N}(x_i \mid \mu_{z_i}, \Sigma_{z_i}) \]
\[ \log p(X, Z \mid \theta) = \sum_{i=1}^{n} \left[ \log \pi_{z_i} + \log \mathcal{N}(x_i \mid \mu_{z_i}, \Sigma_{z_i}) \right] \]
Here’s the key insight! Since \(z_i\) takes on exactly one value from \(\{1, 2, \ldots, K\}\), we can rewrite \(\pi_{z_i}\) using an indicator function:
\[ \log \pi_{z_i} = \sum_{k=1}^{K} \mathbb{1}(z_i = k) \log \pi_k \]
Why? Because only one term in this sum is non-zero (when \(k = z_i\)), and it equals \(\log \pi_{z_i}\).
Similarly: \[ \log \mathcal{N}(x_i \mid \mu_{z_i}, \Sigma_{z_i}) = \sum_{k=1}^{K} \mathbb{1}(z_i = k) \log \mathcal{N}(x_i \mid \mu_k, \Sigma_k) \]
Combining everything:
\[ \log p(X, Z \mid \theta) = \sum_{i=1}^{n} \sum_{k=1}^{K} \mathbb{1}(z_i = k) \left[ \log \pi_k + \log \mathcal{N}(x_i \mid \mu_k, \Sigma_k) \right] \]
Compare it with the observed-data log-likelihood that the lecture tried to maximize directly:
\[ \ell(\theta) = \log p(X \mid \theta) = \sum_{i=1}^{n} \log \left( \sum_{k=1}^{K} \pi_k \, \mathcal{N}(x_i \mid \mu_k, \Sigma_k) \right). \]
The two differ only in where the sum over clusters sits: inside the log in the observed-data version, outside it in the complete-data version. Inside, every parameter is tangled up with every other one in each term. Outside, each term involves a single cluster’s parameters, so the expression separates by cluster.
As mentioned earlier, this form is much easier to work with because: