Proof of TSS Decomposition
Explainer Video on Proof
Only accessible with you BU login.
Proof that TSS = RSS + ESS
Given:
- TSS = \(\sum_{i=1}^n (y_i - \bar{y})^2\) (Total Sum of Squares)
- RSS = \(\sum_{i=1}^n (y_i - \hat{y}_i)^2\) (Residual Sum of Squares)
- ESS = \(\sum_{i=1}^n (\hat{y}_i - \bar{y})^2\) (Explained Sum of Squares)
Proof:
Start by decomposing the deviation from the mean: \[y_i - \bar{y} = (y_i - \hat{y}_i) + (\hat{y}_i - \bar{y})\]
Square both sides: \[ \begin{align} (y_i - \bar{y})^2 &= [(y_i - \hat{y}_i) + (\hat{y}_i - \bar{y})]^2\\ &= (y_i - \hat{y}_i)^2 + 2(y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) + (\hat{y}_i - \bar{y})^2 \end{align} \]
Sum over all \(i\): \[ \sum_{i=1}^n (y_i - \bar{y})^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2 + 2\sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) + \sum_{i=1}^n (\hat{y}_i - \bar{y})^2 \]
Which gives us: \[ \text{TSS} = \text{RSS} + 2\sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) + \text{ESS} \]
The key step: We need to show that the cross-term equals zero: \[ \sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) = 0 \]
Why is this zero?
For OLS regression with an intercept, the residuals \((y_i - \hat{y}_i)\) satisfy two orthogonality conditions:
- \(\sum_{i=1}^n (y_i - \hat{y}_i) = 0\) (residuals sum to zero)
- \(\sum_{i=1}^n (y_i - \hat{y}_i)\hat{y}_i = 0\) (residuals are orthogonal to predictions)
Both are the normal equations \(X^T\mathbf{e} = \mathbf{0}\) read row by row: the row for the column of 1s says the residuals sum to zero (condition 1), and because \(\hat{\mathbf{y}} = X\hat{\boldsymbol\beta}\) is a combination of the columns of \(X\), \(\mathbf{e}\) is orthogonal to it too (condition 2). The proofs below spell this out.
Using condition (2): \[ \sum_{i=1}^n (y_i - \hat{y}_i)(\hat{y}_i - \bar{y}) = \sum_{i=1}^n (y_i - \hat{y}_i)\hat{y}_i - \bar{y}\sum_{i=1}^n (y_i - \hat{y}_i) = 0 - \bar{y} \cdot 0 = 0 \]
Therefore: TSS = RSS + ESS ✓
Geometric Intuition:
This decomposition is a Pythagorean theorem in high-dimensional space! The vectors \((y_i - \bar{y})\), \((y_i - \hat{y}_i)\), and \((\hat{y}_i - \bar{y})\) form a right triangle because the residuals are orthogonal to the fitted values.
Why residuals sum to exactly zero (with intercept)
Condition 1 is satisfied exactly (not just approximately), but this is only true when the model includes an intercept term.
When you include an intercept \(\beta_0\), the normal equations \(X^TX\boldsymbol\beta = X^T\mathbf{y}\) include a row corresponding to the column of 1s in the design matrix.
Proof
Consider a linear model with an intercept. The design matrix has a column of 1s:
\[ X = \begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}, \quad \boldsymbol\beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_p \end{bmatrix}, \quad \mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix} \]
The Normal Equations
The OLS solution satisfies the normal equations: \[ X^T X \boldsymbol\beta = X^T \mathbf{y} \]
Looking at \(X^T\)
The transpose of \(X\) is: \[ X^T = \begin{bmatrix} 1 & 1 & 1 & \cdots & 1 \\ x_{11} & x_{21} & x_{31} & \cdots & x_{n1} \\ x_{12} & x_{22} & x_{32} & \cdots & x_{n2} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ x_{1p} & x_{2p} & x_{3p} & \cdots & x_{np} \end{bmatrix} \]
The first row is all 1s!
The First Normal Equation
The normal equations give us a system of \((p+1)\) equations. The first equation comes from multiplying the first row of \(X^T\) with both sides:
Left side: First row of \(X^T\) times \(X\boldsymbol\beta\) \[ [1, 1, 1, \ldots, 1] \cdot X\boldsymbol\beta = \sum_{i=1}^n (X\boldsymbol\beta)_i = \sum_{i=1}^n \hat{y}_i \]
Right side: First row of \(X^T\) times \(\mathbf{y}\) \[ [1, 1, 1, \ldots, 1] \cdot \mathbf{y} = \sum_{i=1}^n y_i \]
The Result
Therefore, the first normal equation is: \[ \boxed{\sum_{i=1}^n \hat{y}_i = \sum_{i=1}^n y_i} \]
This is exact, not approximate. It’s a direct consequence of having the column of 1s in \(X\).
Interpretation
Dividing by \(n\): \[ \bar{\hat{y}} = \bar{y} \]
The mean of the fitted values equals the mean of the observed values. This is why, with an intercept, the regression line always passes through the point \((\bar{x}, \bar{y})\).
End of proof
Without an intercept
If you force the regression through the origin (no \(\beta_0\) term), then:
- Condition 1 is NOT satisfied - residuals don’t sum to zero
- The TSS = RSS + ESS decomposition doesn’t hold
- \(R^2\) can even be negative!
Key takeaway
For standard OLS with intercept: - \(\sum (y_i - \hat{y}_i) = 0\) ✓ (exactly) - \(\sum (y_i - \hat{y}_i)\hat{y}_i = 0\) ✓ (exactly, from normal equations)
Both are mathematical properties, not approximations!