The ANOVA table and the standard error of the estimate

What does ANOVA do in a regression?

Analysis of variance (ANOVA) splits the variation in Y into the part the regression explains and the part it does not, then compares them. The regression output tabulates the pieces.

Start with the three sums of squares. The sum of squares total (SST) measures how far the Y values are from their mean. The sum of squares regression (SSR, also called the sum of squares model) measures how far the fitted values are from that mean (the explained part). The sum of squares error (SSE) measures how far the actual values are from the fitted line (the unexplained part).

$$SST = SSR + SSE$$

What does the table look like?

For the six retailers from the previous lessons (df stands for degrees of freedom):

Source	Sum of squares	df	Mean square	F
Regression	SSR = 62.727	1	MSR = 62.727	90.47
Error	SSE = 2.773	n − 2 = 4	MSE = 0.693
Total	SST = 65.500	n − 1 = 5

Each mean square is a sum of squares divided by its degrees of freedom. With k independent variables the regression row has k degrees of freedom and the error row n − k − 1. Simple regression has k = 1, so they are 1 and n − 2. The error degrees of freedom are $n-2$ because two coefficients (intercept and slope) were estimated from the data.

The mean square regression (MSR) is the explained variation per degree of freedom.

$$MSR = \frac{SSR}{k} = SSR \ (k = 1)$$

The mean square error (MSE) is the average squared miss.

$$MSE = \frac{SSE}{n-k-1} = \frac{SSE}{n-2}$$

Its square root is the typical size of a miss, the standard error of the estimate , $s_e$. It is also called SEE, the standard error of the regression or the root mean square error.

$$s_e = \sqrt{MSE} = \sqrt{\frac{SSE}{n-2}}$$

For the retailers, $s_e = \sqrt{0.69333} = 0.8327$ percentage points: a typical miss of about 0.8 points of revenue growth. $s_e$ is in the units of Y. $R^2$ has no units.

How is F built from the table?

The F-statistic is the ratio of the two mean squares. It uses mean squares, never the raw sums.

$$F = \frac{MSR}{MSE} = \frac{SSR \,/\, k}{SSE \,/\, (n-k-1)}$$

F is a ratio of two variances. Big is bad for the null: a large F means the line explains far more than the noise. Compare it with the critical F for 1 and $n-2$ degrees of freedom, or read its p-value. To judge whether the model is significant, use F (or the slope’s t-test), not $R^2$.

Total variation, SSTdf = n − 1

• Explained by the line: SSR Memorize df = 1 in simple regression
  • MSR = SSR ÷ k (k = 1) the numerator of F
  • R² = SSR ÷ SST share explained
• Left unexplained: SSE Memorize df = n − 2
  • MSE = SSE ÷ (n − 2) the denominator of F
  • s_e = √MSE typical miss, in the units of Y
• Variance of Y = SST ÷ (n − 1) Use it spread around the mean

How does the standard error of the estimate compare with the spread of Y?

The table also gives the variance of the dependent variable : SST ÷ (n − 1), the spread around the mean. $s_e$ is the spread around the regression line. For the retailers the variance of Y is 65.5 ÷ 5 = 13.1 (standard deviation 3.62) against $s_e$ = 0.83: marketing spend explains most of the spread. $R^2$ is a relative measure of fit (a share); $s_e$ is an absolute one (in the units of Y).

$$s_Y^2 = \frac{SST}{n-1}$$

How do you complete a table with blanks?

The exam gives you $n$ and any two of SSR, SSE and SST, and you fill in the rest. Use $SST = SSR + SSE$ to get the missing sum of squares, the degrees of freedom $1$, $n-2$ and $n-1$, then divide.

Exam tip Check your table with the degrees of freedom: regression plus error must equal total (1 + 20 = 21). A mismatch means you used n instead of n − 2.

Go deeper Why the error degrees of freedom are n − 2

The residuals are computed from a line whose two coefficients were chosen using the same data. Those two choices tie down two of the $n$ residuals (they sum to zero, and they are uncorrelated with X), so only $n-2$ are free to vary. Dividing SSE by $n-2$, not $n$, corrects for that and gives an unbiased estimate of the error variance.