The regression line: slope, intercept and least squares

What is a simple linear regression?

A simple linear regression explains the variation in one variable using one other variable and a straight line. The variable you want to explain is the dependent variable , Y. The variable you use to explain it is the independent variable , X.

$$Y_i = b_0 + b_1 X_i + \varepsilon_i$$

The regression coefficients are the intercept , $b_0$, and the slope coefficient , $b_1$. The error term , $\varepsilon_i$, is everything about $Y_i$ that the line does not capture.

Y = b₀ + b₁X + εone explained variable, one explaining variable

• Dependent variable, Y Memorize the variable you want to explain
  • Also called explained variable, predicted variable
• Independent variable, X Memorize the variable doing the explaining
  • Also called explanatory variable, predictor
• Regression coefficients Memorize estimated from the data
  • Intercept, b₀ predicted Y when X = 0
  • Slope, b₁ change in predicted Y per one-unit rise in X
• Error term, ε Use it what the line leaves unexplained
  • Residual, e the sample estimate of ε: actual Y minus fitted Y

Throughout this module we use one small data set. Six retailers report their marketing spend as a percentage of sales (X) and their revenue growth in percent (Y).

Retailer	A	B	C	D	E	F
Marketing spend, X (%)	2	3	5	6	8	9
Revenue growth, Y (%)	4	5	9	8	12	13

How does least squares choose the line?

Any line gives a fitted value , $\hat{Y}_i = b_0 + b_1X_i$, for each observation. The miss, $e_i = Y_i - \hat{Y}_i$, is the residual : actual minus fitted. It is the sample’s estimate of the error term. Square each residual and add them up to get the sum of squares error , SSE. The least squares line is the one with the smallest SSE.

$$SSE = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2 = \sum e_i^2$$

Squaring does two jobs. Positive and negative misses cannot cancel, and a big miss counts far more than a small one. Calculus then gives a closed-form answer for the two coefficients.

$$b_1 = \frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sum (X_i-\bar{X})^2} = \frac{\text{Cov}(X,Y)}{\text{Var}(X)}$$

$$b_0 = \bar{Y} - b_1\bar{X}$$

The second formula says the fitted line always passes through the point of means, $(\bar{X}, \bar{Y})$.

How is the slope related to the correlation?

Dividing the covariance by both standard deviations gives the correlation, $r$. Dividing by the variance of X alone gives the slope. So the two are linked by the ratio of standard deviations.

$$b_1 = r\,\frac{s_Y}{s_X}$$

For the six retailers $r = 0.9786$, $s_Y = 3.62$ and $s_X = 2.74$, and $0.9786 \times 3.62 \div 2.74 = 1.29$. A positive correlation always means a positive slope, and the slope is in units of Y per unit of X, while $r$ has no units.

What are the residuals telling you?

Put the fitted line next to the data and each retailer has a residual.

Retailer	X	Y	Fitted Ŷ	Residual e
A	2	4	3.97	0.03
B	3	5	5.27	−0.27
C	5	9	7.85	1.15
D	6	8	9.15	−1.15
E	8	12	11.73	0.27
F	9	13	13.03	−0.03

The residuals add up to zero, and their squares add up to SSE = 2.773. No other straight line through these points has a smaller sum of squares.

Exam tip On the exam you will usually be handed the covariance and variance, or the sums of squares. Slope first, then intercept. Check that the slope has the variance of X underneath it.

Go deeper Why squares and not absolute values?

Squares give a smooth function that calculus can minimize in one step, so the coefficients have simple formulas. They also punish large misses heavily, so one wild point pulls the line toward it. That is why a single outlier can tilt a least squares line, a point lesson 2 returns to.

Go deeper Beta is a regression slope

Regress a stock’s returns (Y) on the market’s returns (X) and the slope is the stock’s beta: Cov(stock, market) ÷ Var(market). The same slope formula you just used drives the capital asset pricing model in Portfolio Management. A time-series regression like this uses returns over time; a cross-sectional regression uses many companies at one date. The mechanics are identical.