The regression line: slope, intercept and least squares

What is a simple linear regression?

A simple linear regression explains the variation in one variable using one other variable and a straight line. The variable you want to explain is the dependent variable , Y. The variable you use to explain it is the independent variable , X.

Fitting the line this way is called ordinary least squares (OLS) regression, and we say Y is regressed on X. One independent variable makes the regression simple; two or more make it a multiple regression, which is beyond this module.

$$Y_i = b_0 + b_1 X_i + \varepsilon_i$$

The regression coefficients are the intercept , $b_0$, and the slope coefficient , $b_1$. The error term , $\varepsilon_i$, is everything about $Y_i$ that the line does not capture. Once the data give estimates of the coefficients, we write them with hats, $\hat{b}_0$ and $\hat{b}_1$.

Y = b₀ + b₁X + εone explained variable, one explaining variable

• Dependent variable, Y Memorize the variable you want to explain
  • Also called explained variable, predicted variable
• Independent variable, X Memorize the variable doing the explaining
  • Also called explanatory variable, predictor
• Regression coefficients Memorize estimated from the data
  • Intercept, b₀ predicted Y when X = 0
  • Slope, b₁ change in predicted Y per one-unit rise in X
• Error term, ε Use it what the line leaves unexplained
  • Residual, e the sample estimate of ε: actual Y minus fitted Y

Throughout this module we use one small data set. Six retailers report their marketing spend as a percentage of sales (X) and their revenue growth in percent (Y).

Retailer	A	B	C	D	E	F
Marketing spend, X (%)	2	3	5	6	8	9
Revenue growth, Y (%)	4	5	9	8	12	13

How does ordinary least squares choose the line?

Any line gives a fitted value , $\hat{Y}_i = \hat{b}_0 + \hat{b}_1X_i$, for each observation. The miss, $e_i = Y_i - \hat{Y}_i$, is the residual : actual minus fitted. It is the sample’s estimate of the error term. Square each residual and add them up to get the sum of squares error , SSE. The OLS line is the one with the smallest SSE.

$$SSE = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2 = \sum e_i^2$$

Squaring does two jobs. Positive and negative misses cannot cancel, and a big miss counts far more than a small one. Calculus then gives a closed-form answer for the two coefficients. The slope is the covariance of X and Y, Cov(X, Y), divided by the variance of X, Var(X).

$$\hat{b}_1 = \frac{\sum (X_i-\bar{X})(Y_i-\bar{Y})}{\sum (X_i-\bar{X})^2} = \frac{\text{Cov}(X,Y)}{\text{Var}(X)}$$

$$\hat{b}_0 = \bar{Y} - \hat{b}_1\bar{X}$$

Two properties follow, and both are tested: the fitted line always passes through the point of means, $(\bar{X}, \bar{Y})$, and the residuals always add up to zero.

Both the covariance and the variance use n − 1, so the divisor cancels and the sums give the same slope.

What are the residuals telling you?

Put the fitted line next to the data and each retailer has a residual.

Retailer	X	Y	Fitted Ŷ	Residual e
A	2	4	3.97	0.03
B	3	5	5.27	−0.27
C	5	9	7.85	1.15
D	6	8	9.15	−1.15
E	8	12	11.73	0.27
F	9	13	13.03	−0.03

The residuals add up to zero, and their squares add up to SSE = 2.773. No other straight line through these points has a smaller sum of squares.

Exam tip On the exam you will usually be handed the covariance and variance, or the sums of squares. Slope first, then intercept. Check that the slope has the variance of X underneath it.

Go deeper Slope and correlation

Dividing the covariance by both standard deviations gives the correlation, $r$. Dividing by the variance of X alone gives the slope. So the two are linked by the ratio of standard deviations.

$$\hat{b}_1 = r\,\frac{s_Y}{s_X}$$

For the six retailers $r = 0.9786$, $s_Y = 3.62$ and $s_X = 2.74$, and $0.9786 \times 3.62 \div 2.74 = 1.29$. A positive correlation always means a positive slope, and the slope is in units of Y per unit of X, while $r$ has no units.

Does a regression prove cause and effect?

Know the idea

No. A regression measures association: how Y tends to move with X. It cannot tell you that X causes Y, because a third factor can drive both. Retailers in fast-growing regions may spend more on marketing and also grow faster, so part of the 1.29 slope could reflect local demand rather than marketing. Read a slope as “goes with”, not “causes”. This is the point of correlation versus causation .

What kinds of data does a regression use?

The data come in three shapes. Cross-sectional data are many entities observed in the same period (indexed by $i$). Time-series data follow one entity over many periods (indexed by $t$). Panel data follow the same entities over many periods.

Regression datawho and when

• Cross-sectional Memorize many entities, one period
• Time-series Memorize one entity, many periods
• Panel Memorize the same entities over many periods

The six retailers in one year are cross-sectional data. A regression of one company’s monthly returns on an index’s monthly returns uses time-series data. Forty countries’ bond yields against their debt ratios over ten years are panel data.

What if X is a yes-or-no variable?

An indicator variable (also called a dummy variable) equals 1 when a condition holds and 0 when it does not. With an indicator as X, the intercept is the mean of Y in the 0 group, the slope is the difference between the two group means, and the mean of the 1 group is intercept plus slope.

$$Y_i = b_0 + b_1 I_i + \varepsilon_i, \quad I_i = 1 \text{ or } 0$$

Lesson 3 tests whether that difference is real.

Go deeper Why squares and not absolute values?

Squares give a smooth function that calculus can minimize in one step, so the coefficients have simple formulas. They also punish large misses heavily, so one wild point pulls the line toward it. That is why a single outlier can tilt a least squares line, a point lesson 2 returns to.

Go deeper Beta is a regression slope

Regress a stock’s excess returns (its return minus the risk-free rate) on the market’s excess returns and the slope is the stock’s beta. Lesson 7 runs that regression step by step and turns the beta into an expected return.

Go deeper Running a regression in a spreadsheet

Spreadsheets do the same arithmetic. In Excel and Google Sheets, SLOPE and INTERCEPT return the coefficients, RSQ the R-squared (the share of the variation in Y that the line explains) and CORREL the correlation; LINEST returns the full set of regression statistics; FORECAST.LINEAR (FORECAST in Google Sheets) and TREND return predicted values. Statistical software (R’s lm function, Python’s scikit-learn or statsmodels) prints the same output table. On the exam you calculate by hand or read the output.