Linear Regression

The One Thing

Linear regression uses a scatter diagram to assess whether a linear relationship is reasonable, then applies least squares to fit a regression line for prediction. The correlation coefficient measures the direction and strength of linear association, whereas the regression equation provides the prediction rule.

Definitions and Results

• Bivariate data: Data consisting of paired observations of two variables, usually denoted by $X$ and $Y$.

• Independent variable: The predictor or explanatory variable, commonly represented by $X$.

• Dependent variable: The variable being predicted or explained, commonly represented by $Y$.

• Scatter diagram: A graph in which each paired observation is represented by a point $(x,y)$, showing the possible relationship between two variables.

• Positive correlation: A relationship in which larger values of one variable tend to be associated with larger values of the other variable.

• Negative correlation: A relationship in which larger values of one variable tend to be associated with smaller values of the other variable.

• No correlation: A situation in which the points show no clear linear pattern between the variables.

• Correlation coefficient: The numerical measure of the direction and strength of linear association, represented by $r$, where

$$-1\le r\le 1.$$

• Regression: A statistical method for estimating the value of one variable from the known value of another variable.

• Regression line of $Y$ on $X$: The line used to estimate $Y$ for a given value of $X$.

• Regression line of $X$ on $Y$: The line used to estimate $X$ for a given value of $Y$.

• Least squares method: A method of fitting a line by making the sum of the squared vertical errors as small as possible.

• Residual or error: The difference between an observed value and its predicted value, such as

$$y-\hat{y}.$$

• Regression coefficient: The slope of a regression line, indicating the expected change in the predicted variable for a one-unit change in the predictor variable.

• Coefficient of determination: The quantity $r^2$, representing the proportion of variation in the dependent variable explained by the linear relationship. It lies between $0$ and $1$ and is often expressed as a percentage.

• Extrapolation: The use of a regression equation to predict outside the range of observed data. Such predictions may be unreliable. Interpolation, which predicts within the observed range, is usually safer.

• Sample means: For paired data $(x_i,y_i)$,

$$\bar{x}=\frac{\sum x_i}{n},\bar{y}=\frac{\sum y_i}{n}.$$

• Regression line of $Y$ on $X$:

$$y-\bar{y}=b_{yx}(x-\bar{x}),$$ where $$b_{yx}=\frac{\sum [(x_i-\bar{x})(y_i-\bar{y})]}{\sum [(x_i-\bar{x}{)}^2]}.$$

• Regression line of $X$ on $Y$:

$$x-\bar{x}=b_{xy}(y-\bar{y}),$$ where $$b_{xy}=\frac{\sum [(x_i-\bar{x})(y_i-\bar{y})]}{\sum [(y_i-\bar{y}{)}^2]}.$$

• Relation between regression coefficients and correlation:

$$b_{yx}=r\left(\frac{s_y}{s_x}\right),b_{xy}=r\left(\frac{s_x}{s_y}\right),$$ where $s_x$ and $s_y$ are the standard deviations.

• Product of regression coefficients:

$$b_{xy}{b}_{yx}=r^2.$$

• Mean point: Both regression lines pass through

$$(\bar{x},\bar{y}),$$ called the mean point.

• Least squares condition: The least squares regression line of $Y$ on $X$ minimizes

$$\sum (y_i-{\hat{y}}_i{)}^2,$$ the sum of squared vertical residuals.

• Slope-intercept form of the least squares line:

$$\hat{y}=a+bx,$$ where $b$ is the slope and $$a=\bar{y}-b\bar{x}$$ is the intercept.

• Deviation notation: If

$$u=x-\bar{x},v=y-\bar{y},$$ then the regression equation of $Y$ on $X$ is $$v=b_{yx}u.$$

• Signs of regression coefficients: Each regression coefficient has the same sign as $r$.

• Perfect correlation: If $r=1$ or $r=-1$, the two regression lines coincide and all data points lie on a straight line.

• Zero correlation: If $r=0$, the regression coefficients are zero and the two regression lines are

$$y=\bar{y},x=\bar{x}.$$

• Correlation coefficient from deviations:

$$r=\frac{\sum ((x_i-\bar{x})(y_i-\bar{y}))}{\sqrt{\sum ((x_i-\bar{x}{)}^2)\sum ((y_i-\bar{y}{)}^2)}}.$$

• Alternative computational form of the correlation coefficient:

$$r=\frac{n\sum (xy)-(\sum x)(\sum y)}{\sqrt{[n\sum (x^2)-(\sum x{)}^2][n\sum (y^2)-(\sum y{)}^2]}}.$$

• Choice of regression equation: Estimate $Y$ from the regression line of $Y$ on $X$; estimate $X$ from the regression line of $X$ on $Y$.

• Interpretation: Correlation and regression describe association and prediction. A strong correlation alone does not prove that one variable causes the other.

Worked Methods

1. Assessing linearity with a scatter diagram

• Represent each paired observation $(x,y)$ as a point on a graph.
• Examine the direction of the pattern:

- An upward pattern indicates positive correlation. - A downward pattern indicates negative correlation. - No clear linear pattern indicates no correlation.

• Assess the strength of the relationship from how closely the points follow a straight-line pattern.
• Fit a regression line only when the scatter diagram suggests an approximately linear relationship.

A scatter diagram should therefore be examined before fitting a line. It provides evidence about whether a linear model is reasonable, but it does not establish causation.

2. Calculating the regression line of $Y$ on $X$

For paired data $(x_i,y_i)$:

• Calculate the means:

$$\bar{x}=\frac{\sum x_i}{n},\bar{y}=\frac{\sum y_i}{n}.$$

• Calculate

$$b_{yx}=\frac{\sum [(x_i-\bar{x})(y_i-\bar{y})]}{\sum [(x_i-\bar{x}{)}^2]}.$$

• Substitute the coefficient into

$$y-\bar{y}=b_{yx}(x-\bar{x}).$$

• If required, express the line as

$$\hat{y}=a+bx,$$ using $$a=\bar{y}-b\bar{x}.$$

• Use this line to estimate $Y$ for a specified value of $X$.

Using deviation notation, $$u=x-\bar{x},v=y-\bar{y},$$ the same regression equation is $$v=b_{yx}u.$$

The line minimizes $$\sum (y_i-{\hat{y}}_i{)}^2,$$ the sum of squared vertical residuals.

3. Calculating the regression line of $X$ on $Y$

• Calculate $\bar{x}$ and $\bar{y}$.
• Calculate

$$b_{xy}=\frac{\sum [(x_i-\bar{x})(y_i-\bar{y})]}{\sum [(y_i-\bar{y}{)}^2]}.$$

• Substitute the coefficient into

$$x-\bar{x}=b_{xy}(y-\bar{y}).$$

• Use this line to estimate $X$ for a specified value of $Y$.

This line must not be interchanged with the regression line of $Y$ on $X$: the two lines serve different prediction purposes.

4. Using the correlation coefficient

The correlation coefficient can be calculated from deviations using $$r=\frac{\sum ((x_i-\bar{x})(y_i-\bar{y}))}{\sqrt{\sum ((x_i-\bar{x}{)}^2)\sum ((y_i-\bar{y}{)}^2)}}.$$

Alternatively, use $$r=\frac{n\sum (xy)-(\sum x)(\sum y)}{\sqrt{[n\sum (x^2)-(\sum x{)}^2][n\sum (y^2)-(\sum y{)}^2]}}.$$

Then:

• Check that $-1\le r\le 1$.
• Use the sign of $r$ to determine the direction of association.
• Use the magnitude of $r$ to describe the strength of linear association.
• Calculate $r^2$ when the proportion of explained variation is required.
• Express $r^2$ as a percentage if appropriate.

The regression coefficients can also be obtained from $$b_{yx}=r\left(\frac{s_y}{s_x}\right),b_{xy}=r\left(\frac{s_x}{s_y}\right).$$

Their product provides the check $$b_{xy}{b}_{yx}=r^2.$$

5. Making and interpreting a prediction

• Decide which variable is to be estimated.
• Select the appropriate regression line:

- Use $Y$ on $X$ to estimate $Y$. - Use $X$ on $Y$ to estimate $X$.

• Substitute the known value into the chosen equation.
• Determine whether the prediction is interpolation or extrapolation.
• Interpret the result cautiously, particularly when the correlation is weak or when the value lies outside the observed data range.

Both regression lines should pass through the mean point $$(\bar{x},\bar{y}),$$ which provides a useful check on a calculation.

Where It Goes Wrong

• Fitting a regression line without first examining the scatter diagram; a regression line is useful only when the pattern is approximately linear.

• Using the regression line of $X$ on $Y$ when estimating $Y$, or using the regression line of $Y$ on $X$ when estimating $X$.

• Forgetting that the least squares method minimizes the sum of squared vertical residuals for the regression of $Y$ on $X$:

$$\sum (y_i-{\hat{y}}_i{)}^2.$$

• Omitting the mean point condition: both regression lines must pass through

$$(\bar{x},\bar{y}).$$

• Confusing correlation with prediction or causation: $r$ measures linear association, while the regression equation gives the prediction rule, and even a strong correlation does not prove that one variable causes the other.

• Treating extrapolation as reliable without qualification; predictions outside the observed range may be unreliable, whereas interpolation is usually safer.

What Gets Asked

• Define bivariate data, independent and dependent variables, scatter diagrams, positive correlation, negative correlation, no correlation, regression, residuals, regression coefficients, coefficient of determination, and extrapolation.

• Interpret a scatter diagram and decide whether a linear regression model is reasonable.

• Calculate $\bar{x}$ and $\bar{y}$ from paired data.

• Find the regression line of $Y$ on $X$:

$$y-\bar{y}=b_{yx}(x-\bar{x}).$$

• Find the regression line of $X$ on $Y$:

$$x-\bar{x}=b_{xy}(y-\bar{y}).$$

• Convert a regression equation into the form

$$\hat{y}=a+bx,$$ using $$a=\bar{y}-b\bar{x}.$$

• Calculate a regression coefficient using deviations or standard deviations and the correlation coefficient.

• Calculate $r$ using either the deviation formula or the alternative computational form.

• Use

$$b_{xy}{b}_{yx}=r^2$$ as a check or to determine an unknown quantity.

• Make a prediction using the correct regression line.

• Calculate and interpret $r^2$ as the proportion or percentage of variation explained.

• Check that the regression lines pass through $(\bar{x},\bar{y})$.

• Identify the special cases $r=1$, $r=-1$, and $r=0$.

• Distinguish interpolation from extrapolation and comment on the reliability of a prediction.

• Explain why correlation indicates association rather than proving causation.