Correlation Analysis

The One Thing

Covariance shows how two variables vary together, whereas Karl Pearson’s coefficient of correlation standardizes this relationship into a unit-free value between $-1$ and $+1$. Correlation measures linear association, not necessarily a cause-and-effect relationship.

Definitions and Results

• Bivariate data: Data consisting of paired observations of two variables, such as study hours and marks.

• Correlation: A statistical measure showing the direction and strength of the relationship between two variables.

• Positive correlation: A relationship in which larger values of one variable are generally associated with larger values of the other.

• Negative correlation: A relationship in which larger values of one variable are generally associated with smaller values of the other.

• Zero correlation: A situation in which there is no linear relationship between the two variables.

• Covariance: A measure of how two variables vary together. Its sign shows the direction of association, while its magnitude depends on the measurement units. For paired observations with means $\bar{x}$ and $\bar{y}$, population covariance is

$$\mathrm{Cov}(X,Y)=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{n}.$$ For a sample, covariance is commonly calculated as $$s_{xy}=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{n-1}.$$

• Karl Pearson’s coefficient of correlation: A unit-free measure of linear correlation, usually denoted by $r$, whose value lies between $-1$ and $+1$:

$$r=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{\sqrt{\sum (x-\bar{x}{)}^2\times \sum (y-\bar{y}{)}^2}}.$$

• Computational formula for Karl Pearson’s coefficient:

$$r=\frac{n\sum xy-(\sum x)(\sum y)}{\sqrt{[n\sum x^2-(\sum x{)}^2][n\sum y^2-(\sum y{)}^2]}}.$$

• Relationship between covariance and correlation:

$$r=\frac{\mathrm{Cov}(X,Y)}{{\sigma }_x{\sigma }_y},$$ where ${\sigma }_x$ and ${\sigma }_y$ are the standard deviations of $X$ and $Y$.

• Perfect positive correlation: $r=+1$, where all observations follow an exact increasing straight-line pattern.

• Perfect negative correlation: $r=-1$, where all observations follow an exact decreasing straight-line pattern.

• Mean deviation from the mean: For each observation, the difference between its value and the corresponding mean, such as $x-\bar{x}$ or $y-\bar{y}$.

• Scatter diagram: A graph of paired observations used to obtain a visual indication of the direction and strength of correlation. Points rising from left to right suggest positive correlation, while points falling from left to right suggest negative correlation.

• Arithmetic means of paired observations: For $(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n)$,

$$\bar{x}=\frac{\sum x}{n},\bar{y}=\frac{\sum y}{n}.$$

• Range of the coefficient: The coefficient always satisfies

$$-1\le r\le +1.$$

• Interpretation of the sign: $r>0$ indicates positive correlation, $r<0$ indicates negative correlation, and $r=0$ indicates no linear correlation.

• Interpretation of magnitude: Values close to $1$ or $-1$ indicate strong linear correlation, whereas values close to $0$ indicate weak linear correlation.

• Units: Covariance has units equal to the product of the units of the two variables. Pearson’s coefficient has no units.

• Zero standard deviation condition: If either variable has zero standard deviation, Pearson’s coefficient cannot be calculated because the denominator becomes zero.

• Effect of transformations: Correlation is unchanged if a constant is added to or subtracted from every observation. It is also unchanged when both variables are multiplied by positive constants. Multiplication by a negative constant reverses the sign for that variable.

• Nature of the relationship measured: Pearson’s coefficient measures linear association. A value near zero does not rule out a strong non-linear relationship.

• Interpretive limitations: A positive or negative correlation does not prove that one variable causes changes in the other. Outside factors or coincidence may explain the association.

Worked Methods

Method 1: Calculating covariance

• Identify the paired observations:

$$(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n).$$

• Calculate the arithmetic means:

$$\bar{x}=\frac{\sum x}{n},\bar{y}=\frac{\sum y}{n}.$$

• Calculate the mean deviations:

$$x-\bar{x},y-\bar{y}.$$

• Multiply the paired deviations:

$$(x-\bar{x})(y-\bar{y}).$$

• Add the products of deviations.

• For a population, divide by $n$:

$$\mathrm{Cov}(X,Y)=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{n}.$$

• For a sample, divide by $n-1$:

$$s_{xy}=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{n-1}.$$

• Interpret the sign. A positive covariance indicates that the variables tend to increase together; a negative covariance indicates that one tends to increase as the other decreases. Its numerical magnitude must be interpreted with reference to the units of measurement.

Method 2: Calculating Karl Pearson’s coefficient using deviations from the mean

• Find the means:

$$\bar{x}=\frac{\sum x}{n},\bar{y}=\frac{\sum y}{n}.$$

• Calculate the deviations:

$$x-\bar{x},y-\bar{y}.$$

• Form the products of deviations:

$$(x-\bar{x})(y-\bar{y}).$$

• Calculate the squared deviations:

$$(x-\bar{x}{)}^2,(y-\bar{y}{)}^2.$$

• Substitute the totals into:

$$r=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{\sqrt{\sum (x-\bar{x}{)}^2\times \sum (y-\bar{y}{)}^2}}.$$

• Interpret both the sign and magnitude of $r$. The sign gives the direction, while the magnitude gives the strength of the linear association.

Method 3: Calculating Karl Pearson’s coefficient using the computational formula

• Calculate the following quantities from the paired observations:

$$\sum x,\sum y,\sum xy,\sum x^2,\sum y^2.$$

• Count the number of paired observations, $n$.

• Substitute the values into:

$$r=\frac{n\sum xy-(\sum x)(\sum y)}{\sqrt{[n\sum x^2-(\sum x{)}^2][n\sum y^2-(\sum y{)}^2]}}.$$

• Check that the result lies between $-1$ and $+1$.

• Interpret the result in relation to the data, scatter diagram, units, and context.

Method 4: Interpreting a scatter diagram

• Plot each paired observation as a point.

• Examine the overall pattern:

- Points rising from left to right suggest positive correlation. - Points falling from left to right suggest negative correlation. - A dispersed pattern without a linear trend suggests zero or weak linear correlation.

• Assess the strength from the closeness of the points to an apparent straight-line pattern.

• Check for extreme observations, because unusual values can strongly affect covariance and Pearson’s coefficient.

• Remember that a scatter diagram and Pearson’s coefficient describe association, not causation, and that a weak linear pattern does not exclude a strong non-linear relationship.

Where It Goes Wrong

• Treating covariance as directly comparable across datasets without considering its units; covariance has units equal to the product of the units of the two variables.

• Forgetting to distinguish the population denominator $n$ from the commonly used sample denominator $n-1$.

• Omitting either the product of deviations $(x-\bar{x})(y-\bar{y})$ or the squared deviations $(x-\bar{x}{)}^2$ and $(y-\bar{y}{)}^2$ when applying Pearson’s formula.

• Interpreting the sign of $r$ as its strength, rather than using the sign for direction and the magnitude for strength.

• Attempting to calculate $r$ when either variable has zero standard deviation, making the denominator zero.

• Concluding that correlation proves causation, or assuming that a value near zero rules out a strong non-linear relationship.

What Gets Asked

• Define bivariate data, correlation, covariance, positive correlation, negative correlation, zero correlation, Karl Pearson’s coefficient, mean deviation from the mean, and a scatter diagram.

• Calculate $\bar{x}$ and $\bar{y}$ for paired observations.

• Construct deviation tables containing $x-\bar{x}$, $y-\bar{y}$, $(x-\bar{x})(y-\bar{y})$, $(x-\bar{x}{)}^2$, and $(y-\bar{y}{)}^2$.

• Calculate population covariance using

$$\mathrm{Cov}(X,Y)=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{n}.$$

• Calculate sample covariance using

$$s_{xy}=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{n-1}.$$

• Calculate Pearson’s coefficient using the deviation formula:

$$r=\frac{\sum [(x-\bar{x})(y-\bar{y})]}{\sqrt{\sum (x-\bar{x}{)}^2\times \sum (y-\bar{y}{)}^2}}.$$

• Calculate Pearson’s coefficient using the computational formula:

$$r=\frac{n\sum xy-(\sum x)(\sum y)}{\sqrt{[n\sum x^2-(\sum x{)}^2][n\sum y^2-(\sum y{)}^2]}}.$$

• Explain the relationship

$$r=\frac{\mathrm{Cov}(X,Y)}{{\sigma }_x{\sigma }_y}.$$

• Interpret values of $r$, including $r=+1$, $r=-1$, $r=0$, positive values, negative values, and values close to $1$, $-1$, or $0$.

• Compare covariance with Pearson’s coefficient in terms of standardization and units.

• Interpret a scatter diagram showing points rising or falling from left to right.

• Explain the effects of adding or subtracting constants and multiplying variables by positive or negative constants.

• Identify why Pearson’s coefficient cannot be calculated when either variable has zero standard deviation.

• Explain why correlation does not establish causation and why extreme observations, the scatter diagram, units, context, and possible non-linear relationships must be considered.