Probability

The One Thing

Conditional probability restricts attention to outcomes consistent with known information, while Bayes’ theorem reverses conditional probabilities to update the probability of a cause given observed evidence. Probability distributions then model random outcomes numerically, with expected value and variance summarising their average and spread.

Definitions and Results

• Conditional Probability: The probability of event $A$ occurring when event $B$ has already occurred, written $P(A\mid B)$. For $P(B)>0$,

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$

• Multiplication Theorem:

$$P(A\cap B)=P(A)P(B\mid A)=P(B)P(A\mid B).$$

• Independent Events: Events $A$ and $B$ are independent if occurrence of one does not affect the probability of the other. Thus,

$$P(A\mid B)=P(A),P(B\mid A)=P(B),$$ and $$P(A\cap B)=P(A)P(B).$$

• Complement Rule:

$$P(A^{\prime })=1-P(A).$$

• Addition Rule:

$$P(A\cup B)=P(A)+P(B)-P(A\cap B).$$ If $A$ and $B$ are mutually exclusive, $P(A\cap B)=0$, so $$P(A\cup B)=P(A)+P(B).$$

• Partition of Sample Space: A collection $B_1,B_2,\dots ,B_n$ of mutually exclusive and exhaustive events that covers all possible outcomes.
• Law of Total Probability: If $B_1,B_2,\dots ,B_n$ form a partition of the sample space, then

$$P(A)=\sum _{i}P(B_i)P(A\mid B_i),$$ or explicitly, $$P(A)=P(B_1)P(A\mid B_1)+\cdots +P(B_n)P(A\mid B_n).$$

• Bayes’ Theorem: If $B_1,B_2,\dots ,B_n$ form a partition and $P(A)>0$, then

$$P(B_j\mid A)=\frac{P(B_j)P(A\mid B_j)}{{\sum }_{i}P(B_i)P(A\mid B_i)}.$$ Equivalently, $$P(B_j\mid A)=\frac{P(B_j)P(A\mid B_j)}{P(A)},$$ where $P(A)$ may be found using the law of total probability. Generally, $P(A\mid B)$ and $P(B\mid A)$ have different values.

• Random Variable: A function assigning a real number to each outcome of a random experiment.
• Discrete Random Variable: A random variable taking a finite or countably infinite set of values.
• Probability Distribution: A rule or table giving the probability associated with each possible value of a random variable.
• Probability Mass Function: For a discrete random variable $X$,

$$p(x)=P(X=x).$$ A valid distribution satisfies $$p(x)\ge 0\text{for every }x,\sum p(x)=1.$$

• Mean or Expected Value: For a discrete random variable,

$$E(X)=\sum xp(x).$$ It represents the long-run average value.

• Variance:

$$\mathrm{Var}(X)=E(X^2)-[E(X){]}^2,$$ where $$E(X^2)=\sum x^{2}p(x).$$ Equivalently, $$\mathrm{Var}(X)=\sum x^{2}p(x)-{\left[\sum xp(x)\right]}^2.$$

• Standard Deviation: The positive square root of the variance. It measures spread in the same units as the random variable.
• Bernoulli Trial: A random experiment with exactly two outcomes, usually called success and failure, and a constant probability of success.
• Binomial Distribution: The distribution of the number of successes in $n$ independent Bernoulli trials, each with success probability $p$.
• Parameters of Binomial Distribution: $n$ is the number of trials, $p$ is the probability of success, and

$$q=1-p$$ is the probability of failure.

• Binomial Probability Formula: If $X$ has a binomial distribution with parameters $n$ and $p$, then

$$P(X=r)=\left(\genfrac{}{}{0ex}{}{n}{r}\right)p^r(1-p{)}^{n-r}$$ or $$P(X=r)=\left(\genfrac{}{}{0ex}{}{n}{r}\right)p^r{q}^{n-r},$$ where $r=0,1,\dots ,n$.

• Binomial Mean, Variance and Standard Deviation: For $X\sim \mathrm{Bin}(n,p)$,

$$E(X)=np,\mathrm{Var}(X)=npq,\mathrm{SD}(X)=\sqrt{npq}.$$ The possible values are $0,1,2,\dots ,n$.

Worked Methods

Conditional probability

• Identify the event whose probability is required and the event known to have occurred.
• Restrict attention to the outcomes in the given event $B$.
• Apply

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)},$$ ensuring that $P(B)>0$.

The multiplication theorem can then be used to recover the probability of both events: $$P(A\cap B)=P(B)P(A\mid B)=P(A)P(B\mid A).$$

Complements, unions and independence

• For “not $A$,” use

$$P(A^{\prime })=1-P(A).$$

• For “$A$ or $B$,” use

$$P(A\cup B)=P(A)+P(B)-P(A\cap B).$$

• If the events are mutually exclusive, set $P(A\cap B)=0$.
• If the events are independent, use

$$P(A\cap B)=P(A)P(B),$$ together with $P(A\mid B)=P(A)$ and $P(B\mid A)=P(B)$.

Law of total probability

• Confirm that $B_1,B_2,\dots ,B_n$ are mutually exclusive and exhaustive.
• Find each prior probability $P(B_i)$.
• Find each conditional probability $P(A\mid B_i)$.
• Multiply corresponding terms and add:

$$P(A)=\sum _{i}P(B_i)P(A\mid B_i).$$

Bayes’ theorem

• Identify the possible causes $B_1,B_2,\dots ,B_n$ and the observed evidence $A$.
• Confirm that the causes form a partition.
• Calculate the prior-weighted likelihood

$$P(B_j)P(A\mid B_j).$$

• Calculate the total probability of the evidence:

$$P(A)=\sum _{i}P(B_i)P(A\mid B_i).$$

• Divide to obtain the posterior probability:

$$P(B_j\mid A)=\frac{P(B_j)P(A\mid B_j)}{P(A)}.$$

This method applies to medical testing, quality control, diagnosis, classification, and other situations in which evidence changes an earlier probability.

Constructing and analysing a discrete probability distribution

• List every possible value of the discrete random variable $X$.
• Assign the corresponding probability $p(x)=P(X=x)$.
• Check that $p(x)\ge 0$ for every value.
• Check that

$$\sum p(x)=1.$$

• Calculate the mean:

$$E(X)=\sum xp(x).$$

• Calculate

$$E(X^2)=\sum x^{2}p(x).$$

• Use

$$\mathrm{Var}(X)=E(X^2)-[E(X){]}^2.$$

• Take the positive square root of the variance to obtain the standard deviation.

Binomial probability

• Check the binomial conditions: the number of trials is fixed; trials are independent; each trial has exactly two outcomes; and the probability of success remains constant.
• Define $X$ as the number of successes.
• Identify $n$, $p$, $q=1-p$, and the required value $r$.
• Use

$$P(X=r)=\left(\genfrac{}{}{0ex}{}{n}{r}\right)p^r{q}^{n-r}.$$

• Remember that $r$ can take the values $0,1,2,\dots ,n$.

For a binomial random variable, use the summary results directly: $$E(X)=np,\mathrm{Var}(X)=npq,\mathrm{SD}(X)=\sqrt{npq}.$$

Where It Goes Wrong

• Reversing conditional probabilities: $P(A\mid B)$ and $P(B\mid A)$ generally differ.
• Applying $P(A\mid B)=P(A\cap B)/P(B)$ without checking the required condition $P(B)>0$.
• Treating events as independent without establishing that one event does not affect the probability of the other.
• Omitting the subtraction of $P(A\cap B)$ in the addition rule when events are not mutually exclusive.
• Using the law of total probability or Bayes’ theorem without ensuring that $B_1,B_2,\dots ,B_n$ are mutually exclusive and exhaustive.
• Applying the binomial model when the number of trials is not fixed, trials are not independent, there are more than two outcomes, or the probability of success is not constant.

What Gets Asked

• Calculate a conditional probability from $P(A\cap B)$ and $P(B)$.
• Use the multiplication theorem to find the probability of simultaneous events.
• Apply complement, addition and independence rules.
• Calculate a total probability from a partition of the sample space.
• Use Bayes’ theorem to find a revised probability after observing evidence.
• Identify a random variable and construct or verify a discrete probability distribution.
• Calculate $E(X)$, $E(X^2)$, variance and standard deviation from a probability mass function.
• Determine whether a situation satisfies the conditions for a binomial distribution.
• Calculate $P(X=r)$ using

$$\left(\genfrac{}{}{0ex}{}{n}{r}\right)p^r{q}^{n-r}.$$

• Find the possible values, mean, variance and standard deviation of a binomial random variable.