Probability

The One Thing

Conditional probability evaluates an event under given information, while Bayes’ theorem reverses conditional probabilities to infer the likelihood of a cause from observed evidence. Random variables then represent uncertain outcomes numerically through probability distributions, expectation, variance, and standard deviation.

Definitions and Results

• Conditional Probability: The probability of event $A$ occurring when event $B$ has already occurred, written $P(A\mid B)$. If $P(B)>0$,

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$ The condition $B$ changes the relevant sample space from all outcomes to those in which $B$ has occurred.

• Multiplication Rule: For events $A$ and $B$,

$$P(A\cap B)=P(A)P(B\mid A)=P(B)P(A\mid B).$$

• Independent Events: Events $A$ and $B$ are independent if the occurrence of one does not affect the probability of the other. Thus,

$$P(A\cap B)=P(A)P(B).$$ When the conditional probabilities are defined, $$P(A\mid B)=P(A),P(B\mid A)=P(B).$$ Independence is stronger than mutual exclusivity; mutually exclusive events with positive probabilities cannot be independent.

• Partition of Sample Space: A collection of mutually exclusive and exhaustive events whose union is the entire sample space. The events must be pairwise disjoint, and their probabilities must add to $1$.

• Law of Total Probability: If $B_1,B_2,\dots ,B_n$ form a partition of the sample space, then

$$P(A)=\sum _{i}P(B_i)P(A\mid B_i),$$ or explicitly, $$P(A)=P(B_1)P(A\mid B_1)+\cdots +P(B_n)P(A\mid B_n).$$

• Bayes’ Theorem: The probability of a possible cause $B_i$, given an observed event $A$, is

$$P(B_i\mid A)=\frac{P(B_i)P(A\mid B_i)}{{\sum }_{j}P(B_j)P(A\mid B_j)}.$$ Equivalently, $$P(B_i\mid A)=\frac{P(B_i)P(A\mid B_i)}{P(B_1)P(A\mid B_1)+\cdots +P(B_n)P(A\mid B_n)}.$$ Bayes’ theorem reverses conditional probabilities by combining prior probabilities with likelihoods. It is useful in medical testing, quality control, fault detection, weather prediction, and decision-making when evidence updates prior probabilities.

• Random Experiment: An experiment whose outcome cannot be predicted with certainty in advance, although all possible outcomes are known.

• Random Variable: A function assigning a real number to each outcome of a random experiment.

• Discrete Random Variable: A random variable taking a finite or countably infinite set of values.

• Probability Distribution: A table or rule listing the possible values of a discrete random variable and their corresponding probabilities.

• Probability Mass Function: For a discrete random variable $X$,

$$p(x)=P(X=x).$$ It must satisfy $$p(x)\ge 0$$ for every possible value, and $$\sum p(x)=1.$$

• Cumulative Distribution Function: For a random variable $X$,

$$F(x)=P(X\le x).$$

• Mathematical Expectation: For a discrete random variable $X$ with values $x_1,x_2,\dots ,x_n$,

$$E(X)=\sum x_{i}p(x_i).$$ It is the weighted average, or long-run average, of the outcomes. The expected value need not be one of the actual values taken by $X$.

• Linearity of Expectation: For constants $a$ and $b$,

$$E(aX+b)=aE(X)+b.$$

• Variance: A measure of the spread of values around the mean $\mu$:

$$\mathrm{Var}(X)=E[(X-\mu {)}^2]=E(X^2)-[E(X){]}^2.$$

• Second Moment:

$$E(X^2)=\sum x_i^{2}p(x_i).$$

• Variance Transformation: For constants $a$ and $b$,

$$\mathrm{Var}(aX+b)=a^2\mathrm{Var}(X).$$

• Standard Deviation: The positive square root of the variance:

$$\sigma =\sqrt{\mathrm{Var}(X)}.$$

Worked Methods

Conditional Probability

• Identify the event being sought, $A$, and the event already known to have occurred, $B$.
• Check that $P(B)>0$.
• Use

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}.$$

• If the intersection is required first, use the multiplication rule:

$$P(A\cap B)=P(A\mid B)P(B)$$ or $$P(A\cap B)=P(B\mid A)P(A).$$

Testing Independence

• Calculate $P(A\cap B)$.
• Calculate $P(A)P(B)$.
• Compare the two quantities.
• Conclude that $A$ and $B$ are independent exactly when

$$P(A\cap B)=P(A)P(B).$$ Where defined, this is equivalent to checking $$P(A\mid B)=P(A)\text{or}P(B\mid A)=P(B).$$

Applying the Law of Total Probability

• Confirm that $B_1,B_2,\dots ,B_n$ are mutually exclusive and exhaustive.
• Express the target event $A$ according to the partition.
• Multiply each partition probability by the corresponding conditional probability.
• Add the results:

$$P(A)=\sum _{i}P(B_i)P(A\mid B_i).$$

• For the named partition events, write

$$P(A)=P(B_1)P(A\mid B_1)+\cdots +P(B_n)P(A\mid B_n).$$

Applying Bayes’ Theorem

• Identify the observed evidence $A$ and the possible cause $B_i$.
• Determine the prior probability $P(B_i)$.
• Determine the likelihood $P(A\mid B_i)$.
• Calculate the total probability of the evidence:

$$P(A)=\sum _{j}P(B_j)P(A\mid B_j).$$

• Substitute into

$$P(B_i\mid A)=\frac{P(B_i)P(A\mid B_i)}{{\sum }_{j}P(B_j)P(A\mid B_j)}.$$

• Interpret the result as the probability of cause $B_i$ after evidence $A$ has been observed.

Constructing a Discrete Probability Distribution

• List every possible value $x_1,x_2,\dots ,x_n$ of the discrete random variable $X$.
• Assign the probability

$$p(x_i)=P(X=x_i)$$ to each value.

• Check that

$$p(x_i)\ge 0$$ for every $i$.

• Check that

$$\sum p(x_i)=1.$$

• If required, obtain the cumulative distribution function using

$$F(x)=P(X\le x).$$

Calculating Expectation, Variance, and Standard Deviation

• Use the distribution values and probabilities to calculate

$$E(X)=\sum x_{i}p(x_i).$$

• Calculate the second moment:

$$E(X^2)=\sum x_i^{2}p(x_i).$$

• Calculate the variance:

$$\mathrm{Var}(X)=E(X^2)-[E(X){]}^2.$$

• Calculate the standard deviation:

$$\sigma =\sqrt{\mathrm{Var}(X)}.$$

• For a transformed variable $aX+b$, use

$$E(aX+b)=aE(X)+b$$ and $$\mathrm{Var}(aX+b)=a^2\mathrm{Var}(X).$$

Where It Goes Wrong

• Treating $P(A\mid B)$ as $P(B\mid A)$; Bayes’ theorem is required to reverse the conditional probability.
• Using

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}$$ without checking the condition $P(B)>0$.

• Applying the law of total probability or Bayes’ theorem when $B_1,B_2,\dots ,B_n$ are not mutually exclusive and exhaustive.
• Forgetting that the probabilities in a discrete probability distribution must all be non-negative and must sum to $1$.
• Assuming that mutually exclusive events with positive probabilities are independent; mutual exclusivity and independence are distinct conditions.
• Calculating variance as $E(X^2)$ alone rather than using

$$\mathrm{Var}(X)=E(X^2)-[E(X){]}^2.$$

What Gets Asked

• Calculate a conditional probability $P(A\mid B)$ from an intersection and a marginal probability.
• Use the multiplication rule to calculate $P(A\cap B)$.
• Determine whether two events are independent.
• Verify whether events form a partition of the sample space.
• Apply the law of total probability to calculate $P(A)$.
• Use Bayes’ theorem to calculate $P(B_i\mid A)$ from prior probabilities and observed evidence.
• Define a random experiment, random variable, discrete random variable, probability distribution, or probability mass function.
• Construct or check a discrete probability distribution.
• Calculate a cumulative distribution function $F(x)=P(X\le x)$.
• Calculate $E(X)$, $E(X^2)$, variance, and standard deviation.
• Apply the transformations

$$E(aX+b)=aE(X)+b$$ and $$\mathrm{Var}(aX+b)=a^2\mathrm{Var}(X).$$