IA Probability Lent 2007

2. AXIOMATIC PROBABILITY

2.1 The axioms

The formulation for classical probability in which all outcomes or points in the sample space are equally likely is too restrictive to develop a useful theory of probability. In this section we give a general definition. For the moment we will assume that the sample space of outcomes Ω is either a finite or countable set.

A probability distribution ℙ on Ω is a real-valued function defined on subsets (events) of Ω satisfying the following three conditions:

1. 0 ⩽ ℙ(A) ⩽ 1, for all A ⊆ Ω.

2. ℙ(Ω) = 1.

3. For a finite or infinite collection of disjoint events {A_{i}}_{i}, that is A_{i} ∩ A_{j} = ∅ for

i ≠ j, we have

ℙ (⋃_{i} A_{i}) = ∑_{i} ℙ (A_{i}) .

First observe the following consequence of Axiom 3.

4. ℙ (∅) = 0. This follows because if we take A_{i} = ∅, for each i in Axiom 3 then we get a contradiction unless ℙ (∅) = 0. Note that Axiom 3 here implies the property 3. that we had in Chapter 1 if we take A_{i} = ∅ for i > 2, A_{1} = A and A_{2} = B. This of course extends to n disjoint events A_{1}, …, A_{n} for which ℙ ( ⋃_{i=1}^{n} A_{i}) = ∑_{i=1}^{n} ℙ(A_{i}). An important consequence of this axiom is a continuity property of probabilities in that for events B_{1} ⊆ B_{2} ⊆ B_{3} ⊆ ⋯ we have

ℙ (⋃_{i=1}^{∞} B_{i}) = lim_{i→∞} ℙ (B_{i}) .  (2.1)

This follows by taking A_{1} = B_{1} and for i > 1, A_{i} = B_{i} ∩ B_{i−1}^{c}, then the {A_{i}} are disjoint

with ⋃_{j=1}^{i} A_{j} = B_{i} and ⋃_{i=1}^{∞} A_{i} = ⋃_{i=1}^{∞} B_{i} so that ℙ (B_{i}) = ∑_{j=1}^{i} ℙ (A_{j}), and

ℙ (⋃_{i=1}^{∞} B_{i}) = ℙ (⋃_{i=1}^{∞} A_{i}) = ∑_{i=1}^{∞} ℙ (A_{i}) = lim_{i→∞} ∑_{j=1}^{i} ℙ (A_{j}) = lim_{i→∞} ℙ (B_{i}) .

As in Chapter 1 we may deduce that 5. ℙ(A^{c}) = 1 − ℙ(A).

6. If A ⊆ B then ℙ(A) ⩽ ℙ(B).
7. ℙ(A ⋃ B) = ℙ(A) + ℙ(B) − ℙ(A ∩ B).

Using (2.1) and Property 5. we may see a corresponding continuity property for probabilities on decreasing events, in that for events B_{1} ⊇ B_{2} ⊃ B_{3} ⊇ ⋯, we have

ℙ (⋂_{i=1}^{∞} B_{i}) = lim_{i→∞} ℙ (B_{i}) .

Boole’s Inequality We may deduce from Property 7, that for any events A_{1}, …, A_{n},

ℙ (⋃_{i=1}^{n} A_{i}) ⩽ ∑_{i=1}^{n} ℙ (A_{i}) .

The case n = 2 follows from Property 7 (and Axiom 1) immediately and the case for general n by induction. We may use (2.1) to extend Boole’s Inequality to the case of countably many events A_{1}, A_{2}, … so that

ℙ (⋃_{i=1}^{∞} A_{i}) ⩽ ∑_{i=1}^{∞} ℙ (A_{i}) .

2.2 Inclusion-exclusion formula

The inclusion-exclusion formula gives a method of calculating that at least one of a number of events occurs.

Theorem 2.2 (Inclusion-exclusion) For any events A_{1}, A_{2}, …, A_{n},

\begin{aligned}ℙ (⋃_{i=1}^{n} A_{i}) = ∑_{i} ℙ (A_{i}) − & ∑_{i_{1}<i_{2}} ℙ (A_{i_{1}} ∩ A_{i_{2}}) \\ & + ∑_{i_{1}<i_{2}<i_{3}} ℙ (A_{i_{1}} ∩ A_{i_{2}} ∩ A_{i_{3}}) − ⋯ + (−1)^{n−1}ℙ (∩_{i=1}^{n}A_{i}) . \end{aligned}

Proof. The proof is by induction on n. The case n = 2 is just Property 7. Assume the result holds for any n events and consider A_{1}, …, A_{n+1}. Now see from Property 7. again that

ℙ ( ⋃_{1}^{n+1} A_{i}) = ℙ (⋃_{1}^{n} A_{i}) + ℙ (A_{n+1}) − ℙ ((⋃_{1}^{n} A_{i}) ∩ A_{n+1})
= ℙ (⋃_{1}^{n} A_{i}) + ℙ (A_{n+1}) − ℙ (⋃_{1}^{n}(A_{i} ∩ A_{n+1})) ,  (2.3)

and then apply the inductive hypothesis to the first and the third terms to complete the induction and obtain the conclusion. □

Example 2.4 What is the probability that in a round of Bridge at least one player holds exactly two aces and two kings? Let Ω be the sample space of all possible (unordered) hands for 4 players, then the number of points in Ω is N = \binom{52}{13 13 13 13}. For each player i, i = 1, 2, 3, 4, let A_{i} be the event that player i holds exactly two aces and two kings. Then

ℙ (A_{i}) = \begin{matrix}\binom{4}{2} & \binom{4}{2} & \binom{44}{9} & \binom{39}{13 13 13} \\ ( & {52 \atop 13 13 13 13} & ) \end{matrix} = \begin{matrix}\binom{4}{2} & \binom{4}{2} & \binom{44}{9} \\ ( & {52 \atop 13} & ) \end{matrix};

think of choosing the aces for player i, then the kings and then 9 other cards and finally distributing the remaining 39 cards among the other three players. If i ≠ j, then

ℙ (A_{i} ∩ A_{j}) = \begin{matrix}\binom{4}{2} & \binom{4}{2} & \binom{44}{9 9 26} & \binom{26}{13} \\ ( & {52 \atop 13 13 13 13} & ) \end{matrix} = \begin{matrix}\binom{4}{2} & \binom{4}{2} & \binom{44}{9 9 26} \\ ( & {52 \atop 13 13 26} & ) \end{matrix};

as before, think of picking the aces and kings for player i with the remaining aces and kings going to player j, and then picking 9 other cards for each of i and j. If i, j, k are unequal then A_{i} ∩ A_{j} ∩ A_{k} = ∅, so that by inclusion exclusion we have the required probability is

ℙ (⋃_{i=1}^{4} A_{i}) = 4 × \begin{matrix}\binom{4}{2} & \binom{4}{2} & \binom{44}{9} \\ ( & {52 \atop 13} & ) \end{matrix} − \binom{4}{2} × \begin{matrix}\binom{4}{2} & \binom{4}{2} & \binom{44}{9 9 26} \\ ( & {52 \atop 13 13 26} & ) \end{matrix}.  □

Example 2.5 Suppose that n students leave their n coats outside a lecture room and when they leave they pick up their coats at random. What is the probability that at least one student has his own coat? Let Ω consist of all permutations of 1, …, n, so that if (i_{1}, …, i_{n}) ∈ Ω then i_{j} is the index of the coat got by student j. Denote by A_{i} the event that student i gets his own coat. Then for i_{1} < i_{2} < ⋯ < i_{r}, the probability that all of the students i_{1}, …, i_{r} get their own coats is

ℙ ( ⋂_{k=1}^{r} A_{i_{k}}) = \frac{(n − r)!}{n!},

since the numerator is the number of ways that the remaining n−r coats may be permuted among the remaining n − r students. Then by inclusion-exclusion we have

\begin{aligned}ℙ (⋃_{i=1}^{n} A_{i}) & = ∑_{r=1}^{n} [(−1)^{r−1} ∑_{i_{1}<···<i_{r}} ℙ ( ⋂_{k=1}^{r} A_{i_{k}})] \\ & = ∑_{r=1}^{n}(−1)^{r−1}\binom{n}{r}\frac{(n − r)!}{n!} = ∑_{r=1}^{n}(−1)^{r−1} \frac{1}{r!}, \end{aligned}

so that for large n the probability is approximately equal to

1 − \frac{1}{2 !} + \frac{1}{3 !} − \frac{1}{4 !} + ⋯ = 1 − e^{−1} ≈ 0·632 .  □

Corollary 2.6 (Bonferroni’s Inequalities) For any events A_{1}, A_{2}, …, A_{n} and for any r, 1 ⩽ r ⩽ n,

\begin{aligned}ℙ (⋃_{i=1}^{n} A_{i}) \begin{matrix}⩽ \\ or \\ ⩾ \end{matrix} & ∑_{i} ℙ (A_{i}) − ∑_{i_{1}<i_{2}} ℙ (A_{i_{1}} ∩ A_{i_{2}}) \\ & + ∑_{i_{1}<i_{2}<i_{3}} ℙ (A_{i_{1}} ∩ A_{i_{2}} ∩ A_{i_{3}}) − ⋯ + (−1)^{r−1} ∑_{i_{1}<···<i_{r}} ℙ (A_{i_{1}} ∩ ⋯ ∩ A_{i_{r}}) , \end{aligned}

according as r is odd or even. That is, if the sum in the inclusion-exclusion formula is truncated after r terms it overestimates or underestimates the probability of the union of the n events according as r is odd or even.

Proof. The proof proceeds by induction on n. Assume it is true for n, then for n + 1 events it is true when r = n + 1, by the inclusion-exclusion formula, and for r ⩽ n, apply the inductive hypothesis to probability of the two unions of n events in relation (2.3) to get the result. □

2.3 Conditional probability

Suppose that B ⊆ Ω is an event with ℙ(B) > 0. For any event A ⊆ Ω, the conditional probability of A given B is

ℙ ( A | B) = \frac{ℙ(A ∩ B)}{ℙ(B)};

it is the probability that the event A occurs, that is that the outcome of the ‘experiment’ is in A, if it is known that the outcome is in the event B, i.e., that B occurs. Example 2.7 What is the probability that a Bridge hand contains the ace of hearts given that it contains exactly 5 hearts? Let A be the event that the hand contains the ace of hearts and B the event that the hand contains exactly 5 hearts. Then

ℙ(B) = \begin{matrix}\binom{13}{5} & \binom{39}{8} \\ ( & {52 \atop 13} & ) \end{matrix} and ℙ(A ∩ B) = \begin{matrix}\binom{12}{4} & \binom{39}{8} \\ ( & {52 \atop 13} & ) \end{matrix},

whence ℙ( A | B) = \binom{12}{4}/\binom{13}{5} = \frac{5}{13}. □

The first thing to observe is that ℙ ( · | B) is a probability distribution on the sample space B, because it satisfies the axioms for a probability distribution as follows:

1. For C ⊆ B, ℙ( C | B) = ℙ(C)/ℙ(B), so that 0 ⩽ ℙ(C | B) ⩽ 1.

2. ℙ( B | B) = ℙ(B)/ℙ(B) = 1.
3. For disjoint events C_{1}, C_{2}, … in B,

\begin{aligned}ℙ (⋃_{i=1}^{∞} C_{i} | B) & = \frac{ℙ ( ⋃_{i=1}^{∞} C_{i} ∩ B)}{ℙ(B)} = \frac{ℙ ( ⋃_{i=1}^{∞} C_{i})}{ℙ(B)} \\ & = \frac{∑_{i=1}^{∞} ℙ(C_{i})}{ℙ(B)} = ∑_{i=1}^{∞} ℙ ( C_{i} | B) . \end{aligned}

Multiplication rule The next thing to notice is the multiplication rule that

ℙ(A ∩ B) = ℙ( A | B)ℙ(B),

so that the probability of two events occurring can be broken up into calculating successive probabilities–firstly the probability that B occurs and then given that B has occurred the probability that A occurs. This is one of two central procedures for calculating probabilities (the second is the Law of Total Probability introduced below).

Example 2.8 Suppose that two students are selected, without replacement, from a class of 5 women and 13 men. What is the probability that the first student selected is a man and the second is a woman? Let B be the event that the first is a man and A the event that the second is a woman, then ℙ(A ∩ B) = ℙ( A | B)ℙ(B) = \frac{5}{17} × \frac{13}{18}. □

More generally we can write down the multiplication rule for events A_{1}, …, A_{n},

ℙ (A_{1} ∩ A_{2} ⋯ ∩ A_{n}) = ℙ(A_{1})ℙ( A_{2} | A_{1})ℙ( A_{3} | A_{1} ∩ A_{2}) ⋯ ℙ( A_{n} | A_{1} ∩ ⋯ ∩ A_{n−1}).

Example 2.9 In drawing three cards without replacement from a pack what is the probability of three successive aces? Here A_{i} would be the event that an ace is obtained on draw i, i = 1, 2, 3, then

ℙ (A_{1} ∩ A_{2} ∩ A_{3}) = ℙ(A_{1})ℙ( A_{2} | A_{1})ℙ( A_{3} | A_{1} ∩ A_{2})
= \frac{4}{52} × \frac{3}{51} × \frac{2}{50}.  □

Law of Total Probability A collection {B_{i}}_{i=1}^{∞} of disjoint events for which ⋃_{i=1}^{∞} B_{i} = Ω is said to be a partition of the sample space Ω. For any partition of the sample space, {B_{i}}, and for any event A, we may write

ℙ(A) = ∑_{i=1}^{∞} ℙ (A ∩ B_{i}) = ∑_{i} ℙ ( A | B_{i}) ℙ(B_{i}),  (2.10)

where the second summation extends only over those events B_{i} in the partition for which ℙ(B_{i}) > 0. The identity in (2.10) is known as the Law of Total Probability. It follows immediately from Axiom 3 and the multiplication rule, because the event A may be represented as A = ⋃_{i=1}^{∞} (A ∩ B_{i}).

Example 2.11 An urn contains b black balls and r red balls from which two balls are drawn without replacement. What is the probability that the second ball drawn is black? Let A represent the event that the second ball is black and B the event that the first ball is black. Then B and B^{c} form a partition of the sample space (think of the other events in the partition as being ∅). Then

\begin{aligned}ℙ(A) & = ℙ ( A | B) ℙ(B) + ℙ ( A | B^{c}) ℙ(B^{c}) \\ & = (\frac{b − 1}{b + r − 1}) (\frac{b}{b + r} ) + (\frac{b}{b + r − 1}) (\frac{r}{b + r} ) = \frac{b}{b + r} . \end{aligned}

This probability may be seen to be the same as the probability that the second ball is black when sampling with replacement. □

Example 2.12 A survey of US voters shows the following figures for proportions of voters registered with the main parties and the proportions of voters registered for each of the parties who declare an intention to vote for Dubya.

Registered  Proportion for Bush
Democratic  45%  10%
Republican  35%  60%
Not affiliated  20%  40%

Suppose that a voter is chosen at random, what is the probability (s)he is a Bush voter? Here the partition of the sample space is D (Democrat), R (Republican) and NA (No affiliation), and if B is the event the voter is a Bush supporter then

ℙ(B) = ℙ( B | D)ℙ(D) + ℙ( B | R)ℙ(R) + ℙ( B | NA)ℙ(NA)
= 0·1 × 0·45 + 0·6 × 0·35 + 0·4 × 0·2 = 0·335.  □

Bayes’ Theorem For any events A and B, for which ℙ(A) > 0 and ℙ(B) > 0, we have

ℙ ( B | A) = \frac{ℙ ( A | B) ℙ(B)}{ℙ(A)}.  (2.13)

Example 2.14 In Example 2.12, given that the voter chosen is a Bush supporter, what is the probability that (s)he is a Republican?

ℙ( R | B) = \frac{ℙ ( B | R) ℙ(R)}{ℙ(B)} = 0·6 × 0·35/0·335 ≈ 0·63.  □

Combining the Law of Total Probability with the statement in (2.13), gives the general statement of Bayes’ Theorem:

Theorem 2.15 (Bayes’ Theorem) Suppose that {B_{i}}_{i} is a partition of the sample space and that A is an event for which ℙ(A) > 0. Then for any event, B_{i}, in the partition with ℙ(B_{i}) > 0, we have

ℙ ( B_{i} | A) = \frac{ℙ ( A | B_{i}) ℙ (B_{i})}{∑_{j} ℙ ( A | B_{j}) ℙ (B_{j})},

where the summation in the denominator extends over all j for which ℙ(B_{j}) > 0.

Example 2.16 Consider a diagnostic test for some disease for which the outcome of the test is either positive, +, or negative, −, and which is 99% accurate, so that if D represents the event that the patient has the disease then

ℙ ( + | D) = 0·99 = ℙ ( − | D^{c}) .

Suppose that 0·1% of patients have the disease. A patient is chosen at random and tests positive, what is the probability that (s)he has the disease? Then from Bayes’ Theorem

\begin{aligned}ℙ ( D | +) & = \frac{ℙ ( + | D) ℙ(D)}{ℙ ( + | D) ℙ(D) + ℙ ( + | D^{c}) ℙ(D^{c})} \\ & = \frac{0·99 × 0·001}{0·99 × 0·001 + 0·01 × 0·999} ≈ 0·09 . \end{aligned}

At first sight this conclusion, that the probability that someone testing positive has a somewhat low probability of having the disease, may be counter intuitive. It arises because the probability of a ‘false positive’, ℙ ( + | D^{c}), is large compared to the probability of the disease in the population, ℙ (D). Notice that

ℙ ( D | +) = \frac{1}{1 + (ℙ ( + | D^{c}) ℙ(D^{c})/ℙ ( + | D) ℙ(D))};

typically ℙ(D^{c}) and ℙ ( + | D) will be close to 1 so that ℙ ( D | +) will be large or small according as the ratio ℙ ( + | D^{c}) /ℙ (D) is small or large. This becomes less mysterious when you consider a population of 1,000 patients with just one person suffering from the disease. Of the 999 not suffering from the disease there will be about 10 who will test positive so there will be about 11 in the population who will test positive; given that the person selected has tested positive then there is about a 1 in 11 chance that the person is the one suffering from the disease. A similar calculation to the above shows that ℙ ( D | −) ≈ 0·000001. □

Example 2.17 Simpson’s paradox One example of conditional probability that appears counter-intuitive when first seen is the following situation which can arise frequently. Consider one individual chosen at random from 50 men and 50 women applicants to a particular College. Figures on the 100 applicants are given in the following table indicating whether they were educated at a state school or at an independent school and whether they were admitted or rejected.

All applicants  Admitted  Rejected  % Admitted
State  25  25  50%
Independent  28  22  56%

Note that overall the probability that an applicant is admitted is 0·53, but conditional on the candidate being from an independent school the probability is 0·56 while conditional on being from a state school the probability is lower at 0·50. Suppose that when we break down the figures for men and women we have the following figures.

Men only  Admitted  Rejected  % Admitted
State  15  22  41%
Independent  5  8  38%
Women only  Admitted  Rejected  % Admitted
State  10  3  77%
Independent  23  14  62%

It may now be seen that now for both men and women the conditional probability of being admitted is higher for state school applicants, at 0·41 and 0·77, respectively. This may seem to be a surprising result in that while the aggregate data suggests that independent school applicants have a better chance of being admitted, the individual tables for both men and women show that in both cases state school applicants have a higher acceptance rate. A result of this type in tables of this sort (called contingency tables in statistics) is known as Simpson’s paradox. Strictly, it is not a paradox in that it has an easy explanation. Note that overall women have a much higher acceptance rate (66%) than men (40%) whereas the proportion of men from state schools is 74% with 26% from independent schools while those proportions are reversed for women. This situation is known in statistics as confounding; it occurs when figures for two separate and different populations are aggregated to give misleading conclusions. It may be difficult or impossible to determine whether data such as that presented in the first table arise from two disparate populations and so confounding is present.

The example shows that if A, B, C are three events it is possible to have the three inequalities

ℙ (A | B ∩ C) > ℙ (A | B ∩ C^{c}) , ℙ (A | B^{c} ∩ C) > ℙ (A | B^{c} ∩ C^{c}) , and (2.18)
ℙ (A | C^{c}) > ℙ (A | C) ,  (2.19)

holding simultaneously. In this example, A would be the event ‘being admitted’, B the event ‘being a man’ (with B^{c} being a woman) and C being ‘state school’ (with C^{c} being independent school). One situation where the three inequalities in (2.18) and (2.19) cannot hold simultaneously is when

ℙ (B | C) = ℙ (B | C^{c}) .  (2.20)

To see this, suppose that (2.18) and (2.20) hold, then we see that

ℙ (A ∩ B | C) > ℙ (A ∩ B | C^{c}) and ℙ (A ∩ B^{c} | C) > ℙ (A ∩ B^{c} | C^{c})

and then adding and observing that, for example,

ℙ (A ∩ B | C) + ℙ (A ∩ B^{c} | C) = ℙ (A | C) ,

we obtain ℙ (A | C) > ℙ (A | C^{c}), which contradicts (2.19). In this example it would not be possible to engineer that (2.20) holds, but consider the situation where a clinical trial of a new drug for some illness is being conducted to test its effectiveness against a standard drug. Then A would be the event that a patient recovers from the illness and C would be the event that the patient receives the new drug (C^{c} the event that (s)he receives the standard drug); again B or B^{c} would correspond to the patient being a man or woman, respectively. Then to avoid the sort of situation described in this example, (2.20) would require that the trial be designed so that the relative proportions of men and women that receive the new drug are the same as the relative proportions that receive the standard drug. □

2.4 Independence

We say that two events A and B are independent if

ℙ (A ∩ B) = ℙ(A)ℙ(B).

Example 2.21 Roll a die twice and let A be the event that a 1 is obtained on the first roll and B be the event that a 5 is obtained on the second roll. The sample space Ω = {(i, j) : 1 ⩽ i ⩽ 6, 1 ⩽ j ⩽ 6} has 36 points, all of which have the same probability \frac{1}{36}. Then A ∩ B is just the outcome (1, 5) and ℙ(A) = \frac{1}{6} = ℙ(B) so we can conclude that A and B are independent. □

Notice that if ℙ(B) > 0 then A and B are independent if and only if ℙ( A | B) = ℙ(A). Furthermore, if A and B are independent then

(i) A and B^{c} are independent; (ii) A^{c} and B^{c} are independent; and (iii) A^{c} and B are independent.

For (i),

ℙ(A ∩ B^{c}) = ℙ(A) − ℙ(A ∩ B) = ℙ(A) − ℙ(A)ℙ(B) = ℙ(A)(1 − ℙ(B)) = ℙ(A)ℙ(B^{c}),

and (ii) and (iii) follow from (i).

We need to generalize the notion of independence to more than two events. Events A_{1}, A_{2}, … are independent if for all choices of i_{1} < i_{2} < ⋯ < i_{r}, we have

ℙ (A_{i_{1}} ∩ A_{i_{2}} ∩ ⋯ ∩ A_{i_{r}}) = ℙ (A_{i_{1}}) ℙ (A_{i_{2}}) ⋯ ℙ (A_{i_{r}}) .  (2.22)

The definition in (2.22) implies that if we take any two of the events A_{i} and A_{j} (i distinct from j) then they are independent; so the events are said to be pairwise independent. However, it should be noted that pairwise independence does not imply independence as the next example shows.

Example 2.23 Suppose that a fair coin is tossed twice (by a fair coin we mean the 4 possible outcomes HH, HT, T H and T T are equally likely and equal to \frac{1}{4}, so that a Head or a Tail on either toss has probability \frac{1}{2}). Let A_{1} be the event that a head is obtained on the first toss, A_{2} the event that there is a head on the second toss and A_{3} the event that exactly 1 head is obtained. Then ℙ(A_{i}) = \frac{1}{2} for i = 1, 2, 3. It may be seen that the events are pairwise independent since, for example,

ℙ(A_{1} ∩ A_{3}) = ℙ(A_{1} ∩ A_{2}^{c}) = \frac{1}{4} = ℙ(A_{1})ℙ(A_{3}),

and similarly for A_{2} and A_{3} (and A_{1} and A_{2}). But

ℙ (A_{1} ∩ A_{2} ∩ A_{3}) = 0 ≠ ℙ(A_{1})ℙ(A_{2})ℙ(A_{3}) = \frac{1}{8},

so the three events are not independent. □

We need to see how to model the notion of independent experiments where the outcome of one experiment does not influence the outcome of the other and see how it relates to this definition of independence. Suppose that Ω_{1} and Ω_{2} are the sample spaces for two experiments with probability distributions ℙ_{1} and ℙ_{2} respectively. Let Ω = Ω_{1} × Ω_{2} be the sample space corresponding to both experiments being performed, so that Ω = {(ω_{1}, ω_{2}) : ω_{1} ∈ Ω_{1}, ω_{2} ∈ Ω_{2}}. We can define the probability distribution ℙ on Ω by specifying ℙ for any point (singleton) of Ω (see next section), by letting ℙ ({(ω_{1}, ω_{2})}) = ℙ_{1}({ω_{1}})ℙ_{2}({ω_{2}}). If now A_{i} ⊆ Ω_{i}, i = 1, 2, and both experiments are performed we can identify the event A_{1} with the event A_{1} × Ω_{2} in Ω and the event A_{2} with the event Ω_{1} × A_{2} in Ω, and the intersection A_{1} ∩ A_{2} with A_{1} × A_{2}, so that

\begin{aligned}ℙ(A_{1} ∩ A_{2}) & = ∑_{ω_{1}∈A_{1}} ∑_{ω_{2}∈A_{2}} ℙ (\{(ω_{1}, ω_{2})\}) = ∑_{ω_{1}∈A_{1}} ∑_{ω_{2}∈A_{2}} ℙ_{1} (\{ω_{1}\}) ℙ_{2}(\{ω_{2}\}) \\ & = ∑_{ω_{1}∈A_{1}} ℙ_{1} (\{ω_{1}\}) ∑_{ω_{2}∈A_{2}} ℙ_{2}(\{ω_{2}\}) = ℙ_{1}(A_{1})ℙ_{2}(A_{2}); \end{aligned}

but we then have ℙ (A_{1} ∩ Ω_{2}) = ℙ_{1}(A_{1}), and ℙ (Ω_{1} ∩ A_{2}) = ℙ_{2}(A_{2}), whence we can interpret this as ℙ (A_{1} ∩ A_{2}) = ℙ(A_{1})ℙ(A_{2}) which ties in with the definition of independence given above. This extends to n independent experiments in the obvious way.

2.5 Distributions

We are considering a finite or countable sample space Ω = {ω_{i}}_{i}, and for each point ω_{i} ∈ Ω let p_{i} = ℙ ({ω_{i}}) be the probability of the event consisting of the single point ω_{i}. Then the sequence {p_{i}}_{i} satisfy

p_{i} ⩾ 0, for all i, and ∑_{i} p_{i} = 1,  (2.24)

because Ω = ⋃_{i}{ω_{i}}. There is a one-to-one correspondence between sequences {p_{i}}_{i} satisfying (2.24) and probability distributions as defined in this chapter through the relation ℙ(A) = ∑_{i:ω_{i}∈A} p_{i}. Because of this correspondence the term probability distribution is also applied to sequences satisfying (2.24). We consider a number of particular distributions: Example 2.25 Bernoulli distribution Consider a sample space with just two points Ω = {H, T }, where the two points may be thought of as H = “heads”, and T = “tails”, so we are modelling a coin toss where we will take p as the probability of heads and 1 − p as the probability of tails. Then p = ℙ (H) and 1 − p = ℙ (T), with 0 ⩽ p ⩽ 1. □

Example 2.26 Binomial distribution This models the number of heads obtained in n successive tosses of the coin in the previous example. Then Ω = {0, 1, 2, …, n} and the probability of k heads is

p_{k} = ℙ (k) = \binom{n}{k}p^{k}(1 − p)^{n−k}, 0 ⩽ k ⩽ n.  (2.27)

Notice that this defines a probability distribution since, by the Binomial Theorem

∑_{k=0}^{n} \binom{n}{k}p^{k}(1 − p)^{n−k} = [p + (1 − p)]^{n} = 1.

To see why this corresponds to the probability of k heads in n tosses of the coin–let Ω′ = {(i_{1}, …, i_{n}) : i_{j} = H or T } be the sample space where an outcome records whether a head or a tail is obtained on each toss. Then, if the sequence \underline{i} = (i_{1}, …, i_{n}) represents an outcome in Ω′, let N(\underline{i}) represent the number of indices j with i_{j} = H; that is, N(\underline{i}) is the number of heads in \underline{i}. By independence the probability of the outcome \underline{i} is

ℙ (\{\underline{i}\}) = p^{N(\underline{i})}(1 − p)^{n−N(\underline{i})}.

Now the number of sequences \underline{i} ∈ Ω′ for which N(\underline{i}) = k, that is the number of sequences for which there are exactly k heads, is \binom{n}{k} (think of choosing the k positions for the heads from the n possible positions) and hence we get (2.27). □

Example 2.28 Poisson distribution This distribution is often used to model the number of occurrences of some event in a specified period of time, such as the number of accidents on a particular stretch of road, for example, or the number of customers who enter a particular shop. Here the probability space is Ω = {0, 1, 2, …}, the non-negative integers, and the probability of the point k is

p_{k} = ℙ (k) = e^{−λ} \frac{λ^{k}}{k!} , k = 0, 1, 2, …

for some fixed λ > 0. Check that

∑_{k=0}^{∞} p_{k} = e^{−λ} ∑_{k=0}^{∞} \frac{λ^{k}}{k!} = e^{−λ}e^{λ} = 1.  □

Suppose that we consider customers entering a shop during an hour-long period, (0, 1]. Think of dividing the period into n segments, ((i − 1)/n, i/n], for i = 1, …, n, and suppose that 1 customer enters the shop in each segment with probability p, 0 < p < 1. Then the probability that k customers enter in the hour is the binomial probability

\binom{n}{k}p^{k} (1 − p)^{n−k} = \frac{n!}{k!(n − k)!}p^{k} (1 − p)^{n−k} .

Now suppose that n → ∞ and p → 0 in such a way that np → λ, then we have, for each fixed k,

lim_{n→∞} [\frac{n!}{k!(n − k)!}p^{k} (1 − p)^{n−k}] = lim_{n→∞} [(1 − \frac{λ}{n})^{n−k} \frac{(np)^{k}}{k!} \frac{n!}{n^{k}(n − k)!}] = e^{−λ} \frac{λ^{k}}{k!} ,

because

lim_{n→∞} [\frac{n!}{n^{k}(n − k)!}] = lim_{n→∞} [1 (1 − \frac{1}{n}) ⋯ (1 − \frac{k − 1}{n})] = 1.

This has proved the following result. Theorem 2.29 (Poisson approximation to the binomial) Suppose that n → ∞ and p → 0 so that np → λ, then

\binom{n}{k}p^{k} (1 − p)^{n−k} → e^{−λ} \frac{λ^{k}}{k!} , for k = 0, 1, 2, ….  □

Example 2.30 Geometric distribution This is a distribution which models the number of tosses of a coin required to obtain the first occurrence of a head, when p, 0 < p < 1, is the probability of a head on each toss; for the probability space Ω = {1, 2, …} we have

p_{k} = ℙ (k) = p (1 − p)^{k−1} for k = 1, 2, ….

Check that ∑_{1}^{∞} p_{k} = p/ (1 − (1 − p)) = 1. Note that the term geometric distribution is also applied to the distribution on the probability space Ω = {0, 1, 2 …} with p_{k} = p(1 − p)^{k}, for k ⩾ 0; this would be modelling the number of tails before first obtaining a head, but no confusion should arise between the slightly different usage of the term. □

Example 2.31 Hypergeometric distribution Consider an urn with n_{1} red balls and n_{2} black balls of which n are drawn without replacement, n ⩽ n_{1} + n_{2}. The probability that there are exactly k red balls drawn is

p_{k} = \begin{matrix}\binom{n_{1}}{k} & \binom{n_{2}}{n − k} \\ ( & {n_{1} + n_{2} \atop n} & ) \end{matrix}, for max(0, n − n_{2}) ⩽ k ⩽ min(n, n_{1}).

For example, the probability that there are exactly 5 hearts in a bridge hand is

\begin{matrix}\binom{13}{5} & \binom{39}{8} \\ ( & {52 \atop 13} & ) \end{matrix}.  □

9 January 2007