Initial Concepts of Probability

Definitions

• Probability is the quantification of event uncertainty. For instance, probability of getting (H)eads in a coin toss is \(1/2\). Deterministic models will give the same results given the same inputs (e.g. 2 times 2 is 4), but probabilistic models might yield different outcomes.

• An experiment is a process that generates data. For instance, tossing a coin is an experiment. Outcome is the realization of an experiment. Possible outcomes for a coin toss is Heads and Tails.

• Sample space (\(\mathbb{S}\)) is the collection of all the possible outcomes of an experiment. Sample space of the coin toss is \(\mathbb{S} = \{H,T\}\). Sample space of two coin tosses experiment is \(\mathbb{S} = \{HH,HT,TH,TT\}\). Sample space can be discrete (i.e. coin tosses) as well as continuous (i.e. All real numbers between 1 and 3. \(\mathbb{S} = \{x | 1 \le x \le 3, x \in \mathbb{R}\}\)) (Side note: Sample space is not always well defined.)

• An event is a subset of sample space. While outcome represents a realization, event is an information. Probability of an event \(P(A)\), say getting two Heads in two coin tosses is \(P(A) = 1/4\).

• A random variable represents an event is dependent on a probabilistic process. On the other hand, a deterministic variable is either a constant or a decision variable. For instance, value of the dollar tomorrow can be considered a random variable but the amount I will invest is a decision variable (subject to no probabilistic process) and spot (current) price of the dollar is a constant.

Set Operations

• Complement of an event (\(A^\prime\)) with respect to the sample space represents all elements of the sample space that are not included by the event (A). For instance, complement of event \(A=\{HH\}\) is \(A^\prime=\{HT,TH,TT\}\)

• Union of two events \(A\) and \(B\) (\(A \cup B\)) is a set of events which contains all elements of the respective events. For example, say \(A\) is the set that contains events which double Heads occur (\(A = \{HH,HT,TH\}\)) and \(B\) is the set which Tails occur at least once (\(B = \{TT,HT,TH\}\)). The union is \(A \cup B = \{HH,TH,HT,TT\}\).

• Intersection of two events \(A\) and \(B\) (\(A \cap B\)) contains the common elements of the events. For example, say \(A\) is the set that contains events which Heads occur at least once (\(A = \{HH,HT,TH\}\)) and \(B\) is the set which Tails occur at least once (\(B = \{TT,HT,TH\}\)). The intersection is \(A \cap B = \{TH,HT\}\).

• Mutually exclusive or disjoint events mean that two events have empty intersection (\(A \cap B = \emptyset\)) and their union (\(A \cup B\)) contains the same amount of elements as the sum of their respective number of elements. Also \(P(A \cap B) = 0\) and \(P(A \cup B) = P(A) + P(B)\). For example getting double Heads (\(HH\)) and double Tails (\(TT\)) are mutually exclusive events.

Axioms of Probability

1. Any event \(A\) belonging to the sample space \(A \in \mathbb{S}\) should have nonnegative probability (\(P(A) \ge 0\)).
2. Probability of the sample space is one (\(P(\mathbb{S}) = 1\)).
3. Any disjoint events (\(A_i \cap A_j = \emptyset \ \forall_{i,j \in 1 \dots n}\)) satisfies \(P(A_1 \cup A_2 \cup \dots \cup A_n) = P(A_1) + P(A_2) + \dots + P(A_n)\).

Other Set and Probability Rules

• \((A^\prime)^\prime = A\)
• \(S^\prime = \emptyset\)
• \(\emptyset^\prime = S\)
• \((A \cap B)^\prime = A^\prime \cup B^\prime\)
• \((A \cup B)^\prime = A^\prime \cap B^\prime\)
• \((A \cup B) \cap C = (A \cap C) \cup (B \cap C)\)
• \((A \cap B) \cup C = (A \cup C) \cap (B \cup C)\)
• \((A \cup B) \cup C = A \cup (C \cup B)\)
• \((A \cap B) \cap C = A \cap (C \cap B)\)

• \(A \cup A^\prime = \mathbb{S}\) and \(A \cap A^\prime = \emptyset\) so \(P(A) = 1 - P(A^\prime)\). This is especially useful for many problems. For example the probability of getting at least one Heads in a three coin tosses in a row is \(1 - P(\{TTT\}) = 7/8\), the complement of no Heads in a three coin tosses in a row. Otherwise, you should calculate the following expression.

  \[P(\{HTT\}) + P(\{THT\}) + P(\{TTH\}) + P(\{HHT\}) + P(\{HTH\}) + P(\{THH\}) + P(\{HHH\}) = 7/8 \]

• If \(A \subseteq B\) then \(P(A) \le P(B)\).
• \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\).

• \(P(A \cup B \cup C) = P(A) + P(B) + P(C) - P(A \cap B) - P(B \cap C) - P(A \cap C) + P(A \cap B \cap C)\)

Counting

Counting rules will help us enumerate the sample space. It will include multiplication rule, permutation and combination.

Multiplication Rule

If I have a series of independent events, say \(1\) to \(k\), and number of possible outcomes are denoted with \(n_1\) to \(n_k\); total number of outcomes in the sample space would be \(n_1n_2\dots n_k\).

Take a series of coin tosses in a row. If I toss a coin its sample space consists of 2 elements such as \(\{H,T\}\). If I toss 2 coins the sample space would be 2*2 \(\{HH,HT,TH,TT\}\). If I toss 3 coins, the sample space would be 2*2*2 \(\{HHH,HTH,THH,TTH,HHT,HTT,THT,TTT\}\).

A poker card consists of a type and a rank. There are four types of playing cards (clubs, diamonds, hearts and spades) and 13 ranks (A - 2 to 10 - J - Q - K). Number of cards in a deck is 4*13 = 52.

Permutation Rule

Permutation is the arrangement of all or a subset of items.

• Given a set of items, say \(A = {a,b,c}\) in how many different ways I can order the elements? Answer is n!. In our case it is, \(3! = 3.2.1 = 6\).

  \[A = \{a,b,c\},\{b,a,c\},\{b,c,a\},\{c,a,b\},\{c,b,a\},\{a,c,b\}\]

• Suppose there are 10 (n) participants in a competition and 3 (r) medals (gold, silver and bronze). How many possible outcomes are there? Answer is \(n(n-1)(n-2)\dots (n-r+1) = \dfrac{n!}{(n-r)!} = \dfrac{10!}{(10-3)!} = 720\).

• If there are more than one same type items in a sample, then the permutation becomes \(\dfrac{n!}{n_1!n_2!\dots n_k!}\) where \(\sum n_i = n\).

  For example enumerate the different outcomes of four coin tosses which result in 2 heads and 2 tails. Answer is \(\dfrac{4!}{2!2!} = 6\)

  \[A = \{HHTT,HTTH,HTHT,THTH,THHT,TTHH\}\]

Combination Rule

Suppose we want to select \(r\) items from \(n\) items and the order does not matter. So the number of different outcomes can be found using \(\binom{n}{r} = \dfrac{n!}{(n-r)!r!}\).

Out of 10 students how many different groups of 2 students can we generate? Answer \(\dfrac{10!}{8!2!} = 45\)