Jump to content

Probability/Random Variables

From Wikibooks, open books for an open world

Random variable

[edit | edit source]

Motivation

[edit | edit source]

In many experiments, there may be so many possible outcomes in the sample space that we may want to instead work with a "summary variable" for those outcomes. For example, suppose a poll is conducted for 100 different people to ask them whether they agree with a certain proposal. Then, to keep track of the answers from those 100 people completely, we may first use a number to indicate the response:

  • number "1" for "agree".
  • number "0" for "disagree".

(For simplicity, we assume that there are only these two responses available.) After that, to record which person answer which response, we use a vector with 100 numbers for the record. For example, , etc. Since for every coordinate in the vector, there are two choices: "0" or "1", there are in total different vectors in the sample space (denoted by )! Hence, it is very tedious and complicated to work with that many outcomes in the sample space . Instead, we are often only interested in how many "agree" and "disagree" are there, instead of which person answers which response, since the number of "agree" and "disagree" determines whether the proposal is agreed by majority of them, and thus captures the essence of the poll.

Hence, it is more convenient to define a variable which gives the number of "1"s in the 100 coordinates in every outcome in the sample space . Then, can only take 101 possible values: 0,1,2,...,100, which is much fewer than the number of outcomes in the original sample space.

Through this, we can change the original experiment to a new experiment, where the variable takes one of the 101 possible values according to certain probabilities. For this new experiment, the sample space becomes .

During the above process of defining the variable (called random variable), we have actually (implicitly) defined a function where the domain is the original sample space, and the range is . Usually, we take the codomain of the random variable to be the set of all real numbers . That is, we define the random variable by for every .

Definition

[edit | edit source]

To define random variable formally, we need the concept of measurable function:

Definition. (Measurable function) Let and be measurable spaces (that is, and are -algebras of and respectively). A function is (-)measurable if for every , the pre-image of under

Remark.

  • If is -measurable, then we may also write to emphasize the dependency on the -algebras and .
  • We just consider the pre-image of set in the -algebra since only the sets in are "well-behaved", and hence they are "of interest". Then, the measurability of ensures that the pre-image is also "well-behaved".
  • So, a measurable function preserves the "well-behavedness" of a set in some sense.
  • It also turns out that using pre-image (instead of image) in the definition is more useful.

Definition. (Random variable)

A graphical illustration of random variable.

Let be a probability space. A random variable is a -measurable function .

Remark.

  • Usually, a capital letter is used to represent a random variable, and the small corresponding letter is used to represent the realized value (i.e., the numerical value mapped from a sample point) of the random variable. For example, we say that a realized value of the random variable is .
  • The -algebra is the Borel -algebra on . We will not discuss its definition in details here.
  • Since is -measurable, we have for every set , the pre-image .
  • It is common to use to denote . Furthermore, we use , etc. to denote , etc.
  • We require the random variable to be -measurable, so that the probability (often written instead) is defined for every . (The domain of probability measure is , and because of the -measurability of the random variable .)
  • Generally, most functions (that are supposed to be random variables) we can think of are -measurable. Hence, we will just assume without proof that the random variables constructed here are -measurable, and thus are actually valid.

By defining a random variable from a probability space , we actually induce a new probability space where

  • The induced sample space is the range of the random variable : .
  • The induced event space is a -algebra of . (Here, we follow our previous convention: when is countable.)
  • The induced probability measure is defined by

for every .

It turns out the induced probability measure satisfies all the probability axioms:

Example. Prove that the induced probability measure satisfies all the probability axioms, and hence is valid.

Proof. Nonnegativity: For every event , Unitarity: We have Particularly, we always have for every by the construction of ( is the range of the random variable ).

Countable additivity: For every infinite sequence of pairwise disjoint events (every event involved belongs to ),


After proving this result, it follows that all properties of probability measure discussed previously also apply to the induced probability measure . Hence, we can use the properties of probability measure to calculate the probability , and hence , for every set . More generally, to calculate the probability for every ( does not necessarily belong to ), we notice that , and it turns out that . Hence, we can calculate by considering .

Example. Suppose we toss a fair coin twice. Then, the sample space can be expressed by . Now, we define the random variable to be the number of heads obtained in the tosses of a sample point (this means maps every sample point in the sample space to the number of heads obtained in that sample point). Then, we have Thus, . Hence, we have (The four outcomes in the sample space should be equally likely.) (It is common to write instead of , instead of , etc.)

Clipboard

Exercise. Suppose we toss a fair coin three times, and define the random variable to be the number of heads obtained in the tosses of a sample point. Then, . Calculate the probability for every . Hence, calculate the probability for every . (Hint: We can write . Now, consider .)

Solution

First, we have It follows that we have Since , it follows that we have

Sometimes, even it is infeasible to list out all sample points in the sample space, we can also determine the probability related to the random variable.

Example. Consider the example about the poll discussed in the motivation part. We define the random variable to give the number of "1"s. Here, we assume that every sample point in the sample space is equally likely. Show that for every .

Proof. Since there are sample points that contains "1"s (regard this as placing indistingusihable "1"s into 100 distinguishable cells), the result follows.

Remark.

  • For example, , , and .
  • Plot of the values of for different values of :


A special kind of random variable that is quite useful is the indicator random variable, which is a special case of indicator function:

Definition. (Indicator function)

A graphical illustration of indicator function. The "raised portion" is the set , and the whole red square is the set .

The indicator function of a subset of a set is a function defined by

Remark.

  • Special case: we can regard the indicator function as a random variable by modifying it a little bit:
  • Let be a probability space, and be an event. Then, the indicator random variable of the event is (here, we change the codomain to to match with the definition of random variable) defined by

Example. Suppose we randomly select a citizen from a certain city, and record the annual income (in the currency used for that city) of that citizen. In this case, we can define the sample space as . Assume that the city has a taxation policy such that the annual income is taxable if it exceeds 10000 (in the same currency as ). Now, we let be the taxable income of the recorded annual income. To be more precise, is defined by for every . That is, , denoted by , for every .

Example. Suppose we roll two distinguishable dice, and define to be the sum of the numbers obtained in a outcome from the roll. Then, the sample space is . Here, we can see that the range of is . Calculate for every .

Solution. Notice that there are 1,2,3,4,5,6,5,4,3,2,1 sample points in the sample space where respectively. Thus, we have

Clipboard

Exercise. Calculate the probability that . (Answer: )

Solution

The probability is


Cumulative distribution function

[edit | edit source]

For every random variable , there is function associating with it, called the cumulative distribution function (cdf) of :

Definition.

Three examples of cdf, which are illustrated by the red lines and dots between two blue lines.

(Cumulative distribution function) The cumulative distribution function (cdf) of a random variable , denoted by (or ) is for every .

Example. Consider a previous exercise where we toss a fair coin three times, and the random variable is defined to be the number of heads obtained in the tosses of a sample point. We have calculated . So, the cdf of the random variable is given by Graphically, the cdf is a step function with a jump for every , where the size of the jump is .

An illustration of the "jumps" in the cdf.

We can see from the cdf in the example above that the cdf is not necessarily continuous. There are several discontinuities at the jump points. But we can notice that at each jump point the cdf takes the value at the top of the jump, by the definition of cdf (the inequality involved includes also the equality). Loosely speaking, this suggests that the cdf is right-continuous. However, the cdf is not left-continuous in general.

In the following, we will discuss three defining properties of cdf.

Theorem. (Defining properties of cdf) A function is the cdf of a random variable if and only if

(i) for each real number .

(ii) is nondecreasing.

(iii) is right-continuous.

Proof. Only if part ( is cdf these three properties):

(i) It follows the axioms of probability since is defined to be a probability.

(ii)

(iii) Fix an arbitrary positive sequence with . Define for each positive number . It follows that . Then, It follows that for each with as . That is, which is the definition of right-continuity.

If part is more complicated. The following is optional. Outline:

  1. Draw an arbitrary curve satisfying the three properties.
  2. Throw a fair coin infinitely many times.
  3. Encode each result into a binary number, e.g.
  4. Transform each binary number to a decimal number, e.g. . Then, the decimal number is a random variable .
  5. Use this decimal number as the input of the inverse function of the arbitrarily drawn curve, and we get a value, which is also a random variable, say .
  6. Then, we obtain a cdf of the random variable , if we throw a fair coin infinitely many times.


Sometimes, we are only interested in the values such that , which are more 'important'. Roughly speaking, the values are actually the elements of the support of , which is defined in the following.

Definition. (Support of random variable) The support of a random variable , , is the smallest closed set such that .

Remark.

  • E.g. closed interval is closed set.
  • Closedness will not be emphasized in this book.
  • Practically, (which is the smallest closed set).
  • is probability mass function for discrete random variables;
  • is probability density function for continuous random variables.
  • The terms mentioned above will be defined later.

Example. If then , since and this set is the smallest set among all sets satisfying this requirement.

Remark. etc. also satisfy the requirement, but they are not the smallest set.

Clipboard

Exercise.

Suppose we throw an unfair coin. Define if head comes up and otherwise. Let be the cdf of .

1 Find .

It cannot be determined since the probability that head comes up is not given.

2 Suppose , compute .

0
0.3
0.5
0.7
1

3 Suppose . Which of the following is (are) true?

if the coin is fair instead.


Discrete random variables

[edit | edit source]

Definition. (Discrete random variables) If is countable (i.e. 'enumerable' or 'listable'), then the random variable is a discrete random variable.

Example. Let be the number of successes among Bernoulli trials. Then, is a discrete random variable, since which is countable.

On the other hand, if we let be the temperature on Celsius scale, is not discrete, since which is not countable.

Clipboard

Exercise.

Which of the following is (are) discrete random variable?

Number of heads coming up from tossing a coin three times.
A number lying between 0 and 1 inclusively.
Number of correct option(s) in a multiple choice question in which there are at most three correct options.
Answer to a short question asking for a numeric answer.
Probability for a random variable to be discrete random variable.


Often, for discrete random variable, we are interested in the probability that the random variable takes a specific value. So, we have a function that gives the corresponding probability for each specific value taken, namely probability mass function.

Definition.

An example of pmf. This function is called probability mass function, since the value at each point may be interpreted as the mass of the dot located at that point.

(Probability mass function) Let be a discrete random variable. The probability mass function (pmf) of is

Remark.

  • Alternative names include mass function and probability function.
  • If random variable is discrete, then (it is closed).
  • The cdf of random variable is . It follows that the sum of the value of pmf at each inside the support equals one.
  • The cdf of a discrete random variable is a step function with jumps at the points in , and the size of each jump defines the pmf of at the corresponding point in .

Example. Suppose we throw a fair six-faced dice one time. Let be the number facing up. Then, pmf of is

Clipboard

Exercise.

1 Which of the following is (are) pmf?

. It is given that is countable.

2 Compute such that the function is a pmf.


Continuous random variables

[edit | edit source]

Suppose is a discrete random variable. Partitioning into small disjoint intervals gives In particular, the probability per unit can be interpreted as the density of the probability of over the interval. (The higher the density, the more probability is distributed (or allocated) to that interval).

Taking limit, in which, intuitively and non-rigorously, can be interpreted as the probability over 'infinitesimal' interval , i.e. , and can be interpreted as the density of the probability over the 'infinitesimal' interval, i.e. .

These motivate us to have the following definition.

Definition. (Continuous random variable) A random variable is continuous if for each (measurable) set and for some nonnegative function .

Remark.

  • The function is called probability density function (pdf), density function, or probability function (rarely).
  • If is continuous, then the value of pdf at each single value is zero, i.e. for each real number .
  • This can be seen by setting , then (dummy variable is changed).
  • By setting , the cdf .
  • Measurability will not be emphasized. The sets encountered in this book are all measurable.
  • is the area of pdf under , which represents probability (which is obtained by integrating the density function over the set ).

The name continuous r.v. comes from the result that the cdf of this kind of r.v. is continuous.

Proposition. (Continuity of cdf of continuous random variable) If a random variable is continuous, its cdf is also continuous (not just right-continuous).

Proof. Since (Riemann integral is continuous), the cdf is continuous.

Example. (Exponential distribution) The function is a cdf of a continuous random variable since

  • It is nonnegative.
  • . So, .
  • It is nondecreasing.
  • It is right-continuous (and also continuous).
Clipboard

Exercise.

1 Which of the following is (are) pdf?

2 Compute such that the function is a pdf.

There does not exist such .

3 Compute such that the function is a cdf.

There does not exist such .

4 Which of the following is (are) true?

If the support of a random variable is countable, then it is discrete.
If the support of a random variable is not countable, then it is continuous.
If the support of a random variable is not countable, then it is not discrete.


Proposition. (Finding pdf using cdf) If cdf of a continuous random variable is differentiable, then the pdf .

Proof. This follows from fundamental theorem of calculus:

Remark. Since is nondecreasing, . This shows that is always nonnegative if is differentiable. It is a motivation for us to define pdf to be nonnegative.

Without further assumption, pdf is not unique, i.e. a random variable may have multiple pdf's, since, e.g., we may set the value of pdf to be a real number at a single point outside its support (without affecting the probabilities, since the value of pdf at a single point is zero regardless of the value), and this makes another valid pdf for a random variable. To tackle this, we conventionally set for each to make the pdf become unique, and make the calculation more convenient.

Example. (Uniform distribution) Given that is a pdf of a continuous random variable , the probability

Clipboard

Exercise.

It is given that the function is a pdf of a continuous random variable .

1 Compute .

2 Compute .

3 Compute .



Mixed random variables

[edit | edit source]

You may think that a random variable can either be discrete or continuous after reading the previous two sections. Actually, this is wrong. A random variable can be neither discrete nor continuous. An example of such random variable is mixed random variable, which is discussed in this section.

Theorem. (cdf decomposition) The cdf of each random variable can be decomposed as a sum of three components: for some nonnegative constants such that , in which is a real number, is cdf of discrete, continuous, and singular random variable respectively.

Remark.

  • If and , then is a mixed random variable.
  • We will not discuss singular random variable in this book, since it is quite advanced.
  • One interpretation of this formula is:
  • If is discrete (continuous) random variable, then ().
  • We may also decompose pdf similarly, but we have different ways to find pdf of discrete and continuous random variable from the corresponding cdf.

An example of singular random variable is the Cantor distribution function (sometimes known as Devil's Staircase), which is illustrated by the following graph. The graph pattern keeps repeating when you enlarge the graph.

Cantor distribution function

Example. Let . Let . Then, is a cdf of a mixed random variable , with probability to be discrete and probability to be continuous, since it is nonnegative, nondecreasing, right-continuous and .

Clipboard

Exercise. Consider the function . It is given that is a cdf of a random variable .

(a) Show that .

(b) Show that the pdf of is

(c) Show that the probability for to be continuous is .

(d) Show that is .

(e) Show that the events and are independent if .


Proof.

(a) Since is a cdf, and when ,

(b) Since is a mixed random variable, for the discrete random variable part, the pdf is On the other hand, for the continuous random variable part, the pdf is Therefore, the pdf of is

(c) We can see that can be decomposed as follows: Thus, the probability for to be continuous is .

(d)

(e) If , . Thus, i.e. and are independent.