Probability Foundations for Electrical Engineering

Prof. Krishna Jagannathan

Department of Electrical Engineering
Indian Institute of Technology, Madras

Lecture – 4

Welcome back. So, today we will began the study of probability theory from the basic
axiomatic perspective. So, we will start building on the basic probability axioms and
work towards building probability space, probability measures and so on. So, in this
mathematically theory of probability, its axiomatic theory which starts of assuming that
there are 2 entities which are not defined. So, the 2 entities which are not defined in the
beginning are the following.

(Refer Slide Time: 00:57)

The first is the concept of a random experiment. So, this is we do not define what it is.
So, this is we do not define it what it is we have to understand it just like in English; it is
an experiment its outcome is random alright, which brings me to the second undefined
concept, which is outcome of the random series. So, these 2 concepts are not defined. So,
you cannot make a … just like in geometry you do not define a point right or number
theory you do not define what a number. So, you have to start of some where you can
only define only certain things and certain things you understand, you cannot define
everything right.
So, probability begins with 2 entities which you do not question or try to define. So,
there is a random experiment, so experiment whose outcome is random just as English
says. And it has an outcome; every time you perform a random experiment there is a
outcome right. So, and that is it, you cannot say anything more about these
terminologies, that is all …

Now, everything in probability; that is defined is in infinite terms of infinite, so we build

up from on. So, the first most important concept most basic concept is the concept of a
sample space, sample space of a random experiment. So, it is often denoted by capital
deflator omega big omega. So, it is defined as set of all possible outcomes of the random
experiment.

So if you see, this is definition of sample space. So, I have defined the sample space in
terms of these 2 things I haven’t defined. So, defined in terms of 2 things which I have
not defined. So, that is the way it always progress right. So, you think you understand
these 2 things and move on and define other things in terms of thing you did not define
in the first place, right; that is how it goes. So, this is a sample, just a set; set containing
all possible outcomes of a random experiment. So, which what every outcome is possible
everything should be included, should not leave behind any possible outcome right.

Now, so you can for example, you can always say in a given example. So, let us say toss
a coin once. So, here you want know what the sample space is. So now, the thing I want
emphasis here is; what a sample space is corresponding to random experiments itself
depends on what you are interested in. So, it is not it so happens that I am tossing a coin;
that is my random experiment right. And if I want to know what the face it lands on, then
the sample space will have 2 elements in it; heads and tails right, assuming that is does
not land vertically; it will have only 2 outcomes right, 2 heads and tails.

However, it is not true that the random experiment of tossing a coin once always have to
have these to sample space, it depends on what you are interested in, somebody else may
be interested in finding out how many times the coin tumbles in the air right. In that case
sample space is not head tail, in that case your sample space would be the number of
time all possible number of times it tumbles, so it is probably positive integer right. Yes

So, which is why, so which is why the outcome itself depends on what the experiment
you are interested in. So, I may toss a coin, but you may be interested in counting
number of tumbles and somebody else interested in the face it lands in right. So, it is not,
so what is an outcome itself depends what you want which is why sample space depends
on what you are interested in. So, this is what I want to emphasis. So, this is the
experiment tossing.

(Refer Slide Time: 06:28)

So, this is what tossing a coin once. So, if we are interested in the face that shows, then
omega will be head and tail; will only have 2 elements, only 2 possible outcomes. But
for same random experiment, so if you are interested in the number of tumbles say, in
this case it is reasonable to say that the omega will be some subset of natural number or
natural number themselves, any number of tumbles are possible, then you would want to
choose omega equal to n right.

So, it is not even for a given random experiment it is not as though what is the outcome
or what is set of all possible outcomes, is very much a function of what you are looking
at right. So, it depends on what you are interested in studying or modeling, right. This is
something I want to this is a conceptual point, so I want to make this theory clear. So,
you may also want to you may be interested in the velocity with which the coin strikes
the ground right, it seems to be bit strange may be interested in knowing that right. So, in
this case probably you should choose R plus or some subset of R plus right, some
positive real number right. So, you understand what I am trying to say here.
So, the sample space capital omega depends on not only on random experiment, but what
you are interested in measuring or knowing right it. And as you can see this here we have
seen that the omega could be finite set of all possible outcomes could be finite or infinite,
actually here it is countable finite and here it is uncountable infinite right. So, there all
these possibility exist; you should have an omega which is finite, have an omega which
is infinite countable omega and uncountable omega right, all this is possible depending
what you are interested looking at.

So, an elementary outcome, so an outcome of random experiment, so let us say you are
fixed in what you are interested in a you, let us say you are interested in number of
tumbles in. Then, so the outcome is known as an elementary outcome. So, this is also an
terminology outcome of elementary outcome is 1 and the same. So, the outcome over
elementary outcome is denoted by little omega alright, this is not a w, this is the declator
little omega and is an element of sample space omega and big omega. So now, here little
omega choosing of little omega is what is random alright, this is the thing you have no
control on. In probability theory so this is random number.

So, you can you have a sample space, you have built a sample space set of all possible
outcomes depending on what you are interested in. And this little omega is chosen by a
you can think of it as a gene or goddess of chance who picks it right, you have no control
over it. And every time you run a random experiment, the goddess of chance picks up
little omega from this big omega and that you do not question and you have no way of
controlling. You can think of it as weather tomorrow which is a natural phenomenon or
something else right, something that you have no control over right or full knowledge of
ability to model of something of that sort. So, this is where the random is.

So, the selection of the … you have a big sample space, this is a set of possible outcome
and which particular omega realizes and you run the experiment once is in fact the
source of randomness, which you can think of it being a gene. And every time you run
the experiment, it will be a different little omega picking the samples. And this is the
source of randomness; this is the thing that you do not have control. This is ok, any
questions on this?

Yeah yeah, it is an elementary. So, little omega is elementary which is why it is called
elementary outcome. Outcome is a particular point in omega. So, if you can … so, in this
particular case if your omega is h and t, there are only 2 possible outcomes. So, little
omega must be either h or t, it could be different at different runs of the random
experiment, but it is a single term, this is 1 element of capital omega.

Yeah so, if you are interested in, so ya it really depends on what you are interested in
right. If you want to measure both, you will have an omega big omega correspondingly
right.

No no no that is fine. So, it is see you have to decide what you are interested in, you have
to see you have to build sample space after all, so there is some random experiment
going on where you are running it or nature is running it, but what you consider the
sample space is; obviously, function in what you are interested in determining. And you
may be interested in determining a the velocity or number of tumbles or whatever it is
you want to determine, you put that in sample space, but it should contain all possible
element, should not leave out any possible outcomes right. So, you have to determine
that.

So, in everything I am going insert, building a probability space we have to … our

responsibility to determine what a sample space is and so on right. So, it is our
responsibility to determine what a probability should be also is. So, under that example,
let us say toss a coin n times and you’re interested in determining the faces that show.
Now we will tossing a coin n times and you are interested in this faces that show. So,
you can look at this experiment as n different repeats of this experiment. Or you can just
say all n trials is 1 random experiment; that is valid what we are looking at it. You toss a
coin n time or n coins all at once right and say that string of n out comes a see and n
faces; that they see that is really what I am interested, that is the elementary outcome I
am interested in. So, in that case, so your omega will be all possible heads and tails all 2
power n possibilities of heads and tails. So, it should be h t power n, right.

So, each elementary, each little omega in this sample space will be h t t h t h h some n
strings of heads and tails. And this each of this elementary outcome strings of h t t h t h is
considered as elementary outcome, if you just say that all elementary outcome n toss
corresponds to random experiment, right. That is also valid way of looking at it.

So, this what kind of a sample space is this? This it is the finite sample space; there is
only 2 possibilities here n is fixed n is 2 power 10 or 20 or whatever, but there will be
only 2 power n possible elements in this sample space corresponding to the 2 power n
possible strings of heads or tails right. Now you can also, now let us say 1.

(Refer Slide Time: 16:49)

Toss a coin infinitely many times and then again you are interested in the all the faces
that show. So now, you may get … you may wonder how do I toss a coin infinitely many
times will I every finish and so on, right. Will I every finish according outcome. But
these things you do not worry about and how would shape you right. Again you just say
that you are tossing some coin infinitely many times, so your omega is set of all possible
outcomes is what? Set of all possible strings of heads and tails, infinite strings of heads
or tails right.

So, every trail of random experiment, so I want to emphasis here that 1 trail of random
experiment is not tossing a coin in once right, 1 trail is a particular infinite strings of
heads and tails. So, in this particular case you are, so this the infinite sets of heads and
tails and your omega for example, may be string h h t t t h h h something right, some
string of some infinite string s of heads and tails. This is an elementary outcome; I want
say not that is a elementary outcome. So, now, your random experiment is itself is
tossing a coin infinitely many times. And you do not ask questions like will I ever finish
will I ever be able to count then. You do not ask that right. That is the experiment.

So, what kind of sample space is that? It is an uncountable sample space, because it is
you can say like h is 0, t is 1, so it is like 0 1 power infinite, we are very familiar with
canters are given. So, this guy is an uncountable sample, right. There are many examples
you can give like you can take a line, you through darts as a line and let us take 0 1
interval you are throwing darts at 0 1 interval and the elementary outcome will some real
number between 0 and 1 and sample space will be interval itself another uncountable
sample space right. So, these are some examples.

Positively real numbers, it is an uncountable set.

Yes.

I have not said that right, yeah you are getting ahead of me before the lecture. I am only
talk about the sample space now, right. What are all the set of possible outcome; I have
not said anything about probability so far, have I right. Sample space everybody with
sample space. Ok. So, now, so in probability right, we are often interested in not in
weather a particular elementary outcome has occurred or not, we are often interested in
whether a subset of sample space has occurred or not. So, we may be interested in let us
say toss a coin n times, we may not actually say what is the specific string that came up,
we may say number of heads even, right. So, those are the things that are of actually the
modeling interest to speak right. So, you may be interested in weather discursive sample
space, you may not cap weather this elementary outcome showed up or some other
elementary outcome showed up, you may say did that subset show up, right.

So, in the case of this number of tumbles, you can say the number of tumbles greater
than 5. So, it is not elementary out come in the sense. So, these kinds of subsets of
sample space, which are on interest to the moderator, which are of interest to the person
doing this business of modeling probability modeling is the subsets are called events. So,
this is a … I just a informal definition, because the formal definition requires more build
up. So, I may be I wonder if I have to write this definition down.

So, let us say informally subsets of a subset, we say informally let me just say sub set of
omega which are of interest are called events. So, I may be … so, for example, and the
… so, toss a coin thrice right and we are interested in faces, so your sample space is h t
over 3. You may be interested in, so the event of interest may be this at least 2 heads. In
this case you are really not asking I tossed the coin 3 times, I am not really asking if
weather particular string showed up, I am asking is at least 2 head showed up. So, this
corresponds to more than 1 elementary outcome right. So, in particular, what all the
outcome that correspond to this? So, you have fist of all you can have all heads or you
can have t h h right, h t h right h h t, anything else? That is it right.

So, these are elementary outcomes, elementary outcomes string of 3 string of head and
tails and event of interest, you where asking me the question of at least 2 heads showed
up; that corresponds to these strings, which is actually a sub set of that of the sample
space for all. So, for all colloquial purposes and event is the subset of the sample space,
except that is not the entire story. That is something we get to latter. For now you can
just consider a event as a subset of sample space, but I have put something in quotes
here; they are interest to us.

So, we will see latter that all events are in fact, subsets of sample space events are
necessarily subsets of sample space, but just put a caveat in your mind that not all subsets
of sample space are necessarily considered events, only those subsets which are of
interest which we are interested in assigning probably storing particular are called events.
All events are subsets of omega right, you can call this event a for example, right. This
event is subset of omega, right, all events are subsets of omega, but not all subsets of
omega are necessarily considered events. And this is not a point that will you will
completely appreciate now, we are little bit confused we will get to it right. So, this is
something bear in mind, not all subsets are called events.

So, as I said, so let continue on this slightly informally intuitive line, events are subsets
of sample space which are of interest to me. So, I want to build up a structure of these
subsets omega. So, I want say let us say a which is subset of omega event, it is an interest
to me, so it is an event. So, I want to say that not a is also an interest to me, an
occurrence of interest to me is very reasonable to say that its non occurrence is also
interesting to me, after all 1 determines the other a is interest to me and a complement is
also of interest to me, if a is an event a complement is also an event. Remember a
complement is the subset of omega right. But since I am only saying that events are
subsets which are of interest, I want to impose a certain constrain that the ways of
interest, a complement should be of interest to me. So, it is being perfect sense right. So,
that 1 thing I want in this intuitive notion of interesting subset of sample space.

Another such that I want to impose, let us say there are 2 events a and b both subsets of
omega a and b, I want to know whether a or b, a union b are of interest to me right. If a
and b are interest I want to say that at least 1 of them occur is of interest to me right.
These are things that I want to impose right. And finally, I want to say omega itself
which is subset of itself, which is sample space. That is sure to occur right. If, any time I
run a random experiment, the omega itself will occur right, after all omega contains all
possible outcomes of random experiment.

So, you may say that omega itself should be an interesting event; sample space is
concern subset of itself must be of interesting event right. So, this is the structure we are
building up to these 3 things, omega itself should be interesting, when a is interesting, a
complement should be interesting. And when a and b are interesting, then a union b
should be interesting. So, that is what I am building up.

(Refer Slide Time: 29:00)

So, by the way, so 1 little definition I forgot, here definition an event is said to occur if
omega is in a. So, that is fairly simple definition, I am just saying I considering some
subset of sample space, I say that let us say that is an event say that say that event occurs
if, my elementary outcome lands in that set right. So, goddess of chance picks up little
omega right; that little omega may be in this set, may be some ware else. And if that little
omega so happens to be in this a, then I say that event a occurs. Let us say in this
particular case if the elementary outcome is any of this guys, then I say the event
corresponding to this guy at least 2 guys has occurred. That is just a question of
terminology; it is just a name.
So, going back to what I was saying; I want to impose certain structure to the events
right. I want to say that a is of interest, a complement is of interest, a and b are of
interest, then a union b is of interest a and b are of interest and omega should be of
interest. So, this structure leads to what is known as algebra, this concept is
mathematically known as algebra. So, let us put definition.

(Refer Slide Time: 30:50)

So, I just show omega as sample space collection f not of subsets of omega is called an
algebra if, following 3 satisfies. So, null set should be algebra. If, a is f not, then a
complement is f not. And 3: a is a set b is a set f not, then a union b is in f not. So, this is
scripted. So, there is question of notation I want to clarify right away. So, when I am
talking about sets, I will use capital English letters and when I am talking about not
necessarily English letter capital letter omega is a Greek letter. And when we are talking
about elements of a set, we will use small letters; small omega etc all small letter. So, we
should write statement like that this is an element of set right.

Similarly, when I write something scripted right, it is a collection of set. This is a set of
and you collect many of set, collection of set this is a scripted letter. So, this is a
convention you will follow or will stick to as much as possible. So, I will have x belongs
to a and when I write a subset line, these 2 must be of same type; they must be subset of
same type, here this is an element and that is a set right. For example: I can write a is an
element of f not, f not is collection of sets, here is an element, here is the collection of
set. This is a element, this a set this a set and this is collection of sets. So, that is like a
hierarchy right.

Similarly, I may write some f 1 is subset of f 1 is contained in f not right, f 1 may be

some collection f not may be some other collection, this is contained in that means, all
the sets which are in f 1 will be contained in which are also elements of f not right. So,
this is like a hierarchy, this is something you needs to get used to. So, capital letter are
sets small letters elements and scripted letters are collection of sets. And of course, we
can talk about collection of set and collection of set rare and so on, where this whole
hierarchy. So, I have denoted f not as subset of collection of set big omega and big
omega as sample space, it has many subset; possible infinite, possible finite, possibly
uncountable. So, this has many subsets.

And that collection of subsets of omega said to be an algebra if, these 3 constraints are
satisfied which should have the null always be an element of algebra alright. And when a
is element of algebra and a complement is always element of algebra. Some people write
omega here does not have this, then pie complement of omega, you can either write
omega or pie here no big deal. And here is you are saying a is an element of f not and be
is element of f not, then a union b is element of f not. This is like; this is the structure I
was talking about.

So, any collection …. What is an algebra, after all algebra is collection of subset of
sample space, which has certain properties, it must be closed under complementation and
if I take 2 elements, it must be closed and union and null set should be contained in it.
So, from these from this structure you can easily prove, if you have n elements of algebra
a 1 a 2 a n, then they are all in f not, the union of all this guys will be in f not right. So,
this may be you should do it as a home work.
(Refer Slide Time: 37:30)

So, exercise: let A 1 A 2 A n; they are all subsets of omega. All of these are elements of
algebra. You can show that union A i is in f not and intersection is in n is fixed number
here, any fixed finite number. And you are considering n subset of sample space, which
are all elements of this algebra. Using these which are called actives of algebra, using
this actives of algebra you can prove that all this finite units and finite intersection are in
fact, elements of algebra. Can someone indicate to me how 1 would do this lets take
union first? First 3 elements you can say we have a b c and a 1 b 1 and c 1 let us say a 1
union a 2 is event of f not. So, if you union with a 3 is also union of f not, you should get
parallel element of f not.

If you want to prove to any n, you will use induction right, you will use induction right.
This you can prove that right. How do you prove that? You can use demorgan’s law.
After all this intersection, this can be written as union i equals 1 through n a i
complement the whole complement right. This is by demorgan’s law right. So, a
complement an event if when every a complements and event union of all, what did I say
A i is all elements of algebra, A i complement are elements of algebra, union is a element
of algebra and complement of algebra. So, this is an element, so this is clearly an element
of f not. So, you cleared it.

So, this is the structure you want to impose on the events right and that leads to
mathematical structure called algebra; algebra of absence. So, if you want say it in words
an algebra of subsets, quickly collection of subsets which contains a null set and it
should be closed under finite union, finite intersection complements finite unions. And
therefore, under finite intersections this is consequence of demorgan’s law right. It is a
collection of subset closed under complementation and finite unions, not unions; finite
unions, I want to make it very very clear, finite union and therefore, also finite
intersections because of demorgans law.

So, this is the structure you want to impose for these events ideally. So, events we have
still not defined what it is, we have only said some interesting sub set of sample space
right. So, is this enough, right. Let us take an example.

I have only made statement what algebra is, I have only define algebra, is an algebra
which is collection of subset which is closed under complementation and finite unions
right, this n infinite number, it can be anything it has to be finite and finite intersection
by default.

Which 1? yeah, you can jolly well defined like this 1 also right. In this case some people
define algebra as if A 1 A 2 a n is in f not, then the union of A 1 to A n must be in f not.
If, you define it for 2 elements you will get the n element version or you can just define
as n element union, for all n this must be closed under finite union. And this is something
you do not put it in axiom, because it automatically follows from union and its
complement axiom, it applies both together with demorgan law you will get this, you
will never put in that, because you can prove it right.

No, it is just a collection of subsets of ya, this f not have infinitely many … no problem,
it can have infinitely many subsets of omega right. But all I am saying is it is an algebra
it is called an algebra, since it is closed under complementation’s and finite unions and
finite intersections.

No no, demorgans law is valid always, for all unions and intersection it is always valid,
but I am saying that algebra only closed under finite unions, I am defining an algebra as
a structure among subsets, which has a property of null set contained in it, complement is
closed and complementation is closed under finite unions, finite union it is very very
important. That is what algebra is. So, it turns out, so it feels like it is intuitively enough
right, after all it is closed under any finite unions. So, it turns out though in order to
make, in order to actually study events of interest day to day modeling of probability,
actually it is closer under finite unions is actually little bit short of what we actually need.
In order to make this proper interesting probability theory, closer under finite unions
little bit shorter than what you need, you need little more structure than an algebra.

So, that is what … there are some examples use to motivate it. Let us say tossing, so let
us say your tossing points and you will toss until you see the first head. If, you have a
situation like that; let see what outcomes are. So, this is an experiment. So, you may have
the first may be a head, you will stop. So, the first time you see a head you will stop right
or maybe you will have a head and then a tail you will stop or you may have 2 tails and a
head you may stop and so on right, HT HT TH T TTTH, but you will toss until you see
the first head. That is your experiment, this is the set of all possible out comes right, the
king of faces that show. See you may be interested, let us you may be interested in some
events like this. Let the event is the total number of toss even right. This is the question I
want to answer. And this is what I want to consider an interesting event, this what I am
interested in studying.

Now, we are let us say this is my sample space, let me say that lets say I build my let us
say I take each of my elementary outcome and let us say I try to build an algebra from
here. If, I take all my elementary outcomes, so I take out this is an elementary outcome,
so I have to include its complement, then I have to so then I have take that and include
that its complement so on. And I will also have go on and include all this finite unions
and I have include that union complement and that union complement and so on. That is
how I build my algebra right correct according to my f not.

So, the problem is this guy this event let us say this, lets say a this a consist of following
elements. What does it have? TH TTTH so on, hope this is right, what is argument I
want to make. This is the guys I am interested in, this is the sub set of sample space.
However, if I try to just build a, if I take elementary outcome here little each of the
elementary outcomes include them in the f not and I will have all the complements and
have include all this infinite unions. But if you do that you will notice something like this
is never be contained in your algebra, because what kind of this is an countable infinite
set, this is also countably infinite sub set of this set. This set you can never build by
taking this elementary count outcomes, taking complements and taking finite unions, you
can actually verify that a you can never build this a just by taking this elementary
outcomes and taking complements and finite unions. After all this is infinite union
countably infinite unions of elementary outcomes right. This is see this is that union that
union that, but it is a countably infinite union right.

So, a perfectly reasonably sounding event, this is something which I am may be very
much interested in modeling right, what is the event do the event of total number head
being even, just occur perfectly reasonable event 1 would think and 1 would want to
make it an interesting event. But the structure here imposed, namely the structure of an
algebra is does not seem to be enough bring this from elementary outcomes, just think
about it right, which is why I said that the structure of algebra falls little bit short of than
what is actually needed to make probability theory interesting right.

So, that leaves a stronger notion a slightly stronger structure among subsets know as
sigma algebra. That is something we will cover in next call. So, we have defined algebra
today. So, we will motivated why we need not just finite union, but going to say that
complementation and countably infinite union. And that structure is called sigma algebra
and that is what give you a very rich probability theory and that is what gives you the
ability to model, even fairly elementary events like this, fairly simple events like this
which algebra not able to capture. So, probability operates in the sigma algebra of
substance. That we will see in next class.