Introduction to Markov chains
Caveat emptor!
Warning. Experimental page.
Disclaimer: This is intended to be a simple introduction to
some of the principles of Markov chains. I want to point out that I am no
mathematician and that while I try to be as accurate as possible, I might
very well be way off here. If you find any mistakes or errors, please
contact me so I can make a correction to the page. As for my examples, I
have made up all the numbers, I have not performed any practical studies
in this, and have no idea how accurate my models are.
I do my best here, but feel free to contact me by email with comments
Idea
I am going to try to explain markov chains by first discussing the general
principles, then introduce a few examples of how this could be realized.
You should be aware that there is mathematical magic involved in some of
the steps here, but I will avoid most of the gory details (or, at least
hide the details in their own section at the end). My goal is to write
a simple text aimed at the interested layman. (Feel free to comment on
my success/lack of success).
There are a few links to MathWorld towards the end that might be useful to
get additional, more formal, information.
Principles and the Markov condition
States
First of all a discrete process needs to have distinct states. A
(discrete) state is a distinct state that the system can be in.
For instance, let us assume that the human has three different
states to be in when it comes to relationships. That is we assume that
everyone is one of the three:
- Is single
- Has someone they are dating, but not quite partners
- Has a partner
Now, this is a rather severe restriction on reality, but a suitable one
for making a model that we can use as an example.
Transitions
The way to think about the chain is as "transitions" between different
states. The transitions take place when we go from one instance in time to
another. In the model we will use the event "Having sex with a
stranger" as the event for the transitions. So if we have the event "I had
a partner but I had sex with a stranger one night and when I called my
partner and told him/her about it he/she broke up with me", then we have
the transition (3)->(1). On the other hand the event "I was single but
had sex with a stranger one night, and the next morning we decided to
start seeing each other" is written as "(1)->(2)".
Probabilities
A probability is the chance/risk that an event will occur. P(E) is the
probability that event "E" will occur.
So, if I throw a (fair and six sided) dice then you have P(getting a 5) of
1/6. Of course the same goes for all the other results from 1 to 6. The
chance of not getting a five is 1-P(getting a 5) or 5/6 and is called the
complement event.
All probabilities are between 0 and 1. The value of the probability can be
expressed as a percentage between 0 and 100%.
There are a lot of theorems and and tricks in this area, the area of
conditional probability yields a lot of results that surprise you at first
and is well worth reading up on. I will however resist the temptation to
force you to do so here.
The sum of all probabilities will be 1. This means that there is a 100%
chance that something will happen. (Note: This is not strictly
true, but it is true "enough" for now).
Last, to connect back to our example. What are the odds of getting a
partner by having sex with a total stranger? If we assume that 10% of the
time you will end up with a partner the next morning we can write it as:
P((1)->(3)) = 0.1. That is, "There is a 10% chance that if we are in state
1 we end up in state 3".
The Markov Condition
In plain English the condition for a process to be a Markov process is
that the probabilities for a transition only depends on which states the
transition is between. That is, we assume that your chances to get a
partner from having one night stands is unrelated to how many times you
repeat the procedure.
You sometimes say that the process has no "memory", that is, it only knows
which state it is in, it doesn't care about how it got there.
There are other processes which are not Markov-processes, but I will not
cover them here.
Fleshing out our model
We can build a table of transitions like this :
| From | To | Probability |
|---|
| 1 | 1 | 0.4 |
| 1 | 2 | 0.5 |
| 1 | 3 | 0.1 |
| 2 | 1 | 0.4 |
| 2 | 2 | 0.5 |
| 2 | 3 | 0.1 |
| 3 | 1 | 0.7 |
| 3 | 2 | 0.2 |
| 3 | 3 | 0.1 |
Reading the probability table
You should be aware of what each line means, so go back to the
list of states above and read the text for each state, then read out loud
to yourself for each line something like "Being single and having sex
with strangers give you a 50% chance to start dating".
This information can also be rendered in the form of a matrix, like this:
| From / To | 1 | 2 | 3 |
|---|
| 1 | 0.4 | 0.5 | 0.1 |
| 2 | 0.4 | 0.5 | 0.1 |
| 3 | 0.7 | 0.2 | 0.1 |
To find out the P((1)->(3)) you should read line 1, column (3) to find it
to be a 10% chance.
Mathematically speaking magic is performed on the matrix to arrive at
many interesting and/or surprising results. I will look at some of those
results in the coming sections.
Another good way to represent this is as a graph, I will do that graph
any day now.
Reading the transition matrix
Repeat the exercise above, making sure that you understand what the elements in the matrix symbolize.
Rowsums
Add up all the elements in a row. Examine the value that you get. Do the same
thing to another row. Do you get the same sum? If yes, why? If no, why not?
What if we want to do it more then once?
What if we decide to have sex with two strangers and want to know what our
odds are after that? Well, assume that we are certain of our current state. We
know that we are single. This means that we know that after the first night
we know that we might be in state (1) with the probability 0.4, in state (2)
with probability 0.5 and in state (3) with the probability 0.1. This is simply
the first line from the transition matrix above.
Now we might ask ourselves what the probabilities will be for the
final state. One way to do this would be to analyze the different cases,
something like this:
- What ways do we have to end up in state (1)?
- We could be in state (1) after the first step, and then stay
in state (1) in the second steps. What is the probability for this?
Well, we know that we have a chance of 0.4 to be in state (1) after
the first step, and we know (see the matrix above) that the chance of
(1)->(1) is 0.4, so the TOTAL probability of first going to state (1)
and then staying there will be 0.4*0.4=0.16.
- We could be in state (2) after the first step and then go to state
(1) in the second step. The probability of being in (2) after one step is
0.5, and the transition (2)->(1) has the probability 0.4 which gives us a
total probability of 0.4*0.5=0.2.
- We could be in state (3) after the first step and then go to
state (1) in the second step. The probability of being in (3) after one
step is 0.1, and the probability for (3)->(1) is 0.7, which brings us to a
total probability of 0.7*0.1=0.07
We have now specified all the ways that we can get to (1) in two steps.
The total probability for (1) will be the sum of the individual
probabilities, so that we'll end up with
0.16+0.2+0.07=0.43.
Interpreting in words
Try interpreting "The probability of going from (1) to (1) in two
steps is 0.43" into words.
- What ways do we have to end up in state(2)?
- We could be in state (1) after the first step(probability 0.4)
and then transition (1)->(2)(P=0.5). The probability will be 0.4*0.5=0.2
- We could be in state (2) after the first step(P=0.5) and then do
(2)->(2) (P=0.5) for a total probability of 0.5*0.5=0.25
- We could be in state (3) after the first step(P=0.1) and then do
(3)->(2) (P=0.2) for a total probability of 0.2*0.1=0.02.
Just as for the first case we can sum up the individual probabilitiesto
get 0.2+0.25+0.02=0.47
-
Calculate the final probability
You should calculate the probability of ending up in state (3). (There is
more then one way to do this, one is very easy and quick. Even if you find
the easy way it may be a good exercise to do it the way I outlined above).
Now we have found out what happens if you have sex with two strangers.
This method does work, but it will be quite tedious if you want to find
out what happens after a hundred steps or more. Without further motivation
I will just state that this could be viewed as matrix-multiplication.
Infact, most of the markov-things are viewed in terms of matrix-operations
of various kinds.
What am I not telling you?
First of all, I am ignoring a lot of mathematic formality here. I do
think that I am still correct though. But you should be aware
that there is much more to this then what I am presenting here. I am
trying to make this accesible to people with only a basic understanding of
mathematics.
Well, there is a lot that I have not had the time to write. The topics I
would mainly like to cover in the future are:
- The matrix methods. (But I don't think most people are comfortable
doing matrix operations, so I am not sure how to even start writing
this.
- Another example. I would like to make a separate example of another
model, with all the numbers written up all pretty.
- Convergence. This would tell you in effect what your odds are (in the
long run) of having a partner if you sleep with many strangers.
- Absorption. This can, for instance, tell you how many strangers (at an
average) you have to sleep with before you get your first partner.
- Chapman-Kolmogorov.
The last two points would add completely new material, which would be
interesting. But the first two points would make the material more solid.
I am not sure which course to take here. Or even if anyone is interested.
So if you do read this page and have an opinion on the material. Then
please let me know (for instance by emailing me).
References
(C)Viktor Leijon 1998-2008