Let be a stochastic process, taking on a finite or countable number of values.
is a DTMC if it has the Markov property: Given the present, the future is independent of the past
We define , since has the stationary transition probabilities, this probability is not depend on n.
Transition probabilities satisfy
2. n Step transition Probabilities
3. Example: Coin Flips
4. Limiting Probabilities
Theorem: For an irreducible, ergodic Markov Chain, exists and is independent of the starting state i. Then is the unique solution of and .
Two interpretation for
- The probability of being in state i a long time into the future (large n)
- The long-run fraction of time in state i
- If Markov Chain is irreducible and ergodic, then interpretation 1 and 2 are equivalent
- Otherwise, is still the solution to , but only interpretation 2 is valid.