t-step transition probability: Let be the probability that the system is in state j in t time units, given the system is in state i now.
= (by stationarity)
Lemma 6.2 b:
3. Forward Chapman-Kolmogorov Equations
Let be a stochastic process, taking on a finite or countable number of values.
is a DTMC if it has the Markov property: Given the present, the future is independent of the past
We define , since has the stationary transition probabilities, this probability is not depend on n.
Transition probabilities satisfy
2. n Step transition Probabilities
3. Example: Coin Flips
4. Limiting Probabilities
Theorem: For an irreducible, ergodic Markov Chain, exists and is independent of the starting state i. Then is the unique solution of and .
Two interpretation for
- The probability of being in state i a long time into the future (large n)
- The long-run fraction of time in state i
- If Markov Chain is irreducible and ergodic, then interpretation 1 and 2 are equivalent
- Otherwise, is still the solution to , but only interpretation 2 is valid.
Remove the restriction that two or more customers cannot arrive at the same time, (i.e., remove orderliness property)
be a Poisson Process with rate
, and let
be the i.i.d random variable, then
is a compound Poisson process
Buses arrive according to a Poisson process. Let
be the number of people on bus i, and let
be the total number of people arriving by time t.
Insurance claims arrive according to a Poisson process. Let
be the size of the claim (in dollars), and let
be the total amount due to all claims by time t.
var[X(t)] = var[E[X(t)|N(t)]] + E[var[X(t)|N(t)]]
and we have
1. Definition of Poisson Random Variable
Def: A Poisson random variable with mean A and probability mass function:
2. Binomial Approximate Poisson
be an indicator variable of either 0 or 1. And assume
for all i. Then
Based on the assumptions, we have a binomial distribution:
(A Poisson random variable)
is approximately Poisson with mean 400/365
4. Priliminary Definitions
Def : A stochastic process is a collection of random variable (RV) indexed by time .
- If T is continuous set, the process is a continuous time stochastic process (e.g., Poisson Process)
- If T is countable, then the process is a discrete time stochastic process (e.g., Markov chain)
Def. A counting process
is a stochastic process
Def: A counting process has stationary increments
if the distribution of the number of events in an interval depends on the length of the interval, but not on the starting point of the interval. That is,
does not depend on s. Intuitively, the interval can be “slide” around without changing its stochastic nature.
– Stochastic Process course notes
A random variable X has the memoryless property if
Def`2: A random variable X has the memoryless property if
The exponential distribution is the only distribution that has the memoryless property (Satisfy definition 2)
3. Useful Properties: First occurrence among events
Assume are exponential variable with rate .
Then what is the probability that .
4. Distribution of time of first event
This is the CDF of an exponential RV with rate
5. Distribution of time of last event (maximum)