A Markov model is a mathematical framework that predicts what happens next in a system based entirely on where the system is right now, ignoring everything that happened before. It works by defining a set of possible states, then assigning probabilities to every possible transition between those states. If you know the current state and the transition probabilities, you can forecast how the system will evolve over time. Markov models show up in fields ranging from genomics to economics to artificial intelligence, and understanding the basic idea unlocks all of them.
The Core Idea: Only the Present Matters
The defining feature of a Markov model is something called the Markov property, often described as “memorylessness.” It means the probability of moving to any future state depends only on the current state, not on the sequence of states that came before it. Think of it like a board game where your next move depends only on the square you’re standing on, never on how you got there.
This is a simplification of reality, and an intentional one. Most real systems do carry some history. A patient who has relapsed three times is probably different from a patient who has relapsed once, even if both are currently in the same health state. But the memoryless assumption makes the math tractable and, in many situations, produces results accurate enough to be genuinely useful.
States, Transitions, and the Transition Matrix
Every Markov model has two building blocks: states and transition probabilities. States are the distinct conditions the system can be in. For a weather model, the states might be “sunny,” “cloudy,” and “rainy.” For a disease model, they might be “healthy,” “sick,” and “dead.” The system occupies exactly one state at any given time.
Transition probabilities describe how likely the system is to move from one state to another in a single time step. These probabilities are organized into a grid called a transition matrix, where each row represents a current state and each column represents a possible next state. One firm mathematical rule governs this matrix: every row must sum to 1, because the system has to go somewhere (including possibly staying in the same state).
To estimate these probabilities from real data, you count transitions. If you observe a system in state A a total of 100 times, and 30 of those times it moves to state B next, the estimated probability of transitioning from A to B is 30/100, or 0.3. This counting approach is the standard statistical method for fitting a Markov model to observed data, and it works well when you have enough observations for each state.
Discrete Time vs. Continuous Time
Markov models come in two main flavors depending on how they handle time. In a discrete-time Markov chain, time is chopped into fixed intervals (daily, weekly, yearly), and the system can only change states at those regular checkpoints. This is the most common version and the easiest to work with.
In a continuous-time Markov chain, state changes can happen at any moment. There are no fixed time steps. Instead, the time spent in each state before transitioning follows a specific probability distribution. The concept of a “single-step transition probability” doesn’t apply in continuous time because there is no defined step.
The distinction matters in practice. Discrete-time models can introduce errors by forcing events into rigid time slots. If your model uses yearly cycles but a patient could realistically develop a complication and recover from it within the same year, the model might miss that sequence entirely. Continuous-time models avoid this problem but are harder to build and compute.
Hidden Markov Models
A standard Markov model assumes you can directly observe which state the system is in. A hidden Markov model (HMM) drops that assumption. In an HMM, the true states are invisible. What you observe instead are signals, or “symbols,” that are influenced by the hidden state but don’t reveal it directly.
An HMM is really two layered processes running simultaneously: an invisible process of hidden states that follows a Markov chain, and a visible process of observations whose probability depends on whatever hidden state the system currently occupies. The goal is to use the observable signals to infer the most likely sequence of hidden states.
HMMs are workhorses in biological research. They’re used for gene prediction (figuring out which stretches of DNA encode proteins), sequence alignment (lining up DNA or protein sequences to find similarities), predicting protein structure, identifying non-coding RNA, and modeling DNA sequencing errors. Speech recognition systems also relied heavily on HMMs for decades before deep learning took over.
Markov Decision Processes
A Markov Decision Process (MDP) extends the basic Markov model by adding two ingredients: actions and rewards. In a plain Markov chain, transitions just happen according to fixed probabilities. In an MDP, an agent chooses actions that influence which transitions occur, and each state-action combination produces a reward (or penalty).
An MDP has five components: a set of possible states, a set of possible actions, transition probabilities that depend on both the current state and the chosen action, a reward function that assigns a value to each state-action pair, and a discount factor that controls how much the agent values future rewards compared to immediate ones. The objective is to find a strategy (called a policy) that maximizes total reward over time.
MDPs are the mathematical backbone of reinforcement learning, the branch of artificial intelligence where systems learn by trial and error. Every time you hear about an AI learning to play a game or control a robot, an MDP or something very close to it is usually involved.
How Markov Models Work in Healthcare
One of the most practical applications of Markov models is in health economics, where they simulate how groups of patients move through disease states over time. A simple example: researchers modeling patients on mechanical ventilation used a three-state Markov chain with the states “intubated,” “extubated,” and “dead.” Each day, patients had a 10% probability of being successfully removed from the ventilator, a 2.2% probability of dying, and extubated patients had a 1% probability of needing the ventilator again. After simulating 28 days starting with 100 intubated patients, cumulative mortality reached 35.6%, closely matching the results of the actual clinical study the model was based on.
These models become especially powerful when paired with cost-effectiveness analysis. Each state gets assigned a quality-of-life weight and a cost. A year spent in perfect health equals one Quality-Adjusted Life Year (QALY), while death equals zero, and states involving illness fall somewhere in between. By running the model forward and tallying up QALYs and costs for different treatment strategies, researchers can estimate which approach gives the most health benefit per dollar spent. This kind of analysis directly informs insurance coverage decisions and clinical guidelines.
Where the Memoryless Assumption Breaks Down
The biggest limitation of Markov models is the very thing that makes them elegant: memorylessness. In many real situations, history matters. A cancer patient’s prognosis depends not just on their current stage but on how many rounds of treatment they’ve already received. A machine’s failure risk depends on its cumulative wear, not just its current operating condition.
There are workarounds. The most common is adding “tunnel states,” which are temporary states that track how long a system has been in a particular condition. A patient who has been sick for one month occupies a different state than a patient who has been sick for six months, even though both are “sick.” This solves the memory problem but creates a new one: the number of states can explode, making the model unwieldy.
Discrete-time models face an additional technical issue. Converting real-world rates into the step-by-step probabilities the model needs can introduce subtle errors if done incorrectly. A poorly constructed discrete-time model may rule out event sequences that would genuinely occur in continuous time, effectively disconnecting the model from the reality it’s supposed to represent. Researchers working with Markov models in health technology assessments have flagged this as a common and underappreciated source of error.
Why Markov Models Are So Widely Used
Despite their limitations, Markov models persist across disciplines because they balance simplicity with usefulness. They require relatively little data to parameterize (you just need enough transition counts), they’re computationally cheap to run, and they produce interpretable results. You can look at a transition matrix and immediately understand the dynamics of the system.
They also scale naturally. A two-state model can capture something as simple as a coin flip pattern. A model with hundreds of states can represent the full progression of a complex disease or the behavior of a protein folding into its three-dimensional shape. The underlying math stays the same regardless of the number of states, which means the same framework and software tools work across vastly different problems.