A Markov model is a mathematical way of predicting what happens next in a system based only on where it is right now, not on its history. If you’ve ever seen your phone suggest the next word while you’re typing, you’ve used a product built on this idea. The core principle is simple: the future depends only on the present state, never on the sequence of events that led to it. This “memoryless” property is what makes Markov models both powerful and surprisingly practical across fields from medicine to finance to artificial intelligence.
The Core Idea: Only the Present Matters
Imagine you’re tracking the weather and it’s sunny today. A Markov model would predict tomorrow’s weather based entirely on the fact that it’s sunny right now. It wouldn’t care whether yesterday was rainy, snowy, or also sunny. That single rule, where the next step depends only on the current step, is called the Markov property. Andrei Markov, a Russian mathematician, introduced this concept in 1906, and the family of models that followed bears his name.
This memoryless quality might seem like a limitation, and in some cases it is. But it turns out that many real-world systems behave this way, or close enough that the simplification still produces useful results. A patient’s disease progression often depends more on their current health state than on the exact path they took to get there. A stock market’s next move is shaped more by current conditions than by prices from three months ago. By ignoring history, Markov models become computationally fast and surprisingly accurate for a wide range of problems.
How a Markov Model Is Built
Every Markov model has three ingredients. First, a set of states, which are the possible situations the system can be in. For a weather model, the states might be sunny, cloudy, and rainy. For a disease model, the states might be healthy, mild symptoms, severe symptoms, and recovered.
Second, each model needs transition probabilities, which describe the chance of moving from one state to another. These are usually organized in a grid called a transition matrix. If there’s a 70% chance a sunny day is followed by another sunny day and a 30% chance it’s followed by a cloudy day, those numbers go into the matrix. Every row in this grid adds up to exactly 1, because the system has to go somewhere (including possibly staying in the same state).
Third, there’s a starting distribution, which describes where the system begins. Maybe there’s a 50% chance you start on a sunny day and a 50% chance you start on a cloudy one. Together, these three components fully define how the model behaves over time. You can run it forward step by step, and at each step the transition probabilities determine what happens next.
Reaching a Steady State
One of the most useful properties of many Markov models is that they eventually settle into a stable pattern called an equilibrium distribution. Run the model long enough, and the proportion of time spent in each state stops changing, regardless of where you started. A weather model might settle into spending 60% of its time in sunny, 25% in cloudy, and 15% in rainy, no matter whether the first day was a downpour or a clear sky.
This convergence happens when the model meets two conditions: every state can eventually be reached from every other state, and the system doesn’t get trapped in a repeating cycle. When those hold, the long-run behavior becomes predictable even if any individual step is random. This is what lets researchers answer questions like “over a 20-year period, what fraction of the time will a patient spend in each stage of a disease?”
Hidden Markov Models
In many real situations, you can’t directly see what state the system is in. You can only observe clues that hint at the underlying state. A Hidden Markov Model (HMM) handles exactly this situation. It has two layers: a hidden layer of states that follows the Markov property, and a visible layer of observations that each state produces.
Speech recognition is a classic example. The words someone intends to say are the hidden states. The messy audio signal your microphone picks up is the observation. The model’s job is to figure out the most likely sequence of hidden states (words) given the observations (sound waves). HMMs are sometimes called “doubly stochastic” because randomness operates at both levels: the hidden states shift randomly according to transition probabilities, and each hidden state generates observations with its own set of probabilities.
Text Prediction and Language Models
The autocomplete on your phone is a direct descendant of Markov’s 1906 idea. In a language model, each word (or character) is a state, and the transition probabilities capture how likely one word is to follow another. A bigram model predicts the next word based on the single word before it. A trigram model uses the two previous words. These are still Markov models; they just expand the definition of “current state” to include a small window of recent history.
The tradeoff is straightforward. A bigram model is simple but often produces awkward text because one word of context isn’t much. A trigram model is more accurate but needs far more data to estimate all the transition probabilities reliably, because the number of possible three-word combinations is enormous. Modern large language models have moved well beyond simple Markov chains, but the fundamental insight that you can predict the next word from a limited context still underpins how they work.
Medical and Health Economics Uses
Markov models are a standard tool for simulating how diseases progress over time. Researchers define health states (for example, normal kidney function, early kidney damage, clinical kidney disease, end-stage renal disease, and death) and then assign transition probabilities for moving between them each year. Running the model forward simulates what happens to a group of patients over decades.
A CDC model for type 2 diabetes tracks patient cohorts along five complication paths: kidney disease, nerve damage, eye disease, coronary heart disease, and stroke. At the end of each one-year cycle, portions of the cohort move between states or stay put. The model assigns a quality-of-life score between 0 and 1 to each state. Someone with diabetes-related blindness, for instance, receives a quality-of-life reduction of about 0.16 from a baseline of 1. By multiplying these scores by the time spent in each state and summing across a patient’s lifetime, researchers calculate quality-adjusted life years (QALYs), a single number that captures both how long and how well patients live under different treatment strategies. This is how healthcare systems compare the cost-effectiveness of competing interventions.
Markov Decision Processes in AI
When you add the ability to choose actions and earn rewards to a Markov model, you get a Markov Decision Process (MDP). This is the mathematical backbone of reinforcement learning, the branch of AI where an agent learns by trial and error. An MDP has five components: a set of states, a set of actions, transition probabilities that now depend on which action is taken, a reward function that scores each state-action pair, and a discount factor that controls how much the agent values future rewards versus immediate ones.
A robot navigating a warehouse, a program learning to play chess, or an algorithm optimizing ad placement all operate within this framework. The agent observes its current state, picks an action, receives a reward, and lands in a new state. Over thousands or millions of repetitions, it learns which actions maximize long-term reward. The Markov property makes this tractable: the agent only needs to know its current state, not every move it has ever made.
Financial Market Applications
In finance, Markov regime-switching models capture the idea that markets operate in different modes. A calm, low-volatility period might behave very differently from a turbulent, high-volatility period. Rather than fitting one model to all market data, a Markov-switching model allows the market to flip between regimes, with transition probabilities governing how likely it is to stay in the current regime or shift to a new one.
Recent research on forecasting stock market volatility found that the abrupt-transition approach of Markov-switching models outperformed smoother alternatives. This makes intuitive sense: markets tend to shift quickly between calm and crisis, not gradually. The model’s structure captures that reality by treating each regime as a distinct state with its own statistical behavior, connected by Markov transition probabilities.