The entropy formula most people encounter first is Boltzmann’s: S equals k times the natural log of W, where S is entropy, k is Boltzmann’s constant, and W is the number of microscopic arrangements that produce a given macroscopic state. But that single equation is really the starting point of a much larger family. Shannon borrowed the same logarithmic structure for information theory, von Neumann extended it to quantum mechanics, and Bekenstein applied it to black holes. Each version looks slightly different on paper, yet they all share the same core logic: entropy counts how many hidden possibilities are compatible with what you can actually observe.
Boltzmann’s Formula and What It Actually Counts
Ludwig Boltzmann’s insight was that the thermodynamic entropy Rudolf Clausius had defined through heat flow could be recast as a counting problem. If you know the large-scale properties of a system, like its temperature, pressure, and volume, there are many microscopic configurations of individual particles that could produce those properties. Boltzmann defined entropy as the logarithm of the volume of that set of microstates. The bigger the set of hidden arrangements, the higher the entropy. This definition matches the thermodynamic entropy of Clausius when you’re dealing with a system in local thermal equilibrium and the quantities you’re tracking are the locally conserved ones like energy and particle number.1Physica D: Nonlinear Phenomena. On the (Boltzmann) entropy of non-equilibrium systems
The constant k in the formula (Boltzmann’s constant, roughly 1.38 × 10⁻²³ joules per kelvin) is what gives entropy its physical units. Without it, you’d just have a dimensionless number. The logarithm is there for a practical reason: it makes entropy additive. If you combine two independent systems, the total number of microstates multiplies, but the logarithm turns multiplication into addition, so total entropy is the sum of the parts. That additive property is what makes entropy behave like a well-defined physical quantity you can track as systems change.
Shannon Entropy and the Information Theory Version
In 1948, Claude Shannon needed a way to quantify uncertainty in a message. If you’re waiting for a signal that could be one of several symbols, and some symbols are more likely than others, how surprised should you expect to be? Shannon arrived at H equals the negative sum of each probability times its logarithm. When the logarithm is base 2, the units are bits. When it’s the natural log, the units are called nats.
The structural resemblance to Boltzmann’s formula is not a coincidence. If every microstate is equally likely (which is the assumption in the classic Boltzmann setup), the Shannon formula reduces to the log of the number of microstates, exactly Boltzmann’s expression with the constant stripped off. Shannon entropy is really the general case; Boltzmann entropy is the special case where every possibility has the same probability. This connection has been made precise: for any probability distribution you can compute with an algorithm, the expected Kolmogorov complexity of an outcome equals its Shannon entropy, up to a constant.2Entropy. Entropy Measures vs. Kolmogorov Complexity In other words, the average amount of information you need to specify an outcome and the average “surprise” per outcome are measuring the same thing from two directions.
Gibbs Entropy and Why It Matters for Interacting Systems
Boltzmann’s formula works beautifully when you can assume every microstate is equally probable. Many real systems don’t cooperate. In a gas at equilibrium, that assumption holds reasonably well, but in complex systems with interactions and constraints, some configurations are far more likely than others. The Gibbs entropy formula handles this by working directly with the probability distribution over microstates, using the same negative-sum-of-p-log-p form that Shannon later reinvented. It’s the more general statistical-mechanics version, and it underpins most modern work in the field.
The Gibbs approach extends naturally to systems that aren’t ordinary gases. Researchers have applied it to network ensembles, for instance, where the “microstates” are all the different network wiring diagrams that satisfy a given set of constraints. The Gibbs entropy of such an ensemble is the logarithm of the number of compatible network configurations, and it captures how much structural randomness a real network encodes.3PubMed Central. Gibbs entropy of network ensembles by cavity methods The same formula, applied to a completely different kind of system, still measures the same fundamental thing: how many hidden arrangements are consistent with what you observe.
Quantum Entropy and the Von Neumann Formula
In quantum mechanics, the state of a system isn’t described by a list of particle positions but by a mathematical object called a density matrix. The quantum analog of entropy, introduced by John von Neumann, replaces the classical probability distribution with this density matrix. The formula looks like S equals the negative trace of the density matrix times its logarithm. For a system in a pure quantum state (one with no uncertainty at all), the von Neumann entropy is zero. For a mixed state, where the system could be in several quantum states with various probabilities, the entropy is positive.
Defining entropy this way in quantum settings turns out to be subtle. A given density matrix can be decomposed into pure states in more than one way, and the von Neumann entropy has a special minimality property with respect to those decompositions. Researchers have used this property to build a purely algebraic definition of entropy for quantum systems that resolves ambiguities arising from the multiple possible decompositions.4PubMed Central. Entropy of Quantum States
One striking result from quantum entropy calculations involves topological phases of matter. When you compute a generalized family of entanglement entropies (the Rényi entropies, parametrized by a tunable exponent) for topologically ordered systems, you’d expect different values for different exponents. But for all nonchiral topological phases, the topological part of the entanglement entropy turns out to be the same regardless of which exponent you use. This means the reduced density matrices in these phases have a surprisingly simple structure, and you can’t extract additional universal information by varying how you measure entanglement.5PubMed Central. Topological entanglement Rényi entropy and reduced density matrix structure
Black Hole Entropy and the Bekenstein-Hawking Area Law
Perhaps the most dramatic application of entropy formulas sits at the intersection of gravity and quantum mechanics. In the early 1970s, Jacob Bekenstein proposed that a black hole’s entropy is proportional to the area of its event horizon, not its volume. Stephen Hawking then showed, using quantum field theory in curved spacetime, that black holes radiate thermally, and the proportionality constant works out to one quarter (in appropriate units): S equals A divided by 4.6arXiv. Black Hole Thermodynamics: Established Results, Unresolved Paradoxes, and Speculative Resolutions
This area-entropy relationship was initially derived from the generalized second law of thermodynamics for black holes and leads to a far-reaching idea known as the holographic principle: the entropy inside any region of space is bounded by the area of its boundary, not by the volume enclosed.7Entropy. Entropy Bounds, Holographic Principle and Uncertainty Relation That’s a deeply counterintuitive result. You’d normally expect that packing more stuff into a bigger volume means more possible arrangements and higher entropy. The holographic bound says the surface area is the real limit.
The Bekenstein-Hawking formula isn’t the end of the story. More refined calculations show that beyond the leading area term, there’s a correction proportional to the logarithm of the area, plus further sub-leading terms that form a series in inverse powers of the area.8PubMed. Logarithmic correction to the bekenstein-hawking entropy These corrections are small for large black holes but become significant as the black hole shrinks, and they are a key testing ground for candidate theories of quantum gravity.
Entropy in Ecology and Biodiversity
Ecologists have been using entropy formulas since the mid-20th century to measure biodiversity, though the connection isn’t always made explicit. If you survey a forest and record the relative abundances of different tree species, you can compute Shannon entropy on those abundance proportions. A forest dominated by a single species has low entropy; one with many species in roughly equal numbers has high entropy. The resulting number is commonly called the Shannon diversity index.
But Shannon entropy is only one member of a broader family. The Hill numbers, widely used in ecology, are directly related to both Rényi entropy and Shannon entropy from information theory.9PubMed Central. Mathematical constraints on a family of biodiversity measures via connections with Rényi entropy Hill numbers are parametrized by an order q that determines how much weight rare species get. At q = 0, you’re just counting species (richness). At q = 1, you recover a quantity directly linked to Shannon entropy. At q = 2, rare species matter less, and you get a measure tied to the Simpson index. Because the compositional complexity of an assemblage can’t be captured by a single number, ecologists plot entire profiles of these diversity measures across different values of q to compare sites.10Methods in Ecology and Evolution. Estimating diversity and entropy profiles via discovery rates of new species Two forests might have the same Shannon diversity but differ sharply at other orders, reflecting different patterns of rare-species abundance.
Entropy in Machine Learning
If you’ve trained a neural network, you’ve almost certainly minimized cross-entropy loss, whether you realized it or not. Cross-entropy is built directly from Shannon’s formula. When a classifier outputs predicted probabilities for each category and you compare those to the true labels, the cross-entropy measures how many extra bits of surprise the model’s predictions incur relative to perfect predictions. Minimizing that surprise is what drives the model to produce better probability estimates.
A closely related measure, the Kullback-Leibler divergence (KL divergence), quantifies the difference between two probability distributions. It assesses how much information is lost when one distribution is used to approximate another, which makes it a natural tool for comparing a model’s output to training data.11PubMed Central. Exact Expressions for Kullback-Leibler Divergence for Univariate Distributions KL divergence is not symmetric: the “distance” from distribution p to distribution q is generally different from q to p, which means it isn’t a true distance in the mathematical sense. Despite this, it’s the standard error measure for model fitting across statistics and machine learning.12PubMed Central. Computation of Kullback-Leibler Divergence in Bayesian Networks Cross-entropy loss, it turns out, equals the KL divergence plus the entropy of the true distribution, and since the true distribution’s entropy is fixed during training, minimizing cross-entropy and minimizing KL divergence amount to the same thing.
Landauer’s Principle and the Physical Cost of Erasing a Bit
One of the most surprising consequences of the entropy formula is that erasing information has a thermodynamic cost. Landauer’s principle states that erasing a single bit of information irreversibly produces a minimum entropy of k_B times the natural log of 2, which at room temperature corresponds to a tiny but nonzero amount of energy dissipated as heat.13Nature Physics. Quantum Landauer erasure with a molecular nanomagnet This limit is fundamental: it connects information theory to thermodynamics at the most basic level, and it resolved the old paradox of Maxwell’s demon by showing that the demon must eventually erase its memory, paying the entropy cost that keeps the second law intact.
Reaching the Landauer limit exactly would require an infinitely slow erasure process. Any real operation carried out in finite time costs more energy.14PubMed. Minimal energy cost to initialize a bit with tolerable error Current computer chips dissipate energy per bit operation that’s thousands of times above the Landauer floor, so in practice, engineering inefficiencies swamp the fundamental limit. But as devices shrink and energy budgets tighten, the gap between practical costs and the theoretical minimum keeps narrowing, making the principle increasingly relevant for chip designers thinking about the ultimate limits of computation.
Non-Equilibrium Extensions and the Jarzynski Equality
The classical entropy formulas assume you’re dealing with systems in or near equilibrium. Nature rarely cooperates. Biological cells, weather systems, and industrial processes all operate far from equilibrium, and extending entropy concepts to those settings has been one of the big projects in modern thermodynamics.
A major breakthrough came with the Jarzynski equality, proved in 1997, which relates the irreversible work done on a system to its equilibrium free energy difference. Free energy is the portion of a system’s energy available to do useful work, and the second law of thermodynamics normally says you can’t extract the free energy difference from irreversible measurements, only from idealized reversible ones. Jarzynski showed you can, if you average the exponential of the work over many repetitions of a non-equilibrium process. The result was experimentally confirmed by stretching single RNA molecules both reversibly and irreversibly; applying the equality to the irreversible work trajectories recovered the free energy profile to within about half the thermal energy of the best independent estimate from reversible stretching.15PubMed. Equilibrium information from nonequilibrium measurements in an experimental test of Jarzynski’s equality
Another active research area involves the maximum entropy production (MEP) principle, which hypothesizes that thermodynamic processes far from equilibrium tend to settle into steady states that dissipate energy and produce entropy at the maximum possible rate.16PubMed Central. Maximum entropy production in environmental and ecological systems This principle has been applied to climate science with some success: it appears to describe heat transport in Earth’s atmosphere, and similar models seem to work for Mars, Titan, and even mantle convection.17Reviews of Geophysics. The second law of thermodynamics and the global climate system: A review of the maximum entropy production principle The MEP principle remains controversial, with some researchers reinterpreting it as requiring joint optimization of heat transport and mechanical work production rather than simple entropy maximization.18PubMed. Thermodynamic efficiency and entropy production in the climate system
Entropy in Dynamical Systems and Chaos
In the study of chaotic systems, entropy takes yet another form. The Kolmogorov-Sinai (K-S) entropy measures the rate at which a dynamical system generates new information as it evolves. If you watch a chaotic system unfold and try to predict its future state, the K-S entropy tells you how quickly your uncertainty grows per unit time. It’s related to Lyapunov exponents, which measure how fast nearby trajectories diverge, but captures something slightly different: while Lyapunov exponents track how fast individual trajectories separate, K-S entropy measures how fast the system forgets where it started, in the sense that ensembles starting from different initial positions spread out and occupy the full space of possible states.19Monthly Notices of the Royal Astronomical Society. Chaos in three-body dynamics: Kolmogorov—Sinai entropy
A regular, predictable system has zero K-S entropy. A completely random system has infinite K-S entropy. Chaotic systems sit in between, with a finite, positive value that quantifies just how unpredictable they are. This makes K-S entropy a useful diagnostic in astrophysics, fluid dynamics, and anywhere else you need to distinguish genuine chaos from noise or regular motion.
Entropy as a Clinical Tool
Physicians and biomedical engineers have borrowed entropy measures to analyze physiological signals like heart rate variability. Approximate entropy (ApEn), introduced in the early 1990s, quantifies the regularity of a time series: a perfectly periodic heartbeat has low ApEn, while a healthy heart with complex beat-to-beat variability has higher ApEn. A refinement called sample entropy (SampEn) addresses some statistical biases in ApEn, producing more consistent results on short, noisy clinical data.20PubMed. Physiological time-series analysis using approximate entropy and sample entropy
These measures have clinical relevance because certain diseases reduce the complexity of physiological signals. Heart failure, sepsis, and aging all tend to make heart rate time series more regular, which sounds like a good thing but actually reflects a loss of the adaptive variability that a healthy cardiovascular system maintains. Entropy-based metrics give clinicians a way to quantify that loss and potentially detect deterioration earlier than traditional vital signs alone would suggest.
Residual Entropy and the Ice Rule
Even at absolute zero, some materials retain a nonzero entropy, a phenomenon called residual entropy. Ordinary ice is the classic example. In an ice crystal, each oxygen atom is bonded to two nearby hydrogens and has two more hydrogens from neighboring molecules nearby, forming a tetrahedral arrangement. The “ice rule” dictates which of these four hydrogens sit close to the oxygen and which sit farther away, but there are many valid configurations that all satisfy the rule simultaneously. Linus Pauling estimated the resulting residual entropy in 1935, and the measured value agrees well with his calculation.
The origin of this residual entropy has been interpreted through the lens of fractional exclusion statistics: the chemical constraint of O-H bonds in ice acts as a form of statistical interaction, and the residual entropy reflects genuine accessibility to a macroscopic number of ground states through quantum tunneling between them.21Journal of Physics A: Mathematical and General. Residual entropy of ice: a manifestation of the fractional exclusion statistics in real three-dimensional space The possibility that geometric phase effects suppress tunneling splitting and maintain finite residual entropy makes this an active area of low-temperature physics research, not just a textbook curiosity.
The H-Theorem and the Arrow of Time
Boltzmann’s H-theorem provides a microscopic foundation for the second law of thermodynamics by showing that a certain quantity (the H-functional, essentially the negative of entropy) decreases over time as a gas relaxes toward equilibrium. It’s the formal bridge between the reversible laws governing individual particles and the irreversible behavior we see at the macroscopic level.
The relationship between microscopic reversibility and macroscopic irreversibility has remained conceptually tricky ever since Boltzmann’s time. Recent work has taken a novel approach: training neural networks to infer the arrow of time directly from the positions and velocities of simulated hard-disk particles. A permutation-invariant neural network, given only the task of assigning later states a larger scalar value than earlier states, independently discovers a quantity that closely resembles the Boltzmann H-functional.22arXiv. Machine Learning the H-theorem The fact that a model with no built-in physics arrives at the same structure Boltzmann derived analytically is a striking confirmation that entropy increase really is the signature the data points toward when asked “which direction is time going?”