Diffusion models are a family of generative AI systems that create new data, whether images, audio, video, or molecular structures, by learning to reverse a process of gradual noise corruption. The core idea is borrowed from physics: take a piece of real data, slowly add random noise until it becomes pure static, then train a neural network to undo that destruction step by step. Once the network has learned the reversal, you can feed it pure noise and watch it sculpt something that looks like it belongs in the original dataset. This deceptively simple concept has powered most of the high-profile image generators you have encountered in recent years, and it has quietly spread into drug design, protein engineering, and medical imaging.
How the Forward and Reverse Process Works
Picture dropping ink into a glass of water. Over time, the ink disperses until the water is a uniform murky color. That dispersal is easy to describe mathematically. Now imagine filming the process and playing it backward: the murky water gradually organizes itself until the ink re-forms into a coherent drop. A diffusion model learns to play that tape in reverse. The “forward” diffusion process takes a training image and adds small amounts of Gaussian noise at each of many timesteps, eventually turning it into something indistinguishable from random static. A neural network is then trained to predict, at each timestep, what the slightly less noisy version should look like. The foundational 2015 paper by Sohl-Dickstein and colleagues described this as systematically destroying structure through an iterative forward diffusion process, then learning a reverse process that restores it.1arXiv. Deep Unsupervised Learning using Nonequilibrium Thermodynamics
What makes this work so well in practice is that the network does not need to generate an image from scratch in a single pass. Instead, it makes a long sequence of small, corrective adjustments, each one slightly refining the signal buried in the noise. That iterative refinement turns out to be far more stable and controllable than earlier approaches that tried to generate a full image in one shot. Later theoretical work unified the forward and backward dynamics under the language of stochastic differential equations, showing that several seemingly different formulations of diffusion models were actually special cases of a single mathematical framework.2arXiv. Score-Based Generative Modeling through Stochastic Differential Equations
Researchers in physics have also found the framework illuminating from their own perspective. A 2025 analysis rooted in path integrals and concepts from spin glass theory formulated entropy production, equilibrium measures, and phase transitions within the diffusion generation process, treating the reverse diffusion as a form of statistical inference.3PubMed. Nonequilbrium physics of generative diffusion models In other words, diffusion models are not just engineering tools; they sit at a genuine intersection of machine learning and statistical physics, which is part of why they have attracted attention from such a wide range of disciplines.
From Pixels to Latent Space
Early diffusion models operated directly on raw pixels. For a modest 256×256 image, that means the network is denoising a signal with nearly 200,000 values at every single timestep. Scale that up to the resolutions people actually want, like 1024×1024, and the computational cost becomes brutal. The breakthrough that made diffusion models practical for everyday use was the move to latent space. Instead of adding and removing noise on the full image, you first compress the image into a much smaller representation using an autoencoder, run the diffusion process there, and then decode the result back into pixel space.
This is the idea behind latent diffusion models, introduced by Rombach and colleagues in what became the backbone of Stable Diffusion. Their paper showed that running diffusion in latent space achieved a near-optimal balance between complexity reduction and detail preservation, significantly cutting computational requirements compared to pixel-based approaches while boosting visual fidelity.4arXiv. High-Resolution Image Synthesis with Latent Diffusion Models The practical effect was enormous: suddenly, researchers without warehouse-scale GPU clusters could train and experiment with high-resolution generative models.
The Shift from U-Nets to Transformers
For most of the diffusion model era, the workhorse neural network architecture has been the U-Net, a convolutional network originally designed for medical image segmentation. It processes the noisy input at multiple resolutions through a series of downsampling and upsampling stages, with skip connections that carry fine detail from early layers to later ones. U-Nets work well, but they were designed for pixel grids and do not scale as gracefully as some alternatives.
The architecture trend in 2023 and 2024 shifted toward transformers, the same family of models behind large language models. The Diffusion Transformer (DiT) approach replaces the U-Net with a transformer that operates on patches of the latent representation. Research showed that scaling up the transformer, whether by making it deeper, wider, or feeding it more tokens, consistently improved image quality as measured by standard benchmarks.5arXiv. Scalable Diffusion Models with Transformers This scaling behavior is familiar from language models and suggests that diffusion transformers have not yet hit a performance ceiling.
More recently, researchers have pushed this further using mixture-of-experts techniques, where the model activates only a subset of its parameters for any given input. A sparse DiT variant scaled to 16.5 billion parameters achieved a state-of-the-art FID score of 1.80 at 512×512 resolution, demonstrating that the transformer scaling trend extends to very large models while keeping inference costs manageable through sparsity.6arXiv. Scaling Diffusion Transformers to 16 Billion Parameters For context, FID is a standard metric where lower is better, measuring how closely generated images resemble real ones, and scores under 2.0 represent outputs that are extremely difficult to distinguish from photographs.
How Text Prompts Steer Image Generation
When you type a prompt into an image generator, the words need to influence the denoising process at every timestep. This happens through a mechanism called cross-attention, where the text is first converted into a numerical representation by a language model, and those text features are then injected into the diffusion network’s internal layers. At each denoising step, the network “attends” to different parts of the text, deciding which spatial regions of the emerging image should correspond to which words.
Despite how central this mechanism is, researchers note that the transformation from text embeddings to latent visual features remains only partially understood.7Proceedings of the AAAI Conference on Artificial Intelligence. Mechanistic Dissection of Cross-Attention Subspaces in Text-to-Image Diffusion Models This is part of why prompt engineering can feel unpredictable: small changes in wording sometimes produce dramatic shifts in the output, while seemingly important additions get ignored. The cross-attention layers are doing a complicated dance between linguistic meaning and spatial layout, and the model’s priorities at each timestep are not always obvious from the outside.
The most widely used technique for boosting the effect of text conditioning is classifier-free guidance, where the model generates two predictions at each step, one conditioned on the text and one without, and then amplifies the difference. Turning up the guidance scale makes images more closely match the prompt but can introduce artifacts or oversaturation. Turning it down produces more diverse but potentially less text-faithful results.
The Speed Problem and How It Is Being Solved
The biggest practical drawback of diffusion models has always been speed. A standard diffusion model might need hundreds or even a thousand denoising steps to produce a single image. Each step requires a full pass through the neural network, so generation is orders of magnitude slower than older approaches like GANs, which produce an image in a single forward pass.
The first major speedup came from Denoising Diffusion Implicit Models (DDIMs), which reformulated the reverse process to be deterministic rather than stochastic. This seemingly technical change had a dramatic practical effect: DDIMs could produce high-quality samples 10 to 50 times faster than the original approach, because deterministic trajectories allow you to skip timesteps without the quality collapsing.8arXiv. Denoising Diffusion Implicit Models Further work on generalized versions of this approach provided a clearer theoretical picture of why deterministic sampling outperforms stochastic sampling when you are taking fewer steps.9arXiv. gDDIM: Generalized denoising diffusion implicit models
Consistency models took a more radical approach. Instead of running the full denoising chain, they train a network to jump directly from any noise level to the final clean output in a single step. They can also be run for a few steps if you want higher quality, giving you a knob to trade compute for fidelity. The key advantage is that consistency models can be distilled from pretrained diffusion models, inheriting their quality without their slowness.10arXiv. Consistency Models Latent consistency models applied this idea specifically to latent diffusion, enabling generation of 768×768 images in just two to four steps, with the distillation training taking only about 32 A100 GPU hours, a tiny fraction of the cost of training the original model from scratch.11arXiv. Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference
Hybrid approaches have also shown promise. Denoising diffusion GANs combined the diffusion framework with adversarial training, achieving sample quality competitive with full diffusion models while being roughly 2,000 times faster on standard benchmarks.12arXiv. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs The speed improvements across these different methods have collectively brought generation times from minutes down to seconds or even fractions of a second, which is what made real-time and interactive applications feasible.
Beyond Images: Audio, Video, and 3D
Although images get most of the attention, diffusion models have spread into nearly every generative domain. In audio, diffusion-based systems handle both text-to-speech synthesis and speech enhancement. DiffWave, one of the earlier audio diffusion models, demonstrated that the same denoising framework could generate raw audio waveforms for both conditional and unconditional tasks.13arXiv. DiffWave: A Versatile Diffusion Model for Audio Synthesis More recent work has combined autoregressive generation with diffusion, encoding audio as continuous vector sequences and generating them token by token with a decoder-only diffusion transformer.14arXiv. Autoregressive Diffusion Transformer for Text-to-Speech Synthesis The result is speech synthesis that can handle the prosody and naturalness challenges that plagued earlier concatenative and purely autoregressive systems.
Video generation adds the challenge of temporal coherence: not just making each frame look good, but making them flow together as a believable sequence. Diffusion-based video models extend the spatial denoising process into the time dimension, applying noise and denoising across frames simultaneously to maintain consistency.15arXiv. Video Diffusion Models: A Survey The computational cost scales steeply with frame count and resolution, which is why video generation still lags behind images in terms of accessible quality, but progress over the past two years has been striking.
For 3D content, diffusion models have been adapted through techniques like score distillation, where a pretrained 2D image diffusion model guides the optimization of a 3D representation. A text prompt gets rendered from multiple viewpoints, and the 2D model provides gradient signals about how realistic each view looks. Early versions suffered from over-saturation and loss of detail, but improved methods like variational score distillation have substantially raised the fidelity of text-to-3D generation.16arXiv. ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation
Scientific Applications
Some of the most consequential uses of diffusion models are happening in the sciences, far from consumer-facing image generators. In structural biology, diffusion models have been applied to protein structure generation by treating the backbone as a sequence of angles and “folding” from a random unfolded state toward a stable conformation, mimicking the natural folding process. A transformer-based denoising model trained this way unconditionally generates protein structures with complexity and patterns resembling those of naturally occurring proteins.17PubMed Central. Protein structure generation via folding diffusion This is useful not because we need more copies of existing proteins, but because it points toward designing entirely new ones with specific properties.
Drug discovery represents another active frontier. Diffusion models can generate novel small molecules designed to fit a target protein’s binding pocket, producing candidates with desired physical and chemical properties. One system, MolSnapper, demonstrated the ability to generate 3D molecular structures that reproduce most of the hydrogen bond interactions between a reference drug and its protein target, yielding roughly twice as many valid molecules as competing methods.18PubMed Central. MolSnapper: Conditioning Diffusion for Structure-Based Drug Design A key remaining challenge for small-molecule generation is ensuring that the molecules the model invents can actually be synthesized in a real lab, not just exist as plausible-looking 3D coordinates.19PubMed Central. Diffusion Models at the Drug Discovery Frontier: A Review on Generating Small Molecules Versus Therapeutic Peptides
In medical imaging, diffusion models are being used to reconstruct high-quality images from noisy or undersampled scan data. This matters in clinical settings where faster scans (which collect less data) are safer or more comfortable for patients but traditionally produce lower-quality images. Diffusion-based reconstruction can recover diagnostic-quality images from these degraded inputs, potentially enabling faster scans without sacrificing the information that radiologists need.20PubMed Central. A Systematic Review of Diffusion Models for Medical Image-Based Diagnosis: Methods, Taxonomies, Clinical Integration, Explainability, and Future Directions
Memorization, Privacy, and the Copyright Problem
A persistent concern with any generative model trained on large datasets is memorization: the possibility that the model has not just learned general patterns but has stored specific training examples and can reproduce them verbatim. For diffusion models, this risk turns out to be non-trivial. Research on diffusion language models has established a theoretical relationship between sampling resolution and memorization, showing that increasing resolution strictly raises the probability of extracting exact training data.21arXiv. Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
On the image side, there has been an assumption that unconditional diffusion models (those without text prompts) are relatively safe from data extraction attacks because there is no easy way to target a specific training example. Recent work challenges this directly. The SIDE framework demonstrated that by constructing data-driven surrogate conditions, researchers could successfully extract training data even from unconditional models previously considered safe, and the attack outperformed baselines even on conditional models.22Proceedings of the AAAI Conference on Artificial Intelligence. SIDE: Surrogate Conditional Data Extraction from Diffusion Models The theoretical takeaway is that all forms of conditioning, whether explicit prompts or inferred surrogates, amplify memorization. This has real implications for copyright and privacy: if a model can be coaxed into reproducing a specific photographer’s image or a specific person’s face from its training data, the legal and ethical ground gets complicated fast.
One response to these concerns is watermarking. Rather than trying to prevent memorization entirely, some researchers are focused on making it possible to trace where a generated image came from and whether it has been tampered with. GenPTW, for example, embeds watermarks directly into the latent space during or after generation, supporting both provenance tracing and tamper localization without altering the generative process itself, and is compatible with latent diffusion models as a plug-and-play addition.23Proceedings of the AAAI Conference on Artificial Intelligence. GenPTW: Latent Image Watermarking for Provenance Tracing and Tamper Localization A complementary approach called Dual-Guard combines two watermark channels, one embedded in the initial noise and one in the final denoised output, to create both a global provenance signal and a content-level fingerprint.24arXiv. Dual-Guard: Dual-Channel Latent Watermarking for Provenance and Tamper Localization in Diffusion Images Neither approach solves the fundamental tension between open generative models and content provenance, but they represent the most technically mature attempts so far.
Discrete Diffusion and Non-Image Data
Most of the diffusion models that reach the public operate on continuous data: pixel values, audio waveforms, or molecular coordinates. But there is growing interest in discrete diffusion models that work on categorical data like text tokens or DNA sequences. The challenge is that you cannot just “add a little noise” to a word the way you can to a pixel value. Instead, discrete diffusion corrupts tokens by randomly swapping them for other tokens, gradually turning coherent text into random gibberish, and then trains a model to reverse that corruption.
Speed has been a particular bottleneck here, because discrete diffusion processes typically require even more steps than their continuous counterparts. Discrete non-Markov diffusion models address this by allowing the forward process to skip timesteps in a principled way, enabling a training-free sampling algorithm that significantly reduces the number of neural network calls needed during generation.25NeurIPS Proceedings. Accelerating Discrete Diffusion Models via Non-Markovian Forward Processes Whether discrete diffusion can eventually compete with autoregressive models for text generation remains an open question, but the theoretical unification between the two paradigms, where autoregressive generation can be seen as a limiting case of diffusion with maximal sampling resolution, suggests the boundary between them is fuzzier than it first appears.
Why Diffusion Models Displaced GANs
Before diffusion models took over, generative adversarial networks were the dominant approach for image generation. GANs pit a generator against a discriminator in a minimax game and can produce sharp images in a single forward pass. So why did diffusion models win?
The short answer is mode coverage. GANs are prone to “mode collapse,” where the generator learns to produce a narrow range of outputs that fool the discriminator rather than covering the full diversity of the training data. You might get photorealistic faces, but they all share suspiciously similar features. Diffusion models, by contrast, are trained with a straightforward denoising objective that naturally encourages the model to represent the entire data distribution. The result is both higher diversity and, in most benchmarks, comparable or better perceptual quality. Hybrid models that combine both frameworks have confirmed this tradeoff: denoising diffusion GANs showed better mode coverage and sample diversity than traditional GANs while approaching diffusion-level quality.26arXiv. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
GANs also suffered from training instability. The adversarial dynamic is finicky: if the discriminator gets too good too fast, the generator receives useless gradients and stops improving. If the generator fools the discriminator too easily, quality stalls. Diffusion models sidestep this entirely because there is no adversarial component. The training objective is a regression problem, predicting the noise that was added, which behaves much more predictably during optimization. For large-scale production systems where reliability matters as much as peak quality, that stability advantage proved decisive.
Density Estimation and What Else Diffusion Models Can Tell You
Beyond generating new samples, diffusion models have a less flashy but scientifically valuable capability: they can estimate how likely a given input is under the training distribution. In other words, you can hand a trained diffusion model a new image and ask, “How probable is this according to what you learned?” This is done through the probability flow ordinary differential equation formulation, which provides unbiased density estimates.27arXiv. Investigating the Adversarial Robustness of Density Estimation Using the Probability Flow ODE
This matters for applications like anomaly detection, where you want to flag inputs that are unusual relative to a known distribution. A medical imaging system trained on healthy scans could, in principle, use density estimation to identify scans that look abnormal. It also matters for understanding what the model has actually learned: if certain types of images consistently receive high likelihood scores, that tells you something about the biases and coverage of the training data. The robustness of these density estimates under adversarial perturbations is an active area of study, because if an attacker can slightly modify an image to fool the density estimator, the downstream applications become unreliable.