Amdahl’s Law states that the speedup you can achieve by parallelizing a program is fundamentally limited by the fraction of that program that must still run sequentially. If 5% of your code cannot be split across multiple processors, then no matter how many processors you throw at the problem, you will never get more than a 20× speedup. The law was articulated by computer scientist Gene Amdahl in 1967, and it remains one of the most practically important constraints in computing, from laptop CPUs to warehouse-scale data centers training large language models.
What the Law Actually Says
The idea is deceptively simple. Every program has some portion of work that can be divided among multiple processors running simultaneously and some portion that cannot. The portion that cannot be parallelized, the serial fraction, becomes a hard floor on how fast the program can finish. Double the processors and you roughly halve the time spent on the parallel portion, but the serial portion stays exactly the same length. Triple the processors and the parallel portion shrinks further, but again the serial portion is untouched. Eventually, adding more processors buys you almost nothing because you are spending nearly all of your time waiting on the serial part.
Suppose a task takes 100 seconds on one processor, and 10 of those seconds are inherently sequential. With 10 processors, the parallel portion drops from 90 seconds to 9, giving you a total of 19 seconds and roughly a 5× speedup. With 100 processors, the parallel portion drops to under 1 second, but you still spend 10 seconds on the serial work, so you get about a 10× speedup. With a million processors, the parallel portion is negligible but those 10 serial seconds remain, capping you just under 10×. That ceiling, determined entirely by the serial fraction, is the core lesson of Amdahl’s Law.
Where Serial Bottlenecks Actually Live
People sometimes imagine the serial fraction as one big block of code at the beginning or end of a program, like initialization or writing a final output file. That picture is misleading. In real multithreaded software, serial bottlenecks are scattered throughout the program and often disguised. Critical sections where threads must take turns accessing shared data, barriers where all threads must wait for the slowest one to finish, and pipeline stages where one slow step holds up everything downstream all serialize execution and waste processor cycles.
These bottlenecks are insidious because they are often small individually but collectively significant. A lock that protects a shared counter might only take microseconds each time it fires, but if thousands of threads are competing for that lock millions of times, the aggregate waiting time dominates. When many processors try to synchronize on the same variable, contending writes get serialized, creating a condition that does not scale no matter how much hardware you add.1ACM SIGARCH Computer Architecture News. CASPAR Research into architectures that can break this serialization, allowing queued processors to perform synchronization in parallel rather than one at a time, is an active area precisely because the serial penalty is so punishing at scale.
Bottleneck identification in multithreaded applications has become a discipline of its own. Critical sections, barriers, and slow pipeline stages all serialize execution in different ways, and diagnosing which bottleneck type is limiting a particular application requires careful profiling.2ACM SIGARCH Computer Architecture News. Bottleneck identification and scheduling in multithreaded applications A program might look heavily parallel on paper yet perform dismally on many cores because a handful of synchronization points effectively re-serialize it.
Memory and Hardware Constraints as a Hidden Serial Fraction
Amdahl’s Law is typically discussed in terms of code, but hardware creates its own serial bottlenecks. The most common one is main memory bandwidth. A processor can only compute as fast as it can feed itself data, and when multiple cores are all pulling from the same memory bus, they compete for a shared resource that has a fixed throughput ceiling. This is functionally the same as a serial fraction: one resource that cannot be parallelized, limiting the speedup of everything else.
Research on power-aware computing has shown that main memory bandwidth has a significant effect on the relationship between performance and CPU frequency. By modeling the memory bottleneck using Amdahl’s Law, researchers found they could estimate both the performance speedup and peak power consumption to within about 5% of measured values as CPU frequency changed.3ACM Transactions on Embedded Computing Systems. A case study of a system-level approach to power-aware computing In other words, Amdahl’s Law does not just describe software parallelism limits. It accurately captures hardware bottlenecks too, including the bandwidth wall between processors and the memory they depend on.
This matters for anyone choosing hardware. Buying a chip with twice as many cores will not help if your workload is memory-bound. The serial fraction, in this case, is not a piece of code but a physical bus that can only move so many bytes per second. Network latency plays a similar role in distributed systems: no matter how many servers you add, if they must constantly exchange messages through a finite-bandwidth link, that communication overhead becomes a de facto serial component.
Gustafson’s Reframe
For about two decades after Amdahl’s 1967 paper, the law was widely seen as an argument against massively parallel computers. If real programs have even a few percent serial code, the returns on adding hundreds or thousands of processors looked terrible. Then, in 1988, John Gustafson offered a different way to think about it.
Gustafson observed that in practice, people do not take a fixed-size problem and try to solve it faster. Instead, when given a bigger machine, they solve a bigger problem in the same amount of time. A weather forecasting center that gets a new supercomputer does not run the same forecast faster; it runs a finer-grained forecast that was previously impossible. In that framing, the serial fraction stays roughly constant in absolute time, but the parallel fraction grows as the problem scales up. So the proportion of serial work actually shrinks as you add processors.
This perspective, known as Gustafson’s scaled speedup, reframes the question from “how much faster can I solve this problem?” to “how much more work can I do in the same time?”4Journal of Parallel and Distributed Computing. Scalable Problems and Memory-Bounded Speedup It does not contradict Amdahl’s Law mathematically. Both formulas are correct. They simply answer different questions. Amdahl asks about fixed-size speedup; Gustafson asks about fixed-time scaling. In domains where the problem size is naturally elastic (scientific simulation, rendering, data analytics), Gustafson’s framing is more useful. In domains where the problem size is genuinely fixed (a single user waiting for a web page, a real-time control loop that must finish in a set number of milliseconds), Amdahl’s framing applies directly and the serial bottleneck bites hard.
Heterogeneous Chips and the GPU Question
Modern processors are rarely a uniform grid of identical cores. Most computers now combine different types of processing units: general-purpose CPU cores for serial and branching-heavy work, and specialized accelerators like GPUs for massively parallel number crunching. Amdahl’s Law in its original form assumed a homogeneous set of processors, so researchers have extended it to account for this asymmetry.
In a heterogeneous CPU-GPU system, a program’s execution time can be split into three pieces rather than two: a serial fraction that runs on a single CPU core, a parallel fraction that runs across CPU cores, and a parallel fraction that runs across GPU cores. Because GPU cores are typically much simpler and slower individually than CPU cores but far more numerous, the balance between CPU-parallel and GPU-parallel work, and the relative performance of each core type, changes the speedup calculation. The extended model shows that the theoretical speedup depends on how the parallel work is split between CPUs and GPUs and on the performance ratio between the two core types.5IEEE Xplore. Extending Amdahl’s Law for Heterogeneous Computing
The practical takeaway is that GPUs do not magically escape Amdahl’s constraint. They help enormously for the portion of work suited to their architecture (matrix math, graphics shading, certain kinds of simulation), but any fraction of the workload that must run serially on the CPU still imposes the same ceiling. A workload that is 99% GPU-friendly and 1% serial can still only achieve at most a 100× speedup regardless of how many GPU cores are available. And in practice, the overhead of moving data between CPU and GPU memory eats into the gains further, acting as yet another serial-like cost.
Diminishing Returns in Large-Scale AI Training
If there is a domain where Amdahl’s Law is generating real frustration right now, it is the training of large language models and other deep-learning systems. Training a model like GPT-4 or its successors requires distributing work across tens of thousands of GPUs. The parallel fraction of training (the matrix multiplications that make up forward and backward passes) is enormous, but the communication overhead of synchronizing gradients across all those GPUs is a serial-like bottleneck that grows with scale.
Recent research into hardware scaling trends in distributed training has shown that beyond certain cluster sizes, the overhead from distributed communication strategies causes diminishing returns. Scaling the total number of accelerators for large model training quickly yields poor marginal performance per additional unit of power or GPU-hour, even when hardware and parallelization strategies are properly optimized.6arXiv. Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training This is Amdahl’s Law showing up at industrial scale: the serial-equivalent overhead of coordinating 50,000 GPUs consumes an increasing share of total time as you try to push beyond that scale.
The AI industry has responded with clever engineering to push the ceiling higher. Techniques like gradient accumulation, pipeline parallelism, and asynchronous updates all try to reduce the effective serial fraction of training. But none of them eliminate it. At some point, buying more GPUs stops meaningfully reducing training time, and the money is better spent on algorithmic efficiency or better data. This is a direct, multi-billion-dollar consequence of a law stated in a short 1967 conference talk.
The Supercomputer Ceiling
Amdahl’s Law does not just limit individual programs. It limits entire computing paradigms. Analysis of exascale supercomputers, machines capable of at least a billion billion calculations per second, has shown that the extended and reinterpreted Amdahl model describes the behavior of existing supercomputers surprisingly well. The model explains why some flagship supercomputers achieve only a tiny fraction of their theoretical peak performance on real applications, a phenomenon that has puzzled observers for years.7arXiv. Limitations of performance of Exascale Applications and supercomputers they are running on
The argument goes further: using present technology and the current computing paradigm (processors executing instructions sequentially at their core, then fanning work out to parallel units), only marginal performance improvement is possible. The fundamental obstacle is not the number of processors or the speed of interconnects. It is the 70-year-old paradigm itself, the sequential instruction model that dates back to von Neumann. No amount of parallel scaling can fully escape a paradigm in which some coordination must happen sequentially.
This is why some researchers are interested in radically different computing models, from neuromorphic chips that mimic the brain’s parallel signaling to quantum computing, which sidesteps classical serial-parallel trade-offs entirely for certain problem types. Whether any of these approaches will deliver practical general-purpose computing is an open question, but the motivation is clear: Amdahl’s Law says that within the current paradigm, the walls are closing in.
Neural Network Simulation and Biological Analogs
An unexpected application of Amdahl’s Law has emerged in computational neuroscience. Researchers attempting to simulate large artificial neural networks, or even full-scale brain simulations, on conventional hardware have discovered that the law constrains these simulations just as it constrains any other parallel workload. Both software-based neural simulators and dedicated hardware simulators rely on clock-driven electronic circuits operating within the same sequential-parallel computing paradigm. As a result, they hit the same Amdahl’s Law ceilings that limit other parallelized systems.8PubMed Central. How Amdahl’s Law limits the performance of large artificial neural networks
This finding explains the performance saturation that prior brain simulation studies had observed without fully accounting for. The biological brain, by contrast, does not operate on a global clock or use the same serial-parallel division. Neurons fire asynchronously and communicate through chemical and electrical signals with no central coordinator enforcing sequential order. In a sense, the brain’s architecture naturally avoids the serial bottleneck that Amdahl’s Law describes. Replicating that architecture on hardware designed around the very paradigm the law constrains is a bit like trying to simulate wind by waving individual fans in sequence.
Common Misconceptions
The most widespread misunderstanding of Amdahl’s Law is treating the serial fraction as a fixed property of the algorithm that can never change. In reality, the serial fraction is a property of a specific implementation at a specific time. Clever engineering can sometimes convert serial code into parallel code. Lock-free data structures, speculative execution, and architectural innovations like CASPAR (which enables processors to perform lock-free synchronization in parallel rather than serially) all shrink the effective serial fraction.9ACM SIGARCH Computer Architecture News. CASPAR The law tells you the penalty for whatever serial fraction remains; it does not say that fraction is immutable.
A second misconception is that Amdahl’s Law means parallelism is pointless beyond a few cores. For workloads where the serial fraction is genuinely tiny, like large-scale matrix operations, image rendering, or Monte Carlo simulations, the ceiling is very high and adding hundreds of cores delivers real gains. The law is most devastating for workloads with modest serial fractions that look innocuous in percentage terms. A program that is “only” 2% serial still caps out at 50× speedup. For a workload running on a thousand-core machine, that means 95% of the hardware is idle at the bottleneck. Whether that matters depends on the economics: is 50× fast enough, and is the hardware cost justified?
A third misreading is confusing Amdahl’s Law with a law of nature that applies universally to all forms of speedup. It specifically describes the effect of parallelizing a fixed workload. It says nothing about other ways to make programs faster, like better algorithms, smarter caching, or reduced I/O. A program with a 10% serial fraction and a 10× Amdahl ceiling might still achieve a 100× end-to-end speedup if someone rewrites the serial portion in a more efficient algorithm. Amdahl’s Law constrains the parallelization axis, not every axis of improvement at once.
When Amdahl’s Law Misleads
There are situations where naïvely applying the law gives you the wrong intuition. One is when the serial overhead itself is not constant but grows with the number of processors. In many distributed systems, the coordination cost (exchanging messages, synchronizing state, resolving conflicts) increases as you add nodes. The effective serial fraction is not fixed; it inflates with scale. Amdahl’s original formula assumes a constant serial fraction, so it actually understates the problem in these cases. Real-world performance can degrade faster than the basic law predicts.
Another case is when the workload is not cleanly divisible into “serial” and “parallel.” Some tasks have complex dependency graphs where different stages can overlap partially. A portion of the code might be parallelizable across four threads but not forty. Modeling this as a single serial fraction oversimplifies reality. Researchers have proposed multi-fraction extensions of the law for exactly this reason, and the heterogeneous CPU-GPU model mentioned earlier is one example of splitting the parallel portion into finer categories.10IEEE Xplore. Extending Amdahl’s Law for Heterogeneous Computing
Finally, there is an economic dimension the law does not capture. Even when adding processors delivers only marginal speedup improvements, those margins might justify themselves. If training a language model in 30 days instead of 35 means launching a product a month earlier, that five-day gain has enormous business value even though the per-GPU efficiency is poor. Amdahl’s Law tells you the physics of the situation, but it does not tell you whether the economics still make sense.