GPU architecture is built around a single overriding principle: throw thousands of simple processing threads at a problem simultaneously, rather than making a few threads run as fast as possible. Where a CPU dedicates large amounts of its transistor budget to branch prediction, out-of-order execution, and deep caches that keep a handful of threads blazing fast, a GPU spends those transistors on arithmetic units and scheduling hardware that keep thousands of threads in flight at once. That design choice, which originated in the demands of real-time graphics rendering, has turned GPUs into the dominant hardware platform for machine learning, scientific simulation, and a growing range of data-parallel workloads. Understanding how this architecture actually works, where it shines, and where it struggles gives you a much clearer picture of why modern computing looks the way it does.
How a GPU Runs Code
The execution model at the heart of modern GPUs is called Single-Instruction Multiple-Thread, or SIMT. The idea is straightforward: the hardware groups dozens of threads together into a unit (NVIDIA calls these “warps,” AMD calls them “wavefronts”) and issues one instruction to the entire group at once. Every thread in the warp executes the same instruction on its own piece of data. This lets the GPU share the cost of fetching, decoding, and scheduling that instruction across all the threads in the group, which is far cheaper than giving each thread its own fetch-and-decode pipeline the way a CPU does.1ACM SIGARCH Computer Architecture News. Simultaneous branch and warp interweaving for sustained GPU performance
The catch is divergence. When threads in the same warp hit an if-else branch and some go one way while others go the other, the hardware cannot execute both paths at the same time. Instead, it serializes them: first the threads taking the “if” path run while the others sit idle, then the “else” threads run. The more your code branches, the more execution time you waste with idle threads. This is the fundamental tax GPUs pay for their massively parallel design, and a huge amount of architectural research goes into reducing it.
One approach allows two distinct instructions to be scheduled to different subsets of the same row of execution units, relaxing the strict one-instruction-per-warp rule. This gives the scheduler more flexibility to keep threads busy without introducing extra memory access problems.2ACM SIGARCH Computer Architecture News. Simultaneous branch and warp interweaving for sustained GPU performance Another technique uses a compaction-adequacy predictor to decide, branch by branch, whether it is worth stalling threads to regroup them into fuller warps. Rather than blindly forcing every divergent branch through compaction, the predictor identifies which branches will actually benefit, avoiding unnecessary stalls on threads that would do fine without regrouping.3ACM SIGARCH Computer Architecture News. CAPRI
Why GPU Memory Is Designed Differently Than CPU Memory
If you have used a CPU, you are accustomed to caches that are small but extraordinarily fast, designed to get data to a core in just a few nanoseconds. GPU memory works on a completely different philosophy: bandwidth first, latency second. The GPU’s L1 data caches are optimized for density, capacity, and low power consumption, with access latencies that can be orders of magnitude longer than what a CPU cache delivers.4ACM Transactions on Architecture and Code Optimization. A Low-latency On-chip Cache Hierarchy for Load-to-Use Stall Reduction in GPUs The GPU tolerates this because it has so many threads in flight at once: while one warp is waiting for its data to come back from memory, dozens of other warps can step in and keep the execution units busy. Latency is hidden by parallelism rather than by fast caches.
This trade-off works beautifully for workloads where every thread is doing roughly the same memory access pattern, like rendering pixels or multiplying large matrices. It starts to break down for graph processing, tree traversal, sparse neural networks, and other applications where threads diverge in both their execution paths and their memory access patterns. When effective thread-level parallelism drops, there are not enough ready warps to mask the high cache latency, and performance suffers.5ACM Transactions on Architecture and Code Optimization. A Low-latency On-chip Cache Hierarchy for Load-to-Use Stall Reduction in GPUs
Off-chip, GPUs have also moved away from the memory technologies used by CPUs. High Bandwidth Memory (HBM) stacks memory chips vertically and connects them to the GPU through a wide interface on a silicon interposer, delivering far more bandwidth at lower power than traditional GDDR5 memory. HBM also introduced features like dual-command and pseudo channel modes, which give the memory controller more flexibility in how it schedules reads and writes.6IEICE Electronics Express. Exploring new features of high-bandwidth memory for GPUs This shift to HBM has been especially important for AI training, where models need to move enormous amounts of data between memory and compute units every fraction of a second.
Tensor Cores and Ray Tracing Units
Modern GPUs are no longer just arrays of identical arithmetic units. They now contain dedicated hardware blocks designed to accelerate specific kinds of computation far faster than general-purpose cores could manage.
Tensor cores are the most commercially significant of these additions. They perform small matrix multiplications directly in hardware, executing a fused multiply-add on a matrix tile in a single operation. NVIDIA has iterated on these across several GPU generations, with the V100, A100, H100, and B200 each supporting different combinations of 8-bit, 16-bit, and 19-bit floating-point input formats.7arXiv. Accurate Models of NVIDIA Tensor Cores Because deep learning is built almost entirely on matrix multiplication, tensor cores have become the main driver of AI training throughput. Their use of lower-precision arithmetic is a deliberate trade-off: neural network training turns out to tolerate reduced numerical precision surprisingly well, so the hardware can do more operations per watt by using smaller numbers.
Researchers have pushed this idea further by exploring multiword arithmetic on tensor cores, where a higher-precision matrix is broken into the sum of two or more lower-precision matrices that each get multiplied on the tensor core hardware. The results are then combined to recover more accuracy than a single low-precision multiplication would provide, giving a better balance between speed and numerical fidelity.8SIAM Journal on Scientific Computing. Matrix Multiplication in Multiword Arithmetic: Error Analysis and Application to GPU Tensor Cores
Ray tracing cores serve a different purpose. They accelerate the traversal of bounding volume hierarchies, the tree-like data structures used to figure out which objects a ray of light hits in a 3D scene. Without dedicated hardware, this traversal chews through general-purpose compute units and limits real-time rendering. AMD and NVIDIA both now include ray tracing acceleration, with frameworks like HIPRT relying on hardware-accelerated BVH traversal to make ray tracing practical at interactive frame rates.9Proceedings of the ACM on Computer Graphics and Interactive Techniques. HIPRT: A Ray Tracing Framework in HIP
How GPU Architecture Is Adapting to Large Language Models
The rise of large language models has created workloads that stress GPU architecture in ways that differ from traditional deep learning training. LLM inference, for instance, involves long sequences of dependent operations that do not always saturate thousands of threads the way a big matrix multiply does. Training these models pushes multi-GPU communication to its limits, since gradients need to be synchronized across potentially thousands of GPUs many times per second.
On the execution side, one of the most interesting recent developments is warp specialization. Instead of having every warp do the same type of work, the GPU assigns different roles to different warps: some warps act as producers that fetch data, others act as consumers that process it, and still others handle activation functions. This lets different stages of a computation overlap in time, keeping more of the hardware busy. Simulators that model this fine-grained asynchronous pipeline behavior have shown that warp specialization substantially improves performance for LLM workloads.10arXiv. Sim-FA: A GPGPU Simulator Framework for Fine-Grained Asynchronous Pipeline Analysis
At the system level, scaling AI training across many GPUs requires high-speed interconnects between chips. Modern GPU clusters use multiple communication “rails” per GPU, and the design of collective operations like allreduce (where every GPU needs to combine its data with every other GPU’s data) has to be carefully tailored to the specific interconnect topology and bandwidth of the system.11CrossRef / IEEE. Unified Designs of Multi-Rail-Aware MPI Allreduce and Alltoall Operations Across Diverse GPU and Interconnect Systems The architecture of the GPU itself now extends well beyond the chip boundary, encompassing the network fabric that ties thousands of GPUs into a single training cluster.
Power, Heat, and the Limits of Scaling
A top-end data center GPU can draw several hundred watts under load, and a cluster of thousands of them consumes enough electricity to power a small town. That energy consumption creates a cascading set of engineering problems: a larger carbon footprint, enormous cooling demands, and power fluctuations that can threaten the stability of the local electrical grid.12Future Generation Computer Systems. An in-depth study of GPU frequency-scaling latency and its optimization on modern architectures
CPUs have long used dynamic voltage and frequency scaling to throttle down when full speed is not needed, saving power and reducing heat. The same technique exists for GPUs in principle, but there is a critical knowledge gap about how frequency scaling actually behaves on GPU hardware. The latency of switching frequencies, the relationship between clock speed and power draw, and the impact on real workloads are all less well-characterized for GPUs than for CPUs.13Future Generation Computer Systems. An in-depth study of GPU frequency-scaling latency and its optimization on modern architectures This matters because without a precise understanding of these dynamics, power-saving strategies that work well on CPUs cannot simply be transplanted to GPU workloads. The result is that GPUs often run closer to full power than they need to, wasting energy.
Thermal management has become a first-class architectural concern. Liquid cooling, once exotic in data centers, is increasingly standard for GPU racks. And the trend toward ever-higher power envelopes shows no sign of reversing: each new generation of AI-focused GPUs tends to draw more power than the last, because the demand for more compute per chip outstrips the efficiency gains from smaller transistors.
Silent Data Corruption at Scale
When you run thousands of GPUs for weeks on end training a language model, rare hardware faults that would be invisible on a single machine become a serious concern. Silent data corruption, where the hardware produces a wrong result without flagging any error, is one of the hardest problems in large-scale GPU computing. Unlike a crash or a visible error message, SDC quietly produces bad numbers that propagate through a training run and can corrupt a model without anyone noticing until much later.
A large-scale fault-injection study, consuming over three million simulator hours across 63 workloads on a production-class data center GPU, revealed several surprising things about how SDC actually behaves. The results that produce obviously wrong values like NaN or infinity account for only about one percent of all silent corruptions. That means the common software strategy of scanning outputs for NaN and infinity catches only a tiny fraction of real errors.14arXiv. The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance
The study also found that single-bit flips, which dominate CPU error profiles at rates of 72 to 98 percent, make up less than 40 percent of GPU bit-flip events. GPU corruptions are more complex: they exhibit structured spatial periodicity rather than being randomly scattered across memory addresses, and their patterns vary by functional unit.15arXiv. The Anatomy of Silent Data Corruption: GPU Error Pattern Study and Modeling Guidance These findings matter for anyone designing resilience strategies for GPU clusters. Simple error-detection schemes borrowed from the CPU world may miss most of the corruption that actually occurs on GPU hardware.
Cross-Vendor ISA Differences
Unlike CPUs, where x86 has served as a common instruction set across Intel and AMD for decades, GPUs have no shared instruction set architecture. NVIDIA, AMD, Intel, and Apple each use their own proprietary ISA, and the differences run deeper than just instruction names. The way threads are grouped, the width of SIMD lanes, the available synchronization primitives, and the memory model all vary between vendors. This is a major reason why GPU software portability is harder than CPU software portability, despite standardization efforts like OpenCL, Vulkan compute, and SYCL.
A recent cross-vendor analysis tried to identify hardware-invariant computational primitives that work across architecturally distant GPUs. The researchers validated an abstract execution model on NVIDIA T4 and Apple M1 hardware and found that on five of six benchmark-platform pairs, the abstract model matched or exceeded the performance of native vendor-optimized code. The one outlier, a parallel reduction on NVIDIA that hit only about 63 percent of native performance, revealed that intra-wave shuffle operations (where threads exchange data within a warp without going through memory) are so performance-critical that they must be treated as a mandatory primitive in any cross-vendor model.16arXiv. Toward a Universal GPU Instruction Set Architecture: A Cross-Vendor Analysis of Hardware-Invariant Computational Primitives in Parallel Processors
This kind of work is still early-stage, and for now, getting the best performance out of a GPU almost always means writing code that targets one vendor’s hardware. Libraries like cuDNN (NVIDIA) and rocBLAS (AMD) are vendor-specific for exactly this reason. The architectural differences are not cosmetic; they reflect genuinely different design decisions about how to balance thread occupancy, memory access patterns, and arithmetic throughput.
The Generality-Specialization Tradeoff
GPU architecture has been on a long journey from fixed-function graphics hardware toward general-purpose computation, but the latest generation of designs is bending back toward specialization. The addition of tensor cores, ray tracing units, and video encoding blocks means that a modern GPU is really a collection of specialized accelerators sharing a die with a general-purpose SIMT compute fabric. Each generation adds more fixed-function hardware targeted at specific workload types.
A recent survey of industrial AI accelerators classifies this trend across the broader landscape of data center hardware, connecting advances in arithmetic throughput with changes in precision, data delivery, execution coordination, communication, power delivery, and cooling.17arXiv. Balancing Generality and Specialization: A Survey on AI Datacenter Hardware Architecture The core tension is that specialization delivers massive efficiency gains for the workload it targets, but it costs transistor area and design complexity that could have gone to making the general-purpose portion of the chip more capable. A tensor core sitting idle during a graphics workload is wasted silicon.
This tradeoff plays out differently depending on the market. Data center GPUs lean heavily into AI-specific specialization because their workloads are predictable and their buyers will pay for every percentage point of training throughput. Consumer GPUs for gaming need to balance ray tracing performance, rasterization speed, video decode, and increasingly, local AI inference for features like frame generation and upscaling. The same underlying SIMT architecture serves both markets, but the ratio of general-purpose to specialized hardware varies significantly between a gaming card and a data center accelerator.
The question of how far to push specialization extends beyond individual chips to entire data center systems. As models grow and workload diversity increases, there is growing interest in heterogeneous architectures that pair GPUs with other types of accelerators, each handling the portion of the workload it is best suited for. The GPU’s role in such systems may evolve from being the sole compute engine to being one component in a more complex orchestration, but its architectural DNA, thousands of threads in lockstep chewing through parallel data, remains the foundation that everything else is built on.