GPUs are good for machine learning because they can perform thousands of math operations simultaneously, and training a neural network is essentially millions of small, identical math problems that don’t depend on each other. A modern data center GPU like the NVIDIA H100 has nearly 17,000 cores, while even a high-end server CPU tops out around 128. That difference in raw parallelism is what makes a task that would take days on a CPU finish in hours on a GPU.
Thousands of Cores vs. Dozens
CPUs are designed to handle complex, varied tasks quickly. To do that, they dedicate 70 to 80 percent of their transistor budget to cache memory and prediction logic, the circuitry that figures out what instruction is coming next and keeps data close at hand. Only 20 to 30 percent of a CPU’s silicon actually performs arithmetic. A GPU flips that ratio. It packs its chip with simple arithmetic units and spends relatively little on prediction or caching.
The result is a massive difference in core count. A consumer desktop CPU typically has 8 to 16 cores. A workstation chip might have 24 to 96, and a top-end server CPU reaches about 128 cores per socket. The NVIDIA RTX 4090, a consumer GPU, has 16,384 cores. The H100, built for data centers, has 16,896. Each individual GPU core is simpler and slower than a CPU core, running at around 2 to 2.5 GHz, but when the job is doing the same calculation on thousands of data points at once, sheer numbers win.
Neural Networks Run on Matrix Math
Almost every layer in a neural network boils down to the same operation: multiplying two large grids of numbers together. These are called matrix multiplications, and they show up in fully connected layers, convolutional layers, recurrent networks like LSTMs, and the attention mechanism behind models like GPT and BERT. Training a network means running these matrix multiplications billions of times, adjusting the numbers slightly each round.
GPUs handle this by splitting the output of each matrix multiplication into small tiles and assigning each tile to a separate group of cores. Each group loads its slice of data, multiplies, and accumulates results independently. Because no tile needs to wait for another tile to finish, the entire operation scales almost linearly with the number of cores available. Newer NVIDIA GPUs also include dedicated Tensor Cores, specialized circuits built specifically to accelerate these multiply-and-add operations beyond what general cores can do.
Memory Bandwidth Matters Too
Raw compute power is only useful if data can reach the cores fast enough. This is where GPU memory architecture creates a second major advantage. The H100 uses a type of memory called HBM3, which delivers roughly 3.4 terabytes per second of bandwidth in its highest configurations. HBM3E, the latest generation, reaches around 1.2 terabytes per second per memory stack. By comparison, AMD’s top workstation CPU with a 12-channel DDR5 memory setup peaks at about 461 gigabytes per second. That’s less than half the bandwidth of a single HBM3E stack.
This gap matters because training involves constantly streaming weight matrices and activation data to the cores, computing results, then writing updated values back. If the memory bus can’t keep up, cores sit idle waiting for data. High-bandwidth memory keeps the pipeline full, which is why GPU designers invest heavily in it.
How Much Faster in Practice
Benchmarks on standard models give a concrete sense of the speedup. When researchers moved ResNet-50 training (a widely used image recognition model) from Intel Xeon CPUs to a single NVIDIA V100 GPU, they measured a 12x speedup and an eightfold improvement in energy efficiency. A broader survey of published results found that GPUs deliver, on average, 30x more throughput than CPUs for ResNet-class networks, with 5 to 10x better energy efficiency.
Scaling to multiple GPUs increases the advantage further. One study showed a four to fivefold reduction in training time per epoch when moving from CPUs to a cluster of four GPUs. Even at the consumer level, the differences are significant. An NVIDIA RTX 3060 trained a remote sensing model at 8.12 iterations per second, roughly double the 3.97 iterations per second of an Apple M3 Pro running the same workload. Epoch durations ranged from about 201 seconds on an NVIDIA T4 to 366 seconds on the M3 Pro.
The Software Stack Seals the Deal
Hardware alone doesn’t explain GPU dominance in machine learning. NVIDIA’s software ecosystem plays a critical role. The CUDA programming platform lets developers write code that runs directly on GPU cores, and on top of CUDA sits cuDNN, a library of pre-optimized building blocks for neural networks. cuDNN provides hand-tuned implementations of convolutions, matrix multiplications, attention mechanisms, pooling, and normalization. It automatically selects the best algorithm for a given problem size using built-in heuristics, so developers don’t need to manually tune performance.
cuDNN also supports operation fusion, combining multiple steps into a single pass through memory. For example, instead of running a matrix multiplication, then a bias addition, then an activation function as three separate operations (each requiring a round trip to memory), the library can fuse them into one kernel. This reduces memory traffic and further accelerates training. Specialized fusion patterns exist for common structures like attention blocks, which are the core of transformer models.
Frameworks like PyTorch and TensorFlow call into cuDNN automatically, meaning most machine learning practitioners benefit from these optimizations without writing a single line of GPU code. This mature, deeply integrated software stack is a major reason GPUs remain the default hardware for training and inference, even as alternative accelerators emerge.
Why CPUs Still Can’t Compete
The fundamental issue isn’t that CPUs are slow. A modern server CPU is extremely fast at sequential work, branching logic, and tasks where each step depends on the result of the previous one. But neural network training is the opposite of that workload. It’s millions of independent, identical arithmetic operations repeated across massive datasets. A CPU core that excels at running one complex thread is simply the wrong tool when the job calls for running 16,000 simple threads in parallel.
Energy cost reinforces this gap. Because GPU cores are simpler, they use less power per operation. Getting 30x more throughput at 5 to 10x better energy efficiency means that even if you could match a GPU’s speed by buying enough CPUs, the electricity bill would make it impractical. For large-scale training runs that consume thousands of GPU-hours, energy efficiency isn’t a footnote. It’s a primary cost driver.