For most deep learning workloads, a GPU trains models roughly 5x to 50x faster than a CPU, with some tasks showing even larger gaps. In one benchmark training a deep learning model over 20 epochs, a CPU took about 13 hours while a GPU finished the same job in under 2 hours, and doubling the batch size cut GPU training time further to around 75 minutes. The exact speedup depends on the model architecture, dataset size, batch size, and the specific hardware you’re comparing.
Why the Gap Is So Large
Deep learning is fundamentally a math problem built on matrix multiplication. Training a neural network means multiplying enormous grids of numbers together millions of times, then adjusting weights based on the results. CPUs can do this math, but they process it in a relatively serial fashion. A modern CPU might have 16 to 64 cores. A modern GPU has thousands of smaller cores designed to run simple math operations in parallel.
GPUs use a threading model where hundreds or thousands of threads execute the same instruction simultaneously, each working on a different piece of data. If one thread stalls waiting for memory, other threads keep running, which keeps the hardware busy. CPUs use a different parallel approach that processes chunks of data in lockstep. If any piece of data in that chunk causes a delay, everything in that batch waits. For the highly repetitive, uniform math that defines neural network training, the GPU’s approach is a near-perfect fit.
Memory Bandwidth Is the Other Bottleneck
Raw compute speed is only half the story. Deep learning models need to constantly shuttle data between the processor and memory: loading model weights, storing activations, reading gradients. How fast that data moves matters enormously.
NVIDIA’s flagship H100 data center GPU delivers memory bandwidth exceeding 2 terabytes per second, with the H100 NVL variant reaching 3.9 TB/s. A modern CPU system using DDR5 RAM typically provides 50 to 100 GB/s. That’s a 20x to 40x gap in raw memory throughput. When you’re training a large model with billions of parameters, this difference alone can bottleneck a CPU long before its cores run out of compute capacity. The GPU can feed its thousands of cores with data fast enough to keep them working. The CPU often can’t.
Raw Compute: The Numbers
NVIDIA’s H100 GPU delivers 67 teraflops of single-precision (FP32) compute and 34 teraflops of double-precision. With tensor cores handling the lower-precision formats commonly used in deep learning (FP16, BF16, INT8), throughput climbs much higher. A high-end Intel server CPU, by contrast, delivers single-digit teraflops of FP32 compute. The raw math capacity of one GPU can exceed that of a CPU by 10x or more before any software optimization enters the picture.
This gap widens further with mixed-precision training, which is now standard practice. Training in FP16 or BF16 instead of full FP32 roughly doubles the effective throughput on a GPU because the tensor cores are specifically designed for these formats. CPUs have their own acceleration for lower-precision work through specialized vector instructions that can combine multiple operations into a single step, which improves inference performance on INT8 models. But even with these optimizations, CPUs don’t close the gap for training workloads.
Where CPUs Still Make Sense
CPUs aren’t useless in a deep learning pipeline. Data preprocessing, feature engineering, and loading data from disk all run on the CPU. For very small models or quick prototyping where a training run takes minutes anyway, the overhead of setting up GPU code may not be worth it. Inference on a single input (as opposed to batch inference on thousands) can sometimes run efficiently on a CPU, especially with optimized lower-precision formats.
Some organizations also use CPUs for inference at scale when latency requirements are modest and they want to avoid the cost of GPU hardware. Intel’s deep learning optimizations for its server processors can meaningfully accelerate inference by packing neural network calculations into fewer, more efficient instructions. This narrows the gap for inference specifically, though it doesn’t eliminate it.
How Batch Size Affects the Speedup
The GPU advantage grows with batch size. In the benchmark study that compared 13-hour CPU training against 2-hour GPU training, increasing the batch size from 64 to 128 cut the GPU’s time to about 75 minutes with no loss in accuracy. Larger batches let the GPU fill more of its thousands of cores with useful work on each pass, improving utilization.
On a CPU, increasing batch size helps less because there are far fewer cores to saturate. You hit the ceiling quickly. This is why researchers working with large datasets and large models see the biggest speedups from GPUs. A simple logistic regression on a small dataset might only be 2x to 5x faster on a GPU. Training a transformer on millions of text samples can be 50x to 100x faster, or effectively impossible on a CPU within any reasonable timeframe.
Multi-GPU Scaling
Another factor that widens the practical gap: GPUs scale horizontally in ways CPUs can’t match for this workload. Training large language models and image generators routinely uses 8, 64, or even thousands of GPUs connected with high-speed links. Each GPU adds its full compute and memory bandwidth to the job. While you can add more CPU sockets to a server, the memory bandwidth and compute per node don’t scale nearly as aggressively, and the interconnects between CPUs weren’t designed for this kind of parallel math workload.
For context, training GPT-3 scale models requires thousands of GPU-hours. Doing equivalent work on CPUs would require orders of magnitude more time, pushing training runs from weeks into years. At that scale, the question isn’t really “how much faster” but “is it feasible at all.”
Typical Speedups by Workload
- Image classification (CNNs): 10x to 40x faster on GPU, depending on model depth and image resolution
- Natural language processing (transformers): 20x to 100x+ faster, especially for large sequence lengths and big vocabularies
- Small tabular data models: 2x to 10x faster, sometimes not worth the GPU overhead for very small datasets
- Inference (single inputs): 2x to 10x faster on GPU, though optimized CPU inference can narrow this gap
- Batch inference (thousands of inputs): 10x to 50x faster on GPU, where parallelism pays off again
These ranges are approximate and shift with every hardware generation, but the overall pattern has held steady for a decade: GPUs dominate training performance, and the advantage grows with model size, dataset size, and batch size.