A Vision Transformer (ViT) is a type of artificial intelligence model that applies the transformer architecture, originally designed for processing text, to images instead. It works by splitting an image into small patches, treating each patch like a word in a sentence, and using self-attention to learn relationships between all the patches at once. Introduced by a Google Research team in 2020, the Vision Transformer showed that you don’t need the convolutional neural networks (CNNs) that had dominated computer vision for nearly a decade. A pure transformer, trained on enough data, could match or beat them.
How ViT Turns an Image Into Patches
The core idea behind the Vision Transformer is surprisingly straightforward. Take an image, divide it into a grid of fixed-size patches (typically 16×16 pixels), flatten each patch into a single row of numbers, and feed that sequence into a standard transformer encoder. For a 224×224 pixel image split into 16×16 patches, you end up with 196 patches, each one functioning as a “token” the same way a word would in a language model.
Each flattened patch gets projected into a higher-dimensional space through a linear layer, creating what’s called a patch embedding. In practice, this entire step (splitting, flattening, and projecting) can be done with a single convolution operation where both the kernel size and stride match the patch size. The result is a sequence of vectors that the transformer can process using its standard self-attention mechanism.
One extra token, called the classification token, gets prepended to the sequence. This token doesn’t correspond to any image patch. Instead, it aggregates information from the entire image during processing and serves as the final representation used to make predictions. If the model is classifying an image as “cat” or “dog,” the answer comes from this token.
Why Position Embeddings Matter
Unlike CNNs, which inherently understand spatial relationships because their filters slide across an image in a fixed pattern, transformers have no built-in sense of where things are. The self-attention mechanism is permutation-invariant, meaning it would produce the same output regardless of what order the patches are in. Shuffle all 196 patches randomly, and the transformer wouldn’t know the difference.
To solve this, ViT adds a learnable 1D position embedding to each patch embedding. These position embeddings are vectors of the same size as the patch embeddings, and the model learns them during training. Researchers analyzing trained models have found that the learned position embeddings encode spatial proximity: patches that are physically close in the original image develop similar position embeddings, while distant patches develop different ones. The position information and the visual content of each patch remain surprisingly separable even after being added together, flowing through the model’s attention layers without blending into noise.
How Self-Attention Processes the Image
Once the patches are embedded and position information is added, the sequence passes through a stack of transformer encoder layers. Each layer has two main components: a multi-head self-attention block and a feed-forward network.
Self-attention is what makes transformers powerful. For every patch, the model computes how relevant every other patch is to it. A patch containing part of a dog’s ear can directly attend to a distant patch containing the dog’s tail, recognizing they belong to the same object. This is fundamentally different from CNNs, where information about distant parts of an image can only interact after passing through many layers of progressively larger receptive fields. In a transformer, every patch can relate to every other patch in a single layer.
Multi-head attention means this process runs in parallel across several “heads,” each one learning to focus on different types of relationships. One head might specialize in color similarity, another in spatial proximity, another in texture patterns. Their outputs are combined to form a richer representation.
Performance on Benchmarks
The largest Vision Transformer models have reached remarkable accuracy levels. A two-billion-parameter model called ViT-G/14, developed by scaling up the original architecture, achieved 90.45% top-1 accuracy on ImageNet, a benchmark dataset of over a million images across 1,000 categories. A smaller variant, ViT-H/14, reached 88.55%. These numbers represent the percentage of times the model’s top prediction is correct, and they set new records when published.
These results come with an important caveat: they depend on pre-training on massive datasets first, then fine-tuning on ImageNet. The ViT-G model, for instance, was pre-trained on datasets far larger than ImageNet before being evaluated on it.
The Data Hunger Problem
Vision Transformers are often described as “data-driven learners,” and that label carries a real cost. The original ViT’s strong performance depended on pre-training with JFT-300M, a Google-internal dataset of roughly 300 million labeled images. Without that scale of pre-training, ViT underperformed traditional CNNs on standard benchmarks.
The reason traces back to what makes transformers flexible in the first place. CNNs have built-in assumptions about images (local patterns matter, the same pattern can appear anywhere in the image) that let them learn efficiently from smaller datasets. Transformers lack these assumptions, which means they can learn more abstract and generalizable features, but they need far more examples to do it. When trained on small or medium datasets from scratch, ViTs struggle to match the accuracy of well-designed CNNs. This dependence on massive labeled datasets is one of the most significant practical barriers to adopting Vision Transformers, especially in domains where labeled data is expensive or scarce.
Computational Cost Compared to CNNs
Vision Transformers are also considerably more expensive to run. In a benchmark comparing models on medical image classification, a ViT-Small model with 8×8 patches required 16.76 billion floating-point operations per image and took 184.67 seconds per training epoch. By comparison, GoogLeNet, a well-known CNN, needed only 1.5 billion operations and 17.23 seconds per epoch, while achieving higher accuracy on the same task. An even lighter CNN, ShuffleNetV2, used just 0.04 billion operations and still outperformed the ViT on that particular dataset.
The reason for this gap is that self-attention computes relationships between every pair of patches, and the computational cost grows with the square of the number of patches. Smaller patches mean more tokens, which means dramatically more computation. This makes ViT models slower to train, more expensive to deploy, and more demanding on GPU memory than comparably accurate CNNs for many tasks.
Variants That Address ViT’s Weaknesses
Several architectures have been developed to keep the strengths of vision transformers while reducing their costs. The most influential is the Swin Transformer, introduced in 2021, which makes two key changes. First, it uses a hierarchical design, processing small patches at early layers and progressively merging them into larger regions at deeper layers, similar to how CNNs build up from local to global features. Second, it restricts self-attention to small, shifted windows rather than computing it across the entire image, dramatically reducing computational cost.
Other approaches tackle the data efficiency problem. Data-efficient image transformers (DeiT) use advanced training strategies like knowledge distillation, where a smaller transformer learns to mimic the predictions of a larger, pre-trained model, allowing strong performance without hundreds of millions of training images. Hybrid architectures combine convolutional layers in early stages with transformer layers later, getting the best of both worlds: the sample efficiency of CNNs for low-level feature extraction and the global reasoning ability of transformers for higher-level understanding.
Applications in Medical Imaging
Medical imaging has become one of the most active areas for Vision Transformer research, in part because the global attention mechanism is well-suited to analyzing complex medical scans where relevant features may be spread across an entire image. ViT-based models have been applied to brain tumor segmentation on MRI scans, achieving 89.7% overlap accuracy with expert annotations for whole tumor boundaries. In cardiology, transformer models have been developed to interpret ultrasound videos of the heart, estimating how effectively the left ventricle pumps blood.
Other medical applications include predicting dangerous heart rhythm events from ECG printouts (89% accuracy), scoring the severity of skin diseases like psoriasis and eczema, assessing bone age from hand X-rays with accuracy comparable to expert orthopedic surgeons, and combining tissue slide images with genetic data to predict survival outcomes in colorectal cancer. The common thread across these use cases is that transformers can integrate information across an entire image or even across multiple data types in ways that traditional architectures struggle with.
Beyond Image Classification
While the original ViT was designed for image classification, the architecture has been adapted for nearly every computer vision task. Object detection models use transformer backbones to identify and locate multiple objects within a scene. Semantic segmentation models assign a category to every pixel in an image, useful for autonomous driving and satellite imagery analysis. Video understanding models extend the patch concept into the time dimension, treating short video clips as sequences of spatiotemporal patches.
Vision Transformers have also become foundational components in multimodal AI systems that combine vision and language. Large models that can describe images, answer questions about photos, or generate images from text descriptions frequently use ViT or its variants as their visual processing backbone. The same architectural family that processes text can now process images, making it far easier to build systems that reason across both.