Medical image segmentation is the process of labeling every pixel (or voxel, in 3D scans) in a medical image so that anatomical structures, tumors, or other regions of interest are outlined automatically or semi-automatically. It underpins much of modern diagnostic imaging, from measuring tumor volume on an MRI to guiding a surgeon’s instruments in real time. What was once done painstakingly by hand, with radiologists tracing organ boundaries slice by slice, is now increasingly handled by deep learning models that can process entire scans in seconds. The field has grown explosively over the past decade, and the technology is more capable and more complicated than most people realize.
Why Segmentation Matters in Medicine
When a radiologist looks at a CT scan or MRI, they mentally separate the image into meaningful parts: here is the liver, there is a lesion, that is the aorta. Segmentation automates that mental process by assigning each pixel to a category. This is not just an academic exercise. Accurate segmentation drives treatment planning in radiation therapy, enables volumetric tracking of tumors over time, and allows surgeons to overlay 3D organ maps onto live video feeds during operations. Manual segmentation is not only slow but also inconsistent between experts. One study of prostate ultrasound segmentation found that inter-observer variability was significantly lower when clinicians used a semi-automatic method compared to freehand tracing, with average surface distance dropping from 2.6 mm to 1.5 mm between observers.1PubMed Central. Defining ground truth for prostate segmentation of transrectal ultrasound images: Inter- and intra-observer variability of manual versus semi-automatic methods That kind of inconsistency matters when you are planning radiation beams around a tumor or comparing scan-to-scan changes over months of treatment.
Before Deep Learning
Segmentation did not begin with neural networks. For decades, researchers relied on a toolkit of classical computer vision techniques: thresholding, region growing, active contours (also called “snakes”), and graph-based methods. Region growing, for example, starts from a manually placed seed point and expands outward to neighboring pixels that meet similarity criteria. One semi-automatic liver tumor segmentation system combined adaptive region growing with graph cuts, using a radiologist-selected seed for each tumor and then refining boundaries with gradient information.2Biomedical Signal Processing and Control. Semi-automatic liver tumor segmentation with adaptive region growing and graph cuts
Active contour models, or level set methods, take a different approach. They evolve a curve toward object boundaries by minimizing an energy function that balances smoothness against how well the curve matches image edges. These methods remain relevant because they handle certain challenges well, such as segmenting objects with complex shapes or fuzzy boundaries. Researchers continue to improve them: one recent model replaces conventional gradient-based edge information with a framework that better handles noise, intensity variations, and weak or broken edges in MRI scans.3IET Image Processing. A Neutrosophic Set‐Based Active Contour Model for Medical Image Segmentation Another approach introduced an adaptive edge indicator function that adjusts itself based on local image features, preserving sharp edges while resisting noise.4Scientific Reports. Enhanced medical image segmentation using novel level set evolution and efficient optimization
These classical methods still appear in clinical pipelines, often as preprocessing or refinement steps. But their main limitation is that they typically require manual initialization, careful parameter tuning for each imaging scenario, and struggle with the enormous variability of real clinical images. Deep learning changed the equation by learning features directly from data.
The U-Net Family and Its Descendants
If one architecture defines medical image segmentation, it is U-Net. Introduced in 2015 for biomedical images, U-Net uses a symmetric encoder-decoder structure: the encoder compresses the image into a compact representation, and the decoder expands it back to full resolution to produce a pixel-level label map. What makes U-Net distinctive is its skip connections, which pass fine-grained spatial details from early encoder layers directly to corresponding decoder layers. This lets the network recover sharp boundaries that would otherwise be lost during compression.5PubMed Central. ES-UNet: efficient 3D medical image segmentation with enhanced skip connections in 3D UNet
The original U-Net worked on 2D images, but medical scans are often 3D volumes. 3D U-Net extends the same principle to volumetric data, processing entire CT or MRI volumes at once.6PubMed Central. Deep Learning-Based Segmentation of 3D Volumetric Image and Microstructural Analysis Since then, researchers have explored an enormous space of U-Net variants. One benchmarking study systematically evaluated 18 different 3D U-Net configurations by varying the number of resolution stages, convolutional layers per stage, and the width of the first layer.7Scientific Reports. Revisiting model scaling with a U-net benchmark for 3D medical image segmentation The takeaway from studies like this is that there is no single best architecture; performance depends heavily on the specific imaging task, the size of the training data, and available computing resources.
More recently, transformer-based architectures have entered the picture. Originally designed for natural language processing, transformers excel at capturing long-range relationships across an image, something convolutional networks handle less naturally. A multi-scale attention-based Swin Transformer model, for instance, achieved Dice scores above 0.95 on datasets for gland segmentation, skin lesion segmentation, and gastrointestinal polyp segmentation.8Scientific Reports. A multi-scale attention-based Swin transformer model for medical images segmentation These numbers represent very high overlap between predicted and actual boundaries, though real-world deployment introduces challenges that benchmarks do not always capture.
Where It Gets Used
Medical image segmentation is not one problem but hundreds, each with its own imaging modality, anatomy, and clinical requirements. A few of the most active areas illustrate the range.
Brain Tumors
Brain tumor segmentation from MRI is one of the best-studied applications, largely thanks to the BraTS challenge, which has run annually since 2012 and provides a shared benchmark for comparing methods. The standard approach uses four MRI sequences as input, and algorithms segment three nested tumor regions: the whole tumor including surrounding edema, the tumor core excluding edema, and the active enhancing tumor region.9PubMed Central. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) Automated methods have now reached performance levels considered clinically useful, though they still depend on having all four MRI sequences available.10PubMed Central. The Brain Tumor Segmentation (BraTS) Challenge 2023: Brain MR Image Synthesis for Tumor Segmentation (BraSyn) In practice, patients sometimes lack one or more sequences due to scan protocol differences or patient tolerance, which creates problems addressed by cross-modal techniques discussed later in this article.
Cardiac Imaging
Segmenting heart chambers and the left ventricular wall from CT or MRI enables automated measurement of cardiac volumes and ejection fraction, metrics that cardiologists use constantly to assess heart function. One deep learning system validated against cardiac MRI found strong correlations for left ventricular volumes and myocardial mass, with intraclass correlation coefficients of 0.84 to 0.91 for key left heart measurements.11PubMed Central. Fully Automated Assessment of Cardiac Chamber Volumes and Myocardial Mass on Non-Contrast Chest CT with a Deep Learning Model: Validation Against Cardiac MR Extending segmentation across the cardiac cycle in 4D CT allows assessment of dynamic function, not just static anatomy.12PubMed. Deep learning-based whole-heart segmentation in 4D contrast-enhanced cardiac CT
Digital Pathology
Pathology slides, digitized at extremely high resolution, present a different segmentation challenge: detecting and classifying individual cell nuclei across whole-slide images that can contain millions of cells. Manual assessment is labor intensive and prone to inter-observer variation.13PubMed Central. Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review Newer deep learning methods have dramatically accelerated nucleus segmentation and tumor microenvironment analysis, with one system validated across lung, liver, and breast cancer tissues outperforming existing methods in both accuracy and speed.14PubMed Central. A Deep Learning Approach for Histology-Based Nucleus Segmentation and Tumor Microenvironment Characterization
Foundation Models and the Promise of Universal Segmentation
One of the most significant recent developments is the attempt to build “foundation models” for medical segmentation: single models trained on massive, diverse datasets that can handle many different imaging tasks without being retrained from scratch for each one. MedSAM, adapted from Meta’s Segment Anything Model (SAM) for natural images, was trained on over 1.5 million image-mask pairs spanning 10 imaging modalities and more than 30 cancer types. In validation across 86 internal and 60 external tasks, it demonstrated better accuracy and robustness than specialist models trained for individual modalities.15PubMed Central. Segment anything in medical images
The model works by accepting user prompts, typically bounding boxes or point clicks, that indicate the region of interest. Bounding boxes tend to provide more reliable results because they give the model clearer spatial context, while point prompts can be ambiguous when similar-looking structures sit close together.16Nature Communications. Segment anything in medical images The appeal is flexibility: a single model that a radiologist can prompt for a liver on one scan and a lung nodule on the next, without switching tools.
That said, the story is more nuanced than the headlines suggest. Direct application of SAM to medical images without any fine-tuning has shown mixed results, because medical images differ substantially from the natural photographs SAM was originally trained on.17PubMed. A review of the Segment Anything Model (SAM) for medical image analysis: Accomplishments and perspectives Medical scans have lower contrast, more subtle boundaries, and structures that look similar to the untrained eye. The gains come from fine-tuning on medical data and careful prompt engineering, not from simply downloading SAM and running it on an MRI.
The Annotation Bottleneck
Deep learning models are hungry for labeled data, and creating those labels in medical imaging is expensive. A single 3D scan can have hundreds of slices, each requiring pixel-level annotation by a trained clinician. This is one of the field’s most stubborn bottlenecks, and it has driven a wave of research into reducing annotation burden.
Weakly supervised methods represent one promising approach. Instead of requiring exact pixel boundaries for every training example, these methods accept coarser labels like bounding boxes. One breast cancer segmentation framework trained with bounding box annotations on MRI achieved an average Dice score of about 81%, competitive with methods that use full pixel-level labels, and still managed nearly 79% even on non-mass lesions with irregular shapes that are notoriously difficult to segment.18Biomedical Signal Processing and Control. Bounding boxes for weakly-supervised breast cancer segmentation in DCE-MRI Drawing a box around a tumor takes a fraction of the time that tracing its boundary does, so the practical savings are large.
Semi-supervised learning, self-supervised pretraining, and synthetic data generation are other strategies being explored. The overall trajectory is clear: the field is trying to get comparable performance with less human effort per training example, because the alternative, paying radiologists to trace millions of contours, does not scale.
How Segmentation Performance Is Measured
Two metrics dominate the field. The Dice score (also called the Dice similarity coefficient) measures the overlap between the predicted segmentation and the ground truth, ranging from 0 to 1, where 1 means perfect agreement. The Hausdorff distance measures the worst-case boundary error, capturing how far off the most misplaced prediction is. Researchers have catalogued at least 20 distinct metrics for evaluating 3D segmentation, each capturing different aspects of accuracy.19PubMed Central. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
This matters because a high Dice score does not automatically mean clinically adequate performance. A model might achieve a Dice of 0.90 on average but occasionally produce catastrophic failures, like completely missing a small tumor, that Dice does not penalize heavily because the error involves few pixels relative to the whole image. A clinician cares about those edge cases much more than the average. This mismatch between aggregate metrics and clinical utility is one reason that promising benchmark results do not always translate smoothly to hospital deployment.
The Domain Shift Problem
A model trained on CT scans from one hospital often performs poorly on scans from another, even when the anatomy and task are identical. The reason is domain shift: differences in scanner hardware, imaging protocols, contrast agents, patient demographics, and reconstruction algorithms all change what the images look like at the pixel level.20PubMed Central. Domain Adaptation for Medical Image Analysis: A Survey A network learns to rely on subtle intensity patterns during training, and when those patterns change in a new environment, its predictions degrade.
Domain adaptation techniques attempt to bridge this gap, often by aligning the statistical properties of source and target data so the model treats them as coming from the same distribution. Domain generalization takes this further, aiming to build models robust to unseen domains without ever seeing target data during training. One recent approach uses a diffusion-based generative model designed specifically for single-source domain generalization in medical image segmentation, addressing the common scenario where a hospital has training data from only one source.21PubMed. Domain-Generalized Discrete Diffusion Model for Cross-Domain Medical Image Segmentation This remains an active and largely unsolved problem: no technique fully eliminates domain shift, and many institutions still resort to retraining or fine-tuning models on local data.
Real-Time Segmentation in the Operating Room
Some of the most striking applications involve running segmentation models during surgery. In robot-assisted prostatectomy, researchers have built systems that segment the live endoscopic video feed and overlay a 3D model of the patient’s anatomy in augmented reality, helping surgeons identify critical structures.22PubMed Central. Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance For breast-conserving surgery, one real-time ultrasound segmentation system achieved 16 frames per second, fast enough for smooth surgical guidance.23PubMed. Real-time automatic tumor segmentation for ultrasound-guided breast-conserving surgery navigation Another AI navigation system for vacuum-assisted breast biopsy demonstrated strong potential for clinical application in surgical training and complex cases requiring precise instrument control.24PubMed Central. Real-time deep learning for tumor segmentation and tool tracking: development and validation of an AI navigation system in vacuum-assisted breast biopsy
These systems face constraints that do not exist in offline analysis. Latency must be minimal, because a surgeon cannot wait seconds for a segmentation update while moving an instrument. The model must be robust to unusual views, poor image quality, and the visual clutter of surgical fields. And failure modes carry higher stakes: a badly segmented boundary during radiation planning can be caught and corrected, but an error displayed on a live surgical feed might influence an irreversible cut.
Running on Limited Hardware
Most state-of-the-art segmentation models are designed for powerful GPUs, but many clinical settings, especially in low-resource environments, lack that hardware. Point-of-care ultrasound devices, portable scanners used in field hospitals, and rural clinics need models that run efficiently on CPUs or low-power devices. A recently developed lightweight ultrasound segmentation model called UltraSeg achieved over 50 frames per second on a single-core CPU and maintained nearly 35 frames per second on a mobile device, fast enough for real-time guidance without any GPU at all.25PubMed Central. Enabling Real-Time Point-of-Care Ultrasound Segmentation: A GPU-Free Deployment in Resource-Limited Settings Techniques like model pruning, knowledge distillation, and architecture-level efficiency gains are making this possible, and they will likely determine how widely segmentation tools spread beyond well-funded academic medical centers.
Fairness and Bias in Segmentation
When training data comes predominantly from one patient population, the model can perform unevenly across subgroups. In radiation therapy target segmentation for prostate cancer, for instance, tumors at different clinical stages (T1 through T4) appear at very different frequencies in training datasets. A standard model may perform well on common T2 and T3 cases but poorly on rare T1 or T4 cases, with real consequences for treatment planning in those patients. One research group addressed this by developing a distribution-aware training framework that achieved more consistent performance across all tumor stage subgroups, reaching a Dice score of 0.778 on the underrepresented T4 group compared to 0.685 for a baseline method.26arXiv. Distribution-aware Fairness Learning in Medical Image Segmentation From A Control-Theoretic Perspective
Bias in medical AI extends beyond tumor staging to factors like race, sex, and age, particularly when training sets are not representative of the broader patient population. This is an area where awareness is growing faster than solutions, and regulatory frameworks are still catching up.
Privacy-Preserving Training Across Institutions
Training robust segmentation models ideally requires data from many hospitals, but patient privacy regulations often prevent sharing medical images across institutions. Federated learning offers a workaround: instead of pooling data in one place, each hospital trains a model locally and only shares model updates (not patient images) with a central server that aggregates them. A federated version of the widely used nnU-Net framework was tested across three clinical tasks involving breast MRI, cardiac MRI, and fetal ultrasound, and reliably matched or exceeded the performance of models trained on centralized data.27Scientific Reports. Federated nnU-Net for privacy-preserving medical image segmentation This is significant because it suggests that hospitals can collaborate on building better models without ever sending a patient scan across institutional boundaries.
When Imaging Sequences Are Missing
Clinical protocols vary, and patients do not always receive every scan type a segmentation model expects. Brain tumor segmentation algorithms, for example, are typically designed for four MRI sequences. But a patient might only have one or two available. Cross-modal distillation addresses this by training a model using all available sequences during training, then producing a streamlined model that works with just a single sequence during actual use. One study found that this approach improved the Dice score for whole tumor segmentation from about 74% with a standard single-sequence model to about 79%, a meaningful gain when clinical decisions hinge on accurate tumor volume measurements.28PubMed. Cross-Modal Distillation to Improve MRI-Based Brain Tumor Segmentation With Missing MRI Sequences
A related strategy is cross-modality image synthesis, where a generative model creates a synthetic version of the missing MRI sequence from the available one. This lets existing multi-sequence segmentation models run as designed, even when the patient did not receive the full imaging protocol.29PubMed Central. Cross-modality 3D MRI synthesis via cycle-guided denoising diffusion probability model Both approaches reflect a broader trend in the field: making segmentation tools more forgiving of the messy, incomplete data that characterizes real hospital workflows.
Confidence and Trust
A segmentation model that produces a boundary map but gives no indication of how sure it is about that map creates problems for clinicians. If a model confidently outlines a structure that turns out to be wrong, the clinician who trusted it has made a worse decision than if they had done the work manually. Uncertainty estimation attempts to solve this by having the model flag regions where it is less confident, so the clinician knows where to double-check. Knowing how much confidence there is in a prediction is considered essential for gaining clinician trust.30Computational Intelligence. Estimating uncertainty in deep learning for reporting confidence to clinicians in medical image segmentation and diseases detection Methods for estimating uncertainty include running the model multiple times with slight random variations and observing how much the output changes, or training the model to produce probability maps rather than hard boundaries. Neither approach is computationally free, and integrating uncertainty information into clinical interfaces without overwhelming the user remains an open design challenge.