A bounding box is a rectangle drawn around an object in an image to mark its location. It’s the most common way computers identify where something is in a photo or video, used in everything from self-driving cars spotting pedestrians to medical scans flagging tumors. Each box is defined by just a handful of coordinates, making it a simple but powerful tool at the heart of modern AI.
How a Bounding Box Is Defined
A bounding box needs only four numbers to describe itself, and there are two standard ways to express them. The first uses the coordinates of two opposite corners: the upper-left corner and the lower-right corner of the rectangle. If you know those two points, you know the full shape. The second format uses the center point of the box plus its width and height. Both describe the same rectangle, just from different starting points.
In image coordinates, the origin (0, 0) sits at the top-left corner of the image. Moving right increases the x value, and moving down increases y. This is the opposite of a typical math graph where y goes upward, and it’s a common source of confusion for people new to the field. Every bounding box in computer vision follows this coordinate system.
Along with the four spatial values, a bounding box prediction from an AI model typically includes two more pieces of information: a class label (what the object is) and a confidence score (how sure the model is about the detection). So a single detection might look something like: “dog, 92% confidence, located at these coordinates.”
Axis-Aligned vs. Oriented Bounding Boxes
The simplest and most widely used type is the axis-aligned bounding box, sometimes called an AABB. Its edges are always perfectly horizontal and vertical, aligned with the image’s axes. This works well for objects that are roughly upright, like a person standing or a car seen from the front. But for objects at an angle, like a ship photographed from a satellite or a tilted text block, an axis-aligned box wastes a lot of space and captures background along with the object.
Oriented bounding boxes (OBBs) solve this by adding a fifth value: a rotation angle. Instead of locking to horizontal and vertical, the rectangle can tilt to match the object’s actual orientation. The format becomes center point, width, height, and rotation. This gives a much tighter fit around angled objects, which improves localization accuracy. Oriented detection is common in aerial imagery, document analysis, and any scenario where objects don’t sit neatly upright.
What Bounding Boxes Are Used For
Object detection is the core use case. When you see a photo with colored rectangles around faces, cars, or animals, those are bounding boxes. Every major detection model, from older architectures like Faster R-CNN to current versions of YOLO, outputs bounding boxes as its primary result. In a recent comparison of detection models trained to identify weed species in agricultural fields, the best-performing model (YOLOv9) achieved a mean average precision of 0.935, meaning its bounding boxes matched the true object locations with high reliability across thousands of test images.
In medical imaging, bounding boxes help radiologists by automatically highlighting areas of concern. Brain tumor detection models, for example, predict a bounding box around a tumor region on an MRI scan, giving the coordinates and a confidence score for each detection. This doesn’t replace a radiologist’s judgment, but it draws attention to suspicious areas that might otherwise be missed in a dense scan. Medical imaging standards like DICOM support storing spatial coordinates alongside images, so bounding box data from AI tools can be saved and shared as part of a patient’s imaging record.
Beyond healthcare and agriculture, bounding boxes power features you likely use daily. Smartphone cameras use them to lock focus on faces. Security systems use them to track people through a scene. Retail inventory systems use them to count products on shelves. Autonomous vehicles use them to detect other cars, cyclists, and road signs in real time.
How Accuracy Is Measured
The standard metric for evaluating a bounding box prediction is Intersection over Union, or IoU. The idea is intuitive: take the area where the predicted box and the true box overlap (the intersection), and divide it by the total area covered by both boxes combined (the union). The formula is:
IoU = area of overlap / area of union
An IoU of 1.0 means the predicted box and the ground truth box are identical. An IoU of 0 means they don’t overlap at all. In practice, a prediction is typically considered correct if its IoU exceeds 0.5, though stricter evaluations use thresholds of 0.75 or higher. When researchers report a metric like “[email protected],” they mean the model’s average precision when using 0.5 as the IoU cutoff for a successful detection.
Handling Overlapping Predictions
One practical challenge with bounding box detection is that models often produce multiple overlapping boxes for the same object. A detector might generate five slightly different rectangles around the same cat, each with a different confidence score. Outputting all of them would be cluttered and inaccurate.
The standard solution is a technique called non-maximum suppression, or NMS. It works by keeping the highest-confidence box for each object and removing any other boxes that overlap with it beyond a set threshold. The algorithm loops through all predictions, compares their IoU values, and discards the redundant ones. What remains is a clean set of one box per object, which is what you actually see as the final output of a detection system.
Limitations of Bounding Boxes
Bounding boxes are rectangular, and most real-world objects are not. A bounding box around a banana includes a lot of empty space. For a person with arms outstretched, the box captures far more background than person. This is fine when you just need to know where something is, but it falls short when you need the exact shape of an object.
For pixel-precise boundaries, computer vision uses a different approach called segmentation, which labels every individual pixel as belonging to the object or the background. Segmentation is more computationally expensive and harder to annotate in training data, which is why bounding boxes remain the default for most detection tasks. They strike a practical balance: fast to compute, easy to label, and accurate enough for the vast majority of applications. Many workflows start with bounding box detection and then apply segmentation only within the detected region to get finer detail when needed.