Image Annotation 101

What does it mean to annotate an image?

Image annotation is defined as the task of labeling digital images, typically involving human input and, in some cases, computer-assisted help. Labels are predetermined by a machine learning engineer and are chosen to give the computer vision model information about the objects present in the image. The process of labeling images also helps machine learning engineers hone in on important factors in the image data that determine the overall precision and accuracy of their model.

Example considerations include possible naming and categorization issues, how to represent occluded objects (objects hidden by other objects in the image), how to deal with parts of the image that are unrecognizable, etc.

How do you annotate an image?

From the example image below, a person has used an image annotation tool to apply a series of labels by placing bounding boxes around the relevant objects, thereby annotating the image. In this case, pedestrians are marked in blue and taxis are marked in yellow, while trucks are marked in yellow.

Depending on the business use case and project, the number of image annotations on each image can vary. Some projects will require only one label to represent the content of an entire image (e.g., image classification). Other projects could require multiple objects to be tagged within a single image, each with a different label (e.g., a bounding box).

Annotation software is designed to make image labeling as easy as possible. A good annotation app will include features like a bounding box annotation tool and a pen tool for freehand image segmentation.


Bounding boxes applied to identify vehicle types and pedestrians

What are the different types of image annotation?

To create a novel labeled dataset for use in computer vision projects, data scientists and ML engineers have the choice between a variety of annotation types they can apply to images. Researchers will use an image markup tool to help with the actual labeling. Let’s compare and summarize the three common annotation types within computer vision: 1) classification 2) object detection and 3) image segmentation.

  • With whole-image classification, the goal is to simply identify which objects and other properties exist in an image without localizing them within the image.

  • With image object detection, you go one step further to find the position (established by using bounding boxes) of individual objects within the image.

  • With image segmentation, the goal is to recognize and understand what's in the image at the pixel level. Every pixel in an image is assigned to at least one class, as opposed to object detection, where the bounding boxes of objects can overlap. This is also known as semantic segmentation.


Different types of image annotation

Whole image classification provides a broad categorization of an image and is a step up from unsupervised learning as it associates an entire image with just one label. A distinct benefit is that it is by far the easiest and quickest to annotate out of the other common options. Whole-image classification is also a good option for abstract information such as scene detection and time of day.

Bounding boxes, on the other hand, are the standard for most object detection use cases and require a higher level of granularity than whole-image classification. They provide a balance between quick annotation speed and targeting items of interest.

For specificity, image segmentation is chosen to support use cases in a model where you need to definitively know whether or not an image contains the object of interest and also what isn’t an object of interest. This is in contrast to other annotation types such as classification or bounding boxes that may be faster but less accurate.

Why is image annotation useful?

Image annotation is generally used to prepare images to use in machine learning as part of creating algorithms that perform computer vision tasks. Put more simply, annotated images are used to train computers to annotate images on their own.

This kind of computer vision model is an increasingly important technology. Self-driving is one obvious application. To safely navigate in traffic, a self-driving vehicle relies upon what is essentially a very sophisticated computer vision annotation tool. It is able to label all the objects in the vehicle's environment (cars, pedestrians, bicycles, trees, etc.). The vehicle's computer uses this image data to navigate successfully and safely. 

There are many off-the-shelf computer vision models available. One such model is YOLO, an object detection model that generates bounding box annotations in real time. YOLO stands for "You only look once," indicating that the algorithm analyzes the image and applies image annotations in one pass, prioritizing speed. 

How does a training data platform support complex image annotation? Image annotation projects begin by determining what should be labeled in the images and then instructing annotators to perform the annotation tasks using an image annotation tool. Annotators must be thoroughly trained on the specifications and guidelines of each annotation project, as every company will have different image labeling requirements. The annotation process will also differ depending upon the image annotation tool used.

Once the annotators are trained on proper data annotation procedures for the project, they will begin annotating hundreds or thousands of images on a training data platform dedicated to image annotation. A training data platform is software designed to have all the necessary tools for annotating images. Such software is commonly equipped with multiple image annotation tools which allow you to perform all the tasks mentioned above, from image classification to advanced semantic segmentation.

In addition, training data platforms typically include additional features that specifically help optimize your image annotation projects which include:

  • High-performance annotation tools:

    An important point to consider and test is whether or not the tools provided by the training data platform you are testing can support a high number of objects and labels per image without sacrificing loading times. At Labelbox, our fast and ergonomic drawing tools provide efficiency to help reduce the time-consuming nature of having pixel-perfect labels consistently. Our vector pen tool, for instance, allows you to draw freehand as well as straight lines. When you have the right tool for the job, image annotation is much easier.


Labelbox's pen tool illustrated

  • Customization based on ontology requirements:

    Labelbox image annotation gives you the ability to configure the label editor to your exact data structure (ontology) requirements, with the ability to further classify instances that you have segmented. Ontology management includes classifications, custom attributes, hierarchical relationships, and more. You'll be able to quickly annotate images with the labels that matter to you, without the clutter or irrelevant options.


Configure the label editor to your exact data structure (ontology) requirements.

  • A streamlined user interface which emphasizes performance for a wide array of devices:

    An intuitive design helps lower the cognitive load on image labelers which enables faster image annotation. Moreover, our uncluttered online annotation tool is built to run quickly, even on lower spec PCs and laptops. Both are critical for professional labelers who are working in an annotation editor all day.


A simple, intuitive UI reduces friction

  • Seamlessly connect your data via Python SDK or API:

    Stream data into your training data platform and push labeled data into training environments like TensorFlow and PyTorch. Labelbox was built to be developer-friendly and API-first so you can use it as infrastructure to scale up and connect your computer vision models to accelerate labeling productivity and orchestrate active learning.


Simplified data import without writing and maintaining your own scripts

  • Benchmarks & Consensus:

Quality is measured by both the consistency and the accuracy of labeled data. The industry-standard methods for calculating training data quality are benchmarks (aka gold standard), consensus, and review. As a data scientist in AI, an essential part of your job is figuring out what combination of these quality assurance procedures is right for annotated images used in your ML project. Quality assurance is an automated process that operates continuously throughout your training data development and improvement processes. With Labelbox consensus and benchmark features, you can automate consistency and accuracy tests. These tests allow you to customize the percentage of your data to test and the number of labelers that will annotate the test data.

Benchmarks overview

Benchmarks in action, highlighting the example labeled asset with a gold star

  • Collaboration and Performance Monitoring:

Having an organized system to invite and supervise all you labelers during an image annotation project is important for both scalability and security. A training data platform should include granular options to invite users and to review the work of each one.

With Labelbox, setting up a project and inviting new members is extremely easy, and there are many options for monitoring their performance, including statistics on seconds needed to label an image. You can implement several quality control mechanisms, including activating automatic consensus between different labelers or setting gold standard benchmarks.


Seamless collaboration between data science teams, domain experts, and dedicated external labeling teams

  • Automatic Annotation Tool:

If you are using image annotation to train a machine learning model, Labelbox allows you to use your model to create pre-labeled images for your labeling team using an auto annotation tool. Labelers can then review the output of the computer vision annotation tool and make any necessary corrections or adjustments. Instead of starting from scratch, much of the work is already done, resulting in significant time savings.