Review labels

Updated 2 weeks ago by Alex Cota

This page explains how to use the review tools in Labelbox, how the review queue works, and how to configure the review tool to meet your quality assurance needs.

Review modes

We offer collaborative quality assurance tools to help you review your labeled data. There are two types of review modes.

Queue-based review

Queue-based review refers to the interface that users will see after an admin user configures the review queue to randomly distribute labels for review to a select group of users within a project. A user can do the following things in queue-based review:

  • Review a label (upvote/downvote)
  • Modify a label
Open review

Open review refers to the review interface that appears when an admin user click on a row in the activity list within a project. It presents a streamlined and transparent way to review, modify, copy, and re-enqueue labeled images and track progress. You can do the following things in review mode.

  • Modify a review
  • Modify a label
  • Copy a label URL
  • Set a label as a benchmark (only available in Legacy editor)

About the review queue

The review queue is completely separate from the labeling queue. We have configured the review queue to abide by the following rules to ensure that labeling and reviewing operations can happen concurrently while eliminating the risk of users' interfering with each other's work.

  1. Only images that have been labeled or skipped are entered into the review queue.
  2. Each labeled asset in a reviewer's queue is unique so that only one user may perform a review on a labeled image. A label in the review queue can be reviewed by more than one user, but never more than once by the same user. Once a label is reviewed by the predetermined number of users, via the queue-based review or open review, it will leave the review queue so it will not be reviewed again.

Setting percentage of labels to review

If you do not need to review 100% of your labeled images, you can adjust the percentage of labels to be reviewed by going to Settings > Quality > Review Step. Here you can set the percentage of labels that will enter the review queue.

As an admin, you can bypass the percentage restriction set in Review Step by navigating to Labels > Activity and clicking on any label to enter review mode for all labels in the review queue. Review mode offers the most functionality.

Assessing quality of reviewed labels

The Reviews section at the bottom of the Overview tab contains a simple metric for understanding the general trends in user reviews.

Un-reviewed: labels where upvotes and downvotes cancel each other out

Accepted: labels where the majority upvoted

Declined: labels where the majority downvoted

Ambiguous: no reviews at all


Was this page helpful?