Table of Contents
Updated by Alex Cota
This page explains how to use the review tools in Labelbox, how the review queue works, and how to configure the review tool to meet your quality assurance needs.
We offer collaborative quality assurance tools to help you review your labeled data. There are two types of review modes.
Queue-based review refers to the interface that users will see after an admin user configures the review queue to randomly distribute labels for review to a select group of users within a project. A user can do the following things in queue-based review:
- Review a label (upvote/downvote)
- Modify a label
Open review refers to the review interface that appears when an admin user click on a row in the activity list within a project. It presents a streamlined and transparent way to review, modify, copy, and re-enqueue labeled images and track progress. You can do the following things in review mode.
- Modify a review
- Modify a label
- Copy a label URL
- Set a label as a Benchmark
About the review queue
The review queue is completely separate from the labeling queue. We have configured the review queue to abide by the following rules to ensure that labeling and reviewing operations can happen concurrently while eliminating the risk of users' interfering with each other's work.
- Only images that have been labeled or skipped are entered into the review queue.
- Each labeled asset in a reviewer's queue is unique so that only one user may perform a review on a labeled image. A label in the review queue can be reviewed by more than one user, but never more than once by the same user. Once a label is reviewed by the predetermined number of users, via the queue-based review or open review, it will leave the review queue so it will not be reviewed again.
Setting percentage of labels to review
If you do not need to review 100% of your labeled images, you can adjust the percentage of labels to be reviewed by going to Settings > Quality > Review Step. Here you can set the percentage of labels that will enter the review queue.
As an admin, you can bypass the percentage restriction set in Review Step by navigating to Labels > Activity and clicking on any label to enter review mode for all labels in the review queue. Review mode offers the most functionality.
Assessing quality of reviewed labels
The Reviews section at the bottom of the Overview tab contains a simple metric for understanding the general trends in user reviews.
Un-reviewed: labels where upvotes and downvotes cancel each other out
Accepted: labels where the majority upvoted
Declined: labels where the majority downvoted
Ambiguous: no reviews at all
Enable/disable review queue programmatically
To learn how to use the GraphQL API for enabling and disabling the review queue for a project, see Review queue.