Hybrid & On-prem
Cloud data overview
Restrict data access by IP range
How to generate signed URLs
How to generate non-expiring signed URLs
Creating your first project
Project setup script
Python API FAQ
Intro to the GraphQL API
Data types overview
Model predictions (legacy)
Legacy vs new editor ontology
Legacy vs new editor JSON exports
Legacy vs new editor hotkeys
July 6, 2020
June 22, 2020
June 2, 2020
May 19, 2020
April 14, 2020
April 1, 2020
March 3, 2020
February 18, 2020
February 5, 2020
January 17, 2020
Table of Contents
Benchmarks is a Labelbox QA tool that automatically compares all labels on a data row to a “gold standard” label you set. Once an asset with a Benchmark label gets a human- or computer-generated label, the Benchmark agreement score is automatically calculated.
Global or nested classifications
Labelbox follows a similar methodology for calculating the agreement scores for both Benchmarks and Consensus. The only difference in the calculations is the entity to which the Labels are compared.
Generally speaking, calculating agreement for the polygons of a Label involves Intersection-over-Union and a series of averages to calculate the final agreement between two Labels on an image.
Benchmarks works by interspersing data to be labeled, for which there is a Benchmark label, to each person labeling. These labeled data are compared against their respective Benchmark and an accuracy score between 0 and 100 percent is calculated.
When a Label is created or updated, the Benchmarks score will be recalculated as long as there is one Label on the Data Row. if a Label gets deleted, no benchmark score will appear for that Data Row.
Either Benchmarks or Consensus can be turned on for a project at any given time, but it is not possible to have both on at the same time.
- Create a project.
- Navigate to Settings > Quality and select Benchmarks to turn this QA feature on for your project.
- To mark a label as a Benchmark, select a data row from the Activity table to enter Open review mode, then click the star at the top. When you go back the Activity table, that label you selected as the benchmark will be marked with a gold star.
Benchmark labels are marked with a gold star in the Activity table under the Labels tab.
Under the Labels tab, there is also a Benchmarks table where you’ll see a list of all the Benchmarks labels for that project. Click on “View Results” to see all labels associated with that benchmark label.
When the Benchmarks tool is active for your project, the Individual performance section under the Performance tab will display a Benchmarks column that indicates the average Benchmark score for that labeler.