Table of Contents

Model-assisted labeling (import annotations)

Alex Cota Updated by Alex Cota

Model-assisted labeling workflow

The bulk import annotations workflow allows you to import computer-generated predictions (or simply annotations created outside of Labelbox) and load them as editable annotations on an image. This can be useful for speeding up the labeling process and supporting human labeling efforts.

Note: Model-assisted labeling is only available for Pro and Enterprise customers.

Model-assisted labeling supports the following label types:

Before you start

  1. Make sure you have the proper authentication.
  2. Create a Project.
  3. Create your Dataset and attach Data Rows.
  4. Use the sample script below to get the IDs for your Data Rows.
from labelbox import Client

client = Client(api_key="<LABELBOX_API_KEY>")
project = client.get_project("<PROJECT_ID>")

project.get_dataset("<DATASET_ID>")

for data_row in dataset.data_rows():
data_row.uid
To view a Python notebook end-to-end tutorial for the Model-assisted labeling workflow, click here.

Create the NDJSON file

Your import file must be in newline delimited (NDJSON) format. Each annotation you import will get its own row in the NDJSON file. Include the following values for each annotation, regardless of whether the annotation is an Object or Classification type.

uuid

User-generated UUID for each annotation. See the example below for a sample uuid.

The following UUID formats are supported:

A0EEBC99-9C0B-4EF8-BB6D-6BB9BD380A11

{a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11}

a0eebc999c0b4ef8bb6d6bb9bd380a11

a0ee-bc99-9c0b-4ef8-bb6d-6bb9-bd38-0a11

{a0eebc99-9c0b4ef8-bb6d6bb9-bd380a11}

schemaId

The Feature schema in the ontology that contains all of the information needed for rendering your annotation. To get the value for schemaId, use the sample script below and copy the value for featureSchemaId.

def get_ontology(client: Client, project_id: str) -> Dict[str, str]:
result = client.execute("""
query get_ontology($proj_id: ID!) {
project(where: {id: $proj_id}) {
ontology {
normalized
}
}
}
""", {"proj_id": project_id})
return result['project']['ontology']['normalized']

dataRow

ID of the Data Row where you want the annotation to be attached.

Mask annotations

Each mask color in your import file should match the corresponding mask color on the image. Passing a URI/mask color pair for a mask color that doesn’t exist in the image will generate an error and no annotation will be created.

For example, to import in a mask of the white road markings onto the sample image here, you would indicate a color of: [255, 255, 255].

Below are the two additional values you must provide if you are importing masks.

instanceURI

Where you are hosting the image mask. Can be hosted on any cloud provider. If you are importing multiple mask predictions on one Data Row, each mask should reference the same instanceURI.

colorRGB

An array of RGB values from 0 to 255 that indicates which color represents each given mask. Only 3-channel RGB colors are supported.

In the following sample, 3 mask annotations are attached to the same datarow and they each reference the same instanceURI. Before you import, make sure the masks and data row dimensions match.

Keep in mind that for an ndjson file, each newline is the entirety of a json object. The example below is prettified for legibility and would not be accepted as is. See a full sample import file here.
{ 
"uuid": "45b15f9d-7884-4bb7-ac01-3567e8ed6c36",
"schemaId": "ck68grts29n7w0890wv344dif",
"dataRow": {
"id": "cjxav5aa07r1g0dsq70t9eveg"
},
"mask": {
"instanceURI": "https://api.labelbox.com/masks/feature/ck7a0jw4o0nk80x9o5offz4mc",
"colorRGB": [
255,
255,
255
]
}
}
{
"uuid": "3a95ddcd-3ad0-4dc5-a24e-c05004b4b4d5",
"schemaId": "ck7wi85rnd1050757aac5ba4d",
"dataRow": {
"id": "cjxav5aa07r1g0dsq70t9eveg"
},
"mask": {
"instanceURI": "https://api.labelbox.com/masks/feature/ck7a0jw4o0nk80x9o5offz4mc",
"colorRGB": [
255,
0,
0
]
}
}
{
"uuid": "f8284cbc-ecf3-4363-9e10-138501daf5f7",
"schemaId": "ck7wi85pr1xz6079026yc0hch",
"dataRow": {
"id": "cjxav5aa07r1g0dsq70t9eveg"
},
"mask": {
"instanceURI": "https://api.labelbox.com/masks/feature/ck7a0jw4o0nk80x9o5offz4mc",
"colorRGB": [
0,
0,
0
]
}
}

Vector annotations

The following sample attaches Bounding box, Polygon, Point, and Polyline annotations to the same Data Row. Note: The geometry format in the import file matches the geometry format in the export file for each type. See a full sample import file here.

{
"uuid": "efca0c21-5206-4da6-8cb5-d6ca43649cfa",
"schemaId": "ck67grts29n7x0890atmeiahw",
"dataRow": {
"id": "cjxav4aa07r1g0dsq70t9eveg"
},
"bbox": {
"top": 153,
"left": 34,
"height": 204,
"width": 67
}
}
{
"uuid": "1b5762e9-416c-44cf-9a5f-07effb51f863",
"schemaId": "ck67grts29n7y0890q89jdcyp",
"dataRow": {
"id": "cjxav4aa07r1g0dsq70t9eveg"
},
"polygon": [
{
"x": 2,
"y": 99
},
{
"x": 93,
"y": 5
},
{
"x": 51,
"y": 106
},
{
"x": 176,
"y": 142
}
]
}
{
"uuid": "62e1d949-1c75-47f6-9ea2-e938da17d37c",
"schemaId": "ck68grts29n7z08903nvgaim5",
"dataRow": {
"id": "cjxav5aa07r1g0dsq70t9eveg"
},
"line": [
{
"x": 58,
"y": 148
},
{
"x": 135,
"y": 79
},
{
"x": 53,
"y": 191
}
]
}
{
"uuid": "532953e6-746f-4d74-945d-b4a9c2786479",
"schemaId": "ck68grts29n800890roip3u5d",
"dataRow": {
"id": "cjxav5aa07r1g0dsq70t9eveg"
},
"point": {
"x": 30,
"y": 150
}
}

Classifications

Bulk import annotations also supports global and nested Classifications. Each Classification references a schema for the question and a schema for each answer option. The table below indicates how to specify the answer schemas based on the Classification kind.

Radio

Radio questions can only have one “correct” answer. So, "answer" is singular.

"answer": {
"schemaId": "<answer_schema_id>"
}

Checklist

Checklist classifications can have multiple selected answers, so you should specify each “correct” option. Notice that "answers" is plural.

"answers": [
{
"schemaId": "<answer_1_schema_id>"
},
{
"schemaId": "<answer_2_schema_id>"
}
]

Text

Text classifications are answered with a string. Again, "answer" is singular.

"answer": "<correct_text_answer>"
Note: The import format should mirror the export format.

Global classifications

# Radio Classification
{
"schemaId": "ckd11j3yk000c0z0u4xn6dc4r",
"uuid": "1278daa6-ce64-4363-be24-4fa5eadffb17",
"dataRow": {
"id": "ckd11jg6scq9c0cq43vmh6i07"
},
"answer": {
"schemaId": "ckd11j415000u0z0ubu7ee4w2"
}
}
# Checklist Classification
{
"schemaId": "ckd1295hc00640z0uapvm1xbd",
"uuid": "fb72782d-f6ed-43ba-8677-77b03197392d",
"dataRow": {
"id": "ckd1299m8cqbs0cq43mju1bvp"
},
"answers": [
{
"schemaId": "ckd1295jn00760z0u01hw4yz5"
}, {
"schemaId": "ckd1295hh006g0z0ucbxgfgec"
}
]
}
# Text Classification
{
"schemaId": "ckd1295hg006c0z0u6x41hx0d",
"uuid": "4f1fe322-7b80-49a1-81cb-5914404df378",
"dataRow": {
"id": "ckd1299m8cqck0cq42lsz5khc"
},
"answer": "Text response"
}

Nested classifications

For classifications nested in objects, like in the export, the nested classifications are stored in an array under the key "classifications".

# Radio classification nested in a Bounding Box 
{
"uuid": "6e20a5ec-613d-4ecd-8fe5-34e47a05fea8",
"schemaId": "ckd1295hh006e0z0uh2x32i82",
"dataRow": {
"id": "ckd1299m8cqc00cq4c1by8hza"
},
"bbox": {
"top": 216,
"left": 144,
"height": 69,
"width": 67
},
"classifications": [
{
"schemaId": "ckd1295j9006k0z0udz3rh4mp", # Question Schema Id
"answer": {
"schemaId": "ckd1295jk006y0z0u1sk8h075" # Option Schema Id
}
}
]
},
# Checklist classification nested in a Point
{
"uuid": "9ee22dd1-550b-4686-8590-6cc2d3747911",
"schemaId": "ckd11j3ym000i0z0u288t78kk",
"dataRow": {
"id": "ckd11jg6scq9c0cq43vmh6i07"
},
"point": {
"x": 372,
"y": 19
},
"classifications": [
{
"schemaId": "ckd11j40y000s0z0u0hwse6ru",
"answers": [
{
"schemaId": "ckd11j45m001c0z0ub2dzah43"
}, {
"schemaId": "ckd11j45n001e0z0u2eqh4rlb"
}
]
}
]
},
# Text classification nested in a Polyline
{
"uuid": "a84e9209-6d3a-45a6-9f83-872d12f1eb3d",
"schemaId": "ckd11j3yl000g0z0u4bmz3slk",
"dataRow": {
"id": "ckd11jg6scq9c0cq43vmh6i07"
},
"line": [
{"x": 189, "y": 161},
{"x": 80, "y": 96},
{"x": 5, "y": 214}
],

"classifications": [
{
"schemaId": "ckd11j40w000q0z0ugkerhl3u",
"answer": "a nested text response"
}
]
}

Bulk import

To learn how to bulk import using the upload_annotations method in the Python SDK, see Import annotations.

Wait until the import job is complete before opening the Editor to make sure all annotations are imported properly.

Turn on Model-assisted labeling for the project

Turn on Model-assisted labeling for your project by navigating to “Settings” > “Automation”. When this is on, you will be able to view the imported annotations on an asset when you open it in the Editor. Only admins can toggle on/off Model-assisted labeling.

You can also turn on model-assisted labeling programmatically.

def turn_on_model_assisted_labeling(client: Client, project_id: str) -> None: 
client.execute("""
mutation TurnPredictionsOn($proj_id: ID!){
project(
where: {id: $proj_id}
){
showPredictionsToLabelers(show:true){
id
showingPredictionsToLabelers
}
}
}
""", {"proj_id": project_id})

Start labeling

When an asset is loaded in the Editor, any predictions for that asset will show up as editable annotations for the user.

Segmentation in label interface

Imported annotations will appear when the asset is opened in the Editor as long as the following conditions are met:

  • “Model-assisted labeling” is toggled on under Settings > Automation.
  • The imported annotations are assigned to a Data Row.
  • The asset has not already been labeled in the Labelbox Editor.

Duration (Timer) Currently, if the labeler skips or submits a label without making any changes first, the timer will not start and the duration time recorded will be 0 seconds. This logic may be revised in a future update.

Update/delete imported annotations

When a labeler "skips" an asset with imported annotations, the annotations will not be saved to the asset. Next time a user loads the asset in the labeling interface, the imported annotations will be loaded on the asset again.

When a labeler "submits" an asset with imported annotations, submitted annotations cannot be overwritten with new imported annotations without being deleted first. To update annotations with new imported annotations, you will need to:

  1. Delete the Label from the Data Row.
  2. Import the annotations with the same uuids.
  3. Load the asset again to see the updated annotations.
If you label an asset, delete the Label, then keep the Label as a template, that label template will take precedence over any imported annotations for that asset.

Refer to deleting an import request in order to delete all annotations imported by a specific request.

End-to-end tutorial

Click here for an end-to-end tutorial that walks you through each step in the Model-assisted labeling workflow.

Export format

You can find the export formats for each annotations type here.

Common Issues

Differences from Model predictions (legacy)

The feature outlined on this page is different from the feature outlined in Model predictions (legacy) which is for the legacy editor. Please refer to the Migration guide for migrating from the legacy editor.

The Python SDK references Prediction and PredictionModel which are for interacting with Model predictions (legacy) rather than the feature outlined on this page.

Was this page helpful?

Webhooks setup

Contact