Multistep Labeling

Labelbox is an extremely flexible tool that can be easily configure for a very diverse set of use cases. Many teams with more advanced use cases find it useful to arrive at labels after multiple steps.

In the below example we will be following a data science team that is identifying damage on teeth.

Step 1: Labeling team A is comprised of regular dentists who will either classify the images as either “damaged” or “not damaged”. Step 2: If an asset is labeled as damaged it will be moved to Labeling team B which is comprised of specialists who will put a polygon around exactly where the tooth is damaged.

Here is the project setup.

Here is what a labeler in step 1 would see…

To move images containing damage to step two in real time we can leverage Labelbox’s webhooks.

The below flask app will move assets from step one to step two if “image_contain_tooth_damage?” is true.

flask==1.0.2
graphqlclient==0.2.4
FROM python:3.6

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

# unblock port 80 for the Flask app to run on
EXPOSE 80

COPY . ./
CMD [ "python", "./index.py" ]
from flask import Flask, request
from graphqlclient import GraphQLClient
import json
app = Flask(__name__)

client = GraphQLClient('https://api.labelbox.com/graphql')
client.inject_token('Bearer <YOUR-TOKEN-HERE>')

@app.route('/')
def hello_world():
return 'Hello, World!'

def create_datarow(row_data, dataset_id):
res_str = client.execute("""
mutation CreateDataRowFromAPI(
$rowData: String!,
$datasetId: ID!
) {
createDataRow(data:{
rowData: $rowData,
dataset:{
connect:{
id: $datasetId
}
}
}){
id
}
}
""", {
'rowData': row_data,
'datasetId': dataset_id
})

res = json.loads(res_str)
return res['data']['createDataRow']['id']

@app.route('/step1_to_step2', methods=['POST'])
def move_label_to_step_2_if_damaged():
payload = json.loads(request.data.decode('utf8'))
label = json.loads(payload['label'])
if label['image_contain_tooth_damage?'] == 'yes':
STEP_2_DATASET_ID = 'cjxjg5ujfac7n0846hq4wjvgf'
datarow_id = create_datarow(payload['dataRow']['rowData'], STEP_2_DATASET_ID)
return 'Created Asset to Step 2: ' + datarow_id
else:
return 'no further action needed'

if __name__ == '__main__':
app.run("0.0.0.0", port=80, debug=True)

To run this application locally docker…

# Make sure you have docker installed

# Build the image
docker build -t webhook-app .

# Run the container on port 5000
docker run -p 5000:80 webhook-app

# Now visit localhost:5000 to see 'Hello, World!'

It’s easy to deploy this application to zeit’s now or you could deploy it on your own web server.

➜  multi-step-labeling: now .
> Deploying ~/Downloads/multi-step-labeling under labelboxcom
> Using project multi-step-labeling
> Synced 3 files (1.79KB) [407ms]
> https://multi-step-labeling-rdgskoksnq.now.sh [v1] [in clipboard] (sfo1) [2s]
> Build completed
> Verifying instantiation in sfo1
> ✔ Scaled 1 instance in sfo1 [49s]
> Success! Deployment ready

Your deployment now lives on https://multi-step-labeling-rdgskoksnq.now.sh and your webhook url is https://multi-step-labeling-rdgskoksnq.now.sh/step1_to_step2 as declared in the index.py file.

The last thing you need todo is register this webhook as explained in the webhook docs

mutation CreateWebhook {
createWebhook(data:{
project:{
# This is my projectid
id:"cjygd0pacnkw407946odkbd0b"
},
url:"https://multi-step-labeling-rdgskoksnq.now.sh/step1_to_step2",
secret:"this_will_be_sent_back_as_a_header",
topics:{set:[LABEL_CREATED, LABEL_UPDATED, LABEL_DELETED]}
}){
id
}
}


How did we do?