How to get started in Labelbox Model: Train, evaluate, and improve your ML models
The quality of your data will dictate your model’s performance. ML teams have historically had to rely on manual methods of curating data and debugging model errors. For teams who are looking to go through fast, data-centric iterations, this is not an ideal way to quickly scale and reach production AI.
In order to ship performant models, you need to be able to quickly train models with collaborative data-centric tools. Labelbox Model can help you ship better models faster by leveraging collaborative tools to curate, debug, diagnose, and optimize your machine learning data and models.
This guide will walk you through how to get started with Model in the Labelbox platform. We’ll walk through a COCO object detection example and show you how to get onboarded in Model with your first project, model, and model run.
By the end of the tutorial, you will have learned how to:
Upload and version a dataset in a Labelbox Model run
Export and train an object detection model from a pre-trained model
Visualize model predictions against ground truth annotations
View auto-generated metrics (F1, precision, recall, IOU, confusion matrix, etc.) and the distribution of annotations and predictions
Evaluate model performance and improve your model and data with error analysis and active learning
Key definitions in Labelbox Model:
A model is a directory for all of your experiments for one machine learning problem (e.g object detection on COCO)
A model run is a model training experiment within a model directory. Each model run provides a versioned data snapshot of the data rows, annotations, and training/validation/test splits for that model run
You can specify model run configurations to create, version, and track your hyperparameters and any training-related configurations for a model run
Part 1: Seed a COCO dataset and create a project and model run
For this onboarding tutorial, we’ll be working with a COCO dataset example. You can follow along by using the Google Colab Notebook and the accompanying video tutorials, but feel free to also bring your own dataset into Labelbox Model to create a Labelbox project, model and model run for your use case.
Step 1: Train an object detection model from a pre-trained Faster R-CNN model
Export the labels from the model run you created – the labels will be versioned by the model run
Use the provided helper function to transform the labels into a format that the Pytorch model can accept
We’ve provided a Faster R-CNN model for fine-tuning – this replaces the last box prediction head with a new layer so that it can predict the model ontology and classes that we are concerned with
Train the Faster R-CNN model for 1 epoch (4 minutes) – after this, you’ll have a model that performs decently well for this use case
Step 2: Generate predictions with the trained model
Generating predictions will help us visualize model performance in Labelbox Model and can help identify model errors
Assemble predictions and results of the model into a format that can be ingested back into Labelbox
Turn predictions into NumPy arrays and create annotation payloads for each object
Step 3: Upload predictions back into your Labelbox model run
Upload the model predictions to the Labelbox app in order to visualize how the model is performing
Other than confidence scores, you don’t have to worry about computing metrics. Labelbox Model will auto-compute model metrics such as F1 scores, precision, IOU, confusion matrix, false positives/false negatives/true positives/true negatives, etc
Step 4: Visualize model predictions and auto-generated metrics in Labelbox Model
After you’ve completed the steps above, you should now be able to go to the Models tab to visualize model predictions and model metrics
Click on “Metrics View” to inspect model metrics and the model’s performance on each class
You can run error analysis on specific classes or data rows of interest:
Start by inspecting how the model is doing on each class
If you notice the model is struggling on a specific class, you can click into the histogram to view data rows on which the model is struggling. You can refine the search query to further drill into and inspect model performance
Leverage “detailed view” to better inspect disagreements and find patterns of model failures on images
After running error analysis, you can make more informed decisions on how to iterate and improve your model’s performance with corrective action or targeted data selection
By following this step-by-step tutorial, you’ve now successfully created a model, a model run, and have uploaded model predictions into Labelbox for further analysis.
You can also refer to the below guides for a more in-depth walkthrough on how to improve data selection and model performance: