How to build equipment detection models to improve worker safety and efficiency

With AI-powered object detection, you can now seamlessly integrate the latest advances in foundation models into your warehouse and construction site safety operations. As the demand for better safety monitoring continues to rise, it's essential for teams to maximize protective equipment use to mitigate potential hazards. Labelbox empowers the world’s largest organizations to leverage AI solutions tailored to their unique safety detection challenges.

However, teams can face multiple challenges when implementing AI for large-scale safety detection. This includes: 

• Data quality and quantity: Improving safety detection requires a vast amount of data in the form of images and videos. Orchestrating data from various sources can not only be challenging to maintain, but even more difficult to sort, analyze, and enrich with quality insights.
• Dynamic review landscape: The changing nature and format data from multiple sources poses the challenge for businesses to account for continuous data updates and re-training needs. 
• Cost & scalability: Developing accurate custom AI can be expensive in data, tools, and expertise. Leveraging foundation models, with human-in-the-loop verification and active learning, can help accelerate model development by automating the labeling process.

Labelbox is a data-centric AI platform that empowers businesses to transform their safety detection through advanced computer vision techniques. Instead of relying on time-consuming manual human review, companies can leverage Labelbox’s AI-assisted data enrichment and flexible training frameworks to quickly build task-specific models that uncover actionable insights for the prevention of supply chain mistakes.

In this guide, we’ll walk through an end-to-end workflow on how your team can leverage Labelbox’s platform to build a powerful task-specific model to improve safety detection using personal protective equipment as an example. Specifically, this guide will walk through how you can explore and better understand your visual assets to make more data-driven business decisions for worker safety.

See it in action: How to build equipment detection models to improve worker safety and efficiency

The walkthrough below covers Labelbox’s platform across Catalog, Annotate, and Model. We recommend that you create a free Labelbox account to best follow along with this tutorial.

Part 1: Explore and enhance your data with Catalog and Foundry

Part 2: Create a model run and evaluate model performance

You can follow along with both parts of the tutorial below via:

• Colab Notebook

Part 1: Explore and prepare your data

Follow along with the tutorial and walkthrough in the Colab Notebook. If you are following along, please make a copy of the notebook. 

Ingest data into Labelbox 

For this tutorial, we’ll be working with a dataset that detects workers wearing safety equipment  – with the goal of quickly curating data and finding protective equipment (e..g, helmets, goggles, reflective vests, gloves, masks, etc) from high-volumes of images.

The first step will be to gather data:

Please download the dataset and store it in an appropriate location on your environment. You'll also need to update the read/write file paths throughout the notebook to reflect relevant locations on your environment. You'll also need to update all references to API keys, and Labelbox ontology, project, and model run IDs

• If you wish to follow along and work with your own data, you can import your data as a CSV.
• If your images sit as individual files in cloud storage, you can reference the URL of these files through our IAM delegated access integration. 

Once you’ve uploaded your dataset, you should see your image data rendered in Labelbox Catalog. You can browse through the dataset and visualize your data in a no-code interface to quickly pinpoint and curate data for model training. 

Search and curate data

You’ll now be able to see your dataset in Labelbox Catalog. With Catalog, you can contextualize your data with custom metadata and attachments to each asset for greater context. 

In this demo, we'll be using Catalog to find relevant images of equipment for our dataset with the goal of annotating bounding boxes for the personal protective equipment using foundation models.

Leverage custom and out-of-the-box smart filters and embeddings to quickly explore product listings, surface similar data, and optimize data curation for ML. You can: 

• Search across datasets to narrow in on data containing specific attributes (e.g metadata, media attributes, datasets, project, etc.) 
• Automatically find similar data in seconds with off-the-shelf embeddings
• Filter data based on natural language and flexibly layer structured and unstructured filters for more granular data curation

Using Foundry to pre-label bounding boxes

In this next step, we'll walk through how you can take a human-in-the-loop approach to iterate or modify pre-labels and speed up the annotation process.

Model Foundry enables teams to choose from a library of models and in this case, we'll be using GroundingDINO to generate previews and attach them as pre-labels.

With Model Foundry, you can automate data workflows, including data labeling with world-class foundation models. Leverage a variety of open source or third-party models to accelerate pre-labeling and cut labeling costs by up to 90%.

Part 2: Train your model and generate predictions

The next step is to take all of our labeled data and train our model on it. This allows us to make predictions on this data and for Labelbox to calculate evaluation metrics so that we can see where the model is going wrong and improve model performance.

View model predictions within the Labelbox UI to evaluate and diagnose model effectiveness

As a last step, let's compare model inferences with ground-truth annotations to see where the model may be underperforming. A disagreement between model predictions and ground truth labels is typically due to either a model error (poor model prediction) or a labeling mistake (ground truth is wrong). 

• After running the notebook, you’ll be able to visually compare ground truth labels (in green) to the model predictions (in red).
• Use the ‘Metrics view’ to drill into crucial model metrics, such as confusion matrix, precision, recall, F1 score, and more, to surface model errors.
• Model metrics are auto-populated and interactive. You can click on any chart or metric to open up the gallery view of the model run and see corresponding examples.
• Use Labelbox Model for 10x faster corner-case detection – detect and visualize corner-cases where the model is underperforming.

After running error analysis, you can make more informed decisions on how to iterate and improve your model’s performance with corrective action or targeted data selection and additional labeling.

Summary

By analyzing high-volumes of images and videos using foundation models and human alignment, Labelbox provides teams with the ability to inject valuable insights for delivering better protective equipment detection models for warehouses and construction sites that allow you to improve operational efficiency, compliance and overall worker safety.

Labelbox is a data-centric AI platform that empowers teams to iteratively build powerful task-specific models. To get started, sign up for a free Labelbox account or request a demo.