How to automatically ingest data from Databricks into Labelbox_Guide_Graphic

How to automatically ingest data from Databricks into Labelbox

How to boost retail profitability with AI-powered shelf object detection_Guide_Header

How to boost retail profitability with AI-powered shelf object detection

Zero-Shot Learning vs. Few-Shot Learning vs. Fine-Tuning: A technical walkthrough using OpenAI's APIs & models_Guide_Header

Zero-Shot Learning vs. Few-Shot Learning vs. Fine-Tuning: A technical walkthrough using OpenAI's APIs & models

How to evaluate object detection models with Model Foundry_Guide_Header

How to evaluate object detection models with Model Foundry

How to build a content moderation model guide header

How to build a content moderation model to detect disinformation

How to analyze customer reviews and improve customer care guide header

How to analyze customer reviews and improve customer care with NLP

In this guide, we’ll walk through how your team can leverage Labelbox’s platform to build an NLP model to improve customer care. Specifically, this guide will walk through how you can explore and better understand review topics and classify review sentiment to make more data-driven business decisions around customer care initiatives. 

Labeling automation library card

labeling-automation-guide

Labeling vast quantities of data quickly, efficiently, and accurately is an immense challenge, but advanced machine learning teams have found a way to cut both labeling time and costs with an innovative solution: labeling automation. Download The Labelbox Guide to Labeling Automation to learn more about why training large models on large datasets without automation is so challenging, why Model Assisted Labeling is the labeling automation strategy proven to reduce time and effort, and real-world use cases.

texas

Achieving breakthroughs in baseball data with analytics and AI

RLAIF vs RLHF

RLHF-vs-RLAIF-library

Learn how RLAIF and RLHF differ, the pros and cons of each, and how to choose the right method for your fine-tuning project.

fine tune gpt image

How to fine-tune OpenAI’s GPT-3.5 Turbo using Labelbox

How to leverage Open AI’s GPT-3.5 and Labelbox to simplify the fine-tuning process, allowing you to rapidly iterate and refine your models’ performance on specific data.

walmart-library-hero

walmart-customer-story-library

Walmart's team wanted to find faster ways to annotate conversational text from shopping chatbots and label inventory images for their object detection and classification models, which included tens of millions of diverse product SKUs.

Dialpad Hero Image

dialpad-customer-story-library

Learn how Dialpad turned to the Labelbox platform and Boost services for higher quality training data and reduced time requirements from their own data scientists for labeling project design and review. In addition, see how Labelbox enabled the team to collaborate more smoothly with labelers and produce higher-quality labeled data for their NLP and LLM use cases.

Accelerate retail ROI webinar social card

How to accelerate time-to-value when building AI for retail and e-commerce

How to use Labelbox and Google Cloud to deliver successful AI systems for improving merchandising, enhancing customer care, optimizing marketing strategies in significantly less time.

labeling-observability-hero

labeling-ops-observability

How labeling quality can be measured and dramatically improved with the right observability practices for labeling operations.

GPT-4-enterprise-hero

GPT-4-enterprise

Labelbox Model Foundry provides access to a range of popular models, including third-party APIs and open source models. Categorized by their intended uses, you can quickly browse and select a model that is the best fit for your desired use case. After selecting a model, such as GPT-4, you can use our prompt-based interface to easily create effective prompts for generating accurate pre-labels.

BRT-data-engine-library

Learn how Blue River Technology’s ML teams can now spend more of their time focusing on training, monitoring, and maintaining their computer vision models. Their data scientists can pull updated, refined, relevant datasets for every use case and model within minutes via Labelbox Catalog.

DBx integration

Seamlessly integrate Databricks data pipelines with Labelbox

Use the Labelbox Connector for Databricks to quickly and easily connect their Databricks pipeline to upload their unstructured data to Labelbox.

Flex card thumbnail for Guide to unlocking AI with your data guide.

Guide to unlocking AI with your data_Thumbnail

The-guide-to-unlocking-AI-with-your-data

Less than a decade ago, generating entire software programs, marketing campaigns, and works of art in a matter of minutes or hours would have been impossible. Today's workforce, however, is armed with AI solutions like Copilot, ChatGPT, Stable Diffusion and more — with new and more powerful tools launching every day. AI has become an integral part of our day-to-day lives.


For enterprises, it's more important than ever to use AI effectively and quickly to build and maintain their competitive advantages. Today's AI teams have more computing power, more data, and more pre-trained models than before, but they also face more challenges and setbacks. An AI platform offers the tools and infrastructure to help mitigate these challenges and accelerate AI development. Read the guide to learn how.

Model foundry header

Model foundry_automating data tasks with foundation models_library

Model Foundry brings the power of foundation models into Labelbox to perform complex tasks such as data labeling and data enrichment. Early results indicate that over 88% of labeling tasks can be meaningfully accelerated by one or more foundation models. You will be able to access the world's best AI models from all major providers and open source alternatives. These models will act as a copilot for subject matter experts to accelerate model development.

Improve review efficiency with custom workflows



improve-review-efficiency-with-custom-workflows

improve-review-efficiency-with-custom-workflows-webcast

Welcome to ML Unboxed - the first webinar series designed for AI and ML professionals! You’ll learn about essential ML best practices and emerging concepts and then participate in a hands-on workshop guiding you through putting them into practice in Labelbox.

Product manager Pranoy Chaudhuri will teach you how to structure workflows that help you speed up review time, guide SME reviewers to high-impact data, and improve communication and collaboration with labeling teams.

To help ensure labeling quality, teams need to review labeled data to identify errors and trends leading to poor labeling performance. To do this at scale, you need an efficient workflow that most effectively balances labeling time, review and rework time, and quality. In this tutorial, we’ll walk through specific workflows to help improve quality and collaboration with review and labeling teams, including ways to:

- Customize your review queue based on label status, specific annotations, or specific labelers
- Quickly identify common issues faced by your labeling teams that lead to errors or inaccuracy
- Focus SME reviewer time on high-value labeled data where they’re needed most

With these advanced workflows, you’ll be able to increase the throughput, efficiency, and quality of your labeling and review.

How Deque uses data prioritization and model diagnostics to unlock AI breakthroughs in digital accessibility

how-deque-uses-data-prioritization

how-cape-analytics-uses-active-learning-to-get-to-production-ai-faster



Cape Analytics enables insurers and other property stakeholders to access valuable property attributes during underwriting, by using computer vision algorithms to extract information from geospatial imagery. The power of this solution lies in combining the accuracy and detail of property information traditionally relegated to in-person inspections, with the lightning speed of a living database covering the entire property base of the US, and delivering this information in a matter of seconds.

optimize-labeling-operations-webcast

Learn how Labelbox’s world-class workforce and dedicated labeling expertise can help you achieve both time and cost savings. In our team’s 21 years of experience with machine learning data labeling project management, we’ve managed hundreds of projects, optimizing productivity, cost control, and flexibility.

Watch now to get practical insights on how a managed solution can improve your team’s:

Productivity: Workflow set up and ontology defined in just days. Deep experience in project and instruction design to limit mistakes. Trained labeling teams that are ready to go.

Cost control: A team dedicated to driving efficiency and supporting growth while managing cost. Consistent monitoring of workflows and labelers’ performances to minimize risk and maximize quality.

Flexibility: Ability to evolve approach as needs shift or workflows gain efficiency. Combine external teams alongside in-house or existing resources. Weekend and shift work.

M&E whitepaper visual

get-more-value-unstructured-data

About 80% of enterprise data is now unstructured. But most media and entertainment companies struggle to realize the full promise and value of their video, audio and image files. Don’t let your business leave money on the table. Find out how to develop and launch your unstructured data strategy in this whitepaper.

Harnessing multiple data sources to build innovative AI products

harnessing-multiple-data-sources-to-build-innovative-ai-products-webcast

AI teams today use a variety of different data types for their projects, including text, computer vision, and specialized formats such as OCR, geospatial imagery, and more. Compiling enough high-quality data to train an algorithm can be a challenge with any of these data types, as they can be expensive and/or hard to find.

In this webinar, you’ll learn how leading AI teams tackle these bottlenecks by combining multiple sources of data to gain valuable insights, leverage both external and internal data, and ultimately develop more robust AI products. We’ll also cover:

When to employ different sources to improve predictions in a given ML iteration cycle

Examples of applied use cases and how companies are combining both subjective and objective information from sources such as OCR, images, and text

What cutting-edge AI teams are accomplishing and what’s possible via a training data platform

A walkthrough of how to use products like Catalog to understand and search all of your unstructured data (e.g., consumer reviews for consumer products)

How to combine workflows and data types when your data is unstructured by using ML frameworks like Databricks

How VirtuSense built an AI data engine that dramatically increased model performance

how-virtusense-built-an-AI-data-engine-that-dramatically-increased-model-performance

When VirtuSense founder Deepak Gaddipati’s active and healthy grandmother had a fall, broke her hip, and passed away within days due to resulting complications, it left the entire family shocked — even the physicians in the family. No one had known that she was a fall risk, as she’d never fallen before. The tragedy led Gaddipati to research how hospitals and care facilities could better identify fall risks and prevent similar deaths. He began his research in 2012, and by 2014, he had founded VirtuSense, an AI-based system of sensors that monitors the activity of patients and alerts caretakers when their movements indicate they may be at risk for a fall. 
Every hour, seven people die as a result of fall-related injuries in the United States. Today, VirtuSense is the #1 fall prevention system in the world. In the past two and a half years, the system has successfully prevented over 100,000 falls, saving tens of thousands of lives.
How it works
VirtuSense places LiDAR sensors in each hospital room, ambulance, or care facility room. These sensors generate data of patients’ movements as they lay in bed, sit up, stand, and walk around. The company’s AI models, placed within these edge devices, are trained to find and track movement patterns that, based on peer-reviewed research, identify potential falls 30-65 seconds before they occur. The system will then generate alerts for nearby hospital staff when the model detects fall risks. Expanding upon their initial success, VirtuSense systems now also detects risks for pressure ulcers and alerts bed-ridden individuals and/or their caregivers to shift their postures and prevent ulcers from forming.

The problem of trust
Developing an AI-based solution in healthcare came with one huge challenge: gaining the trust of healthcare professionals. Nurses who experience too many false alarms tend to take alerts less seriously. With current technology, they receive anywhere from ten to twenty alerts a day, a large percent of which are false alarms. To increase the effectiveness of these warning systems and gain the trust of hospital staff, VirtuSense had to ensure that their models achieved a high level of accuracy. Within a couple of years, their technology generated an average of only seven alerts per day, and an average of two were false alarms. As the team worked to increase the accuracy of their models beyond this, however, they hit a roadblock. They needed a more diverse, higher quality training dataset, which was becoming more and more challenging to generate.
The team used a mix of internal and open source labeling tools, which they found difficult to manage as the amount of training data required scaled. The team struggled to distribute unstructured data to the labeling team and track progress in an efficient manner. With their open source tools, the lack of immediate support when issues cropped up became yet another blocker for the team. The VirtuSense team chose Labelbox to help them scale their labeling workflows and generate more high-quality training data.
"False alarms should ideally occur less than 99.99% of the time, even when the models are running 365 days a year, 24/7, at 30 frames a second. Otherwise people won't trust the technology." — Deepak Gaddipati, Founder & CTO of Virtusense
Building a data engine
Rather than adopting Labelbox one step at a time, the VirtuSense team laid out a plan for creating an AI data engine to maintain and raise the accuracy of their models. An AI data engine is a seamless process that brings unstructured data from storage and into a labeling platform, enables labelers to annotate data quickly and easily, incorporates quality assurance workflows, and sends high quality training data into the model training system. As a model trains and enters production, a data engine also includes model error diagnosis and curating subsequent training datasets based on errors and low confidence areas, creating a more effective and efficient iterative loop for the model.

The VirtuSense team designed their data engine to automate every aspect of the model iteration loop, so that the only human involvement required is that of their labelers as they annotate data. The team’s plan also involved labeling LiDAR data not only for their current use cases (which, at the time, included only detecting falls with patients who were getting up from their beds), but also future use cases, such as detecting potential pressure ulcers. The team then implemented their vision by integrating Labelbox seamlessly into their MLOps. 
First, the VirtuSense team finds areas of low confidence in their AI models by running them on a specially curated testing dataset that includes edge cases and rare situations. The system then identifies data relevant to these low confidence areas and “cleans” it of metadata that attaches it to a particular sensor, facility, and patient to uphold confidentiality and security requirements and prevent biases from cropping up. This data is then pre-labeled by a purpose-built model and programmatically imported into Labelbox, a setup made possible with the Labelbox Python SDK. An internal labeling team then reviews and corrects the pre-labeled data within Labelbox, after which the datasets are programmatically exported from Labelbox and into the model training system on VirtuSense servers. 
Once a model is trained, it’s automatically pushed to thousands of endpoint edge devices, where it detects fall-risk movements and sends alerts accordingly. VirtuSense models are now retrained every few weeks with this data engine to increase and maintain their accuracy.
“We were very pleasantly surprised by how easy [Labelbox] is to use. People got the hang of it just like that. And it's not just about how easy it is to label data. We really like its ability to distribute data to multiple people and keep track of what's happening — it gives us a base of operations to help traverse through all that data….but the real beauty of it, the reason we really love Labelbox, is that it integrates so well with our tech. From a thousand-foot view, it looks like an AI labeling engine, which feeds into the deep learning system and spits out trained networks.” — Deepak Gaddipati, Founder & CTO of VirtuSense
Automating the labeling process
VirtuSense uses pre-labeling as a key component of their data engine. This labeling automation method imports model output (from either an off the shelf algorithm, a previous version of the model in training, or another model built by the ML team, depending on the use case) into the training data platform as pre-labeled data, which labelers can correct or edit instead of labeling unstructured data from scratch. 
When the VirtuSense team decided to leverage pre-labeling, they knew that their existing models, built to exist on edge devices with only a few parameters, would be too small to pre-label data according to their needs. Instead, they built a large model specifically for pre-labeling, with enough parameters to label all necessary data as accurately as possible. Once the model has labeled training data, the dataset is imported into Labelbox, where the small internal labeling team only needs to review and correct the labels before the assets are pulled into the model training workflow.
Since they adopted Labelbox, VirtuSense has been able to increase the amount of labels created by 5X. Their team has now produced over one million labeled assets. Their false alarm rates have fallen from roughly 28% to 6%, and the daily average number of alerts have dropped from seven to five. Their average accuracy has increased by over 20%.
“As a CTO, you usually only usually hear about the technologies your team uses when things are going wrong. With Labelbox however, the only time I dealt with it is the day I signed the contract. Since then, my team has been using the technology seamlessly without any issues, and this gives me a lot of peace of mind as a technical and engineering leader.” — Deepak Gaddipati, Founder & CTO of VirtuSense
Due to their new data engine and improved labeling capabilities, the VirtuSense team is planning to expand beyond LiDAR data to detect potential stroke risks, arrhythmia, and other dangerous conditions.
“Out of all the platforms we explored, Labelbox is the easiest to use and the best for managing labelers and workflows and monitoring performance. Labelbox also keeps offering more interesting and helpful features. I find the Catalog feature very interesting. We haven’t been able to use it yet, but we are planning to update our workflow to include it in the future.” — Hithesh Reddivari, Tech Lead, VirtuSense
Watch this on-demand webinar to learn how a data engine can improve your enterprise AI projects. You can also try Labelbox's annotation and data curation features for free.

active learning header image

Get started with active learning

The concept of data-centric machine learning, promoted widely by leaders in AI such as Andrew Ng in recent months, is quickly taking root in the enterprise ML community. Enterprises are treating their training data as IP because it’s the only component of machine learning that requires creativity and domain expertise to collect, organize, and curate. In this post, we’ll dive into some of the specific challenges of data-centric ML, and how your team can address them by employing active learning methods.

how-deque-uses-data-prioritization-and-model-diagnostics-to-unlock-ai-breakthroughs-in-digital-accessibility

How Deque uses data prioritization and model diagnostics to unlock AI breakthroughs in digital accessibility


Introduction

Deque has a storied journey on their mission of delivering digital equality to all. From the early days of the internet in the nineties, they made it their priority to pioneer and democratize digital accessibility. Digital accessibility is the practice of making digital documents, web and mobile apps accessible to everyone, including people with disabilities. Now, the power of machine learning is enabling the Deque team to lead the next generation of accessibility testing. Building out the components of their ML program has been challenging, but they have developed a sophisticated data engine that’s capable of prioritizing the most performant classes of data, discovering model errors quickly, and fueling their iterations with high-quality data. 

How Deque uses AI


In 2015, Deque made a bold decision to help advance automated accessibility testing. They decided to open source their most valuable asset, their accessibility testing rules library called axe-core. This rules engine powers Deque’s axe tools, the accessibility checker in Google’s Lighthouse and thousands of other notable projects around the globe amassing over half a billion downloads. Through automated testing alone, Deque’s axe can detect 57% of accessibility issues. While this might not sound like a lot if you’re new to this space, it’s more than double what the rest of the industry can offer and makes practicing digital accessibility a manageable task. To extend testing coverage even further, Deque turned to computer vision and machine learning to automatically catch even more accessibility issues, which were previously only detectable manually, using highly educated experts armed with various assistive technology and lots of time.

The Deque team is currently using their axe DevTools application to generate a pipeline of data that feeds into the machine learning system powered by Labelbox. The models that are generated are then being leveraged to expand coverage with an eventual target of being able to find 100% of accessibility issues in an automated fashion, thereby making true digital equality for all easy to achieve.


Challenge

It takes an average of about 10 minutes to fully annotate a single webpage screen. Between the thousands of web datasets and mobile datasets, the Deque team amassed a trove of useful data and looked to Labelbox and the Labelbox Workforce team for guidance and manpower. Prior to implementing Labelbox, the Deque team mostly relied on a combination of disparate open-source annotation tools, and hacking together Jupyter Notebooks with Google Sheets. 


Solution

“Before using Model Diagnostics in Labelbox to target the model’s weaknesses, we had to visualize the predictions on our own and it was a much more manual process,” said Noé Barrell, ML Engineer at Deque. “We had to calculate all the metrics on our own and it was a disjointed and difficult process. Being able to convert that process into something we could now do simply within the Labelbox platform made diagnosing errors a much more streamlined experience. It’s become so much easier to iterate.”



The combination of Model Diagnostics and Catalog has allowed the Deque team to reduce their 
data requirements by more than 50%.


Noé and the ML team at Deque were able to make considerable improvements to model performance by seeing the ability to evaluate and visualize model performance. “We detected some noise issues in our dataset and thanks to Model Diagnostics, we were able to filter out about one-third of data points we considered less trustworthy,” said Barrell. “By doing so, performance of the model went up 5%. We re-labeled some data and we saw the performance went up again after we added the re-labeled points. It was challenging data for humans to label and for the model to understand. Being able to target the data we already had in Labelbox and make changes and fixes to it was really helpful to us as a team to save time and target where we knew it would make a difference in our model’s performance.”

The Deque team made huge leaps in several areas via Model Diagnostics and were able to target their data collection in a way that addresses model failures more quickly. For example, they boosted performance on classes of data such as improving detection of checkboxes from 47% accuracy to 75% accuracy, presentational tables from 66% accuracy to 79% accuracy and radio buttons from 37.9% accuracy to 74% accuracy. 

In another time saving measure, the Deque team found they could search, discover, and prioritize the right data with Catalog. “The Catalog feature in Labelbox is also huge for us. Pre-Catalog, for our data selection process, we’d look at the performance metrics of our model and let’s say, for example, we discovered it was indicating 50% accuracy on modals, we would have to tediously and manually collect data surrounding that. But with the Catalog feature in Labelbox, we can target data collection on modals easily and quickly. Embeddings allow us to do unsupervised classification of modals and select a lot of modals. It’s just easier to create batches and sample around that. It takes a lot of the time and effort out of the data selection process,” said Barrell.




The Catalog feature in Labelbox allows the Deque team to prioritize the right data to boost model performance and make their iterations more meaningful and efficient. 


“Being able to reduce the data requirement is huge because you can see the same amount of improvement in the model’s performance in half the time and with half the effort. That was enabled through targeting the model’s weaknesses with Model Diagnostics and then being able to prioritize the right data through Catalog. Before, if we were doing a scattershot data collection, we would have been roughly labeling twice as much data and with twice as much effort,” said Barrell.


Source: Deque’s Guide to Digital Accessibility


Building accessible sites or apps can be tricky without the right guidance. The Web Content Accessibility Guidelines, or WCAG, published by the W3C, is the standard that defines what makes an application accessible. Developers, testers, app owners and accessibility experts from around the world rely on these standards for proper accessibility testing direction. Not only is accessibility a requirement for compliance with many laws, but accessibility, by ensuring all information is available in text form, also increases your website’s SEO and can improve the user experience for all users. For example, video captions don’t just assist those with hearing impairments but can be used by viewers in noisy environments or in environments that are not conducive to listening to audio. Accessibility features are for everyone and create a more inclusive world. 



Source: Deque What Is Digital Accessibility? 


To learn more about the Deque team and their mission, visit their website.

To understand how Labelbox can help you in your AI journey and help your team develop its own data engine, request a free demo now. 


6 best practices to save time when creating AI data

6-best-practices-to-save-time-when-creating-ai-data

saving-time-webcast

Wading through all your unstructured data to accurately annotate assets requires a tremendous amount of patience, organization, and time. This webinar will cover six key time-saving practices implemented by leading AI teams when handling AI data. Join us to discover how to:

Annotate faster with a dynamic queueing system

Improve communication, collaboration, and consensus between teams

Utilize a programmatic-approach for quicker access to data

Leverage software optimized for speed

Incorporate automation through model-assisted labeling

Utilize active learning and prioritize the right data

6 best practices to save time when creating AI data

The training data practices used by the most successful AI teams

training-data-practices-used-by-the-most-successful-teams-webcast

Having helped hundreds of AI teams across a wide range of verticals, we have discovered some commonalities in their ML workflows that lead to long-term project success and greater scalability. Seeing first-hand the key roadblocks and best practices of the most successful ML teams, we wanted to share how they’ve utilized certain processes, standards, and tools to more skillfully master how they work with training data.

In this webinar, you will learn some of the training data practices used by the most successful AI teams and how these have been applied in production use cases. We’ll also be answering live Q&A from the audience so that you can tailor these lessons into your own MLOps and accelerate your AI journey.

Learn how leading AI teams accomplish more by leveraging:

Advanced collaboration between domain experts, labelers, and data scientists

APIs for smooth integrations and delegated access for better access to data

Interpreting model performance and prioritizing areas for iteration

Monitoring on labeling performance across project levels and individual levels

Optimal iteration loop with Matthew Mcauley

optimal-iteration-loop-matthew-mcauley

ML practitioners have realized that iteration is the key to performant models, but perfecting and tightening the loop still remains a challenge for even the most advanced teams. Watch this session to hear the practical insights on how to get models to production-level performance quickly, how to best structure your training data pipeline and create the optimal iteration loop for production AI.

guide-to-training-data-platforms

guide-to-buying-a-training-data-platform

A Training Data Platform (TDP) is an investment in your models' maturity, accuracy, and endurance — so choosing one from the available options can be a difficult decision, especially with taking your use cases, stakeholders, existing processes, and future goals into account. That's why we've created a comprehensive checklist of features and capabilities that a TDP should have to best serve your specific requirements as your ML needs scale. The list includes:

build-or-buy

Until recently, most ML teams have had to build their own labeling tools or adopt a hodgepodge of open-source tools or service-based labeling workflows. Now however, a new category of tooling has emerged: the data engine for AI. A data engine offers best-in-class technology — and are purpose built for AI teams — to combine data, people, and processes into one seamless training data creation experience, enabling ML teams to produce performant models faster and more efficiently.  Technical leaders must now weigh the benefits of a data engine with the potential of building (or continuing to build) their own training data management software, and they have many factors to consider before making a decision. 

Functions and embeddings are a powerful way to quickly search and explore your unlabeled and labeled data. Using them can help you break down silos across datasets so you can focus on curating and labeling the data that will most significantly improve model performance.

When it comes to producing high-quality training data for data-centric AI, labeling is only one part of the solution. The most successful training datasets are carefully curated to increase model performance at each iteration. For example, if a model being trained to recognize various fruits in images has subpar performance in identifying apples, the next dataset used to train that model should contain more images of apples.

However, combing through a large unstructured dataset for just the right assets can be a challenge. Advanced AI teams have discovered a solution: using embeddings to sort through both labeled and unlabeled data.

How to use embeddings to create high-quality training data

how-to-use-embeddings

Functions and embeddings are a powerful way to quickly search and explore your unlabeled and labeled data. Using them can help you break down silos across datasets so you can focus on curating and labeling the data that will most significantly improve model performance.

When it comes to producing high-quality training data for data-centric AI, labeling is only one part of the solution. The most successful training datasets are carefully curated to increase model performance at each iteration. For example, if a model being trained to recognize various fruits in images has subpar performance in identifying apples, the next dataset used to train that model should contain more images of apples.

However, combing through a large unstructured dataset for just the right assets can be a challenge. Advanced AI teams have discovered a solution: using embeddings to sort through both labeled and unlabeled data.

What is an embedding?
An embedding, or feature vector, is an array of numbers assigned to an asset by a neural net. Assets that have similar content will also have similar embeddings. For example, in a dataset comprising images of apples and oranges, an appropriate embedding used for image similarity will show that all the vectors corresponding to apples have similar values. The vectors for all images of oranges will also be grouped together.

A visual representation of vectors in a dataset. Each cluster denotes a similar set of assets.
A visual representation of vectors in a dataset. Each cluster denotes a similar set of assets.
These similarity embeddings are typically generated by a convolutional neural network (CNN) and have dimensions in the order of thousands. Once you have implemented embeddings, similarity search is just a matter of comparing the distances between these vectors.

You can visualize embeddings by using certain techniques to “project” the points from their high dimensional space onto a plane. This can be especially useful to determine patterns in the data, like images from the same class forming clusters.

You can try the Labelbox embedding projector for free:

Sign up for a free account and switch to the Public Demo area
Navigate to Catalog
Select “Pomological Watercolors” in the “Datasets” filter on the left
Click on the point cloud icon on the top right to activate the embedding projector
In this example, the embeddings are automatically grouped by each of the five types of fruit present in the dataset.
In this example, the embeddings are automatically grouped by each of the five types of fruit present in the dataset.
How to generate embeddings for images and text
As explained above, similarity embeddings typically come out of a neural network. The diagram below shows a simplified version of what a typical classification CNN looks like.

Near the end of a neural network, before the last layer that determines the class the network will select, there’s usually one or more fully connected layers. To generate a similarity embedding for an image, input that image into the neural network and then take the values from the last fully connected layer.


The code below uses Tensorflow to load an EfficientNet pre-trained neural network and generate an embedding vector for an image:

import numpy as np
from tensorflow.keras.applications.efficientnet import EfficientNetB2
from tensorflow.keras.applications.efficientnet import preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing import image


efnet = EfficientNetB2(weights='imagenet')

# This is the image file we'll process
file = 'pepe.jpg'
img = image.load_img(file, target_size=(260, 260))
# Preprocess image
x = preprocess_input(np.expand_dims(img.copy(), axis=0))
# Make a new model where the output is the layer we care about
bottleneck_model = Model(inputs=efnet.input, outputs=efnet.get_layer('avg_pool').output)

# Then extract the features, which is basically a 1408-long vector
bottleneck_features = bottleneck_model.predict(x).reshape(1408)
print (bottleneck_features.shape)
print (bottleneck_features)
To generate embeddings for text data, the process is similar, except that we use a neural network trained on text.

In the example below, the code is a lot more compact because the model was trained specifically to generate embeddings, so it’s not necessary to “tweak” a classification model.

import tensorflow_hub as hub
import numpy as np
import tensorflow_text

text = ['The quick brown fox jumps over the lazy dog']

model = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual/3")

# Compute embeddings.
embedding = model(text)
How to use embeddings to find similar data in Labelbox
Labelbox’s embeddings projector makes it easy for AI teams to identify groups of similar assets within a dataset, filter them accordingly, and carefully curate their next training dataset based on where their model needs to improve. Embeddings can be imported via the Labelbox SDK (using precomputed embeddings is also an option).


Teams can then use the Catalog to conduct a similarity search to sort the assets by selecting one or more data rows and then selecting the “View Similar Data Rows'' option from the blue drop down menu on the top right. You can also select which embedding to use for the similarity search, including any custom embeddings previously uploaded with the SDK. Chosen assets can then be sent into a labeling project as a batch for labeling.

How to use functions in Labelbox to automatically find edge cases
Identify edge cases or low-performing classes in your model
Create functions to find similar low-performing data
Use functions to find similar data when new data is uploaded
A function is a saved similarity search that you can use as a filter to find visually similar data. Functions are a useful way to find relevant new data rows as they are added to your workspace. To create a new function in Labelbox, navigate to the Catalog and select the “View Similar Data Rows” option in the drop-down menu. The actively selected data rows and the chosen embedding metadata field will be used to define the function. Once you have created the function, it will start a background task to score all of the data rows that share the embedding.


A function can also be used as a filter in the left hand side panel in Catalog. You can mix function queries with all other filter types in Catalog. For additional details, please refer to the help section on Functions in our documentation.

Start curating high-value data faster
Functions and embeddings are a powerful way to quickly search and explore your unlabeled and labeled data. Using them can help you break down silos across datasets so you can focus on curating and labeling the data that will most significantly improve model performance. These features are available to all users on Labelbox. Sign in or create a free account to try it today!

high quality data meaning

what-does-it-mean-to-create-high-quality-training-data

In recent years, the machine learning community at large has shifted its focus from model to the quality of data used to train the model. Improving the quality of training data, however, can mean something different for every use case, model, and even iteration cycle.

Some teams will need to focus on enhancing their annotation pipeline with quality management methods, updated ontologies, and better collaboration. Others may need to look more closely at the performance of their model in training to better understand its specific needs. And some may need to simply work on expanding their training dataset to include certain classes and edge cases. In this post, we’ll look at all three areas of training data improvement.

Improving annotations
For many ML teams, the biggest hurdle to building a high-performance model is creating training data. It can be a challenge to ensure that each asset is labeled accurately and labeled in a way that will best “teach” the model what it needs to do.

To tackle these challenges, advanced ML teams have developed workflows that improve both labeling quality and the ontology as they better understand the use case and model requirements. While these workflows vary based on the problem at hand, they typically have these goals in common.

An iterative approach to producing training data
Many ML teams label their data in large chunks, or even all at once. This “waterfall” approach to labeling can make it difficult to ensure accuracy, as labeling requirements often evolve as an ML project progresses. Instead, leading ML teams label their data in smaller batches, giving more supervision and feedback to labelers at the beginning of the project to ensure they understand the task at hand, build a strategy to tackle any edge cases that arise, and adjust the ontology as both labelers and other stakeholders develop a better understanding of the project.

With an iterative approach that prioritizes two-way collaboration between the labelers and the ML team, labelers become faster, more efficient, and more accurate with their labels over time.

Ways to correct and improve labeled data
Many advanced ML teams implement quality management workflows into their labeling pipelines to evaluate and improve their training data. This can include benchmarking, which compares labeled assets to ground truth labels, consensus, which compares multiple labelers’ work on the same asset to ensure that everyone is performing the task the same way, and review queues, which enable other labelers or domain experts to review, approve, or leave feedback on labelers’ work.

Any of these workflows can go a long way to improving the quality of training data. Teams can also use metadata to track how a labeled asset has improved over time to assess the efficacy of these workflows.

Expert input
For some use cases in specialized fields such as healthcare or agriculture, a model needs to be trained on tasks that were traditionally done only by those with years of experience and training. Developing training data for these use cases may require these experts to do the labeling, which can be a lengthy and expensive process due to other demands on their time.

Genentech, a large biotech company, experienced this challenge first hand when training ML models to accurately read medical imagery such as radiology scans. Putting together a labeling team consisting entirely of trained radiologists and medical experts was an almost impossible task, so the team came up with an alternative approach. Their medical experts trained ordinary labelers on the task and reviewed their work in an iterative process. This workflow required less time from domain experts while still ensuring that the training data produced by the team was of high quality. Learn more about Genentech’s labeling workflow.

Understanding your model
Developing high-quality training data often requires a more nuanced understanding of the model after every iteration. While ML teams may look at aggregate summary metrics to track model improvements and trends, many teams don’t drill further down into the data. As a result, they can miss biases and performance variances across classes that may produce less favorable results unless the model is given data that rectifies them in the next iteration cycle.

Once a training cycle is complete, teams should look carefully at how the model performs with every class to identify potential edge cases, biases, and other issues. Teams should then choose the next training dataset according to their findings. For example, when building a model that uses satellite imagery to find and track ships, an ML team may find that the model performs well with a certain class when the image comes from one satellite, but performance is lower when presented with a similar task on an image from a different satellite. If the team were to only look at aggregated data, this issue might slip their notice entirely, but drilling into the variance of performance within this class will help them identify it. They can then include more images sourced from the second satellite in their next set of training data to improve performance. Watch this on-demand webinar, How to diagnose and improve model performance, to learn more about this workflow.

Building a better dataset
Datasets compiled according to performance after each iteration will go a long way to help your model make more significant leaps in performance, but this workflow only applies to teams who have an abundant, diverse dataset to work with in the first place. Enterprise ML teams often have to work with limited, proprietary data. Teams may also have niche or highly specialized use cases requiring data that cannot be found among large, publicly available datasets. To address these challenges, advanced ML teams are exploring new ways to round out their training datasets.

Synthetic data
ML teams are increasingly turning to synthetic data to meet their needs. Teams can build a model to generate this data or turn to a vendor for synthetic data. This approach has the added benefit of protecting IP or sensitive data during the model training process as the dataset won’t contain any real information, which can be important if a team’s labeling pipeline includes external teams or less secure infrastructure. Synthetic data can also simulate edge cases and conditions that aren’t represented in real data, helping teams fill in the gaps in their dataset.

However, producing high quality synthetic data can be a challenge. Machine-generated data can be unreliable, misrepresent the real world, and perpetuate biases that your team needs to reduce or eliminate with your model. While synthetic value can certainly provide value for specialized use cases, it’s important to weigh the advantages against potential pitfalls and come up with additional solutions to mitigate them.

Expanding data types and sources
Tackling a lack of data can require some creative problem solving for data scientists and ML engineers. Depending on the use case at hand, your team may be able to fill some of the gaps in data by casting a wider net. A computer vision use case, for example, may be augmented with tabular or text data. You may also be able to use weak supervision methods to sort through a dataset and add metadata or other structure to unstructured data to find patterns within a dataset that contains multiple types of data. Watch our on-demand webinar for a technical demonstration on how you can use multiple data types.

Whether you are enhancing your labeling pipeline, trying to gain a more nuanced understanding of your model as it trains, growing your dataset, or committing to a combination of these improvements, Labelbox can help. Watch our short demo to learn how our training data platform can transform your labeling operations.

10 training data practices used by the most successful AI teams 

10-training-data-practices

Machine learning teams today are confronted by a variety of challenges around the development of training, such as building an efficient and optimized labeling pipeline, improving the quality of labeled data, incorporating advanced methodologies such as automation, and more.
At Labelbox, we’ve partnered with hundreds of leading AI teams across industries to develop solutions to these challenges. In this ebook, you’ll discover:
-Common roadblocks faced by AI teams
-The collaborative workflows used by highly successful ML teams
-Processes, standards, and tools that enable the creation of high-quality training data and AI breakthroughs
-How teams can prepare for and address unexpected issues such as edge cases

How to diagnose and improve model performance

how-to-diagnose-and-improve-model-performance-webcast

Welcome to Labelbox's new webinar series, ML Unboxed. Geared towards ML practitioners, you'll learn essential ML best practices and concepts, coupled with hands-on workshops on using the Labelbox platform. 

In this webinar, product manager Gareth Jones will take you through a hands-on tutorial to teach you how to diagnose model errors and prioritize data that will dramatically improve performance. In a recent blog post, we discussed best practices to help teams spend less time labeling data that doesn’t lead to meaningful improvements in performance. 

Gareth walks through specific workflows to help increase your iteration velocity and improve performance faster, including how to: 

Use an open-source model or the model of your choice for transfer learning (we’ll use the Detectron2 Panoptic Segmentation model)
Diagnose initial model performance with Model Diagnostics to understand where to focus your first batch of labeling
Pre-label data with Similarity Search and labeling functions
Prioritize high-value data for labeling based on model performance 
Retrain your model with a new batch of ground truth labels and compare its performance with the previous version 
With this end-to-end workflow you’ll be able to iterate faster and measurably improve your model’s performance while labeling less data.

How leading teams get full visibility and control over their training data

how-leading-teams-get-full-visibility

How leading AI teams get full visibility and control over their training data
AI is supposed to transform businesses, yet it's taking enterprises much longer than expected to achieve this impact. Creating high-quality training data with consistency and transparency is at the heart of the problem. 

Brian Rieger and Kelly Anderson discuss how model assisted labeling and model diagnostics and can lead to more accurate models, faster implementation and 70% cost savings. 

Watch this video to learn what workflows and strategies leading AI teams use to prioritize the right data at the right stage of their project’s life cycle in order to iterate quickly.

CAPE Analytics tackles catastrophe and property analyses for insurance with AI

CAPE-Analytics-tackles-catastrophe-and-property-analyses

You’re probably doing data discovery wrong

youre-probably-doing-data-discovery-wrong

Building an efficient and intuitive data pipeline for developing AI systems has never been a simple task but the ability to quickly and easily search for data has continued to be a significant obstacle for ML operations. Labeled data is the foundation of your AI system, but is one of the hardest assets to store, evaluate, and visualize efficiently. Even best-in-class ML teams historically haven’t had a perfect solution for interacting with their unstructured and structured data.

How to improve model performance with active learning 

how-to-improve-model-performance-with-active-learning

how-much-training-data-webcast

After working with hundreds of companies deploying AI models in production, we've learned that ML teams often overestimate the amount of labelled data they think need to get a working model. On the other hand, teams also underestimate the amount of labeled data in order to get a first-class model. 
To share how teams get to good and then great, Labelbox lead machine learning engineer Randall Lin discusses how to think through the amount of training data needed for your projects, how to scale these efforts, and will take live questions from the audience.

How to improve model performance with less data

how-to-improve-model-performance-with-less-data

Machine learning has evolved past the traditional method of iterating on the model with a single large training dataset. Today, advanced ML teams extend their focus to carefully selecting their training data, training the model, examining its performance, and modifying the next training dataset accordingly. This data-centric ML method can, according to a recent study from Stanford University researchers, result in anywhere from a 10% to 50% reduction in the amount of training data, depending on the ML task at hand. This can translate into significant time and cost savings for ML teams.

farmers-optimize-crop-production

how-xarvio-uses-labelbox-to-help-farmers-optimize-crop-production

Food production will have to nearly double as the global population reaches 10 billion by mid-century, even as land under cultivation shrinks. How can the world feed all those people sustainably One answer to this is improved efficiencies through digital farming.

designing optimal iteration loop with Manu Sharma

designing-optimal-iteration-loop-manu-sharma

Learn how the latest generation of Labelbox products will redefine how you structure training data pipelines and help you label less data to achieve meaningful performance gains.

NASA-case-study-solar-system


NASA JPL case study: Employing ML to find signs of life in our solar system
Mark Wronkiewicz and Jake Lee, both Data Scientists at NASA JPL, will share the story of how they developed an algorithm that helps scientists look for signs of extraterrestrial life. Learn why their ML team decided to invest in a training data platform after using disparate in-house tools for different use cases as well as best practices for managing non-technical labelers.

blue river technologies model assisted labeling

blue-river-technologies-model-assisted-labeling

Learn how Blue River Technologies leveraged model-assisted labeling to achieve significant time and cost savings.

ImageBiopsy lab uses model assisted labeling

ImageBiopsy-lab-uses-model-assisted-labeling

Since 2016, ImageBiopsy Lab has been building AI applications to help physicians and the medical community at large better understand and diagnose musculoskeletal diseases (MSK), which affect 1.7 billion people worldwide. Today, the team focuses on optimizing the radiological workflow via automation and standardization to enable the early detection and prevention of MSK. This automated decision support allows orthopedists and radiologists to assess and predict bone health by turning images into actionable information.

The team has optimized their ML development process by using AI to pre-sort massive datasets and employing a model-assisted labeling workflow to pre-annotate filtered datasets. As a result, the team has reduced their time spent per label by 160%.


how-to-optimize-your-entire-ml-pipeline

Labelbox’s annotation and automation capabilities have resulted in significant time and cost savings for many of our customers as they generate training data. However, creating quality training data is just one (important) part of the machine learning workflow. ML teams often rely on a variety of providers and platforms for data storage, data management, model development and deployment — all essential components of their process. That’s why Labelbox has partnered with Snowflake, Databricks, DataRobot, and Google Cloud Platform (GCP) to ensure seamless connections as teams store and manage data as well as train and deploy their models.

stop-labeling-data-blindly

One of the hardest questions for ML teams to answer is often “what data will improve model performance fastest?” In data-centric AI systems, improving the quality of your training data is the most important, and most challenging, objective leading to performant models. While the cost of labeling data is largely commoditized, teams can gain dramatic competitive advantages through how they identify model errors, track performance across iterations, and prioritize the type of data needed to improve performance faster so they can ultimately save time and cost through labeling less data.To help teams iterate faster and more efficiently, we’ve outlined best practices for improving data-centric AI systems. Through adopting these best practices, teams can iterate on their models faster and focus more resources on building IP instead of labeling low-impact data.

complete-guide-to-image-annotation

Today’s enterprise and startup AI teams rely on accurate training data sets to train, refine and maintain a wide spectrum of machine learning models. For teams working with image-based machine learning models, training datasets are often labeled using image segmentation – but not all image segmentation meets the quality or standards that machine learning teams require. Download the Complete guide to image annotation from Labelbox to better understand what image annotation is, the different types of image annotation, and how a training data platform supports complex image annotation.

labeling-operations-webinar

Whether you’re just starting a new ML project or are looking to take your labeling operations to the next level of maturity, it’s crucial to understand what metrics are most important for your objectives. Apply best practices from manufacturing and software development to reduce cost, increase quality, and get to production faster. Labelbox co-founder Brian Rieger and Sr. Product Manager David Liang discuss
How throughput, efficiency, and quality can guide your decision making
What common misconceptions in labeling operations often lead to delays or low performance
How to apply best practices learned from working alongside Enterprise AI organizations and leading AI startups

genentech-breakthrough-labeling-process

Genentech is a biotechnology enterprise that has developed breakthroughs in medical research and medicine since 1976. Today, Genentech researchers are building convolutional neural networks to help diagnose illnesses and aid medical professionals.

Genentech has revolutionized their ML process by having domain experts train teams of labelers on medical imagery annotation tasks. With regular labelers creating annotations, which are then sampled and reviewed for quality by experts, Genentech has developed a much faster and less expensive way to create training data and get their life-saving algorithms to production.

The guide to saving time when creating AI data

guide-to-saving-time-when-creating-AI-data

Wading through vast amounts of unstructured data to accurately annotate assets requires a tremendous amount of patience, organization, and time. Drawing from the experiences of hundreds of AI teams across industries, we are sharing six time-saving practices for ML teams to implement when handling AI data. In this guide, you’ll discover how to:
Annotate faster with a dynamic queueing system 
Improve communication, collaboration, and consensus between teams
Utilize a programmatic-approach for quicker access to data 
Leverage software optimized for speed
Incorporate automation through model-assisted labeling
Utilize active learning and prioritize the right data

These strategies will unblock key barriers and speed up overall processes and capabilities for a quicker path to production AI.


Your competitive advantage should be data with Cathie Wood and Manu Sharma 

your-competitive-advantage-should-be-data

With over 40 years of experience identifying and investing in innovation, ARK Invest CEO, Cathie Wood discusses what she believes it takes to win in AI with Labelbox CEO Manu Shama.

how-the-best-achieve-AI-breakthroughs

Labelbox has seen how more than 200 organizations label, manage, and improve their training data. Come learn what sets apart those who move the fastest, realize the largest efficiencies, and find the greatest success in their ML programs.

4 strategies for successful labeling operations

4-strategies-for-successful-labeling-operations

how-to-build-trust-in-your-ml-models

It’s no secret that AI and ML are integral to our future, with teams building algorithms for everything from robotic surgery to sorting fresh produce. But many of us are also skeptical about predictions made by deep learning models. Consumers often find recommendation algorithms as mysterious and daunting as they are helpful. On the enterprise side, leadership and stakeholders place less trust on model outputs when they don’t know how the model produced them. This presents a tough challenge for ML teams who want to build transformative models to support major operations throughout the organization. Let’s explore a few ways that teams can ensure that their models are trusted within their enterprise.

Non-expert labeling teams and training data for medical use cases 

create-high-quality-training-data

new-study-finds-that-non-expert-labeling-teams-can-create-high-quality-training-data

Producing high quality training data for any ML use case can be costly, and it’s doubly so for specialized use cases that require experienced domain experts to review (and often label) training data. In the agriculture industry, for example, ML teams training models to recognize weeds or diseased crops need to have their training data examined and corrected by trained agronomists and horticulturists, whose time is not only expensive, but also limited.

build-scale-labeling-operations

We’re seeing that more ML teams are shifting their focus from iterating on models to improving the quality of their training data, labeling operations is becoming a central aspect of their larger MLOps strategy. Creating, managing, and maintaining an efficient labeling pipeline that produces high-quality training data is no easy task. Teams without the right people, processes, and tools in place will likely struggle to generate the training data they need at a cost-effective rate .

6 time saving keys to creating AI data

6-key-best-practices-for-AI-teams-to-save-time-when-creating-AI-data

6 time saving keys to creating AI data
Annotate faster with a dynamic queuing system
Improve collaboration and consensus between teams
Utilize a programmatic approach for quicker access to data
Leverage software optimized for speed
Incorporate automation through model-assisted labeling
Utilize active learning and prioritize the right data
One thing all ML teams share is the need to make their data useful for their models. Wading through a vast amount of unstructured data to accurately annotate assets requires a tremendous amount of patience, organization, and time. According to Algorithmia's State of ML Survey, early 96% of companies encounter delays getting to production and 78% of machine learning projects stall before deployment. Moving faster is the top pain point we hear from customers; it controls their ability to meet project goals and gain advantage over competitors. Speed is essential to not only get models to production but do so on timelines that meet these increasingly aggressive expectations.

Annotate faster with a dynamic queuing system  
AI teams can benefit from speedier annotations by employing a dynamic queuing system that routes labeling tasks to the right team members so they are never sitting idle. With dynamic queuing, teams are able to create labeling workflows where low confidence predictions are visible and prioritized to their data scientists and labelers. This allows you to more quickly analyze and correct labels.

In the illustration above, Labelbox’s dynamic queuing process is compared to a standard queue model. Without this automation in place, labelers will sit idle while waiting on their next task.
In the illustration above, Labelbox’s dynamic queuing process is compared to a standard queue model. Without this automation in place, labelers will sit idle while waiting on their next task.
Dynamic queuing allows you to save time by automatically routing labeling tasks to the right team members.

Only active labelers will be assigned assets to annotate to eliminate duplication and waiting
Dynamic queuing reserves multiple assets at a time to make sure your labeling team isn’t waiting around in between assets
The team shares a backlog that’s automatically distributed among labelers and reviewers

Improve collaboration and consensus between teams
To enable better collaboration during the labeling process, teams should look to implement processes and dedicated tooling to make repetitive tasks effortless, such as adding and clarifying feedback on a labeled asset. In order to elicit and clarify feedback, the ability to quickly raise issues and add comments on a labeled asset provides a simple and reliable channel for escalating questions to reviewers or subject matter experts.

A labeler can create an issue to ask a question, submit the label completed to the best of their ability, and receive feedback and clarification during the label review process. This streamlined way of collaborating on labels provides teams with an expedited way to handle inevitable issues that arise during labeling.

With an issues and comments tool, you can leave a comment directly on an issue for your workforce or your internal labeling team by tagging them. They can reply back or mark it as resolved.
With an issues and comments tool, you can leave a comment directly on an issue for your workforce or your internal labeling team by tagging them. They can reply back or mark it as resolved.
To improve consensus amongst team members, you can quickly drive toward an agreement as a team with a consensus tool that helps you arrive at a shared understanding of what good looks like. This tool allows you to automatically compare annotations on a given asset to all other annotations on that asset. Consistency is measured by the average of the label agreement between labelers. For example, when the consensus is set to 3, every label is grouped with two similar labels and the average of the three becomes the consensus score.

The consensus tool works in real-time so you can take immediate and corrective actions towards improving your training data and model performance.
The consensus tool works in real-time so you can take immediate and corrective actions towards improving your training data and model performance.
Utilize a programmatic approach for quicker access to data
Teams can now speed up the data import and labeled data export process through the use of a programmatic approach via SDKs and/or APIs rather than slowly handling data transfer methods manually. By connecting your team’s data and automating bulk operations, labeled data (including all the metadata, how the dataset was created, as well as labeler feedback) becomes much easier and quicker to manage and track as the complexity of your projects grows over time. This approach offers endpoint flexibility, so you can plug and play into your existing workflows without losing any valuable aspects of your training data.

One example of how this can be accomplished is through the use of a Python SDK, which offers more control over your data while simplifying and speeding up data import. For example, it simplifies the data import process so that you can use bulk DataRow creation. This process is asynchronous, so you don't have to wait for bulk creation to finish before continuing with other tasks (though you can wait if you want to). In some cases, you can also create projects and datasets programmatically, export labels, and add metadata to your assets all in an object-oriented way with a Python SDK — complete with all relationships between objects, speeding up your labeling workflow. By utilizing this approach, you can even export newly created training data automatically for an active learning workflow and adjust your labeling queue to focus on improving confidence in specific classes.


Leverage software optimized for speed
Many ML teams spend a considerable amount of time attempting to build out custom infrastructure for the training data that doesn’t scale and leads to disjointed development efforts. According to research firm Cognilytica, ML teams spend up to 80% of their time building and maintaining training data infrastructure.

Source: Cognilytica
Source: Cognilytica
Building such custom infrastructure can be both difficult and expensive, potentially costing millions of dollars in engineering resources to build and maintain. As a time-saving alternative, many teams have found the answer in adopting a training data platform (TDP). A TDP is purpose-built software engineered for speed and ease of use in handling AI data. It allows teams to move faster with best practices already built in place, rather than home-grown tools and recreating the wheel. By incorporating a TDP, teams save exponential time on data preparation and labeling, so ML teams can spend more time on their core
competency: building production models.

But not all TDPs are created equal. You can learn more about what TDPs offer ML teams in our TDP 101 guide, and how to choose the best one for your team by reading our buyer’s guide.

Incorporate automation through model-assisted labeling
Accurately annotating the tens of thousands if not millions of assets necessary to achieve a comprehensive data set has blocked many promising AI initiatives from ever reaching production-ready models. That’s why automation through model-assisted labeling is one of the simplest ways to reduce the waste of precious capital. Model-assisted labeling uses your own model to make labeling easier, more accurate, and faster. In some cases, we’ve seen ML teams save 50-70% on their entire labeling budget by utilizing this method.

Source: Aggregate labeling budget from publicly-traded Labelbox customer, Q4 2019 - Q2 2021.
Source: Aggregate labeling budget from publicly-traded Labelbox customer, Q4 2019 - Q2 2021.
Utilize active learning and prioritize the right data
Your team can achieve considerable gains in productivity and efficiency by employing active learning and prioritizing the right data. We’ve seen customers reduce their iteration cycles by up to 8X and use less training data by curating their datasets. Data selection for active learning typically involves selecting rare data or difficult cases. In the case of rare data, models need a minimum number of examples in order to perform well on a chosen task. Without a sufficient number of examples to feed the model, it can be too inefficient to sample data randomly. Using tools like a similarity function with model embeddings can help you to discover more matching data examples. Teams can visually find patterns and identify edge cases in data through the use of model embeddings. By clustering visually similar data, teams can better understand trends in model performance as well as data distribution. While teams can calculate and plot clusters manually, some use cases in ML require more time-sensitive trend detection and a quicker approach. This is where a visual embeddings tool can be helpful to boost model performance and support an active learning workflow.

Model embeddings, shown here in Labelbox, can help visualize trends in data and cluster similar data, helping you uncover outliers faster for an active learning workflow.
Model embeddings, shown here in Labelbox, can help visualize trends in data and cluster similar data, helping you uncover outliers faster for an active learning workflow.
Additional resources
For a deeper dive into the best practices outlined in this post and see demos of how they work in Labelbox, watch our on-demand webinar, the six best practices to save time when creating AI data. You can also download our guide which goes deeper into the six time-saving practices above.

vb-transform-2021

AI is supposed to transform your business but enterprises are finding that it is taking much longer than anticipated to get to full impact. Data labeling and data management are at the heart of the problem. The work of creating and managing training data does not stop after an initial model is trained and deployed. To accelerate development, learn how leading AI teams adopt not just a data-centric approach, but the best teams are prioritizing the right data at the right stage of their project’s life cycle in order to iterate on these models quickly. This approach has been reinforced by new tools such as model assisted labeling and model diagnostic which Manu will dive into. He will also discuss how AI teams are using a training data platform to enhance their overall labeling operations.

complete-guide-to-image-segmentation

For commercial AI teams, an enormous amount of time and energy is spent on the development and curation of custom datasets. This is because, unlike academic machine learning models, the use of open-source datasets for a commercial application will not likely yield an accurate representation of the real-world. Instead, teams rely on the constsant creation of new data sets for their applications. Download our Complete guide to image segmentation to learn more about real-world use cases and the advantages of managing your annotation processes using a training data platform.

ontology-management-with-labelbox

Labelbox customers across industries count their ontologies as a critical part of the IP for their AI application. A detailed ontology is integral to a successful labeling process. An ontology ensures that all the labels in a project follow the same set of rules, reducing the possibility of confusion for the AI/ML model. Being able to quickly create, edit, and reuse ontologies is key to faster iteration cycles in machine learning.

move ai is revolutionizing video content creation

move-ai-is-revolutionizing-video-content-creation

Creating high quality video content, whether it’s a movie, a video game, or a live broadcast, is typically cost-prohibitive. The best content is made by the studios with the best equipment and animation talent. Now however, an emerging AI company, Move.ai, is set to change the status quo, making processes like motion capture and key point recognition easier, faster, and less expensive — for individuals and studios alike.

labeling workforce hero blog library

Blue River Technology’s ML teams can now spend more of their time focusing on training, monitoring, and maintaining their computer vision models. Their data scientists can pull updated, refined, relevant datasets for every use case and model within minutes via Labelbox Catalog.

using labelboxs model assisted labeling to speed up annotation

using-labelboxs-model-assisted-labeling-to-speed-up-annotation

Labelbox is committed to partnering with researchers at universities globally to inspire cutting-edge research and to find new ways to help humans advance AI. One of the ways that we've accomplished this is by offering our software license for free for non-commercial research. Our goal is to empower the academic community to perform their work more efficiently and to collaborate with other experts from around the world using Labelbox's training data platform.

Build vs. Buy: The benefits of a home-grown vs. external labeling solution

build-vs-buy-benefits

This panel dives into a question many ML teams consider: should they build their own labeling solutions or invest in a training data platform? Panelists will discuss tactics they used for creating and improving training data before a TDP, the decision to implement a TDP, and what has changed since implementation.

Lessons learned from my first year with a training data platform

lessons-learned-first-year-training-data-platform

Learn how organizations are successfully moving from a proof of concept to large scale deployment of their ML programs utilizing a training data platform.

How data unlocks AI with Peter Levine of Andreessen Horowitz and Manu Sharma  

how-data-unlocks-ai

How data unlocks AI - Fireside chat, Peter Levine (Andreessen Horowitz) and Manu Sharma (Labelbox)

accelerating-ai

Watch our on-demand webcast on how Labelbox accelerates high-performing AI into production as a training data platform How Labelbox accelerates high-performing AI into production as a training data platform



automating computer vision annotation let your model do the work

automating-computer-vision-annotation-let-your-model-do-the-work

Almost anyone working on computer vision applications these days struggles with the same problem: how to speed up data labeling, the most expensive and time-consuming part of developing a supervised machine learning system. It doesn’t stop when a machine learning system goes into production. In fact, it can grow. The answer, of course, is automation - letting the computer do most of the work so that humans only need to complete it.

customizing an optimal training data workflow using an api first approach

customizing-an-optimal-training-data-workflow

customizing-an-optimal-training-data-workflow-using-an-api-first-approach

Cape Analytics is a leading provider of AI-powered geospatial property data. The company uses deep learning and geospatial imagery to provide instant property intelligence for buildings across the United States. Their pioneering work for assessing risk and property value is tied to common property factors (pools, roof construction, solar panels) but also emerging risks like vegetation coverage, detailed roof condition, and wildfire zones. Creating performant machine learning models to extract intelligence from properties requires enormous amounts of high quality, labeled data. To accomplish this feat, Cape’s engineering and data science team searched extensively for ways to build a complete and automated training data solution

How to build world-class AI applications with Labelbox 

product-webcast-apr-2020

AI-focused product leaders are spending a disproportionate amount of their time processing, labeling and augmenting training data to feed their AI applications. In this webinar, we’ll be delivering a demo of the Labelbox product while sharing insights on how a training data platform can free up time so you can focus on improving your AI application, while getting smart about managing labeling costs and ensuring data quality.

model-assisted-labeling-webcast

Working with hundreds of AI companies currently deploying models in production, we've discovered that many teams aren't initially considering or harnessing model-assisted labeling strategies to help solve one of their core bottlenecks: creating and iterating on high-quality training data to feed into their ML models.To provide some inspiration and best practices, Labelbox’s lead machine learning engineer, Randall Lin will share a few proven ways on how to use model-assisted labeling for your projects that may be helpful, as well as take live questions from the audience.

Library

The guide to unlocking AI with your data

Discover a smarter, easier approach to building AI products faster.

Labelbox essentials for getting started

What great looks like

Explore the latest resources

Try Labelbox today