[marketo] leaderboard blog hero (1)

Beyond benchmarks 

We are going beyond traditional benchmarks to measure the likability and performance of leading AI models. Labelbox leaderboards measure model capabilities by using its data factory: platform, scientific process and expert humans.

labeling workforce hero blog library

Multimodal support for GenAI 

Labelbox's new multimodal chat solution empowers GenAI developers by providing advanced tools for creating, evaluating, and optimizing model responses across various data modalities. It streamlines the data labeling process and offers access to subject matter experts, enabling teams to improve AI models with high-quality, human-evaluated data.

active learning header image

Evaluating text-to-image models

This blog post presents a comprehensive evaluation of three text-to-image models (DALL-E 3, Stable Diffusion, and Imagen 2) using both human preference ratings and automated evaluation techniques. The results reveal that while Stable Diffusion generally ranked highest in human evaluations, DALL-E excelled in prompt adherence, and Imagen 2 produced the highest technical image quality, highlighting the complex relationship between quantitative performance and human perception of quality.

Model foundry header

Model foundry_automating data tasks with foundation models_library

Model Foundry brings the power of foundation models into Labelbox to perform complex tasks such as data labeling and data enrichment. Early results indicate that over 88% of labeling tasks can be meaningfully accelerated by one or more foundation models. You will be able to access the world's best AI models from all major providers and open source alternatives. These models will act as a copilot for subject matter experts to accelerate model development.

How to automatically ingest data from Databricks into Labelbox_Guide_Graphic

How to automatically ingest data from Databricks into Labelbox

walmart-library-hero

walmart-customer-story-library

Walmart's team wanted to find faster ways to annotate conversational text from shopping chatbots and label inventory images for their object detection and classification models, which included tens of millions of diverse product SKUs.

DBx integration

Seamlessly integrate Databricks data pipelines with Labelbox

Use the Labelbox Connector for Databricks to quickly and easily connect their Databricks pipeline to upload their unstructured data to Labelbox.

Accelerate retail ROI webinar social card

How to accelerate time-to-value when building AI for retail and e-commerce

How to use Labelbox and Google Cloud to deliver successful AI systems for improving merchandising, enhancing customer care, optimizing marketing strategies in significantly less time.

labeling-observability-hero

labeling-ops-observability

How labeling quality can be measured and dramatically improved with the right observability practices for labeling operations.

Dialpad Hero Image

dialpad-customer-story-library

Learn how Dialpad turned to the Labelbox platform and Boost services for higher quality training data and reduced time requirements from their own data scientists for labeling project design and review. In addition, see how Labelbox enabled the team to collaborate more smoothly with labelers and produce higher-quality labeled data for their NLP and LLM use cases.

BRT-data-engine-library

Learn how Blue River Technology’s ML teams can now spend more of their time focusing on training, monitoring, and maintaining their computer vision models. Their data scientists can pull updated, refined, relevant datasets for every use case and model within minutes via Labelbox Catalog.

Harnessing multiple data sources to build innovative AI products

harnessing-multiple-data-sources-to-build-innovative-ai-products-webcast

AI teams today use a variety of different data types for their projects, including text, computer vision, and specialized formats such as OCR, geospatial imagery, and more. Compiling enough high-quality data to train an algorithm can be a challenge with any of these data types, as they can be expensive and/or hard to find.

In this webinar, you’ll learn how leading AI teams tackle these bottlenecks by combining multiple sources of data to gain valuable insights, leverage both external and internal data, and ultimately develop more robust AI products. We’ll also cover:

When to employ different sources to improve predictions in a given ML iteration cycle

Examples of applied use cases and how companies are combining both subjective and objective information from sources such as OCR, images, and text

What cutting-edge AI teams are accomplishing and what’s possible via a training data platform

A walkthrough of how to use products like Catalog to understand and search all of your unstructured data (e.g., consumer reviews for consumer products)

How to combine workflows and data types when your data is unstructured by using ML frameworks like Databricks

M&E whitepaper visual

get-more-value-unstructured-data

About 80% of enterprise data is now unstructured. But most media and entertainment companies struggle to realize the full promise and value of their video, audio and image files. Don’t let your business leave money on the table. Find out how to develop and launch your unstructured data strategy in this whitepaper.

designing optimal iteration loop with Manu Sharma

designing-optimal-iteration-loop-manu-sharma

Learn how the latest generation of Labelbox products will redefine how you structure training data pipelines and help you label less data to achieve meaningful performance gains.

build-or-buy

Until recently, most ML teams have had to build their own labeling tools or adopt a hodgepodge of open-source tools or service-based labeling workflows. Now however, a new category of tooling has emerged: the data engine for AI. A data engine offers best-in-class technology — and are purpose built for AI teams — to combine data, people, and processes into one seamless training data creation experience, enabling ML teams to produce performant models faster and more efficiently.  Technical leaders must now weigh the benefits of a data engine with the potential of building (or continuing to build) their own training data management software, and they have many factors to consider before making a decision. 

NASA-case-study-solar-system


NASA JPL case study: Employing ML to find signs of life in our solar system
Mark Wronkiewicz and Jake Lee, both Data Scientists at NASA JPL, will share the story of how they developed an algorithm that helps scientists look for signs of extraterrestrial life. Learn why their ML team decided to invest in a training data platform after using disparate in-house tools for different use cases as well as best practices for managing non-technical labelers.

How Deque uses data prioritization and model diagnostics to unlock AI breakthroughs in digital accessibility

how-deque-uses-data-prioritization

how-deque-uses-data-prioritization-and-model-diagnostics-to-unlock-ai-breakthroughs-in-digital-accessibility

How Deque uses data prioritization and model diagnostics to unlock AI breakthroughs in digital accessibility


Introduction

Deque has a storied journey on their mission of delivering digital equality to all. From the early days of the internet in the nineties, they made it their priority to pioneer and democratize digital accessibility. Digital accessibility is the practice of making digital documents, web and mobile apps accessible to everyone, including people with disabilities. Now, the power of machine learning is enabling the Deque team to lead the next generation of accessibility testing. Building out the components of their ML program has been challenging, but they have developed a sophisticated data engine that’s capable of prioritizing the most performant classes of data, discovering model errors quickly, and fueling their iterations with high-quality data. 

How Deque uses AI


In 2015, Deque made a bold decision to help advance automated accessibility testing. They decided to open source their most valuable asset, their accessibility testing rules library called axe-core. This rules engine powers Deque’s axe tools, the accessibility checker in Google’s Lighthouse and thousands of other notable projects around the globe amassing over half a billion downloads. Through automated testing alone, Deque’s axe can detect 57% of accessibility issues. While this might not sound like a lot if you’re new to this space, it’s more than double what the rest of the industry can offer and makes practicing digital accessibility a manageable task. To extend testing coverage even further, Deque turned to computer vision and machine learning to automatically catch even more accessibility issues, which were previously only detectable manually, using highly educated experts armed with various assistive technology and lots of time.

The Deque team is currently using their axe DevTools application to generate a pipeline of data that feeds into the machine learning system powered by Labelbox. The models that are generated are then being leveraged to expand coverage with an eventual target of being able to find 100% of accessibility issues in an automated fashion, thereby making true digital equality for all easy to achieve.


Challenge

It takes an average of about 10 minutes to fully annotate a single webpage screen. Between the thousands of web datasets and mobile datasets, the Deque team amassed a trove of useful data and looked to Labelbox and the Labelbox Workforce team for guidance and manpower. Prior to implementing Labelbox, the Deque team mostly relied on a combination of disparate open-source annotation tools, and hacking together Jupyter Notebooks with Google Sheets. 


Solution

“Before using Model Diagnostics in Labelbox to target the model’s weaknesses, we had to visualize the predictions on our own and it was a much more manual process,” said Noé Barrell, ML Engineer at Deque. “We had to calculate all the metrics on our own and it was a disjointed and difficult process. Being able to convert that process into something we could now do simply within the Labelbox platform made diagnosing errors a much more streamlined experience. It’s become so much easier to iterate.”



The combination of Model Diagnostics and Catalog has allowed the Deque team to reduce their 
data requirements by more than 50%.


Noé and the ML team at Deque were able to make considerable improvements to model performance by seeing the ability to evaluate and visualize model performance. “We detected some noise issues in our dataset and thanks to Model Diagnostics, we were able to filter out about one-third of data points we considered less trustworthy,” said Barrell. “By doing so, performance of the model went up 5%. We re-labeled some data and we saw the performance went up again after we added the re-labeled points. It was challenging data for humans to label and for the model to understand. Being able to target the data we already had in Labelbox and make changes and fixes to it was really helpful to us as a team to save time and target where we knew it would make a difference in our model’s performance.”

The Deque team made huge leaps in several areas via Model Diagnostics and were able to target their data collection in a way that addresses model failures more quickly. For example, they boosted performance on classes of data such as improving detection of checkboxes from 47% accuracy to 75% accuracy, presentational tables from 66% accuracy to 79% accuracy and radio buttons from 37.9% accuracy to 74% accuracy. 

In another time saving measure, the Deque team found they could search, discover, and prioritize the right data with Catalog. “The Catalog feature in Labelbox is also huge for us. Pre-Catalog, for our data selection process, we’d look at the performance metrics of our model and let’s say, for example, we discovered it was indicating 50% accuracy on modals, we would have to tediously and manually collect data surrounding that. But with the Catalog feature in Labelbox, we can target data collection on modals easily and quickly. Embeddings allow us to do unsupervised classification of modals and select a lot of modals. It’s just easier to create batches and sample around that. It takes a lot of the time and effort out of the data selection process,” said Barrell.




The Catalog feature in Labelbox allows the Deque team to prioritize the right data to boost model performance and make their iterations more meaningful and efficient. 


“Being able to reduce the data requirement is huge because you can see the same amount of improvement in the model’s performance in half the time and with half the effort. That was enabled through targeting the model’s weaknesses with Model Diagnostics and then being able to prioritize the right data through Catalog. Before, if we were doing a scattershot data collection, we would have been roughly labeling twice as much data and with twice as much effort,” said Barrell.


Source: Deque’s Guide to Digital Accessibility


Building accessible sites or apps can be tricky without the right guidance. The Web Content Accessibility Guidelines, or WCAG, published by the W3C, is the standard that defines what makes an application accessible. Developers, testers, app owners and accessibility experts from around the world rely on these standards for proper accessibility testing direction. Not only is accessibility a requirement for compliance with many laws, but accessibility, by ensuring all information is available in text form, also increases your website’s SEO and can improve the user experience for all users. For example, video captions don’t just assist those with hearing impairments but can be used by viewers in noisy environments or in environments that are not conducive to listening to audio. Accessibility features are for everyone and create a more inclusive world. 



Source: Deque What Is Digital Accessibility? 


To learn more about the Deque team and their mission, visit their website.

To understand how Labelbox can help you in your AI journey and help your team develop its own data engine, request a free demo now. 


vb-transform-2021

AI is supposed to transform your business but enterprises are finding that it is taking much longer than anticipated to get to full impact. Data labeling and data management are at the heart of the problem. The work of creating and managing training data does not stop after an initial model is trained and deployed. To accelerate development, learn how leading AI teams adopt not just a data-centric approach, but the best teams are prioritizing the right data at the right stage of their project’s life cycle in order to iterate on these models quickly. This approach has been reinforced by new tools such as model assisted labeling and model diagnostic which Manu will dive into. He will also discuss how AI teams are using a training data platform to enhance their overall labeling operations.

Build vs. Buy: The benefits of a home-grown vs. external labeling solution

build-vs-buy-benefits

This panel dives into a question many ML teams consider: should they build their own labeling solutions or invest in a training data platform? Panelists will discuss tactics they used for creating and improving training data before a TDP, the decision to implement a TDP, and what has changed since implementation.

Lessons learned from my first year with a training data platform

lessons-learned-first-year-training-data-platform

Learn how organizations are successfully moving from a proof of concept to large scale deployment of their ML programs utilizing a training data platform.

How data unlocks AI with Peter Levine of Andreessen Horowitz and Manu Sharma  

how-data-unlocks-ai

How data unlocks AI - Fireside chat, Peter Levine (Andreessen Horowitz) and Manu Sharma (Labelbox)

how-to-optimize-your-entire-ml-pipeline

Labelbox’s annotation and automation capabilities have resulted in significant time and cost savings for many of our customers as they generate training data. However, creating quality training data is just one (important) part of the machine learning workflow. ML teams often rely on a variety of providers and platforms for data storage, data management, model development and deployment — all essential components of their process. That’s why Labelbox has partnered with Snowflake, Databricks, DataRobot, and Google Cloud Platform (GCP) to ensure seamless connections as teams store and manage data as well as train and deploy their models.

guide-to-training-data-platforms

guide-to-buying-a-training-data-platform

A Training Data Platform (TDP) is an investment in your models' maturity, accuracy, and endurance — so choosing one from the available options can be a difficult decision, especially with taking your use cases, stakeholders, existing processes, and future goals into account. That's why we've created a comprehensive checklist of features and capabilities that a TDP should have to best serve your specific requirements as your ML needs scale. The list includes:

6 best practices to save time when creating AI data

6-best-practices-to-save-time-when-creating-ai-data

saving-time-webcast

Wading through all your unstructured data to accurately annotate assets requires a tremendous amount of patience, organization, and time. This webinar will cover six key time-saving practices implemented by leading AI teams when handling AI data. Join us to discover how to:

Annotate faster with a dynamic queueing system

Improve communication, collaboration, and consensus between teams

Utilize a programmatic-approach for quicker access to data

Leverage software optimized for speed

Incorporate automation through model-assisted labeling

Utilize active learning and prioritize the right data

6 best practices to save time when creating AI data

6 time saving keys to creating AI data

6-key-best-practices-for-AI-teams-to-save-time-when-creating-AI-data

6 time saving keys to creating AI data
Annotate faster with a dynamic queuing system
Improve collaboration and consensus between teams
Utilize a programmatic approach for quicker access to data
Leverage software optimized for speed
Incorporate automation through model-assisted labeling
Utilize active learning and prioritize the right data
One thing all ML teams share is the need to make their data useful for their models. Wading through a vast amount of unstructured data to accurately annotate assets requires a tremendous amount of patience, organization, and time. According to Algorithmia's State of ML Survey, early 96% of companies encounter delays getting to production and 78% of machine learning projects stall before deployment. Moving faster is the top pain point we hear from customers; it controls their ability to meet project goals and gain advantage over competitors. Speed is essential to not only get models to production but do so on timelines that meet these increasingly aggressive expectations.

Annotate faster with a dynamic queuing system  
AI teams can benefit from speedier annotations by employing a dynamic queuing system that routes labeling tasks to the right team members so they are never sitting idle. With dynamic queuing, teams are able to create labeling workflows where low confidence predictions are visible and prioritized to their data scientists and labelers. This allows you to more quickly analyze and correct labels.

In the illustration above, Labelbox’s dynamic queuing process is compared to a standard queue model. Without this automation in place, labelers will sit idle while waiting on their next task.
In the illustration above, Labelbox’s dynamic queuing process is compared to a standard queue model. Without this automation in place, labelers will sit idle while waiting on their next task.
Dynamic queuing allows you to save time by automatically routing labeling tasks to the right team members.

Only active labelers will be assigned assets to annotate to eliminate duplication and waiting
Dynamic queuing reserves multiple assets at a time to make sure your labeling team isn’t waiting around in between assets
The team shares a backlog that’s automatically distributed among labelers and reviewers

Improve collaboration and consensus between teams
To enable better collaboration during the labeling process, teams should look to implement processes and dedicated tooling to make repetitive tasks effortless, such as adding and clarifying feedback on a labeled asset. In order to elicit and clarify feedback, the ability to quickly raise issues and add comments on a labeled asset provides a simple and reliable channel for escalating questions to reviewers or subject matter experts.

A labeler can create an issue to ask a question, submit the label completed to the best of their ability, and receive feedback and clarification during the label review process. This streamlined way of collaborating on labels provides teams with an expedited way to handle inevitable issues that arise during labeling.

With an issues and comments tool, you can leave a comment directly on an issue for your workforce or your internal labeling team by tagging them. They can reply back or mark it as resolved.
With an issues and comments tool, you can leave a comment directly on an issue for your workforce or your internal labeling team by tagging them. They can reply back or mark it as resolved.
To improve consensus amongst team members, you can quickly drive toward an agreement as a team with a consensus tool that helps you arrive at a shared understanding of what good looks like. This tool allows you to automatically compare annotations on a given asset to all other annotations on that asset. Consistency is measured by the average of the label agreement between labelers. For example, when the consensus is set to 3, every label is grouped with two similar labels and the average of the three becomes the consensus score.

The consensus tool works in real-time so you can take immediate and corrective actions towards improving your training data and model performance.
The consensus tool works in real-time so you can take immediate and corrective actions towards improving your training data and model performance.
Utilize a programmatic approach for quicker access to data
Teams can now speed up the data import and labeled data export process through the use of a programmatic approach via SDKs and/or APIs rather than slowly handling data transfer methods manually. By connecting your team’s data and automating bulk operations, labeled data (including all the metadata, how the dataset was created, as well as labeler feedback) becomes much easier and quicker to manage and track as the complexity of your projects grows over time. This approach offers endpoint flexibility, so you can plug and play into your existing workflows without losing any valuable aspects of your training data.

One example of how this can be accomplished is through the use of a Python SDK, which offers more control over your data while simplifying and speeding up data import. For example, it simplifies the data import process so that you can use bulk DataRow creation. This process is asynchronous, so you don't have to wait for bulk creation to finish before continuing with other tasks (though you can wait if you want to). In some cases, you can also create projects and datasets programmatically, export labels, and add metadata to your assets all in an object-oriented way with a Python SDK — complete with all relationships between objects, speeding up your labeling workflow. By utilizing this approach, you can even export newly created training data automatically for an active learning workflow and adjust your labeling queue to focus on improving confidence in specific classes.


Leverage software optimized for speed
Many ML teams spend a considerable amount of time attempting to build out custom infrastructure for the training data that doesn’t scale and leads to disjointed development efforts. According to research firm Cognilytica, ML teams spend up to 80% of their time building and maintaining training data infrastructure.

Source: Cognilytica
Source: Cognilytica
Building such custom infrastructure can be both difficult and expensive, potentially costing millions of dollars in engineering resources to build and maintain. As a time-saving alternative, many teams have found the answer in adopting a training data platform (TDP). A TDP is purpose-built software engineered for speed and ease of use in handling AI data. It allows teams to move faster with best practices already built in place, rather than home-grown tools and recreating the wheel. By incorporating a TDP, teams save exponential time on data preparation and labeling, so ML teams can spend more time on their core
competency: building production models.

But not all TDPs are created equal. You can learn more about what TDPs offer ML teams in our TDP 101 guide, and how to choose the best one for your team by reading our buyer’s guide.

Incorporate automation through model-assisted labeling
Accurately annotating the tens of thousands if not millions of assets necessary to achieve a comprehensive data set has blocked many promising AI initiatives from ever reaching production-ready models. That’s why automation through model-assisted labeling is one of the simplest ways to reduce the waste of precious capital. Model-assisted labeling uses your own model to make labeling easier, more accurate, and faster. In some cases, we’ve seen ML teams save 50-70% on their entire labeling budget by utilizing this method.

Source: Aggregate labeling budget from publicly-traded Labelbox customer, Q4 2019 - Q2 2021.
Source: Aggregate labeling budget from publicly-traded Labelbox customer, Q4 2019 - Q2 2021.
Utilize active learning and prioritize the right data
Your team can achieve considerable gains in productivity and efficiency by employing active learning and prioritizing the right data. We’ve seen customers reduce their iteration cycles by up to 8X and use less training data by curating their datasets. Data selection for active learning typically involves selecting rare data or difficult cases. In the case of rare data, models need a minimum number of examples in order to perform well on a chosen task. Without a sufficient number of examples to feed the model, it can be too inefficient to sample data randomly. Using tools like a similarity function with model embeddings can help you to discover more matching data examples. Teams can visually find patterns and identify edge cases in data through the use of model embeddings. By clustering visually similar data, teams can better understand trends in model performance as well as data distribution. While teams can calculate and plot clusters manually, some use cases in ML require more time-sensitive trend detection and a quicker approach. This is where a visual embeddings tool can be helpful to boost model performance and support an active learning workflow.

Model embeddings, shown here in Labelbox, can help visualize trends in data and cluster similar data, helping you uncover outliers faster for an active learning workflow.
Model embeddings, shown here in Labelbox, can help visualize trends in data and cluster similar data, helping you uncover outliers faster for an active learning workflow.
Additional resources
For a deeper dive into the best practices outlined in this post and see demos of how they work in Labelbox, watch our on-demand webinar, the six best practices to save time when creating AI data. You can also download our guide which goes deeper into the six time-saving practices above.

How to build world-class AI applications with Labelbox 

product-webcast-apr-2020

AI-focused product leaders are spending a disproportionate amount of their time processing, labeling and augmenting training data to feed their AI applications. In this webinar, we’ll be delivering a demo of the Labelbox product while sharing insights on how a training data platform can free up time so you can focus on improving your AI application, while getting smart about managing labeling costs and ensuring data quality.

accelerating-ai

Watch our on-demand webcast on how Labelbox accelerates high-performing AI into production as a training data platform How Labelbox accelerates high-performing AI into production as a training data platform



The training data practices used by the most successful AI teams

training-data-practices-used-by-the-most-successful-teams-webcast

Having helped hundreds of AI teams across a wide range of verticals, we have discovered some commonalities in their ML workflows that lead to long-term project success and greater scalability. Seeing first-hand the key roadblocks and best practices of the most successful ML teams, we wanted to share how they’ve utilized certain processes, standards, and tools to more skillfully master how they work with training data.

In this webinar, you will learn some of the training data practices used by the most successful AI teams and how these have been applied in production use cases. We’ll also be answering live Q&A from the audience so that you can tailor these lessons into your own MLOps and accelerate your AI journey.

Learn how leading AI teams accomplish more by leveraging:

Advanced collaboration between domain experts, labelers, and data scientists

APIs for smooth integrations and delegated access for better access to data

Interpreting model performance and prioritizing areas for iteration

Monitoring on labeling performance across project levels and individual levels

Improve review efficiency with custom workflows



improve-review-efficiency-with-custom-workflows

improve-review-efficiency-with-custom-workflows-webcast

Welcome to ML Unboxed - the first webinar series designed for AI and ML professionals! You’ll learn about essential ML best practices and emerging concepts and then participate in a hands-on workshop guiding you through putting them into practice in Labelbox.

Product manager Pranoy Chaudhuri will teach you how to structure workflows that help you speed up review time, guide SME reviewers to high-impact data, and improve communication and collaboration with labeling teams.

To help ensure labeling quality, teams need to review labeled data to identify errors and trends leading to poor labeling performance. To do this at scale, you need an efficient workflow that most effectively balances labeling time, review and rework time, and quality. In this tutorial, we’ll walk through specific workflows to help improve quality and collaboration with review and labeling teams, including ways to:

- Customize your review queue based on label status, specific annotations, or specific labelers
- Quickly identify common issues faced by your labeling teams that lead to errors or inaccuracy
- Focus SME reviewer time on high-value labeled data where they’re needed most

With these advanced workflows, you’ll be able to increase the throughput, efficiency, and quality of your labeling and review.

10 training data practices used by the most successful AI teams 

10-training-data-practices

Machine learning teams today are confronted by a variety of challenges around the development of training, such as building an efficient and optimized labeling pipeline, improving the quality of labeled data, incorporating advanced methodologies such as automation, and more.
At Labelbox, we’ve partnered with hundreds of leading AI teams across industries to develop solutions to these challenges. In this ebook, you’ll discover:
-Common roadblocks faced by AI teams
-The collaborative workflows used by highly successful ML teams
-Processes, standards, and tools that enable the creation of high-quality training data and AI breakthroughs
-How teams can prepare for and address unexpected issues such as edge cases

build-scale-labeling-operations

We’re seeing that more ML teams are shifting their focus from iterating on models to improving the quality of their training data, labeling operations is becoming a central aspect of their larger MLOps strategy. Creating, managing, and maintaining an efficient labeling pipeline that produces high-quality training data is no easy task. Teams without the right people, processes, and tools in place will likely struggle to generate the training data they need at a cost-effective rate .

labeling-operations-webinar

Whether you’re just starting a new ML project or are looking to take your labeling operations to the next level of maturity, it’s crucial to understand what metrics are most important for your objectives. Apply best practices from manufacturing and software development to reduce cost, increase quality, and get to production faster. Labelbox co-founder Brian Rieger and Sr. Product Manager David Liang discuss
How throughput, efficiency, and quality can guide your decision making
What common misconceptions in labeling operations often lead to delays or low performance
How to apply best practices learned from working alongside Enterprise AI organizations and leading AI startups

genentech-breakthrough-labeling-process

Genentech is a biotechnology enterprise that has developed breakthroughs in medical research and medicine since 1976. Today, Genentech researchers are building convolutional neural networks to help diagnose illnesses and aid medical professionals.

Genentech has revolutionized their ML process by having domain experts train teams of labelers on medical imagery annotation tasks. With regular labelers creating annotations, which are then sampled and reviewed for quality by experts, Genentech has developed a much faster and less expensive way to create training data and get their life-saving algorithms to production.

optimize-labeling-operations-webcast

Learn how Labelbox’s world-class workforce and dedicated labeling expertise can help you achieve both time and cost savings. In our team’s 21 years of experience with machine learning data labeling project management, we’ve managed hundreds of projects, optimizing productivity, cost control, and flexibility.

Watch now to get practical insights on how a managed solution can improve your team’s:

Productivity: Workflow set up and ontology defined in just days. Deep experience in project and instruction design to limit mistakes. Trained labeling teams that are ready to go.

Cost control: A team dedicated to driving efficiency and supporting growth while managing cost. Consistent monitoring of workflows and labelers’ performances to minimize risk and maximize quality.

Flexibility: Ability to evolve approach as needs shift or workflows gain efficiency. Combine external teams alongside in-house or existing resources. Weekend and shift work.

scaling-labeling-operations-guide

As your organization grows in its development and use of AI-based applications, so too does your need for scaling your data labeling to produce high-quality training data. In this guide, we will cover some of the key factors to consider when evaluating different data labeling services to reduce costs, increase agility and ensure data quality across your labeling operations.

ontology-management-with-labelbox

Labelbox customers across industries count their ontologies as a critical part of the IP for their AI application. A detailed ontology is integral to a successful labeling process. An ontology ensures that all the labels in a project follow the same set of rules, reducing the possibility of confusion for the AI/ML model. Being able to quickly create, edit, and reuse ontologies is key to faster iteration cycles in machine learning.

move ai is revolutionizing video content creation

move-ai-is-revolutionizing-video-content-creation

Creating high quality video content, whether it’s a movie, a video game, or a live broadcast, is typically cost-prohibitive. The best content is made by the studios with the best equipment and animation talent. Now however, an emerging AI company, Move.ai, is set to change the status quo, making processes like motion capture and key point recognition easier, faster, and less expensive — for individuals and studios alike.

How VirtuSense built an AI data engine that dramatically increased model performance

how-virtusense-built-an-AI-data-engine-that-dramatically-increased-model-performance

When VirtuSense founder Deepak Gaddipati’s active and healthy grandmother had a fall, broke her hip, and passed away within days due to resulting complications, it left the entire family shocked — even the physicians in the family. No one had known that she was a fall risk, as she’d never fallen before. The tragedy led Gaddipati to research how hospitals and care facilities could better identify fall risks and prevent similar deaths. He began his research in 2012, and by 2014, he had founded VirtuSense, an AI-based system of sensors that monitors the activity of patients and alerts caretakers when their movements indicate they may be at risk for a fall. 
Every hour, seven people die as a result of fall-related injuries in the United States. Today, VirtuSense is the #1 fall prevention system in the world. In the past two and a half years, the system has successfully prevented over 100,000 falls, saving tens of thousands of lives.
How it works
VirtuSense places LiDAR sensors in each hospital room, ambulance, or care facility room. These sensors generate data of patients’ movements as they lay in bed, sit up, stand, and walk around. The company’s AI models, placed within these edge devices, are trained to find and track movement patterns that, based on peer-reviewed research, identify potential falls 30-65 seconds before they occur. The system will then generate alerts for nearby hospital staff when the model detects fall risks. Expanding upon their initial success, VirtuSense systems now also detects risks for pressure ulcers and alerts bed-ridden individuals and/or their caregivers to shift their postures and prevent ulcers from forming.

The problem of trust
Developing an AI-based solution in healthcare came with one huge challenge: gaining the trust of healthcare professionals. Nurses who experience too many false alarms tend to take alerts less seriously. With current technology, they receive anywhere from ten to twenty alerts a day, a large percent of which are false alarms. To increase the effectiveness of these warning systems and gain the trust of hospital staff, VirtuSense had to ensure that their models achieved a high level of accuracy. Within a couple of years, their technology generated an average of only seven alerts per day, and an average of two were false alarms. As the team worked to increase the accuracy of their models beyond this, however, they hit a roadblock. They needed a more diverse, higher quality training dataset, which was becoming more and more challenging to generate.
The team used a mix of internal and open source labeling tools, which they found difficult to manage as the amount of training data required scaled. The team struggled to distribute unstructured data to the labeling team and track progress in an efficient manner. With their open source tools, the lack of immediate support when issues cropped up became yet another blocker for the team. The VirtuSense team chose Labelbox to help them scale their labeling workflows and generate more high-quality training data.
"False alarms should ideally occur less than 99.99% of the time, even when the models are running 365 days a year, 24/7, at 30 frames a second. Otherwise people won't trust the technology." — Deepak Gaddipati, Founder & CTO of Virtusense
Building a data engine
Rather than adopting Labelbox one step at a time, the VirtuSense team laid out a plan for creating an AI data engine to maintain and raise the accuracy of their models. An AI data engine is a seamless process that brings unstructured data from storage and into a labeling platform, enables labelers to annotate data quickly and easily, incorporates quality assurance workflows, and sends high quality training data into the model training system. As a model trains and enters production, a data engine also includes model error diagnosis and curating subsequent training datasets based on errors and low confidence areas, creating a more effective and efficient iterative loop for the model.

The VirtuSense team designed their data engine to automate every aspect of the model iteration loop, so that the only human involvement required is that of their labelers as they annotate data. The team’s plan also involved labeling LiDAR data not only for their current use cases (which, at the time, included only detecting falls with patients who were getting up from their beds), but also future use cases, such as detecting potential pressure ulcers. The team then implemented their vision by integrating Labelbox seamlessly into their MLOps. 
First, the VirtuSense team finds areas of low confidence in their AI models by running them on a specially curated testing dataset that includes edge cases and rare situations. The system then identifies data relevant to these low confidence areas and “cleans” it of metadata that attaches it to a particular sensor, facility, and patient to uphold confidentiality and security requirements and prevent biases from cropping up. This data is then pre-labeled by a purpose-built model and programmatically imported into Labelbox, a setup made possible with the Labelbox Python SDK. An internal labeling team then reviews and corrects the pre-labeled data within Labelbox, after which the datasets are programmatically exported from Labelbox and into the model training system on VirtuSense servers. 
Once a model is trained, it’s automatically pushed to thousands of endpoint edge devices, where it detects fall-risk movements and sends alerts accordingly. VirtuSense models are now retrained every few weeks with this data engine to increase and maintain their accuracy.
“We were very pleasantly surprised by how easy [Labelbox] is to use. People got the hang of it just like that. And it's not just about how easy it is to label data. We really like its ability to distribute data to multiple people and keep track of what's happening — it gives us a base of operations to help traverse through all that data….but the real beauty of it, the reason we really love Labelbox, is that it integrates so well with our tech. From a thousand-foot view, it looks like an AI labeling engine, which feeds into the deep learning system and spits out trained networks.” — Deepak Gaddipati, Founder & CTO of VirtuSense
Automating the labeling process
VirtuSense uses pre-labeling as a key component of their data engine. This labeling automation method imports model output (from either an off the shelf algorithm, a previous version of the model in training, or another model built by the ML team, depending on the use case) into the training data platform as pre-labeled data, which labelers can correct or edit instead of labeling unstructured data from scratch. 
When the VirtuSense team decided to leverage pre-labeling, they knew that their existing models, built to exist on edge devices with only a few parameters, would be too small to pre-label data according to their needs. Instead, they built a large model specifically for pre-labeling, with enough parameters to label all necessary data as accurately as possible. Once the model has labeled training data, the dataset is imported into Labelbox, where the small internal labeling team only needs to review and correct the labels before the assets are pulled into the model training workflow.
Since they adopted Labelbox, VirtuSense has been able to increase the amount of labels created by 5X. Their team has now produced over one million labeled assets. Their false alarm rates have fallen from roughly 28% to 6%, and the daily average number of alerts have dropped from seven to five. Their average accuracy has increased by over 20%.
“As a CTO, you usually only usually hear about the technologies your team uses when things are going wrong. With Labelbox however, the only time I dealt with it is the day I signed the contract. Since then, my team has been using the technology seamlessly without any issues, and this gives me a lot of peace of mind as a technical and engineering leader.” — Deepak Gaddipati, Founder & CTO of VirtuSense
Due to their new data engine and improved labeling capabilities, the VirtuSense team is planning to expand beyond LiDAR data to detect potential stroke risks, arrhythmia, and other dangerous conditions.
“Out of all the platforms we explored, Labelbox is the easiest to use and the best for managing labelers and workflows and monitoring performance. Labelbox also keeps offering more interesting and helpful features. I find the Catalog feature very interesting. We haven’t been able to use it yet, but we are planning to update our workflow to include it in the future.” — Hithesh Reddivari, Tech Lead, VirtuSense
Watch this on-demand webinar to learn how a data engine can improve your enterprise AI projects. You can also try Labelbox's annotation and data curation features for free.

Flex card thumbnail for Guide to unlocking AI with your data guide.

Guide to unlocking AI with your data_Thumbnail

The-guide-to-unlocking-AI-with-your-data

Less than a decade ago, generating entire software programs, marketing campaigns, and works of art in a matter of minutes or hours would have been impossible. Today's workforce, however, is armed with AI solutions like Copilot, ChatGPT, Stable Diffusion and more — with new and more powerful tools launching every day. AI has become an integral part of our day-to-day lives.


For enterprises, it's more important than ever to use AI effectively and quickly to build and maintain their competitive advantages. Today's AI teams have more computing power, more data, and more pre-trained models than before, but they also face more challenges and setbacks. An AI platform offers the tools and infrastructure to help mitigate these challenges and accelerate AI development. Read the guide to learn how.

The guide to saving time when creating AI data

guide-to-saving-time-when-creating-AI-data

Wading through vast amounts of unstructured data to accurately annotate assets requires a tremendous amount of patience, organization, and time. Drawing from the experiences of hundreds of AI teams across industries, we are sharing six time-saving practices for ML teams to implement when handling AI data. In this guide, you’ll discover how to:
Annotate faster with a dynamic queueing system 
Improve communication, collaboration, and consensus between teams
Utilize a programmatic-approach for quicker access to data 
Leverage software optimized for speed
Incorporate automation through model-assisted labeling
Utilize active learning and prioritize the right data

These strategies will unblock key barriers and speed up overall processes and capabilities for a quicker path to production AI.


customizing an optimal training data workflow using an api first approach

customizing-an-optimal-training-data-workflow

customizing-an-optimal-training-data-workflow-using-an-api-first-approach

Cape Analytics is a leading provider of AI-powered geospatial property data. The company uses deep learning and geospatial imagery to provide instant property intelligence for buildings across the United States. Their pioneering work for assessing risk and property value is tied to common property factors (pools, roof construction, solar panels) but also emerging risks like vegetation coverage, detailed roof condition, and wildfire zones. Creating performant machine learning models to extract intelligence from properties requires enormous amounts of high quality, labeled data. To accomplish this feat, Cape’s engineering and data science team searched extensively for ways to build a complete and automated training data solution

Optimal iteration loop with Matthew Mcauley

optimal-iteration-loop-matthew-mcauley

ML practitioners have realized that iteration is the key to performant models, but perfecting and tightening the loop still remains a challenge for even the most advanced teams. Watch this session to hear the practical insights on how to get models to production-level performance quickly, how to best structure your training data pipeline and create the optimal iteration loop for production AI.

Functions and embeddings are a powerful way to quickly search and explore your unlabeled and labeled data. Using them can help you break down silos across datasets so you can focus on curating and labeling the data that will most significantly improve model performance.

When it comes to producing high-quality training data for data-centric AI, labeling is only one part of the solution. The most successful training datasets are carefully curated to increase model performance at each iteration. For example, if a model being trained to recognize various fruits in images has subpar performance in identifying apples, the next dataset used to train that model should contain more images of apples.

However, combing through a large unstructured dataset for just the right assets can be a challenge. Advanced AI teams have discovered a solution: using embeddings to sort through both labeled and unlabeled data.

How to use embeddings to create high-quality training data

how-to-use-embeddings

Functions and embeddings are a powerful way to quickly search and explore your unlabeled and labeled data. Using them can help you break down silos across datasets so you can focus on curating and labeling the data that will most significantly improve model performance.

When it comes to producing high-quality training data for data-centric AI, labeling is only one part of the solution. The most successful training datasets are carefully curated to increase model performance at each iteration. For example, if a model being trained to recognize various fruits in images has subpar performance in identifying apples, the next dataset used to train that model should contain more images of apples.

However, combing through a large unstructured dataset for just the right assets can be a challenge. Advanced AI teams have discovered a solution: using embeddings to sort through both labeled and unlabeled data.

What is an embedding?
An embedding, or feature vector, is an array of numbers assigned to an asset by a neural net. Assets that have similar content will also have similar embeddings. For example, in a dataset comprising images of apples and oranges, an appropriate embedding used for image similarity will show that all the vectors corresponding to apples have similar values. The vectors for all images of oranges will also be grouped together.

A visual representation of vectors in a dataset. Each cluster denotes a similar set of assets.
A visual representation of vectors in a dataset. Each cluster denotes a similar set of assets.
These similarity embeddings are typically generated by a convolutional neural network (CNN) and have dimensions in the order of thousands. Once you have implemented embeddings, similarity search is just a matter of comparing the distances between these vectors.

You can visualize embeddings by using certain techniques to “project” the points from their high dimensional space onto a plane. This can be especially useful to determine patterns in the data, like images from the same class forming clusters.

You can try the Labelbox embedding projector for free:

Sign up for a free account and switch to the Public Demo area
Navigate to Catalog
Select “Pomological Watercolors” in the “Datasets” filter on the left
Click on the point cloud icon on the top right to activate the embedding projector
In this example, the embeddings are automatically grouped by each of the five types of fruit present in the dataset.
In this example, the embeddings are automatically grouped by each of the five types of fruit present in the dataset.
How to generate embeddings for images and text
As explained above, similarity embeddings typically come out of a neural network. The diagram below shows a simplified version of what a typical classification CNN looks like.

Near the end of a neural network, before the last layer that determines the class the network will select, there’s usually one or more fully connected layers. To generate a similarity embedding for an image, input that image into the neural network and then take the values from the last fully connected layer.


The code below uses Tensorflow to load an EfficientNet pre-trained neural network and generate an embedding vector for an image:

import numpy as np
from tensorflow.keras.applications.efficientnet import EfficientNetB2
from tensorflow.keras.applications.efficientnet import preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing import image


efnet = EfficientNetB2(weights='imagenet')

# This is the image file we'll process
file = 'pepe.jpg'
img = image.load_img(file, target_size=(260, 260))
# Preprocess image
x = preprocess_input(np.expand_dims(img.copy(), axis=0))
# Make a new model where the output is the layer we care about
bottleneck_model = Model(inputs=efnet.input, outputs=efnet.get_layer('avg_pool').output)

# Then extract the features, which is basically a 1408-long vector
bottleneck_features = bottleneck_model.predict(x).reshape(1408)
print (bottleneck_features.shape)
print (bottleneck_features)
To generate embeddings for text data, the process is similar, except that we use a neural network trained on text.

In the example below, the code is a lot more compact because the model was trained specifically to generate embeddings, so it’s not necessary to “tweak” a classification model.

import tensorflow_hub as hub
import numpy as np
import tensorflow_text

text = ['The quick brown fox jumps over the lazy dog']

model = hub.load("https://tfhub.dev/google/universal-sentence-encoder-multilingual/3")

# Compute embeddings.
embedding = model(text)
How to use embeddings to find similar data in Labelbox
Labelbox’s embeddings projector makes it easy for AI teams to identify groups of similar assets within a dataset, filter them accordingly, and carefully curate their next training dataset based on where their model needs to improve. Embeddings can be imported via the Labelbox SDK (using precomputed embeddings is also an option).


Teams can then use the Catalog to conduct a similarity search to sort the assets by selecting one or more data rows and then selecting the “View Similar Data Rows'' option from the blue drop down menu on the top right. You can also select which embedding to use for the similarity search, including any custom embeddings previously uploaded with the SDK. Chosen assets can then be sent into a labeling project as a batch for labeling.

How to use functions in Labelbox to automatically find edge cases
Identify edge cases or low-performing classes in your model
Create functions to find similar low-performing data
Use functions to find similar data when new data is uploaded
A function is a saved similarity search that you can use as a filter to find visually similar data. Functions are a useful way to find relevant new data rows as they are added to your workspace. To create a new function in Labelbox, navigate to the Catalog and select the “View Similar Data Rows” option in the drop-down menu. The actively selected data rows and the chosen embedding metadata field will be used to define the function. Once you have created the function, it will start a background task to score all of the data rows that share the embedding.


A function can also be used as a filter in the left hand side panel in Catalog. You can mix function queries with all other filter types in Catalog. For additional details, please refer to the help section on Functions in our documentation.

Start curating high-value data faster
Functions and embeddings are a powerful way to quickly search and explore your unlabeled and labeled data. Using them can help you break down silos across datasets so you can focus on curating and labeling the data that will most significantly improve model performance. These features are available to all users on Labelbox. Sign in or create a free account to try it today!

Your competitive advantage should be data with Cathie Wood and Manu Sharma 

your-competitive-advantage-should-be-data

With over 40 years of experience identifying and investing in innovation, ARK Invest CEO, Cathie Wood discusses what she believes it takes to win in AI with Labelbox CEO Manu Shama.

how-the-best-achieve-AI-breakthroughs

Labelbox has seen how more than 200 organizations label, manage, and improve their training data. Come learn what sets apart those who move the fastest, realize the largest efficiencies, and find the greatest success in their ML programs.

How leading teams get full visibility and control over their training data

how-leading-teams-get-full-visibility

How leading AI teams get full visibility and control over their training data
AI is supposed to transform businesses, yet it's taking enterprises much longer than expected to achieve this impact. Creating high-quality training data with consistency and transparency is at the heart of the problem. 

Brian Rieger and Kelly Anderson discuss how model assisted labeling and model diagnostics and can lead to more accurate models, faster implementation and 70% cost savings. 

Watch this video to learn what workflows and strategies leading AI teams use to prioritize the right data at the right stage of their project’s life cycle in order to iterate quickly.

ImageBiopsy lab uses model assisted labeling

ImageBiopsy-lab-uses-model-assisted-labeling

Since 2016, ImageBiopsy Lab has been building AI applications to help physicians and the medical community at large better understand and diagnose musculoskeletal diseases (MSK), which affect 1.7 billion people worldwide. Today, the team focuses on optimizing the radiological workflow via automation and standardization to enable the early detection and prevention of MSK. This automated decision support allows orthopedists and radiologists to assess and predict bone health by turning images into actionable information.

The team has optimized their ML development process by using AI to pre-sort massive datasets and employing a model-assisted labeling workflow to pre-annotate filtered datasets. As a result, the team has reduced their time spent per label by 160%.


4 strategies for successful labeling operations

4-strategies-for-successful-labeling-operations

how-to-build-trust-in-your-ml-models

It’s no secret that AI and ML are integral to our future, with teams building algorithms for everything from robotic surgery to sorting fresh produce. But many of us are also skeptical about predictions made by deep learning models. Consumers often find recommendation algorithms as mysterious and daunting as they are helpful. On the enterprise side, leadership and stakeholders place less trust on model outputs when they don’t know how the model produced them. This presents a tough challenge for ML teams who want to build transformative models to support major operations throughout the organization. Let’s explore a few ways that teams can ensure that their models are trusted within their enterprise.

Non-expert labeling teams and training data for medical use cases 

create-high-quality-training-data

new-study-finds-that-non-expert-labeling-teams-can-create-high-quality-training-data

Producing high quality training data for any ML use case can be costly, and it’s doubly so for specialized use cases that require experienced domain experts to review (and often label) training data. In the agriculture industry, for example, ML teams training models to recognize weeds or diseased crops need to have their training data examined and corrected by trained agronomists and horticulturists, whose time is not only expensive, but also limited.

Discover a smarter, easier approach to building AI products faster.

Library featured image

Library featured content one

Library 

Learn which enterprises are leading the way by innovating faster with AI.

librar featured 2

Library featured content 

Discover how leading AI teams do training data and see real-life use cases.


library featured 3

Library featured content three

Featured library

The scientific approach to generating the highest-quality data 

Labeling automation library card

Building the data factory

Labelbox has built the modern AI data factory by expanding its Labeling Services and enhancing its core platform to accelerate data delivery, generate high-quality data, and improve control over labeling tasks. The new multimodal editor, offline multimodal conversations, and other features streamline the creation and evaluation of complex AI models, especially for Generative AI and RLHF applications.

fine tune gpt image

Programmatically launch jobs

Labelbox's Python SDK simplifies the orchestration of human data labeling projects, enabling programmatic control over every step, from data import to export. This empowers AI teams to efficiently manage labeling jobs, incorporating high-quality labeled data into their workflows and accelerating their AI development process.

Building labeling operations into your core MLOps strategy

Inner Image

The people, processes, and tools your ML team needs to create high-quality training data

mlops social image

Thumbnail

Learn how an AI platform accelerates AI development by offering the tools and infrastructure for AI teams to explore data, label data, train models, and more.

The guide to unlocking AI with your data

Guide to unlocking AI with your data cover image

Labelbox - The complete guide to data engines for AI

Complete guide to data engines hero

Learn how an AI data engine is the foundational infrastructure for how team members interface with data and models in order to build better AI products — and do it remarkably fast.

Complete guide to data engines social

The complete guide to data engines for AI

10-training-data-practices-2022 

top10cover

Discover the four common roadblocks faced by AI teams across industries and use cases and 10 processes and strategies to help overcome them.

10 training data practices used by the most successful AI teams

Buy-build-2022

buy-v-build-cover

The decision to build your own data labeling infrastructure depends on multiple different factors, including model complexity, annotation volume, and opportunity costs. Read our guide to see if you should build it yourself — or buy one.

Should I build or buy an AI data engine?

The Labelbox Guide to Labeling Automation

Labelbox-automation-guide 01

Download The Labelbox Guide to Labeling Automation to learn more about how advanced labeling teams cut labeling time and costs with a robust labeling automation strategy.

Download The Labelbox Guide to Labeling Automation

Labelbox-automation-guide 02

VBT 01

How designing a workflow involving automation and iteration can lead to more accurate models, faster implementation and 70% cost savings

Building a better AI data engine

VBT 02

The definitive guide to buying a training data platform

The definitive guide to buying a training data platform - Image

We've created a comprehensive checklist of features and capabilities that a TDP should have to best serve your specific requirements as your ML needs scale.

Library

Learn more about our approach to human-centric AI evaluation

Generate high-quality, multimodal chat data

Guide to evaluating text-to-image models

Automate data tasks with foundation models

How to automatically ingest data from Databricks into Labelbox

How Walmart drives innovation for their conversational AI and LLM applications

Seamlessly integrate Databricks data pipelines with Labelbox

How to accelerate time-to-value when building AI for retail and e-commerce

How to set up your labeling ops for success with observability

Try Labelbox today