How a Fortune 500 retailer improves training data quality for their conversational AI applications

Problem

The data science team wanted to find faster ways to annotate conversational text from shopping chatbots and label inventory images for their object detection and classification models, which included tens of millions of diverse product SKUs.

Solution

Labelbox Annotate, which provided intuitive text and image editors with the ability to tag entity relationships (NER) for conversations. Integrations with Google BigQuery via the Labelbox Python SDK to automate manual workflows and data import.

Result

Labeled data accuracy improved by an estimated 25% through Labelbox's quality assurance systems, review workflows and real-time collaboration. Labelbox’s Boost team was able to deliver high-quality data (with 95% accuracy in labeled data) and at a 25% reduction in turnaround time compared to similar services.

A leading Fortune 500 enterprise that operates global hypermarkets and ecommerce properties was looking for a way to improve the way they produce labeled data for their conversational AI applications. The data science team wanted to find faster ways to annotate conversational text from shopping chatbots and label inventory images for their object detection and classification models, which included tens of millions of diverse product SKUs. 


The company previously relied on tech-enabled BPOs to create and manage their labeled data but found the decision to be suboptimal because of the BPO’s process lacking transparency, giving them sparse visibility into the quality of the training data being produced. In addition, tech-enabled BPOs typically did not provide dedicated software that would easily give team members the ability to collaborate on the training data iteration process itself. This encompassed a range of stakeholders including natural language data-scientists, ML engineers, data engineers, linguists and software engineers, etc. The relationship with these vendors were more of a black box approach, omitting important metrics such as individual labeler and project-level analytics reporting, while in-house subject matter experts could not closely collaborate with external service providers. 


Labelbox offered a clearer and more efficient alternative, by providing an end-to-end in-app labeling workflow for the company’s conversational AI efforts. The team adopted Labelbox, which delivered intuitive text and image editors with the ability to tag named entity recognition (NER) relationships for conversations. The company's unstructured data for these chatbots existed as a mix of intertwined natural voice commands, text messages, images, and traditional GUI interactions. Labelbox’s data engine also allowed the company’s data science team to work with any labeling vendor (whether internal or external) and collaborate easily with their in-house domain experts. This regularly fed into a labeling process whereby reviewers could check and ensure quality benchmarks in training data were being met. 


In addition, Labelbox’s Annotate product provided the ability to automate a lot of the manual orchestration via the use of the Python SDK. This allowed labeling workflows to be initiated from Google BigQuery, which the enterprise heavily relied on, and had set up as a core part of its existing data infrastructure. Labels could now be easily pulled and pushed from BigQuery and tables for structured data could be easily created from the Labelbox and Google Cloud integration. 


By choosing Labelbox, the enterprise saw dramatic efficiency gains, because they were now able to get full visibility into their labeling pipeline via in-depth project analytics. Model-based pre-labeling (also known as model-assisted labeling) also sped up their labeling process by allowing their team to adjust annotations as opposed to creating ground-truth labels from scratch. The company is now able to draw insights about labeling performance, taking actions that directly translates into improvements in label throughput, efficiency and quality. 


In terms of ROI, the company’s labeled data accuracy improved by an estimated 25% through Labelbox's quality assurance systems, review workflows and real-time collaboration, while Labelbox’s Boost team was able to deliver high-quality data (with 95% accuracy in labeled data) and at a 25% reduction in turnaround time compared to similar services. The company is now planning to leverage Labelbox’s Annotate and Catalog products to build more AI-powered applications that enable richer customer experiences (both online and in-store), which span use cases such as customer care, employee assistance, and customer facing mobile apps.