How to generate industry-specific data for AI training with Labelbox

Generative AI models are becoming increasingly sophisticated thanks to advances in post-training and model alignment tasks. As a result, the demand for models that not only understand language but also grasp the nuances of specific industries is skyrocketing. 

While general-purpose large language models (LLMs) have made significant strides, they often fall short when faced with tasks requiring deep domain expertise. Many industries require AI systems capable of industry-specific reasoning that can navigate complex, domain-specific scenarios with expert-level understanding. 

The journey to creating AI that truly "gets" a particular field goes beyond simply feeding it more data. It requires a targeted approach that focuses on enhancing the model's ability to understand and apply the unique language, concepts, and problem-solving frameworks inherent to a particular field. This is where industry-specific reasoning comes in, and it is the key to unlocking the true potential of AI across sectors like finance, law, medicine, and insurance. By building and using higher quality training data, companies are creating a competitive advantage and opening the door to new opportunities.

This guide will walk you through the process of generating domain-specific data with the Labelbox data factory to train your LLMs and AI models on industry-specific reasoning. We'll explore real-world examples in finance and law, delve into the intricacies of creating post-training datasets, and provide a step-by-step walkthrough of creating a project in the Labelbox platform. We will also discuss the importance of selecting the right AI trainers and crafting clear instructions and ontology. 

By the end of this guide, you'll be equipped with the knowledge and tools to embark on your own journey toward creating powerful training data to help you build AI models that are not just intelligent, but industry-smart.

Examples of how industry-specific reasoning is becoming a reality

The power of industry-specific reasoning in AI is not merely theoretical; it's being realized today by forward-thinking companies leveraging advanced tools and expert human insights. Before explaining how to use the Labelbox platform to build a dataset for training models on industry-specific reasons, let's examine two real-world examples where Labelbox has helped clients in the finance and legal sectors expand the capabilities of foundational models. 

These examples showcase how targeted data labeling, guided by domain experts, can enhance an LLM's ability to support different industries. And while each example discusses a specific industry’s use case, we’ll later see how the process and usage of Labelbox can be extended to a much broader range of domains and uses, paving the way for a deeper dive into the practical steps involved in creating your own successful projects.

Finance: Training models on financial analysis

A leading AI lab wanted to improve their LLM’s performance in financial analysis and argumentation. They hoped to train the LLM to provide meaningful insights on any public company when provided a ticker symbol and the latest financial reports of a company. In addition, they wanted the model trained and prepared to answer the most common questions financial analyst might ask. 

To reach this level of financial expertise, they needed training on detailed, domain-specific datasets that were accurately labeled by a team of Chartered Financial Analysts (CFAs) and financial experts. The CFAs needed to have advanced industry knowledge and experience reviewing and analyzing financial models and details. 

However, they faced two major challenges: (1) sourcing financial experts who could accurately evaluate and rank responses through multi-step analyses, and (2) managing a complex process where each piece of data could take an hour or more to properly analyze and fully label. 

The company turned to Labelbox to help source experts with financial expertise. Labelbox leveraged their Alignerr network of expert human labelers to quickly recruit and onboard a skilled team of financial experts with backgrounds in CFA, MBA, and Masters in Finance. The team built a customized project consisting of a complex ontology, detailed instructions, and numerous attachments per dataset to review and prepare hypothetical questions and scenarios around. 

Legal: Automating case evaluation and data discovery

A forward-thinking legal tech company wants to revolutionize how plaintiff firms operate. By using AI-driven tools with industry-specific reasoning, they want to significantly enhance their ability to evaluate cases, analyze key facts and claims, and ultimately deliver results for their clients more efficiently and transparently.

To achieve this, they needed to imbue their foundational AI model with deep legal expertise, particularly in understanding long legal documents, interpreting insurance and medical bills, and assessing case value.

The company partnered with Labelbox to create specialized datasets for post-training their model. Labelbox's network of legal experts reviewed a long list of example prompts that were each associated with many multi-page legal documents. They were tasked with identifying key information based on the prompt, crafting well-reasoned responses supported by evidence found in the documents, and evaluating the models responses for accuracy, safety, and reasoning. They were also used to build a dataset based on understanding insurance and medical billing details and learning how to extra the correct details.

Similar to the finance example above, this project involved creating a custom ontology and using the flexibility of the Labelbox text editor to generate responses to sample prompts from industry experts, identify key data in attached documents, and use their own expertise to properly interpret complex documents. 

Importance of high-quality training datasets 

The above finance and legal examples, while distinct in their domain-specific challenges, underscore a fundamental truth about building industry-specific reasoning into AI: the underlying approach remains consistent across industries, whether it's finance, medicine, insurance, or any other specialized field. 

At its core, the process revolves around creating unique and differentiated datasets that capture the specific nuances, knowledge, and reasoning patterns of the target domain. These datasets serve as the bedrock for training models to perform the desired capabilities, enabling them to go beyond general understanding and develop true expertise. 

Strategies to remember when planning your project

Before kicking off a project or starting the process of building a post-training dataset for industry-specific reasoning, organizations should take the time to think through the following steps:

• Identifying key domain-specific competencies required
• Defining clear evaluation criteria for expert knowledge
• Gathering supporting documentation and examples to attach to the project
• Writing hypothetical scenarios and questions you want the model to answer
• Establishing metrics for measuring improvement in domain understanding
• Setting realistic scope and scale for the training dataset
• Preparing a model evaluation process for side-by-side comparisons

This list may be shortened by some users and for others there may be other key considerations to add; however, we’ve learned that the most successful projects involve advanced planning and often a fair amount of preparation to gather necessary documents, built prompt examples, and think through the key scenarios that require examples and labeled data to sufficiently train the model. 

Selecting the right AI trainers

The next big AI breakthroughs will be fueled by unique, high-quality data. The massive quantity of available data from the internet has been used to train all of the most popular AI models—meaning they all have similar core capabilities and areas of weakness. 

To properly train these models on new domain-specific knowledge and reasoning it’s imperative that we tap into new information and capture data from domain experts. As a result, the focus must be on properly identifying and recruiting AI trainers with unique skills—like the Alignerr network of highly educated industry experts—to label data, generate new responses, evaluate models, and perform critical model post-training alignment tasks.

Based on our experience from leading the Alignerr network and working with AI labs and model builders to form teams of experts, it’s important to keep the following in mind: 

• Domain expertise: Look for annotators with relevant qualifications, certifications, and experience in the target industry (e.g., CFA, JD, MD).
• Analytical skills: Choose individuals who can demonstrate strong analytical and reasoning abilities.
• Attention to detail: Ensure the annotators are meticulous and capable of identifying subtle nuances in the data.
• Communication skills: Select annotators who can clearly articulate their reasoning and provide constructive feedback.
• Previous success metrics: For annotators that have worked on AI training before and used a complete AI platform, then you should be able to review past performance metrics to assist in the evaluations. 
• AI interviews: Abilities can only be determined so much by reviewing resumes and profiles, so using interviews tailored to their specific background provides the most powerful data. 

Key Labelbox features for generating industry-specific training data

Having outlined strategic considerations to review before starting a project as well as the keys to selecting the right domain experts, the next crucial step is understanding how to operationalize this process efficiently and effectively. This is where a robust and versatile platform like Labelbox becomes indispensable. 

Labelbox Platform is an industry-leading software tool with the flexibility and advanced features needed to translate your vision into a concrete, well-structured labeling project tailored for industry-specific reasoning. 

From selecting the appropriate editor—often the flexible text editor for tasks involving complex analysis and document reviews—to building a comprehensive ontology with a mix of selection tools and free-form text inputs, Labelbox can capture the nuances of your target domain. In addition, the built-in quality assurance and project management capabilities ensure that your project stays on track, maintains high accuracy, and ultimately delivers a dataset that enhances your AI model's performance. 

Let's briefly explore the key components of the Labelbox platform that we see used most often in industry-specific reasoning tasks and discuss how they can be leveraged to build the foundation for your industry-specific AI.

Data types & editors

Labelbox supports a range of data types, including text, video, image, audio, PDFs, multimodal, and more. For each data type, a customized editor exists to serve as the core interface for labeling that specific data type. While tailored to specific datatypes, the editors are extremely flexible and can be customized to help label and capture the specific data needed for a given industry. 

For most industry-specific reasoning projects, we have seen the core text editor serve as the key editor. It was the editor used in both the finance and legal examples shared earlier. The editor supports these annotation types: entity, relationships, radio classification, checklist classification, and free-form text classification. The latter is often used to capture detailed information and sample responses from domain experts on a given prompt or to provide detailed information on specific information found or extracted from an attached document. 

Ontology

The Labelbox ontology defines the structure and categories for labeling data. Ontologies can be reused across different projects and they are required for data labeling, model training, and evaluation. When you are in the editor, the ontology appears in the Tools panel.

Ontologies can be customized for any given project and supports classifications, object detection, segmentation, relationships, messaging ranking, prompt rating, step-by-step rating, and more. The available ontology tasks will vary based on the selected editor and data type. 

Ontologies also allow for hierarchical relationships between classes, allowing you to create complex labeling tasks with detailed information. 

Quality assurance

Labelbox offers a suite of quality assurance features to deliver high-quality labeled data, which is particularly crucial when dealing with complex, industry-specific reasoning tasks. Consensus allows you to measure the agreement between multiple labelers on the same data point, providing a statistical measure of confidence in the assigned labels. Benchmarks enable you to incorporate known ground-truth data into your project, allowing you to directly assess labeler accuracy against established standards. 

Calibration tools help identify and correct for systematic biases that individual labelers might exhibit, further refining the consistency of your dataset. Moreover, Labelbox Monitor provides real-time insights into your labeling operations, allowing you to track key metrics, identify trends, and quickly spot any outliers in labeler performance. With Monitor, you can proactively address issues and make adjustments as needed, ensuring that your project stays on course.

Project management

Labelbox provides robust project management tools to manage and track your industry-specific reasoning projects from start to finish. Unlike many labeling services and solutions, the Labelbox platform offers transparency into project progress, allowing you to monitor the status of individual tasks, track labeler performance, and gain a clear overview of the entire project's health. Real-time communication features enable seamless interaction with your team of expert labelers, facilitating quick clarification of instructions, addressing questions, and providing feedback directly within the platform. 

Labelbox also allows you to build customized workflows tailored to your specific review and quality control processes, ensuring that each piece of data undergoes the appropriate level of scrutiny before being incorporated into your training set. Furthermore, robust data versioning capabilities provide a comprehensive history of all changes made to your data and ontology, allowing for easy rollback if needed and providing a clear audit trail for maintaining data integrity. 

Creating an industry-specific reasoning project in Labelbox

Having explored the core features of the Labelbox platform, let’s examine the key steps to creating and executing on a project to generate new data for training your AI models and apps on industry-specific reasoning.

This section provides a practical roadmap, walking you through each stage of the process, from crafting clear and comprehensive instructions for your expert labelers to setting up your project within the Labelbox environment, configuring the ideal ontology, managing the labeling operation, and conducting thorough reviews to ensure the highest level of data quality. 

Using these steps as a framework for your project, you can start exploring how the Labelbox Platform can be used for your unique project. You’ll learn what it takes to build a new training dataset for your specific needs.

1. Identify desired outcomes and goals

Before diving into the mechanics of using Labelbox, it's crucial to establish a clear understanding of your project's objectives. Training frontier models for industry-specific reasoning requires a well-defined target. What specific problem are you trying to solve? What capabilities do you want your model to possess? Answering these questions will guide your data collection and annotation strategy, ultimately determining the success of your project. For example, are you aiming to build a model that can summarize legal documents, predict financial market trends, or diagnose medical conditions? Each of these scenarios demands a different approach to data and annotation.

This initial phase focuses on defining the desired outcomes and the key capabilities you want your model to achieve. Let's consider a practical example: building a model to assist legal experts in reviewing case files. The desired outcome might be to reduce the time spent on initial case review by automating the identification of key legal arguments and relevant precedents. This translates into key capabilities like: understanding legal terminology, identifying relationships between different parts of a case file, and summarizing complex legal arguments. We need to capture training data that reflects these capabilities. This might include labeled examples of legal arguments, summaries of past cases, and annotations highlighting the relationships between different legal concepts within a document.

2. Write clear instructions

The quality of your training data hinges on the clarity and completeness of the instructions provided to your AI trainers. Ambiguous guidelines lead to inconsistent annotations, which ultimately undermines the performance of your frontier model. This section outlines the key components of effective instruction design, ensuring your team can accurately and efficiently generate the high-quality data you need.

A comprehensive instruction set begins with a clear overview of the task. Explain the project's overall goal and how the annotations contribute to achieving that goal. Frame the task within the context of the larger project, emphasizing the importance of accurate labeling. For our legal case review example, this overview might explain how the labeled data will train a model to assist lawyers in quickly identifying key information within case files, ultimately saving time and resources.

Next, provide a detailed explanation of the ontology and all relevant terminology. Clearly define each label, category, or rating, avoiding jargon or technical terms that the labelers might not understand. Use simple language and provide real-world examples to illustrate each concept. For instance, instead of simply defining "legal precedent," explain it with a concrete example: "Legal precedent refers to a previous court decision that serves as a guide for similar cases in the future. For example, the 1954 Supreme Court case Brown v. Board of Education established a precedent for desegregating public schools." Provide multiple examples for each category to cover a range of scenarios. For financial analysis, this might include defining terms like "bull market," "bear market," and "market volatility," each with illustrative examples from real-world financial news.

Addressing edge cases and exceptions is crucial. Anticipate situations where the correct label might be unclear or ambiguous. Provide specific guidance on how to handle these situations. And it’s important to provide guidance on what a trainer should do if they don’t feel qualified or confident in a given task. The instructions should clearly state: "If you are unsure about the correct label, or if you encounter a situation not covered in these guidelines, do not guess. Instead, flag the case for review by a subject matter expert." This emphasis on accuracy over completeness is essential for maintaining data quality.

Finally, establish a clear process for feedback, questions, and clarification. Provide a designated channel for labelers to ask questions, report issues, or suggest improvements to the guidelines. Regularly review and incorporate feedback to refine the instructions and address any ambiguities that arise during the annotation process. This iterative approach ensures that the labeling process becomes more accurate and efficient over time. A well-defined communication channel also empowers labelers to contribute valuable insights based on their experience with the data.

3. Create a new project and select the right editor

You're ready to create your project within the Labelbox platform. Begin by logging into your Labelbox account. Once logged in, click “Create project.”

The first step is to select the appropriate data modality or task type for your project. This choice is crucial as it determines the tools and interface available to your labelers. For industry-specific reasoning projects involving long-form text analysis, such as reviewing legal documents or analyzing financial reports, the Text editor is often the most suitable option. 

Create a new project in Labelbox by selecting the appropriate data modality or task type

Labelbox also supports other modalities like image, video, audio, and multimodal chat, allowing you to adapt to various data types. For example, if your project involves evaluating the quality of responses from a large language model in a live chat conversation, you may use the multimodal chat editor. If your project involves processing audio recordings of earnings calls, you may select audio. Select the modality that best reflects the nature of your data.

Provide your project with a descriptive and informative name. This will help you easily identify and manage your project within Labelbox. For our legal case review example, a name like "Legal Case Review - Contract Analysis" would be appropriate. For financial analysis, it might be “Financial Report Analysis - Risk Assessment.”

Finally, upload any existing data that you plan to use in the project. Labelbox supports various data formats, making it easy to import your data. This might include PDFs, plain text files, CSV files, or even connections to cloud storage. Consider organizing your data into logical batches or datasets before uploading to simplify project management. While uploading, ensure that your data is properly formatted and structured for optimal use within the Labelbox platform. For large datasets, consider using Labelbox's data import capabilities to streamline the process.

4. Customize your ontology

A well-defined ontology is the backbone of your labeling project. It provides the structure and vocabulary for your AI trainers, ensuring consistency and accuracy in their annotations.

Navigate to the "Ontology" tab within your newly created project. This is where you'll define the building blocks of your annotation schema.

Example of an ontology configured to collect both ratings and free responses on a given datarow

Clean, thoughtful ontologies help create high-quality labeled data with minimal errors and inconsistencies. Ontologies are an essential part of the Labelbox labeling platform. Every time you create a project or a model in Labelbox, you will need to select an ontology.

Along with the core objects, classifications, and relationships that you’ll use in the text editor, you can also establish hierarchical relationships between classes if it makes sense for your domain. This allows you to create a more structured and organized ontology. For instance, "Breach of Contract" could be a subclass of a more general class called "Contractual Issue." Hierarchical relationships can help your model learn more general concepts from specific examples. They can also make the labeling process more efficient by allowing labelers to quickly navigate through related concepts. For complex ontologies, consider using Labelbox's hierarchical labeling features to simplify the annotation process.

For most of today’s complex tasks for generative AI, consider incorporating free text fields to capture richer insights from your AI trainers. These fields give the trainers the ability to rewrite prompts or responses, provide detailed feedback, and explain the rationale behind their ratings or labels. This qualitative information can be invaluable for understanding model behavior and improving its performance. 

For instance, a free text field could allow a labeler to explain why they chose a particular label, highlight ambiguities in the data, or suggest improvements to the ontology. This feedback loop is crucial for refining your model and ensuring it aligns with your desired outcomes.

5. Execute the project: label, rate and align 

With your project configured, data loaded, and a robust ontology in place, you're ready to invite your team to begin executing on the tasks and generating new training data.

First, invite your selected AI trainers to the project. Labelbox makes it easy to add team members and assign them specific roles. Ensure that each annotator has the necessary training and understands the project's goals, the ontology, and the annotation guidelines. Consider providing a brief onboarding session to familiarize annotators with the Labelbox platform and the specific requirements of your project.

Next, assign data batches to the annotators. Organize your data into manageable batches to streamline the annotation workflow. Labelbox provides tools for batch management, allowing you to distribute data evenly among your team members. Consider assigning smaller batches initially to allow for early feedback and adjustments to the annotation process. As annotators become more proficient, you can increase the batch size.

Monitor the labeling progress regularly. The Labelbox workspace monitor provides dashboards and reporting tools that allow you to track the progress of each annotator and identify any potential bottlenecks or issues. Regular monitoring also allows you to provide timely feedback to annotators and address any questions or concerns they may have. This proactive approach helps maintain data quality and ensures that the project stays on track.

Utilize Consensus, Benchmarks, and Calibration to ensure data quality. These features are essential for maintaining consistency and accuracy in your annotations. Consensus involves having multiple annotators label the same data points and then comparing their annotations. Discrepancies can be discussed and resolved, leading to higher quality data. 

Benchmarks are pre-labeled data points that serve as a gold standard for evaluating annotator performance. Regularly testing annotators on benchmarks can help identify areas where they may need additional training or guidance. 

Calibration is the process of adjusting the annotation guidelines or training materials based on feedback from annotators and insights gained from consensus and benchmark analysis. This iterative approach ensures that your data quality continuously improves throughout the project lifecycle. By actively managing your team and implementing these quality control measures, you can generate the high-quality training data needed to power your frontier models.

6. Review and perform QA

Even with well-defined instructions and diligent annotators, a robust review process is essential for guaranteeing the highest quality training data.

Establishing a strong review process within Labelbox allows you to implement multiple layers of quality control. You can configure reviews on a per-project basis, tailoring the process to the specific needs of your task. For example, you might require all annotations to be reviewed by a subject matter expert before being accepted into the training dataset. Alternatively, you could implement a tiered review system, where a subset of annotations are reviewed by a senior annotator, and only those that meet a certain quality threshold are then passed on to a subject matter expert for final review. Labelbox's review workflows can be customized to fit your specific requirements, allowing you to create a scalable and efficient quality control system. Consider implementing a system where annotations are reviewed by a different annotator than the one who originally labeled the data. This helps to catch potential biases or inconsistencies.

Beyond manual review, Labelbox offers AutoQA features that can significantly improve data quality. AutoQA leverages machine learning models to automatically identify potential errors or inconsistencies in your annotations. These features can flag annotations that deviate significantly from the consensus, highlight areas where annotators disagree, or identify annotations that are inconsistent with pre-defined rules or constraints. By proactively identifying potential issues, AutoQA allows you to focus your review efforts on the most critical areas, saving time and resources. 

By combining manual review with AutoQA, you can create a comprehensive quality assurance system that ensures your training data is accurate, consistent, and reliable. This, in turn, will lead to better performing frontier models capable of industry-specific reasoning. 

Disrupting your industry with advanced AI capabilities

The pursuit of AI excellence demands a shift from generic to specialized. As we've explored in this guide, building industry-specific reasoning into LLMs and AI models is not just a technical challenge, but a strategic imperative. By leveraging platforms like Labelbox and embracing a data-centric approach, companies can unlock the true potential of AI and create models that are not just intelligent, but also insightful, reliable, and tailored to the unique demands of their respective industries.

This guide has provided a roadmap for embarking on this transformative journey. From crafting clear instructions and building robust ontologies to selecting the right AI trainers and leveraging the powerful features of the Labelbox platform, you now have the foundational knowledge to create your own industry-specific reasoning projects. Remember that the key to success lies in a meticulous, iterative approach, where continuous learning and improvement are paramount.

As you venture forth, keep in mind that the landscape of AI is constantly evolving. Stay curious, embrace new challenges, and never stop refining your approach. The future of AI is not just about building smarter models, but about building models that truly understand the world in all its specialized complexity. And with Labelbox as your partner, you're well-equipped to lead the charge toward a future where AI is not just a tool, but a true industry expert.

Additional resources:

• Labelbox documentation
• Labelbox customer stories