What we’ve been up to

We’ve heard loud and clear from industry leaders that improvements to frontier and task-specific AI models depend on high-quality, differentiated data. Training on existing data is no longer enough. AI labs and high-tech, digital native enterprises need to tap into human experts to build new datasets across a wide range of subject matters to support RLHF, supervised fine tuning, and other training techniques. 

In response to that feedback, we’ve been quietly building and powering GenAI labeling during the last few quarters! This past quarter, we delivered major innovations and continued to execute on our goal of delivering the data factory for generating high-quality data for GenAI solutions. Manu Sharma, Labelbox Co-founder and CEO, recently shared his insights on the future of AI and Labelbox with Cerebral Valley in a must read article.

To support the vision outlined by Manu, this quarter we’ve expanded our Labelbox Labeling Services and enhanced our core Labelbox platform to help AI-focused companies do three key things: 

• Accelerate the speed of data delivery with our expanded labeling services composed of highly-skilled, on-demand experts
• Generate high-quality data through new platform innovations to simplify three key RLHF jobs: LLM evaluation, preference rankings, and fine tuning dataset creation
• Improve control of labeling tasks with enhanced usability and streamlined operations

Accelerate the speed of data delivery

We understand the challenges AI companies face in generating new, differentiated datasets to train and improve existing models, whether building the next frontier model or adapting existing LLMs for a task-specific application. Existing high-quality data has essentially all been used. Building new datasets is costly and time-consuming. Unfortunately, existing offerings in the market can lack the quality needed by advanced AI teams and offer limited control or insights into the data generation process. 

In response to those challenges, we continue to invest in growing our Labelbox Labeling Services to meet the throughput and scale demands of the most demanding projects. Labeling services are supported by the rapidly growing Alignerr network, a community operated by Labelbox and composed of subject matter experts from numerous disciplines. Alignerrs are available on-demand to align AI models and create high-quality data in their field of expertise. The community spans nearly every major discipline of sciences, industries and languages, worldwide.

The labeling services are designed to take you from initiation to production-scale labeling in just a few days. We aim to deliver a 24 hour turnaround time on calibration data and production data in 5 days.

The Alignerrs use the same Labelbox Platform you are familiar with, providing advanced capabilities around control through real-time collaboration, unified platform for internal and external raters, and advanced metrics and analytics. 

The services specialize in critical tasks needed by today’s AI leaders: 

• Rating & Ranking: assess multiple AI-generated options and determine the most suitable model responses, ensuring the content reaches a high-quality standard for review and providing a rationale for the choices based on specific criteria or factors.
• Open Rewrite: use an initial premise or context for an AI model to elaborate on, depending on the intended outcome and task requirements. These prompts typically encompass straightforward sentences, inquiries, or detailed descriptions aimed at improving a response from the AI model.
• Multimodal: assist in training AI models to process and categorize data from various sources like images, sound, or video. For instance, experts might teach a model to create a story or description based on what it sees in pictures.

These tasks are top of mind for companies building and training frontier models. In addition, digital native enterprises delivering solutions powered by AI also need help building industry- or domain-specific datasets to further train and refine the capabilities of their preferred LLM. 

Generate high-quality data for critical RLHF jobs

As AI builders shift their focus to enhancing and improving existing LLMs, they not only need the services of expert raters to create data, but they need a powerful platform with the tools, controls and analytics to deliver the highest quality data possible. Real-time collaboration is essential to quality. Automation and AI-assistance increase efficiency and time to value. 

The Labelbox Platform is the essential software that powers our tech-enabled services. This quarter we focused on capabilities for key RLHF jobs:  

• LLM evaluation: Evaluate the performance of multiple models together in a multi-modal, multi-turn evaluation either online or offline.
• Preference rankings: Choose the preferred response from multiple model outputs or generate a new, preferred response when needed. 
• Fine tuning dataset creation: Engage with highly-skilled teams to generate an ideal response to a provided prompt, such as for complex reasoning. 

New platform features include the following:

• Enhanced live multimodal editor: These updates are the direct result of new multimodal models, like GPT-4o and Gemini 1.5 Pro, that require multimodal data labeling. The new multimodal editor is a multimodal, multi-turn, and multi-model tool, meaning that you can submit prompts about images, video, documents (PDF) and more; converse in a seamless, multi-turn conversation; and interact with multiple models simultaneously.
• Offline multimodal conversations: Expanding on the initial feature set of the multimodal editor, we added the ability to import conversational data with rows to review and annotate multi-turn conversations with an LLM offline. 
• LaTeX support: LaTeX, a powerful typesetting system for creating complex mathematical formulas and expressions, allows users to seamlessly integrate mathematical notation into their training data, enhancing the capabilities of AI models for tasks such as scientific document analysis, mathematical problem solving, and formula recognition.
• Benchmark & consensus support for LLM training: Benchmark and consensus are two core tools designed to evaluate annotation quality. This past quarter we extended those features to our LLM response evaluation editor to further improve the quality when doing tasks like LLM preference ranking.  
• Llama 3.1 and Claude 3.5 Sonnet support: We continue to test and integrate the latest industry standard models, like Llama 3.1 and support for Claude 3.5 Sonnet, into our catalog of models to ensure users can use and compare to the latest models on the market. 

• Granular video labeling: For video projects, you can now assign free-form text classifications to individual frames or a range of frames.

These features work together to create a powerful data factory. They help both internal and external raters deliver high-quality data efficiently and quickly, empowering teams to meet their RLHF timelines and continue the rapid pace of AI innovation that we expect. 

Improve the control of labeling with enhanced usability 

We also recognize the need for program managers and leaders to have explicit control and visibility over their data labeling and generation projects. This requires a unified platform that enables everything from real-time collaboration with labeling teams to an intuitive user interface that simplifies onboarding anyone, whether they are AI and ML experts, new labelers, or SME (subject matter expert) reviewers. 

We remain committed to a software-first approach in order to deliver the best tech-enabled services on the market. A number of enhancements, big and small, were made to the platform over the past quarter to streamline operations and improve usability. These enhancements include:

• Exports V2 priority queue: We further enhanced our new scalable, performant method for exporting data by implementing a fair (priority) queueing system that allows all exports to proceed equally, regardless of size. We also put a limit on the resources Exports uses at any given time to prevent additional stress.
• Simple annotations import: Inconsistent naming, artificial throttling, nested annotations, and more make importing existing annotations challenging. We enhanced the in-product experience, added new methods to our SDK for simple uploads, and added auto-code generation for importing ground truth or predictions as the ontology level. To see the updated import formats, click through the links at the top of the import ground truth doc.
• Resizable editor side panels: A small but mighty feature, this UI tweak provides a bit more flexibility and control to users so that they can see and access the information and fields they need to, helping increase efficiency and satisfaction. 
• Improved tenant and admin management: Admins can define their own user roles with any set of permissions that Labelbox provides and allow those roles to be assigned to any user. We also introduced a new role called tenant admin, which has the permission to create new workspaces and invite team members to any workspace. To learn more, see our docs on Member roles.  
• Custom roles for enterprises: Before, roles were static in Labelbox. With this new feature, customers can define their own roles, with any set of permissions that Labelbox can provide. These roles can be assigned to any users.  

These advancements create a more efficient and scalable AI development workflow for AI leaders.

Build with us: Training and guides

To further support our customers in their AI development journey, we've created a wealth of resources available online and on-demand:

Technical guides 

Our in-depth guides help you navigate various aspects of AI development. This quarter we delivered these new guides: 

• Improving data quality via AutoQA & advanced labeler review: Learn how you can use Labelbox to accelerate the quality review process for creating better data for generative AI use cases using autoQA and advanced labeler reviews.
• Using multimodal chat to enhance a customer’s online support experience: See how Labelbox can be used to collect training data for an ecommerce chatbot that responds to customer queries about online shopping.
• Working with videos using Gemini 1.5 and multimodal models: Learn how easy it is to convert videos to 1FPS and upload them to Labelbox Catalog for generating predictions in Model Foundry. 
• AI foundations: Understanding embeddings: Uncover how to utilize embeddings for data vector representations and discover key use cases at Labelbox, including uploading custom embeddings for optimized performance.
• What is multimodal data labeling?: Discover why, thanks to new models like GPT-4o and Gemini 1.5 Pro, multimodal data labeling is now fundamental to the AI development process and how you can use it to improve your AI applications. 

Interactive product tours 

For anyone who wants to see the Labelbox platform in action but only has a few minutes, our new interactive product tours are just the answers. 

These interactive tours only last one or two minutes, so check them out to see what you can do in Labelbox:

• Multimodal chat editor for high-quality GenAI data
• GenAI single LLM response evaluation
• GenAI prompt/response generation & labeling
• Request Labelbox Labeling Services 

Connect with us: webinars, events & community 

We offer various opportunities for users to connect, learn, and engage with the AI community through informative webinars, industry events, and a thriving user community.

Webinars 

• Developing better chatbots with AI-assisted labeling & LangSmith: In this webinar, we explore how Labelbox's AI-assisted annotation capabilities and integration with LangSmith can help you build and optimize chatbot applications more effectively. 

Events 

• CVPR (IEEE / CVF Computer Vision and Pattern Recognition Conference): Read through our blog that provides a recap of CVPR 2024 and an insightful look at what the focus on multimodal, generative, and 3D AI means for the future at CV.
• Data + AI Summit by Databricks: If we missed you at the event, then don’t worry. You can reach out to us anytime for a quick chat or ready about how to seamlessly integrate Databricks pipelines with Labelbox.

Community

And finally, did you know we have a vibrant Community of users sharing invaluable tips, technical assistance, and insightful How-To guides on building cool projects with custom data? For example, discover how to convert LangSmith results to Labelbox conversation data or how to fine-tune your models. Sign up for a free account and join the conversation here!

Conclusion

Our Q2 innovations and releases help organizations build and deploy cutting-edge AI systems with high-quality, speed, and control. We’ve taken a big step forward in helping AI companies address the need for new, differentiated datasets for training and RLHF through our expanded Labelbox Labeling Services, which is now supporrted by the worldwide network of highly-skilled Alignerrs.  

In the platform, we’ve made significant enhancements around our multimodal capabilities for GenAI and built advanced workflows to enable online and offline LLM evaluations, preference rankings, and new dataset creation for LLMs. 

As we continue to push the boundaries of what's possible in AI development, organizations can expect even more groundbreaking features and enhancements in the future, cementing Labelbox's position as the data factory for GenAI builders and users.