Customer service systems are prime targets for multimodal chat as this new technology leverages an intuitive interface for answering user questions  while enabling companies to unlock previously hidden insights about customer journeys and sentiment.

With Labelbox, you can streamline the process of labeling and managing data for various use cases, including intent classification, dialogue retrieval, state tracking, and model evaluation. 

In this guide we’ll show how Labelbox can be used to collect training data for an ecommerce chatbot that responds to customer queries about online shopping. We’ll illustrate the various labeling tasks that Labelbox can accomplish to ensure the chatbot is intelligent and effective, providing customers with relevant and timely answers. 

Labelbox offers a number of editor types, including the live multimodal chat editor.

Labeling Tasks

Multimodal Intent Classification, Dialogue Retrieval and State Tracking

Part 1: Generate prompt-response pairs for supervised fine-tuning of agentic workflows.

Label the customer’s intent and sentiments in a given turn of the chat, so you can identify the appropriate next steps accordingly.

The customer appears frustrated, so the sentiment is labeled as “frustrated.”

Furthermore, the customer appears to be reporting their order was damaged and is requesting a new one, so both of those requests are labeled using checklist classifications.

With these labels, you know the customer’s intent (get a new piece) and can also record the current state of the conversation (damage reported but no action taken yet). As the conversation proceeds, you can update the state based on what actions are taken and keep checking the intent and tone of the customer to ensure they are satisfied.

Labelbox allows you to label every aspect of every turn-by-turn conversation, enabling you to develop fine-grained data sets of annotated real-world conversations.

You can use these labels to train your agent to retrieve similar past interactions and follow similar response patterns or simply be inspired by them.

Model evaluation, ranking and selection

You trained a few models offline with the previously generated data using supervised fine-tuning or RLHF. How do you know which of the models is the best performing, and how often do they make mistakes?

Labelbox allows you to connect these models hosted behind your custom endpoints. Then, with the Live Multimodal Chat editor, labelers can interact with all of them simultaneously by feeding prompts, comparing and contrasting responses, and even rewriting the best ones to be even better.

Part 2: Evaluate a single model, generate multiple responses and write your own gold standard response.

When evaluating a single model, Labelbox allows you to generate multiple responses and also add your own response. This is very useful for use cases where all of the model’s responses are lacking in quality and a human-written response provides the gold standard for further downstream training. Furthermore, in the course of the labeling conversation, you can choose one of these responses to be passed in as prior context into later turns, allowing you to choose the path of the conversation in real-time for better training data generation.

Part 3: Evaluate multiple models at once in “chat arena” style.

Labelbox also supports evaluating multiple models in a “chat arena” style setup.

Say you have trained candidate models and now want to compare them against each other and also foundation models like GPT 4o. Connect your own models with Labelbox as custom models, and easily choose any latest and greatest foundation model from Foundry within Labelbox. The chat arena excludes information about which response was generated from which model so as not to bias the labeling process.

Conclusion

With Labelbox's multimodal chat functionalities, you can generate high-quality datasets to train, evaluate, and test your custom LLMs or pit off-the-shelf LLMs like GPT4o and Gemini 1.5 against each other in a Chatbot Arena-style competition to see which one suits your needs best.

Part 4: Survey of all LLM capabilities in Labelbox

As the landscape of LLMs evolves rapidly, Labelbox will continue to build functionality to ensure our customers can align models to their use cases quickly and reliably. With differentiated data, streamlined operations, and transparent orchestration, you can build and refine foundation models that meet your specific needs.

Evaluate multimodal chat data today for free

Ready to deliver on the next-generation of generative AI? Sign up for a free Labelbox account to try it out or contact us to learn more and we’d love to hear from you.