Introduction

A leading AI lab developing cutting-edge LLMs sought to enhance its models' industry-specific complex reasoning capabilities to improve performance, trustworthiness, and accuracy in responding to financial-related queries. The company specifically sought to train the LLM to provide meaningful insights on any public company using a ticker symbol and the latest financial reports. In addition, they wanted the model to accurately answer common questions financial analysts might ask.

However, the complexity and domain-specific nature of the data labeling tasks posed a significant challenge for the company. Lacking the necessary expertise at scale and facing a tight deadline, they turned to Labelbox Labeling Services to source, onboard, and manage a complex project with financial experts, who all needed to have advanced qualifications including CFA, MBAs, Masters and PhDs in Finance, to evaluate and rank response quality. 

Creating a team of financial experts

Labelbox’s Labeling Services, powered by Alignerr, covers all major industry-specific domains and supports a wide range of languages. With the ability to quickly source experts, execute a 24-hour calibration period, and manage projects from start to finish, the AI lab relied on Labelbox to deliver high-quality, differentiated data for their financial use case. 

The unique nature of this financial project provided opportunities for domain experts to apply their knowledge and expand their skills by engaging with the larger AI community.

“As someone with a PhD in finance, I was intrigued by the opportunity to apply my financial expertise to help train AI models. I've found the work both flexible and intellectually stimulating. While the financial tasks are technically challenging, they have been incredibly rewarding and have provided a welcome mental challenge.” - Shaun C, PhD Finance 

Delivering a unique and industry-specific dataset

The AI lab shared their detailed vision for the project, which included an extensive set of documents and multi-layered instructions. Together, the AI lab and Labelbox team developed the initial task instructions and mapped out the best approach to building the project in the Labelbox platform. Using the Labelbox text editor, a custom ontology was built that included classifications and sub-classifications with numerous free-text inputs required from the labelers. 

Next came building the team. The Labelbox team reviewed over 50 candidates to assemble the most qualified group for the project, and then selected experts were tasked with ranking various aspects of the model’s generated outputs to complex, hypothetical prompts on a scale of 1 to 5. These rankings included evaluations of hypotheses—such as probability, importance, and feasibility—as well as assessments of argument quality, covering categories like conclusiveness and causality. 

Throughout each evaluation phase, the AI lab was able to view and monitor performance and quality metrics to ensure a successful end result. The teams customized and adjusted the project workflows as needed and implemented feedback quickly. 

In the end, the AI lab acquired high-quality financial datasets within their tight timeframe. The datasets were used to boost the performance of their LLM and improve the accuracy and reliability of its outputs. Now, with a dedicated team of skilled financial specialists in place, the company is well-positioned to continue advancing its AI’s capabilities in tackling industry-specific, complex reasoning tasks like financial argumentation.