Nayya's models recommend the right benefits plan from a ten-minute survey. Labelbox produced the signal and the actuary-in-the-loop evaluation that keeps recommendations trustworthy.

Note: This post is a shortened recap of a virtual talk from Ishan Babbar, Lead Data Scientist at Nayya during Labelbox Accelerate (Nov 2022).

The challenge

Nayya is an AI-first company that helps people understand and choose their employer benefit plans. A ten-minute survey yields a plan recommendation tuned to the individual, with machine learning powering decision support, financial education, and bundled recommendations. To keep recommendations appropriate, Nayya runs an automated feedback loop between its customer success team and benefit providers to evaluate and explain model outputs. The hard part is the signal: producing labeled data for offline training, live prediction evaluation, and verification by subject matter experts — across insurance plans, pharmaceutical data, geospatial data, and financial wellness insights.

The approach

Nayya built a streamlined annotation workflow on Labelbox, and used the resulting signal (via a feature store) for both training and evaluation to keep up with fast-growing needs.

The annotation workflow within Labelbox allows us to do this at scale and build a repeatable process for our data scientists as well as any subject matter experts that work with us,” says Ishan Babbar, lead data scientist at Nayya.

An SDK-driven approach let its actuaries — the subject matter experts — share how the model generates predictions and evaluate them.

We use Labelbox's Python SDK almost religiously. Part of the main value we get from the platform is being able to interact with how labeling is going, performance quality metrics, and loop that information back to our actuaries and subject matter experts,” says Ishan.

With this, Nayya built reproducible pipelines into its MLOps infrastructure.

The outcome

Nayya uses Labelbox as a data engine to train models faster through streamlined signal production and collaboration. For never-labeled data, a strong QA process speeds cataloging and extraction from disparate sources. The team can visualize its data, maintain quality, save time, and train and test new models to improve its AI-first products.

Where this goes

Benefits is a high-stakes recommendation problem. Expert-verified signal, with the people who understand the domain in the loop, is what makes an AI recommendation one you can trust.