Predicting brand trust by encoding expert judgment into models
Problem
The ML team needed scalable ways to collect text from the web and enrich it into signal that would help its models learn and accurately predict consumer trust in brands.
Solution
Edelman DxI used Labelbox to produce its training signal — selecting contributors, guiding what to annotate, and assessing signal quality and performance.
Result
Edelman now has the infrastructure to continuously collect and iterate on its training signal as projects scale, with one solution serving its data science and ML team and releasing projects on tight timelines without delays.
Edelman is building proprietary models to score consumer trust in brands. Labelbox produced the training signal and folded in domain-expert judgment about what trust looks like.
Note: This post is a shortened recap of a virtual talk from David Bartram Shaw (SVP of Global Head of Data Science & Machine Learning at Edelman) during Labelbox Accelerate (Nov 2022).
The challenge
For more than 20 years, Edelman has pioneered trust research, turning complex data into real-world insights. Its analytics consultancy, Edelman Data & Intelligence (DxI), is building models to find what drives consumer trust in brands and whether ML can detect it from the vast unstructured text on the web. Strong trust-scoring models don't yet exist, so DxI is building proprietary trust algorithms trained on thousands of data points from earned media, social content, and marketing research. The challenge is signal: collecting and enriching that data at scale, and building a representative sample to reduce overfitting and classification bias.
The approach
Edelman DxI used Labelbox to produce its training signal — selecting contributors, guiding what to annotate, and assessing label quality and performance. The Python SDK tied into its AWS data lake, so projects and datasets could be created programmatically and metadata added in an object-oriented way. The key step was folding in internal domain experts who carry institutional knowledge of brand trust, answering questions like “what indicates trust?” and “how do you spot a crisis?” to refine the signal.
one of the most important components is around domain expertise. Labeling is expensive. It's less expensive than it used to be thanks to platforms like Labelbox, but it’s still a time intensive process, so we first need to incorporate the domain expertise that we have across our business and our clients and then pull that into ML projects. This enables us to not just rely on labeled data, but we're also relying on our subject matter experts when we incorporate their insights into our ML models.
— David Bartram Shaw, SVP, Global Head of Data Science & Machine Learning at Edelman
The outcome
Using weak supervision and intelligent sampling, Edelman DxI built a data-centric process to produce high-quality signal with minimal effort. It trained multiple production-grade models on tight timelines and now has the infrastructure to continuously collect and iterate on its signal as projects scale — one solution serving the whole DxI data science and ML team.
Where this goes
Trust is a judgment, not a label. Encoding expert judgment as signal — not just annotations — is how you train a model to predict something as human as trust.