Hardening an LLM's STEM reasoning with expert multimodal signal
Problem
A leading AI lab wanted to find where its LLM failed on K-12 STEM. To do that, it needed original, domain-specific multimodal prompts and expert-graded answers that push the model's limits — and the deep STEM expertise to produce them at scale, feeding a real-time training loop that lacked a consistent source of qualified feedback.
Solution
Through its Alignerr network, Labelbox's platform produced the signal — drawing on 150 vetted STEM experts with advanced degrees (PhDs and Master's) in fields like chemistry, biology, and engineering. They created original multimodal (text and image) prompts and accurate answers to evaluate and improve the model's responses.
Result
Labelbox's platform consistently produced unique multimodal reasoning prompts that exposed the model's limitations, letting the lab target key areas and drive performance gains. Labelbox is now a fully integrated part of the lab's real-time loss training workflow, delivering high-quality, domain-specific STEM signal.
A frontier AI lab needed to find and fix where its LLM failed on K-12 STEM. Labelbox's platform produced adversarial multimodal prompts and expert-graded answers that fed the lab's real-time training loop.
The challenge
A leading AI lab wanted to find where its LLM failed on elementary, middle, and high school (K-12) STEM. The plan: generate complex, differentiated multimodal (image and text) prompts that pinpoint exactly where the model breaks, and feed them into a real-time loss workflow. That required original, domain-specific image-text pairs across biology, physics, engineering, and earth sciences — and deep expertise the lab couldn't source at scale on its own. The workflow also lacked a consistent source of qualified human feedback to evaluate the model and surface its weaknesses.
The approach
Labelbox produced the signal. Through its Alignerr network — spanning industry domains, languages, and experts worldwide — the platform captured judgment from 150 vetted STEM specialists with PhDs and Master's degrees in fields like engineering, math, and physics, selected from hundreds screened. A 24-hour calibration period and end-to-end project management got the signal flowing fast. This was original thinking, not simple labeling: experts generated complex multimodal prompts across STEM fields and grade levels, tuned them until they pushed the LLM to its limits, then wrote accurate responses to train the model. Prompts had to be original — not searchable or available online. A prompt only counted as a "winning label" if the model answered incorrectly multiple times. Labelbox's multimodal chat editor carried the expert feedback, encoded clear instructions, and let experts evaluate generated prompts directly against the lab's own LLM.
My advanced mathematics degree and AP teaching experiences helped me craft nuanced and novel questions that challenged existing AI models. It was exciting to generate multimodal datasets and challenged me to think about certain topics differently. My domain expertise was crucial to creating impactful datasets that will help advance the capabilities of AI.
— Derek H., Math master's and AP math teacher
The outcome
After multiple rounds of evaluation and expert review, Labelbox delivered a new multimodal dataset that significantly improved the lab's model performance on complex STEM questions. The lab now has an efficient real-time loss workflow that continuously identifies weaknesses in STEM queries and feeds precise improvements back into the LLM. Labelbox is a fully integrated part of that loop.
Where this goes
This is the learning loop frontier labs run on: expert-graded signal that finds a model's failure modes, then trains them away. Human expertise becomes structured signal becomes model improvement.