A hierarchical optical coherence tomography annotation workflow with crowds and medical experts

Challenge: Researchers from Genentech are researching solutions to the heavy burden it takes to annotate clinical images due to the cost and availability of medical experts. To address this problem, they proposed a hierarchical annotation workflow in which medical experts review aggregated crowdsourced annotations, using dense annotation of optical coherence tomography (OCT) images from age-related macular degeneration (AMD) patients as an example.

Findings: The proposed hierarchical annotation workflow with crowds and medical experts could reduce the burden on medical experts in extensive clinical annotation tasks.

How Labelbox was used: All annotation was performed on Labelbox which distributed images to remote crowds and medical experts. OCT B-scans with ≥ 9 averages from 20x20° volume scans of AMD patients were randomly selected. Two medical experts annotated 25 representative B-scans with rich pathology. In a training session, 27 labelers read through an annotation guideline and practiced on 15 of the B-scans. B-scans with color-coded agreements and disagreements were presented to the crowd, visualizing their discrepancy to the expert annotations.

Read the full paper here.