Evaluating terrain-dependent performance for martian frost detection in visible satellite observations

Summary: Researchers from Jet Propulsion Labs recently studied seasonal frosting and defrosting on the surface of Mars which is hypothesized to drive both climate processes and the formation and evolution of geomorphological features such as gullies.

Challenge: Past studies have focused on manually analyzing the behavior of the frost cycle in the northern mid-latitude region of Mars using high-resolution visible observations from orbit. Extending these studies globally requires automating the detection of frost using data science techniques such as convolutional neural networks. However, visible indications of frost presence can vary significantly depending on the geologic context on which the frost is superimposed.

Unlike the Earth, the atmosphere of Mars is comprised primarily of carbon dioxide (CO2) and this volatile constitutes most of the frost, falling as snow or condensing at the surface due to surface temperatures falling to the CO2 frost point. A small amount of water frost will also form when temperatures are below the water (H2O) frost point, but only if the local concentration of H2O vapor in the atmosphere is high enough.

Findings: The research team from JPL presented a novel approach for spatially partitioning data to reduce biases in model performance estimation, illustrate how geologic context affects automated frost detection, and propose future work to further mitigate observed biases in automated frost detection work.

They found that geologic context bias is present and significant for this model’s performance on the test set, specifically for dune fields often found in northern mid-latitude craters. Interestingly, for human annotators, dunes often provide strong evidence of frost due to the striking visual appearance of defrosting marks which expose dark basalt sand beneath light-color frost.

They also saw a large degree of diversity in frost appearance on the underrepresented terrain type, both inherently and due to differing illumination and observational conditions, making the concept challenging for the classification model to learn.

Future improvements will permit the training of models better suited for full-planet frost detection, thereby facilitating the creation of global frost maps

How Labelbox was used: Labelbox was used to annotate polygonal boundaries around regions with visible evidence of frost. For each polygon, they collected additional information from the labeler including the applicable visible indicators as well as geologic context, which is either “dunes,” “gullies,” “crater rim/wall,” or “other.” The geologic context categories was mutually exclusive, so labelers could only pick one geologic context per frost polygon.

In addition, they used the geologic context information to investigate terrain-dependent bias in classifier performance. To document the labeling process, they performed an iterative series of labeling sessions with both data science and science domain experts.

Domain expert labeling guidance and clarifications at each iteration were captured in a labeling guide, included with the publicly available dataset2. A total of 6 subframes, detailed in the released dataset, were excluded due to contamination with excess instrument noise or cloud cover. Each subframe was labeled by three different annotators.

Read the full PDF here.