A Sensor Fusion Platform for Semantic Segmentation of Outdoor Ice Scenes

Abstract

Due to climate change and global warming, Arctic sea ice coverage has been on a downwards trend that is expected to continue late into this century, opening up new shipping routes through the Arctic. These routes provide opportunities for much shorter transoceanic journeys compared to currently used routes, offering significant savings in transportation time and resource cost. However, travelling through ice-covered waters still provides a constant source of risk from collisions with ice damaging or even sinking the vessel. Thus, modern tools such as ice charts, satellite imagery, and near-field observations from on-board experts are still used to avoid dangerous collisions with ice. In adverse weather and lighting conditions, near-field human observations can be uncertain, causing inaccuracies in determining nearby ice conditions.

This thesis introduces a system composed of an optical RGB camera, a thermal infrared camera, and a polarization camera to aid in such near-field observations for the purposes of determining nearby ice conditions in a variety of outdoor conditions. The novel sensor suite was tested during field trials conducted in February 2025, collecting river ice data across multiple days and at different locations along the Ottawa and St. Lawrence rivers. After image registration and post processing, a first-of-its-kind, fully labelled semantic segmentation dataset of 118 images across 5 scenes was created to test different methods of sensor fusion. For testing, the test set was further split into easy and hard subsets based on perceived clarity to the human eye. Early, middle, and late fusion networks using different combinations of sensor inputs were created based on modifying the widely used fully convolutional neural network U-Net architecture with a pre-trained ResNet-18 backbone.

Experimental results show that early fusion methods performed consistently worse than the baseline case of only using the optical RGB data on both the easy and hard test sets, regardless of the other input sources used, most likely due to the higher levels of noise being introduced by the additional sources. Middle fusion with polarization and thermal managed to outperform the baseline on the hard test set, and late fusion with polarization managed to outperform on the easy test set. While both middle and late fusion methods show improvements over the baseline through extracting useful information even from noisy sources, late fusion with polarization had the highest overall mIoU improvement on the full test set due to an enhanced ability to differentiate between brash ice and water.

Ultimately, sensor fusion shows potential for improving sea ice classification accuracy while being robust to a variety of different environmental and lighting conditions. These preliminary results serve to support continued development of sensor fusion platforms as tools to aid in the tracking and identification of nearby ice conditions.