Earth Systems Lab 2024 - SAR-FM: A Foundation Model for SAR

Project Background

Although large amounts of satellite imagery are available globally, the task of manually labeling this data for machine learning applications is both time-consuming and expensive. Many regions, especially in developing countries, suffer from an imbalance in data availability. For example, regions such as the United States and Europe benefit from vast amounts of labeled satellite data, while parts of Africa, South America, and Asia remain underrepresented in Earth observation (EO) datasets. This imbalance in data availability makes it difficult to develop accurate, region-specific EO models, which are crucial for tasks such as vegetation monitoring, urban growth tracking and environmental assessments. However, Earth observation foundation models (EO-FMs), which have been pre-trained on large datasets from data-rich regions, have the potential to perform well in low-data environments by transferring knowledge.

The challenge of generalising an FM across spatial regions is inherently complex due to the vast differences in environmental features between regions. For instance, vegetation in tropical regions such as Colombia behaves differently from vegetation in arid regions or mountainous regions. A model trained on data from one region may fail to capture the specific characteristics of vegetation, climate and topography when applied to another region, leading to poor performance. The difficulty in generalising across regions is further exacerbated by the variability in the spectral characteristics of satellite imagery, which can differ based on the type of sensor used. For example, SAR data from the Sentinel 1 satellite and optical data from the Sentinel 2 satellite have different properties, making it challenging for a model to perform equally well on both types of data.

Project Results

The team performed two ablation studies - firstly by using the three large AOIs for training models, and testing on target AOIs; and secondly, by using part of the data in the target AOIs for training, and testing on the remaining target AOI data. This resulted in around 18,000 models when using external AOIs for training, and around 7,000 models when splitting target AOIs for train and test. Each experiment was repeated 20 times, resampling the train and test data each time and reporting the mean and standard deviation of the metric, to result in a total of some 500,000 trained models. The below diagram shows the results of these experiments for each of the eight EO-FMs against the seven classifications of lancer for each AOI. Bluer positions represent greater correlation between predictions and targets, with redder positions representing worse correlation. The black horizontal line splits indicate data derived from S1 (top) and S2 (bottom).

These plots showed tree cover and permanent water exhibited reasonable generalisability to non-US and non-European AOIs, however, models for permanent water trained in Europe seem to show better correlation in California and Texas, but only if data from S2 was used. Other results proved very AOI specific - cropland showed some signal when transferred between the US, Europe and Turkey; and models trained in China applied well in India. The team believed this may be due to the inherent differences between AOIs where classes probably gather a large variety of land features - for example, crops of palm trees look very different from crops of soy. Shrubland showed very low performance and no spatial generalisability at all and grassland had limited signal within the US and Europe. The team also observed some results that seemed inherently dependent on the sensor. For instance, generalisability in Peru and India for bare / sparse vegetation occurred when using S2 FMs regardless where the models were trained, and buildup seems more generalisable with S1 FMs (bluer above the black horizontal line), which was in line with the expected interaction of a SAR signal with man-made objects.

As previously mentioned, to address adaptability of the eight EO-FMs, the team used a LoRA fine-tuning experiment where numerous ranks (r-8, r64, r-256, r-512 and r-1024) were applied to the base model. The rank of a model represents the maximum number of linearly independent rows or columns in the model - the higher the rank, the more parameters in the model and the more computational power will be required. They found LoRA fine-tuning demonstrated superior performance across all rank values, but at the cost of increased GPU memory usage and trainable parameters. However, the performance uplift was not linear, with rank 256 achieving the best overall performance, with an accuracy of 94.29%.