Meta-learning to address diverse Earth observation problems across resolutions

Earth science relies heavily on increasingly abundant satellite imagery data at diverse spatial, spectral, and temporal resolutions. Extracting knowledge from this raw data often involves deep learning models, which typically necessitate large annotated datasets. While some Earth observation problems (e.g., building footprint segmentation or land cover classification) benefit from extensive annotated data, many others (e.g., marine debris detection) suffer from label scarcity. Furthermore, covariate and concept shifts in satellite imagery due to geographical variations require region-specific datasets, demanding significant annotation efforts. Existing approaches often treat each Earth observation problem independently, neglecting shared knowledge between seemingly disparate tasks. This paper addresses this inefficiency by proposing METEOR, a meta-learning methodology that leverages shared knowledge between source and target problems. Transfer learning aims to utilize common knowledge between source and target problems, while meta-learning extends this by learning the learning algorithm itself. Model-based transfer learning, encoding common knowledge in deep learning model weights, has gained prominence. Recent foundation models (RingMo, SSLTransformerRS) pre-train large models on heterogeneous datasets for fine-tuning on specific downstream tasks. Similarly, meta-learning pre-trains a smaller model on a dataset of many source problems to adapt to new tasks. Two main meta-learning categories exist: metric-based and optimization-based. Metric-based approaches pre-train a feature extractor to match test samples to class prototypes. Optimization-based approaches fine-tune the entire pre-trained model to downstream tasks. While successful in specific problem families (high-resolution aerial imagery or medium-resolution multispectral imagery), current meta-learning approaches often lack the ability to transfer knowledge across highly heterogeneous problem families. METEOR aims to overcome these limitations by systematically learning across diverse Earth observation problems.

The paper reviews existing meta-learning and transfer learning approaches in remote sensing. Metric-based meta-learning, often utilizing pre-trained feature extractors, has shown promise in high-resolution imagery, particularly for addressing concept shift. Optimization-based approaches, exemplified by MAML and its variants, have been applied to medium-resolution multispectral imagery for mitigating concept and covariate shifts. However, most research focuses on single homogeneous problem families (e.g., only high-resolution aerial imagery or only medium-resolution multispectral imagery). The authors highlight the limitations of existing methods and position METEOR as a solution that bridges the gap between homogeneous and heterogeneous transfer learning, addressing the challenge of learning across different resolutions, spectral bands, and the number of classes in various remote sensing problems.

METEOR is an optimization-based meta-learning approach using a small deep learning model (ResNet-12) with instance normalization replacing batch normalization to improve performance on class-imbalanced datasets. Three key modifications enable heterogeneous transfer learning: 1) Instance normalization replaces batch normalization to handle class imbalances effectively. 2) Convolutional kernels are dynamically adjusted to adapt to varying numbers of spectral bands. 3) Problems with different class numbers are addressed using a binary meta-model, fine-tuned for each class separately and then ensembled as a one-vs-all classifier. The meta-model is pre-trained using the model-agnostic meta-learning (MAML) algorithm on land cover classification tasks from the Sen12MS dataset. The task-model, initialized with the meta-model's parameters, is then fine-tuned on the specific downstream task using few labeled examples. For segmentation tasks, the task-model is modified by replacing the final linear layer with 1x1 convolutions and removing global average pooling, generating a segmentation map that is then upscaled. The experiments involved pre-training on the Sen12MS dataset and evaluating performance on diverse downstream tasks from multiple datasets (DFC2020, EuroSAT, NWPU-RESISC, Floating Marine Objects, DENETHOR, AnthroProtect). The performance of METEOR was compared with several state-of-the-art methods, including self-supervised learning approaches (SSLTRANSRS, SSL4EO, SeCo, SWAV, DINO), a supervised baseline, and MOSAIKS. Comparisons were made using average rank and accuracy, with statistical significance testing (Wilcoxon signed rank test). Qualitative analysis, including occlusion sensitivity analysis, was also conducted to interpret METEOR's predictions.

The key findings demonstrate METEOR's superior performance and versatility across various remote sensing tasks. Experimentally, instance normalization was shown to significantly outperform batch normalization and other normalization methods (TaskNorm) on realistic, class-imbalanced datasets, highlighting its importance in addressing the challenges of real-world data. METEOR consistently achieved high accuracy and low average rank across different downstream tasks, including land cover classification, deforestation detection, urban scene classification, change detection, and marine debris segmentation. Specifically: * **Land Cover Classification:** METEOR performed comparably to the best multi-spectral methods (SSL4EO) and significantly outperformed contrastive RGB approaches and the supervised baseline. * **Heterogeneous Tasks:** METEOR achieved the best average rank across diverse tasks, highlighting its adaptability to different spatial resolutions, spectral bands, and numbers of classes. It performed particularly well on land cover classification (DFC2020-KR) and floating object detection. * **Qualitative Analysis:** Qualitative analysis using occlusion sensitivity demonstrated the interpretability of METEOR's predictions and provided insights into the model's decision-making process across different applications. The examples showcased METEOR's capacity to effectively use limited training data to achieve high performance in various tasks. * **Computational Efficiency:** Fine-tuning METEOR on downstream tasks is computationally efficient, requiring minimal training time. However, pre-training the meta-model with MAML is more computationally expensive than self-supervised approaches. The results consistently show that METEOR, with its three key modifications, effectively learns to learn from a diverse set of tasks and adapts well to new, unseen tasks with minimal labeled data. Only METEOR and MOSAIKS showed consistent high performance across both homogeneous and heterogeneous tasks.

The results of this study address the research question of whether a single meta-learning model can be effectively used across diverse Earth observation tasks. METEOR's success in various applications, from land cover mapping to marine debris detection, demonstrates that it's possible to leverage shared knowledge among seemingly disparate remote sensing problems. This approach significantly reduces the need for large labeled datasets for each individual task, greatly enhancing the efficiency and feasibility of applying deep learning in Earth science. The findings are highly relevant to the field, emphasizing the potential of meta-learning to address challenges related to data scarcity, concept drift, and covariate shift in remote sensing. The ability to adapt a single model to various tasks makes it a powerful tool for researchers working with limited resources.

METEOR provides a versatile and efficient framework for addressing diverse Earth observation problems with limited labeled data. The three key modifications—instance normalization, dynamic channel adaptation, and one-vs-all ensemble—significantly enhance the model's adaptability across heterogeneous tasks. While achieving high accuracy and outperforming several state-of-the-art methods, limitations exist, particularly with a large number of classes. Future work should focus on addressing these limitations, such as exploring alternative strategies for handling a large number of classes and investigating more sophisticated pre-training strategies to further enhance the model's performance. The open-source availability of METEOR promotes broader adoption and contributes to the development of more efficient and robust deep learning solutions for Earth science applications.

While METEOR shows promising results, some limitations should be noted. The one-vs-all ensemble approach may not be optimal for tasks with a very large number of classes, impacting performance in some scenarios (e.g., NWPU-RESISC with 45 classes). Pre-training the meta-model using MAML is computationally more demanding than self-supervised learning methods. Although the meta-model was pre-trained on land cover data, exploring pre-training with a broader range of tasks might further improve its adaptability and performance on downstream tasks. Further investigation into the impact of different pre-training datasets on the meta-model's performance is warranted.