A human-machine collaborative approach measures economic development using satellite imagery

Reliable economic activity measures are difficult to obtain in developing countries, hindering economic research and policy analysis. Many countries lack recent agricultural or population census data, with North Korea being an extreme example, where the last official county-level population statistics were from 2008. Existing alternative methods, such as interviews, news articles, and nightlight imagery, offer limited precision and coverage. Computer vision models analyzing satellite imagery have shown promise in inferring socioeconomic status in other regions. However, these models typically require ground-truth data, which are often unavailable in low-income countries. This research addresses this limitation by proposing a novel human-machine collaborative deep neural network model that assesses economic development at a grid level (2.45 x 2.45 km²) using visual patterns from satellite imagery. The model captures economic development through observable settlement patterns, with features like higher building density and infrastructure indicating higher development.

Existing literature highlights the challenges of measuring economic activity in data-scarce regions, particularly in developing countries. While recent advancements in computer vision and machine learning have shown potential for using satellite imagery to infer socioeconomic indicators, these methods largely rely on ground-truth data, limiting their applicability in many contexts. Studies have explored alternative data sources such as Wikipedia entries and mobile phone data to enhance prediction accuracy, but the need for ground truth remains a significant constraint. This paper directly addresses this gap by proposing a novel method that leverages human expertise to overcome the limitations of data scarcity.

The proposed model consists of three stages (Figure 1). Stage 1 involves clustering satellite images based on vectorized visual features using the DeepCluster algorithm. Uninhabited areas are pre-processed and separated to improve efficiency. Silhouette analysis is used to determine the optimal number of clusters. Stage 2 involves human annotators who subjectively rank these image clusters based on their perceived level of economic development. These rankings are represented as a partial order graph (POG). Ten annotators, grouped by expertise (economists, satellite imagery experts, North Korean defectors), participated. The POGs from individual annotators are then ensembled into a single representative POG. Stage 3 trains a rank-wise score model (convolutional neural network) to assign a numeric score (siScore) between 0 and 1 to each satellite grid image, consistent with the ensembled POG. The model aims to maximize the Spearman correlation between the assigned scores and the cluster order in the POG. A differentiable ranking function is used to approximate the rank due to the non-differentiable nature of ranks. The model was applied to North Korea and five least developed countries in Asia using Sentinel-2 satellite imagery at 10m per pixel resolution. For validation in North Korea, building footprint data, company counts from North Korean news outlets, and 2008 population density data were used as proxies for economic development. For the LDCs, official census and survey data were used. Four baseline models (nightlight regression, nightlight-guided POG, land cover-guided POG, and relative wealth index) were also compared.

The human-machine collaborative model generated a spectrum of siScores for North Korea, revealing distinct regional development patterns (Figure 2). Western plains and coastal areas showed high siScores, while central and northern mountainous regions had low scores. The model provided a higher resolution picture than nightlight imagery, capturing refined variations within regions (Figure 2). Model predictions were comparable to government-produced land cover and building footprint maps, but generated with substantially fewer resources. The model's applicability extended to the five LDCs in Asia (Figure 3), showing that the approach is not limited to North Korea. The human-machine collaborative model outperformed or showed comparable performance to four baseline models in predicting economic development proxies (Figure 4). In North Korea, the model exhibited strong correlations with building area (Spearman's ρ = 0.77 at the grid-level and R² = 0.83 at the district-level) and other proxies. Analysis of yearly predictions (2016-2019) indicated development concentrated in Pyongyang and areas with state-led projects (Figure 5). Grad-CAM analysis (Figure 6) revealed which pixels contributed to siScore, highlighting infrastructure development in specific regions like the Samjiyon development project and land reclamation in Ryongyon County. Regression analysis showed that distance from major cities negatively correlated with development, while designated economic development zones (EDZs) for agriculture or tourism showed higher development. Nighttime luminosity, often used as a proxy, showed discrepancies with siScore, potentially reflecting different aspects of economic development and limitations in nightlight data in less developed areas. The siScore seems to capture the physical urban development better than nightlight.

The human-machine collaborative model provides valuable insights into regional economic development, particularly in data-scarce environments. The findings for North Korea reveal development concentrated in and around the capital, Pyongyang, and areas with government-led projects in agriculture and tourism. The model's superior performance compared to baselines, and its applicability across various developing countries, demonstrate its potential for policy design and implementation. The ability to generate grid-level economic indicators complements existing remote sensing methods and offers a more granular understanding of regional development disparities. The integration of human expertise improves model accuracy and makes it applicable to regions lacking ground-truth data, addressing a crucial limitation of existing machine learning approaches.

This study introduces a novel human-machine collaborative model for measuring economic development using satellite imagery. The model's ability to generate fine-grained predictions in data-scarce regions, its superior performance compared to existing methods, and its broad applicability across different countries represent significant contributions. Future research could focus on integrating additional data sources, refining the model's training process, and expanding its application to a global scale. This approach promises to improve resource allocation and policy design in developing countries.

While the model demonstrates significant potential, several limitations should be considered. The subjective nature of human annotations could introduce bias, although the ensemble approach mitigates this. Cloud cover in satellite imagery may obscure features and affect prediction accuracy. The static nature of satellite imagery may not fully capture dynamically changing economic factors. Additionally, privacy concerns associated with high-resolution satellite imagery require careful consideration.