Machine learning and phone data can improve targeting of humanitarian aid

The COVID-19 pandemic significantly impacted low- and middle-income countries, leading to widespread food insecurity and reduced living standards. Governments and humanitarian organizations responded by distributing social assistance to over 1.5 billion people. A major challenge is effectively targeting this aid to those most in need. Traditional data sources, such as household income data, are often unavailable or unreliable in these settings. This paper explores the use of mobile phone network data and machine learning to address this challenge. The research leverages recent advances in machine learning which demonstrate its ability to predict wealth from mobile phone usage. It also draws upon the existing literature on social assistance targeting mechanisms. The study focuses on the Novissi program in Togo, a flagship emergency cash transfer program implemented in response to the pandemic.

The paper draws upon several streams of existing literature. First, it acknowledges the significant economic hardship caused by the COVID-19 pandemic, particularly in low- and middle-income countries and the subsequent large-scale deployment of social assistance programs. Second, it highlights the challenges associated with targeting these programs in the absence of comprehensive and up-to-date data on household income or wealth, common in many low-income contexts. Third, it cites prior research demonstrating the potential of machine learning to predict wealth and poverty from various data sources, including satellite imagery and mobile phone metadata. Fourth, it references a rich body of economics literature on the design of mechanisms for effectively targeting social assistance, including proxy means tests and geographic targeting.

The study uses data from the Novissi program in Togo, an emergency cash transfer program launched in response to the COVID-19 pandemic. The methodology involves two main steps. First, they created poverty maps using publicly available satellite imagery and machine-learning algorithms to estimate the relative wealth of each 2.4 km by 2.4 km region in Togo. Second, they predicted the daily consumption of each mobile phone subscriber using machine-learning algorithms trained on a large representative sample of phone subscribers. This training used survey data that measured the wealth and consumption of subscribers and matched them to mobile phone metadata. The resulting algorithms predict wealth and consumption from the high-dimensional vector of mobile phone features. The performance of the phone-based approach was compared to several counterfactual scenarios: geographic targeting using the poorest prefectures or cantons; occupation-based targeting (including Novissi’s original targeting of informal workers and an optimized version); and a simpler phone-data-based approach that doesn't utilize machine learning. The evaluation used two datasets. The first evaluated the actual rural Novissi program using a 2020 phone survey, with a proxy means test (PMT) serving as the ground truth for poverty. The second simulated a hypothetical nationwide program, using nationally representative household survey data from 2018-2019 and consumption as the ground-truth measure of poverty. The study also assessed the fairness of different targeting methods across gender and ethnic subgroups, using measures of demographic parity.

The phone-based approach significantly reduced exclusion errors compared to feasible alternatives in the actual rural Novissi program (53% vs 59-78%). In a simulated nationwide program, phone-based targeting outperformed most feasible alternatives but was slightly less effective than an optimized occupation-based approach. The phone-based approach's benefits were greatest when the population was more homogeneous and when less variation in factors like place of residence existed. Targeting performance improved as the focus narrowed to the most extreme poor. A simpler phone expenditure model performed substantially worse than the machine-learning model. Model accuracy decreased when data were 18 months out of date. Compared to hypothetical social registry methods, the phone-based approach was comparable to asset-based targeting but less accurate than a perfectly calibrated PMT or a poverty probability index. Improved targeting led to higher social welfare. The phone-based approach showed no systematic bias against women or specific ethnic groups although minor differences in accuracy were observed across different demographic subgroups. Six main sources of exclusion were identified: possessing a SIM card and mobile phone access; recent SIM card usage; voter registration; program awareness and successful registration; and targeting errors from the machine learning algorithm.

The findings demonstrate the potential of mobile phone data and machine learning to improve the targeting of humanitarian aid, particularly in crisis situations where traditional data are scarce or outdated. The phone-based approach offers a rapid and cost-effective way to supplement existing methods, reducing exclusion errors. However, the limitations of relying on mobile phone data, including data access and privacy concerns, must be considered. The study highlights the importance of using current and representative data to train machine learning models and the need to address potential bias and fairness concerns in algorithmic targeting. The research contributes to a growing body of work demonstrating the potential of non-traditional data sources and machine learning in development contexts.

This study shows that combining mobile phone data with machine learning can significantly improve the targeting of humanitarian aid. The phone-based approach reduced exclusion errors compared to existing methods in the Togo Novissi program. Future research should explore the integration of real-time data with traditional methods for more comprehensive and inclusive social protection systems. Addressing the limitations related to data access, privacy, and algorithm bias is crucial for responsible implementation.

The study acknowledges several limitations. First, the phone-based approach is not perfect and can still lead to errors of exclusion and inclusion. Second, there are practical limitations related to data access and privacy concerns. Third, the study's findings might not generalize perfectly to all contexts, particularly those with different levels of mobile phone penetration or data availability. Fourth, the performance of a hypothetical, perfectly calibrated PMT may overestimate the performance of real-world PMTs. Fifth, the model's accuracy decreases with older data, highlighting the need for frequent updates.