The Effects of Non-pharmaceutical Interventions on COVID-19 Mortality: A Generalized Synthetic Control Approach Across 169 Countries

The COVID-19 pandemic prompted governments worldwide to implement various NPIs, including lockdowns, school and workplace closures, travel restrictions, and mask mandates. While these measures aimed to reduce COVID-19 mortality, existing research focusing primarily on infection rates and early 2020 data yielded mixed results. These earlier studies suffered from limitations such as underreporting, poor data quality, and low variation in intervention timing. This study aimed to address these limitations by analyzing daily COVID-19 death data per capita from July 1, 2020, to September 1, 2021, across 169 countries (covering 98% of the world's population). The researchers utilized the GSC method, which mitigates selection bias and allows for flexible modeling of post-intervention trajectories. This approach also allowed for the inclusion of 10 different NPIs from the Oxford COVID-19 Government Response Tracker (OxCGRT) while controlling for other factors such as weather conditions, vaccination rates, and NPI-residualized COVID-19 cases. The study's comprehensive dataset and advanced analytical technique offer a more robust evaluation of NPI effectiveness in reducing COVID-19-related deaths.

Previous research on the effectiveness of NPIs on COVID-19 outcomes has primarily focused on infection rates, overlooking the crucial goal of reducing fatalities. Early studies using data from the first half of 2020 often showed mixed results due to underreporting and limited variation in intervention timing and types. While some studies found significant mitigating effects of various NPIs on COVID-19 deaths, these results were largely based on data from the initial wave of the pandemic. The current study aims to enhance the existing literature by employing a larger dataset encompassing a longer period, a more diverse range of countries, and a more sophisticated analytical approach.

The study employed a generalized synthetic control (GSC) method, a combination of synthetic control and difference-in-differences approaches. This method was chosen to address potential biases related to selection into treatment and to allow for flexible modeling of heterogeneous treatment effects over time. The researchers analyzed daily data on confirmed COVID-19 deaths per capita from Our World in Data and ten different NPIs from the OxCGRT. The NPIs were coded as binary indicators (implemented or not). The GSC method involves constructing a synthetic control group for each treated unit (country) based on pre-treatment characteristics. The method estimates the average treatment effect on the treated (ATT) by comparing the post-treatment trajectory of the treated unit to its synthetic control. The analysis included various controls: the stringency index for other NPIs, cumulative vaccination rates, monthly average temperature and its square, cloud cover, specific humidity, precipitation, and NPI-residualized COVID-19 cases (using a regression-with-residuals approach to control for the potential endogeneity of COVID-19 cases and NPIs). The researchers addressed the issue of staggered treatment adoption by dividing the data into country-period splits, treating each transition between treatment and control as a new unit. Statistical inference was based on non-parametric bootstraps, clustered at the country level. The R package 'gsynth' was used for estimation.

The study's main finding is the lack of substantial and consistent COVID-19-related fatality-reducing effects for any of the ten NPIs investigated. While a tentative downward trend in deaths was observed approximately 30 days after the implementation of strict stay-at-home rules and, to a lesser extent, workplace closures, these effects were not statistically significant. In contrast, a robust and statistically significant reduction in COVID-19 fatalities was observed following high vaccination rates (above 80% of the population). The model successfully identified a substantial mitigating effect of vaccination on COVID-19 deaths, demonstrating the method's capacity to detect significant effects. The researchers conducted several robustness checks to ensure the reliability of their findings, including: controlling for spatial spillovers from neighboring countries, altering the coding of intervention (using the two highest stringency categories), controlling for the number of implemented interventions instead of the stringency index, comparing the findings to first-wave data only, and testing for effects in early versus late adopters of interventions. The results largely remained stable across these checks, although some nuances were observed related to the timing of intervention implementation. The finding regarding stay-at-home orders suggests that these might have been implemented reactively to address steep increases in cases. Supplementary figures provide further details on these robustness checks.

The study's findings contrast with some previous research that found significant effects of NPIs on COVID-19 deaths. These discrepancies may stem from methodological differences, variations in data quality and reporting, and focusing on the initial pandemic wave. However, the current study aligns with recent research that found no significant effects of shelter-in-place orders on COVID-19 cases. The lack of strong effects found in this study does not necessarily negate the possibility that NPIs prevented exponential growth in deaths, a hypothesis the study cannot definitively test. The consistent and significant effect of vaccination highlights the importance of pharmaceutical interventions in combating the pandemic. The effectiveness of vaccination is dependent on ongoing efforts to adapt vaccines to new variants and increase vaccination rates. The study acknowledges that differences in compliance with NPIs across populations, simultaneous implementation of multiple NPIs, data quality limitations, and local context might affect the findings. The age gradient in COVID-19 mortality and associated multi-morbidities may also explain the differential impact of NPIs on cases versus deaths.

This study, using a robust methodology and a comprehensive dataset, does not find significant fatality-reducing effects from most NPIs. However, the research does demonstrate the substantial effectiveness of COVID-19 vaccination in reducing mortality. Future research should investigate compliance with NPIs, the effects of simultaneous implementations of multiple NPIs, and explore further the potential for NPIs in preventing exponential growth in fatalities. The study's findings provide valuable insights for informing future public health responses to pandemics.

The study acknowledges several limitations that may affect the interpretation of its results. These include variations in compliance with NPIs across populations, difficulties in disentangling the effects of simultaneous NPI implementations, potential limitations related to data quality and misreporting of COVID-19 deaths, and the inability to definitively test whether NPIs prevented exponential growth in deaths. Additionally, the broad categories used by the OxCGRT might not capture subtle variations in the implementation of NPIs. Despite these limitations, the study provides valuable insights for informing future public health strategies.