The Effects of Non-pharmaceutical Interventions on COVID-19 Mortality: A Generalized Synthetic Control Approach Across 169 Countries

Governments implemented a wide range of non-pharmaceutical interventions (NPIs) in response to COVID-19, including public information campaigns, testing, contact tracing, isolation of infected/vulnerable individuals, mask mandates, closures of schools, workplaces, and public transport, restrictions on public events and gatherings, stay-at-home requirements, and movement restrictions. While stringent NPIs are intended to mitigate COVID-19-related fatalities and prevent health system overload, they can entail significant economic, social, and public health externalities, making proportionality judgments dependent on consistent empirical evidence of their effectiveness. Prior empirical research has often focused on infections rather than fatalities, despite fatalities being the key outcome and less susceptible to underreporting. Early analyses from the first half of 2020 face issues including underreporting, reporting delays, and limited variation in treatment type and timing, complicating robust causal identification. The current study addresses these limitations by analyzing fatalities (not infections), focusing on the period July 1, 2020 to September 1, 2021 when treatment timing varied more across countries, expanding coverage to 169 countries (approximately 98% of the world population), and applying the Generalized Synthetic Control (GSC) method to estimate average treatment effects on the treated while attenuating selection bias and allowing flexible post-treatment trajectories.

Prior studies have primarily evaluated NPIs’ effects on infections, with comparatively few investigating fatalities. Among 37 studies reviewed, only 10 examined effects on COVID-19-related deaths. Early causal analyses using first-wave data reported mitigating effects of lockdowns, school closures, workplace closures, cancellations of public events, stay-at-home orders, travel restrictions, and mask mandates. However, these findings may be influenced by early-pandemic underreporting and limited variation in policy timing. One later study (August 2020–January 2021) using a hierarchical Bayesian transmission model across 7 countries/114 subnational units found substantial mitigating effects for school and workplace closures and restrictions on gatherings. Results across the literature vary due to heterogeneity in data, methods, NPIs evaluated, time spans, and geographic coverage. Methodologically, difference-in-differences designs can suffer from selection on pre-trends and treatment effect heterogeneity; newer approaches (e.g., synthetic control/GSC) have been proposed to mitigate these issues. A related study using bias-attenuating strategies reported no significant impacts of shelter-in-place orders on cases in the US, aligning conceptually with the present study’s approach.

Data: The study aggregates country-level data for 169 countries from July 1, 2020 to September 1, 2021. COVID-19 daily confirmed deaths per million are sourced from Our World in Data (OWID). Ten NPIs—school closing, workplace closing, public transport closure, stay-at-home rules, restrictions on internal movement, international travel restrictions, protection of the elderly, testing policy, contact tracing, and mask mandates—are from the Oxford COVID-19 Government Response Tracker (OxCGRT). The main outcome is daily new deaths per million; days with negative reported deaths (retroactive corrections) are treated as missing. For main analyses, each NPI is recoded as a binary indicator equal to 1 if the country is at the highest available stringency category on a given day, 0 otherwise. Analytical Strategy: To address unobserved heterogeneity, selection into treatment, and flexible dynamic effects, the study employs the Generalized Synthetic Control (GSC) method, which combines features of synthetic control and difference-in-differences within a factor-augmented framework. The model: Y_it = δ_it D_it + X_it β + L_it + ε_it, where Y_it is deaths per capita, D_it is the binary treatment for a specific NPI, δ_it is the time- and unit-varying treatment effect (with interest in the ATT), X_it are observed controls, and L_it captures unobserved time-varying factors via matrix completion (nuclear norm regularization) rather than explicit factor estimation. Estimation proceeds by: (1) estimating β and L on control units with 10-fold cross-validation over 20 candidate λ values; (2) predicting counterfactuals for treated units Ŷ_it(D=0) = X_it β̂ + L̂_it; and (3) obtaining ATT estimates δ̂_it = Ŷ_it(D=1) − Ŷ_it(D=0). Inference uses 1,000 non-parametric bootstrap runs clustered at the country level. Due to staggered and reversible treatments (implementation and relaxation), the data are partitioned into country-period splits so that each transition from treatment to control (and vice versa) defines a new analytic unit, preventing conflation of implementation effects with relaxation effects. Controls: For each target NPI i, all remaining NPIs j≠i enter as OxCGRT-style normalized sub-index scores (Index_j). Additional controls include cumulative vaccinations per capita (OWID); monthly average weather variables—temperature (and squared), cloud cover, specific humidity, and precipitation—from ERA5 reanalysis; and multiple temporal lags of residualized COVID-19 cases to avoid overcontrol bias. Residualized cases are obtained via a regression-with-residuals approach: cases are first regressed on multiple lags of all NPIs, country and time fixed effects; residuals, orthogonal to past NPIs (7–35 days), are then used. The main models include the 7-day backward rolling average of these residualized cases at lags t−7, t−14, t−21, t−28, and t−35 with polynomial terms up to cubic order. Software: Estimation is implemented in R using the gsynth package (v1.1.9). Sensitivity and robustness checks include accounting for spatial spillovers (neighboring countries’ NPIs), alternate treatment codings (highest two categories), replacing NPI indices with counts of concurrent NPIs, restricting to first-wave data, and stratifying by early vs late adopters based on fatalities at intervention.

- Across 169 countries from July 2020 to September 2021, none of the ten strictly implemented NPIs exhibited substantial and consistent reductions in COVID-19-related deaths. Post-treatment ATT trajectories generally did not differ significantly from zero. - Tentative (borderline) trend changes in deaths appear around 30 days after strict stay-at-home requirements and, to a lesser extent, workplace closures, but these effects are not statistically significant. - Proof-of-concept: Using the same framework, high COVID-19 vaccination uptake (treatment defined as ≥80 doses per 100 inhabitants) shows a consistent, statistically significant reduction in COVID-19 deaths from approximately day 45 to day 110 post-threshold. Magnitude example: In a country of 60 million people, vaccinations are estimated to prevent about 90 deaths per day starting around 45 days after reaching the 80/100 threshold. - Robustness checks: • Spatial spillovers (neighboring countries’ NPIs) do not materially change results. • Coding the top two NPI categories as treatment yields similar null results overall; an exception appears for school closures but is unstable due to limited control units. • Using the count of concurrent NPIs instead of indices yields consistent findings. • Restricting to first-wave data (pre-September 2020) shows a possible downturn after internal movement restrictions, but with large uncertainty. • Stratifying by adoption timing indicates the stay-at-home signal disappears for early adopters and is stronger for late adopters, suggesting implementation as a last-resort response during steep increases.

The study aimed to causally assess whether NPIs reduced COVID-19 mortality using a generalized synthetic control approach that mitigates selection bias and accommodates heterogeneous dynamic effects. The findings indicate no substantial, consistent mortality reductions attributable to any of the ten NPIs studied during July 2020–September 2021, though weak trend changes around 30 days after stay-at-home rules and workplace closures were observed without statistical significance. These results diverge from several early first-wave studies reporting mortality reductions from school and workplace closures, stay-at-home orders, travel restrictions, and mask mandates—differences that may reflect early-pandemic underreporting/timeliness issues, limited variation in NPI timing during the first wave, and methodological differences (e.g., DiD vs GSC). The results align with conceptually similar analyses that, when addressing selection into treatment and flexible dynamics, find limited or no significant impacts of certain NPIs on outcomes like cases. Importantly, the analysis robustly identifies a significant mortality reduction from high vaccination coverage, underscoring the effectiveness of pharmaceutical interventions. While the null mortality findings for NPIs could suggest limited direct impact on deaths, it remains possible that NPIs mitigated potential exponential growth rather than producing declines in total deaths—an effect not directly testable with the available framework and data. Overall, the study provides cross-national evidence informing public health decision-making about the relative effectiveness of NPIs versus vaccinations on mortality.

This study expands the evidence base on NPIs’ effectiveness by analyzing COVID-19 mortality across 169 countries over July 2020–September 2021 using a generalized synthetic control approach. The main contribution is the absence of substantial, consistent mortality-reducing effects for the ten NPIs examined, contrasted with a clear, significant mortality reduction associated with high vaccination uptake. These findings suggest that while NPIs may not reliably reduce deaths directly across diverse contexts, they could still help prevent uncontrolled growth in fatalities. Policy implications emphasize the critical role of vaccination campaigns and the need for high coverage. Future research should address simultaneous NPI interactions, incorporate more granular implementation and compliance data (including subnational variation), improve mortality data quality (especially in non-hospital settings), and develop methods/data to assess whether NPIs avert exponential growth rather than lowering absolute deaths.

- Compliance heterogeneity: Variation in population-wide compliance likely attenuates observed effects, particularly for mask mandates, contact tracing, testing policies, protection of the elderly, and domestic travel restrictions. Early pandemic periods may have seen higher compliance than later periods. - Concurrent interventions: The statistical approach complicates inference for simultaneous NPI bundles; pairwise combinations tested did not yield consistent significant effects. - Dynamic counterfactual uncertainty: GSC attenuates pre-treatment selection but cannot rule out that treated countries would have experienced post-treatment exponential growth absent the NPIs. Thus, statistically insignificant effects do not preclude NPIs mitigating potential surges. - Data quality and measurement: Potential misreporting of COVID-19 deaths (e.g., non-hospitalized), broad OxCGRT categories that may not capture local implementation nuances or subnational variation, and cross-country heterogeneity in reporting practices may bias estimates toward null findings. - Generalizability across phases: Effects may vary by epidemic phase and adoption timing; stratified analyses suggest stay-at-home orders may be implemented as a last resort during steep increases. - Limited ability to test prevention of exponential growth directly with available data and methods.