Nations significantly influence individual life outcomes and cultural variations. Cross-national analyses, which investigate the drivers of variation in national outcomes, are prevalent in social sciences, offering insights beyond Western populations. However, a critical statistical flaw undermines the validity of these analyses: nations are not statistically independent. Spatial proximity leads to spatial non-independence because neighboring nations share characteristics due to cultural diffusion and environmental factors. Shared cultural ancestry, or cultural phylogenetic non-independence, causes further non-independence because related nations inherit similar traits, creating a form of pseudoreplication. These dependencies violate the assumption of independent and identically distributed residuals in regression analyses, inflating false positive rates and leading to spurious correlations. While other fields, like ecology and anthropology, have long acknowledged and addressed this issue, cross-national research in economics and psychology largely neglects it.

The authors conducted a comprehensive review of the 100 most highly cited cross-national studies focusing on economic development and cultural values. The analysis revealed a concerning lack of controls for non-independence in the literature. Only a small percentage of studies attempted to control for non-independence, primarily using regional fixed effects, spatial distance controls, or shared cultural history controls. Surprisingly, the use of these controls was not consistently higher in higher-impact journals or more recent publications. This suggests that there are either systemic issues in the incentive structures within publishing or a lack of awareness among researchers about this important methodological issue.

To quantify the degree of non-independence, the authors employed Bayesian multilevel models to analyze the relationship between spatial proximity, shared cultural ancestry, and variables representing economic development (Human Development Index, GDP per capita, GDP growth, Gini index) and cultural values (traditional vs. secular values, survival vs. self-expression values, cultural tightness, individualism). Results indicated that a substantial proportion of national-level variation was explained by spatial proximity and/or shared cultural ancestry. Subsequently, a simulation study was conducted to assess the efficacy of common methods for controlling non-independence. Simulated datasets with varying degrees of spatial or cultural phylogenetic autocorrelation were generated. Various regression models—including naive regressions, regressions with latitude, longitude, continent fixed effects, language family fixed effects, and Bayesian models incorporating geographic and linguistic proximity matrices—were fitted to these datasets. The false positive rates were calculated as the proportion of models that falsely inferred a relationship when none existed. The statistical power was assessed by determining the proportion of models that correctly identified a relationship when it truly existed. Finally, twelve previous cross-national analyses were reanalyzed, incorporating global geographic and linguistic proximity matrices as controls for spatial and cultural non-independence. These reanalyses aimed to assess the robustness of the original findings when accounting for the non-independence of nations.

The analyses revealed several key findings: 1. National-level economic development and cultural values exhibit significant spatial and cultural non-independence. 2. Most cross-national studies fail to account for non-independence. 3. Commonly used control methods are insufficient to reduce false positive rates significantly. 4. In the reanalysis of twelve previous studies, half of the correlations were no longer significant after controlling for non-independence. 5. The simulation study demonstrated that naive regression models yielded high false positive rates under moderate to strong autocorrelation. Simple controls like latitude, longitude, or continent fixed effects did not effectively reduce these rates. However, Bayesian models incorporating geographic and linguistic proximity matrices proved much more effective at reducing false positives while maintaining high statistical power. In particular, Bayesian models incorporating linguistic proximity successfully eliminated false positives in the presence of strong cultural phylogenetic autocorrelation.

The findings highlight a critical methodological oversight in cross-national research. The infrequent use and often inadequate nature of controls for non-independence raise serious concerns about the reliability of existing findings. The authors explain the reasons for this widespread issue might be related to incentives within academic publishing, lack of awareness, or perceived complexity of more appropriate methods. The authors suggest several actions to address this, including increased awareness through replication studies, the use of causal models in research design, and increased accessibility of appropriate statistical software. By emphasizing the importance of properly addressing non-independence, the study aims to improve the rigor and reliability of future cross-national research.

This paper demonstrates the crucial need for improved methodological rigor in cross-national research. The widespread neglect of non-independence leads to inflated false positive rates and potentially spurious findings. The authors strongly advocate for employing more sophisticated methods, such as Bayesian models with covariance matrices based on geographic and linguistic proximity, to accurately analyze cross-national data. Future research should continue to refine methods for dealing with various sources of non-independence and should broadly replicate existing findings using these improved approaches.

While the study comprehensively addresses spatial and cultural phylogenetic non-independence, other potential sources of non-independence (e.g., modern connections through information flow, shared colonial history) were not explicitly simulated. Furthermore, the reanalysis was limited to a subset of 12 studies, preventing complete generalization of the findings. The study's focus on economic development and cultural values may also limit generalizability to other domains of cross-national research.