Online searches to evaluate misinformation can increase its perceived veracity

The proliferation of misinformation online poses a significant threat to democratic legitimacy and public health. While the role of social media in spreading misinformation has been extensively studied, the impact of search engines, which act as crucial gatekeepers in the digital information ecosystem, remains relatively underexplored. This study directly addresses the understudied question of how searching online to evaluate news (SOTEN) affects belief in misinformation. Given that many media literacy interventions encourage online fact-checking via search engines to combat misinformation, understanding the actual effect of this practice is critical. The prevailing assumption is that searching online for information should decrease belief in misinformation; however, this study tests the counterintuitive hypothesis that SOTEN may actually increase belief in misinformation. This is important because it challenges the common wisdom underlying many search-based media literacy recommendations and has significant implications for the design and effectiveness of these initiatives.

Existing research has focused on various factors influencing belief in misinformation, including ideological congruence, psychological factors, and digital media literacy. However, the role of search engines in this context has received limited attention, despite their growing importance as information gatekeepers and the widespread practice of online fact-checking. While some research has investigated partisan biases in search engine results, there's a lack of empirical evidence on the direct impact of SOTEN on belief in misinformation. This gap highlights the need for a deeper understanding of how the process of online fact-checking, specifically through search engines, influences individuals' acceptance or rejection of information presented to them.

The study employs five separate experiments to measure the effect of SOTEN on belief in misinformation and true news. Four experiments use survey experiments, while the fifth combines survey data with digital trace data gathered using a custom browser extension. In the survey experiments, participants are randomly assigned to either a treatment group (encouraged to search online before evaluating articles) or a control group (not prompted to search). The articles presented in all experiments are selected based on their popularity and are subsequently evaluated by professional fact-checkers to establish their veracity. Both categorical and ordinal scales are used to assess participants' belief in the articles' truthfulness. The fifth experiment enhances the data collection by employing a custom browser plug-in to record search queries and URLs visited by the treatment group. This addition allows for the analysis of the quality of search results and their relationship to changes in belief. The study utilizes various statistical methods, including ordinary least squares (OLS) regression with article-level fixed effects and standard errors clustered at the individual and article level, to analyze the data and determine the treatment effects. Data from NewsGuard, which provides reliability scores for news sources, is used to assess the quality of information returned in search results.

Across the five studies, a consistent finding emerged: SOTEN increased belief in misinformation. This effect was robust across various experimental designs, time periods, and topics. In the study that combined survey and digital trace data, the researchers found a strong correlation between exposure to lower-quality information in search results and a higher probability of believing misinformation. Exposure to higher-quality information, however, did not show a similar effect. Furthermore, the study showed a similar search effect on belief in true news, specifically increasing the likelihood of believing true news from low-quality sources. However, there was no consistent effect observed on the belief in true news from mainstream sources. The analysis of search terms revealed that using the headline or URL of the original article as a search query significantly increased the likelihood of encountering unreliable information in the search results. Lower digital literacy was linked to a higher probability of using such search terms, contributing to exposure to unreliable information. Although ideological congruence also shows a potential impact, its effect is less substantial. The magnitude of the search effect on belief in misinformation varied across different experimental setups; the effect was larger in studies with stronger search treatments and more direct measures of compliance.

These findings challenge the conventional wisdom that encouraging online search to verify information will reduce belief in misinformation. The results highlight the existence of "data voids" where search engines return corroborating evidence from low-quality sources, particularly for recently published misinformation. This underscores the complexity of the online information environment and the need for more nuanced approaches to digital media literacy. The inconsistent effect on belief in true news from mainstream sources suggests a potential ceiling effect, as many participants already correctly rated these articles as true before any search. The finding of a strong effect on low-quality sources suggests that the search effects are particularly harmful for news from those sources, since those sources may already be seen as less credible before searching.

This research demonstrates that simply encouraging individuals to search online to evaluate news, a common strategy in media literacy interventions, can have the unintended consequence of increasing belief in misinformation. This counterintuitive effect is strongly related to the quality of information returned by search engines and the search strategies used by individuals. The findings emphasize the need for empirically-grounded media literacy strategies and for search engines to address the challenges identified. Future research should investigate alternative search-based strategies that might mitigate this effect and explore the behavioral impact of disseminating digital media literacy guides in the real world. Also, further research should investigate whether self-selection of news articles might affect this analysis.

The study's design does not allow participants to self-select the news articles they evaluate, which could limit the generalizability of the results. The reliance on highly popular articles, while capturing a significant part of online news consumption patterns, may not fully represent the heterogeneity of effects across all types of news. Also, the study's analysis of the quality of search results is post-treatment, so it does not establish causal relationships; therefore, correlation should not be interpreted as causation. Finally, the study did not capture the effect of those users who use online search engines to evaluate the veracity of news independently, without explicit encouragement.