Misinformation and higher-order evidence

The paper addresses how misinformation affects belief formation in networked communities of rational agents and how agents should respond to higher-order evidence (HOE) about the reliability of their informational environment. In a 21st-century context dominated by online and social media, misinformation and disinformation can influence attitudes and behaviors on issues such as climate, democracy, and health. Philosophically, HOE concerns evidence about evidential relations, such as evidence that sources or reasoning processes are unreliable. The authors bring the HOE debate into contact with computational models of misinformation to evaluate two strategies: denying the import of HOE (trusting/gullible) versus accommodating it (discounting/aligned). They aim to assess truth-conduciveness (accuracy) and timeliness (efficiency) of belief convergence under varying levels and types of unreliability, both in stylized complete networks and a real-world email communication network. The central research questions are: How do misinformation and disinformation alter community epistemic outcomes? Which HOE response strategy better balances accuracy and efficiency?

The study builds on and contrasts several literatures: (a) HOE in epistemology, including debates on how to respond to evidence about one’s evidence (Christensen 2010; Feldman 2005; Fitelson 2012; Kelly 2005; Skipper & Steglich-Petersen 2019; Horowitz 2022). Positions range from denying HOE’s import to accommodating it via doxastic adjustment. (b) Network epistemology and learning from neighbors: Bala & Goyal (1998) formalize social learning; Zollman (2007) explores how network structure trades off accuracy and efficiency; Rosenstock et al. (2017) examine epistemic networks. (c) Models of testimonial trust and homophily: O’Connor & Weatherall (2018) use Jeffrey updating to model mistrust based on belief distance (homophily). The present work instead discounts evidence due to known source unreliability (HOE) rather than homophily. (d) Propaganda and misinformation/disinformation mechanisms including selective reporting (Weatherall et al. 2020). The paper integrates these strands by operationalizing HOE-processing strategies within network simulations and contrasting misinformation (neutral noise) with disinformation (adversarial lies).

Overview: Communities are modeled as graphs where nodes are agents and edges are communication channels. Each agent holds a credence in hypothesis H (“B is better than A,” with true success probability 0.5 + ε; alternative 0.5 − ε). Decision rule: at each step, agents with credence(H) > 0.5 choose action B for a fixed number of trials; otherwise they choose A. Agents share outcome reports with neighbors and update credences. Model variants and updating:

• Baseline (Bala-Goyal): All agents reliable; agents fully trust reports and update by Bayes’ rule using all first-order evidence (own trials + neighbors’ reports). Stopping conditions: consensus on B (all credences > 0.99), consensus on A (all credences < 0.5, so experimentation ceases), or step cap.
• Higher-order evidence strategies under unreliability:
  1. Gullible (deny HOE import): Agents behave as if all reports are reliable; full Bayes’ conditionalization on all testimony despite knowing some fraction are unreliable at the population level.
  2. Aligned (accommodate HOE): Agents discount testimony via Jeffrey’s rule by aligning P(evidence) to the known network reliability r (i.e., set P(e) = r), dampening the evidential force of any given report uniformly to reflect aggregate unreliability. Unreliable agent behaviors (parameter: reliability ∈ {1.0, 0.75, 0.5, 0.25} gives probability a node is reliable):
• Misinformants (binomial noise): Reports are unrelated to actual outcomes and distributed as Binomial with p = 0.5 per trial; neutral overall with respect to H.
• Disinformants (negative epsilon): Adversarial lies; reports invert outcomes so that aggregate reporting distribution matches p = 0.5 − ε, tending to disconfirm the true H when B is better. Simulation settings (small artificial networks):
• Network: complete graph, size 64 nodes. Trials per step per experimenting agent: 64. ε = 0.001. Initial credences drawn uniformly at random in [0, 1]. 500 independent simulations per parameter configuration.
• Parameter grid: Strategy ∈ {Bala-Goyal baseline (equivalent to reliability 1.0), Gullible, Aligned}; Unreliability type ∈ {Binomial misinformation, Negative-ε disinformation}; Reliability ∈ {0.75, 0.5, 0.25}. Max steps: 20,000. Consensus criteria as above.
• Statistical analyses: Accuracy assessed by proportion of converged runs reaching correct consensus (B). Efficiency assessed by steps to convergence for runs reaching B. Tests: chi-square (proportions), Mann–Whitney U (steps), Kolmogorov–Smirnov for distributional differences where applicable. Significance threshold p < 0.05. Large real-world network simulations:
• Network: EU Email Core network (Leskovec & McAuley 2012), 1005 nodes, 24,929 directed edges; information flows along email direction. Out-degree centrality computed; distribution heavy-tailed.
• Runs: 10 simulations per configuration, max 25,000 steps. Due to computational limits, accuracy via convergence not emphasized; instead, track average (mean) credence over time (logged every 100 steps).
• Conditions examined: (a) Increasing random binomial misinformation levels under Gullible strategy with reliability ∈ {1.0, 0.75, 0.5, 0.25}. (b) Structurally targeted unreliability: top 10 out-degree-central nodes set unreliable, acting as (i) binomial misinformants or (ii) negative-ε disinformants, with remaining agents reliable; Gullible processing (since aligned would barely discount at <1% unreliable). Compare mean-credence trajectories vs Bala–Goyal baseline. Implementation: Simulations implemented in a custom framework (PolyGraphs project); numerous hyperparameters available but fixed as above for reported experiments.

Small complete networks (n=64):

• Convergence distributions: Presence of mis/disinformation produces long right-tailed step distributions; many runs take far longer than the median; some reach the 20,000-step cap without consensus.
• Accuracy (proportion of converged runs reaching B; from Table 4): • Baseline (Bala–Goyal, reliability 1.0): 97.8% (489/500) correct. • Gullible + Binomial misinformation: 97.2% (r=0.75), 86.2% (r=0.5), 70.0% (r=0.25). Significant accuracy degradation except at r=0.75 (ns). • Gullible + Negative-ε disinformation: 87.2% (r=0.75), 39.8% (r=0.5), 1.8% (r=0.25). Severe collapse at lower reliability. • Aligned + Binomial misinformation: 97.4% (r=0.75), 97.7% (r=0.5), 96.9% (r=0.25). Near-baseline accuracy even with substantial unreliability. • Aligned + Negative-ε disinformation: 97.6% (r=0.75), 98.4% (r=0.5), 93.8% (r=0.25). Small but significant drop at the lowest reliability (p = 0.00, χ = 9.05), still high accuracy. • Direct comparison: Aligned significantly outperforms Gullible on accuracy in all cases except binomial misinformation at r=0.75 (97.2% vs 97.4%, not significant).
• Efficiency (steps to reach B among correct runs): • Both strategies experience significantly more steps to converge in the presence of either misinformation or disinformation; more unreliability ⇒ more steps (Mann–Whitney U, p < 0.05). • Between strategies, where differences are significant, Aligned is typically slower (larger mean steps) than Gullible, indicating an accuracy–efficiency trade-off; one anomaly noted for negative-ε at r=0.5. Large real-world EU Email Core network (n=1005):
• Randomly distributed binomial misinformation under Gullible processing slows belief growth substantially (mean credence over time): • r=1.0 (baseline): mean credence ~0.8 by ~3,200 median logged steps; ~0.9 by ~10,200 median steps (Table 5). • r=0.75: ~0.9 reached only after >15,000 steps (median ~18,450); slower progression at earlier thresholds as well. • r=0.5: ~0.8 median at ~11,300 steps; typically does not reach 0.9 within 25,000 steps. • r=0.25: typically does not reach 0.8 within 25,000 steps. Kolmogorov–Smirnov tests show distributional differences across reliability levels (p < 0.05).
• Structurally targeted unreliability (top 10 out-degree-central nodes unreliable, Gullible): • Both binomial and negative-ε cases suppress mean-credence growth vs baseline; disinformation suppresses more strongly (KS tests p < 0.05). • Median logged steps to thresholds increase notably compared to baseline (Table 6): up to +27% at the 0.6 threshold with 10 central binomial misinformants; up to +60% at the 0.7 threshold with 10 central disinformants. • These effects are of similar order to those from much higher volumes of randomly distributed misinformants (e.g., up to +81% at 0.9 threshold with ~25× more unreliable nodes), highlighting the potency of strategically positioned unreliable nodes.

The simulations show that even rational Bayesian/Jeffrey agents suffer degraded epistemic performance in contaminated informational environments. With HOE about unreliability, two natural strategies diverge: denying HOE (Gullible) preserves evidential force and yields faster convergence but becomes highly vulnerable to adversarial disinformation and lower reliability, often converging incorrectly. Accommodating HOE (Aligned) by discounting testimony to the known reliability preserves high accuracy across a broad range of unreliability, including disinformation, but slows convergence to truth. Thus, addressing the research questions: misinformation and especially disinformation markedly hinder truth-finding by increasing time to consensus and, under trusting processing, by increasing error rates. Regarding the HOE debate, the results expose a consequentialist trade-off: accuracy versus efficiency. If the epistemic goal prioritizes avoiding error, Aligned is preferable; if timeliness is paramount, Gullible may be favored in high-reliability, non-adversarial settings. In real-world network structure, even minimal but centrally located unreliability depresses community confidence in truth, suggesting that structural properties mediate the impact of misinformation and can be leveraged for interventions (e.g., mitigating or monitoring highly central nodes).

The study contributes by (1) operationalizing HOE-response strategies within network epistemology, (2) distinguishing impacts of neutral misinformation vs adversarial disinformation, and (3) demonstrating an empirical trade-off between accuracy (Aligned superior) and efficiency (Gullible faster) under unreliability. In small complete networks, misinformation/disinformation slows convergence; Gullible processing suffers severe accuracy losses in disinformation-rich, low-reliability settings, whereas Aligned preserves near-baseline accuracy but at the cost of speed. In a large real-world email network, both widespread and targeted (central) unreliability depress mean credence trajectories, with targeted disinformation especially impactful, indicating that community-level, structurally informed interventions may be effective. Future research directions include: systematic variation of network topology; richer unreliable-behavior models (e.g., selective reporting, biased noise); alternative performance metrics (e.g., supermajority thresholds, central-node persuasion time); and policy simulations targeting influential nodes or trust-weighting schemes.

• Model idealizations: agents are fully rational Bayesian/Jeffrey updaters with simplified decision thresholds and uniform initialization; real cognition and heterogeneity are ignored.
• Unreliable behaviors restricted to two stylized types (neutral binomial noise and adversarial negative-ε inversion); other mechanisms (e.g., selective reporting, strategic timing) not explored here.
• Parameter space coverage is selective (e.g., ε fixed at 0.001; complete graph of size 64); results may vary with alternative settings and network sizes/topologies.
• Stopping criteria and step caps (20,000 for small networks; 25,000 for large) may bias observed convergence proportions and timing; censored runs likely underrepresent late convergences.
• Large-network analyses based on only 10 runs per configuration, limiting statistical power; accuracy assessed via mean-credence trajectories rather than convergence proportions.
• Real-world network abstraction (email edges as testimony channels) omits content, timing, and multi-channel effects; results may not generalize to all social platforms.
• Aligned strategy assumes accurate knowledge of global reliability r and applies uniform discounting; in practice, agents may have uncertain or local estimates and heterogeneous trust rules.