Predictability Limit of Partially Observed Systems

The paper addresses how partial observation (temporal sampling) affects the predictability of complex dynamic systems such as epidemics, social media activity, and cyber-security events. Forecasting often relies on time-series models trained on observed data, yet in practice only a subset of events are recorded due to platform limitations, reporting delays, or deliberate manipulation. The research question is: how does data loss due to sampling impact measures of predictability and the accuracy of models learned from such data? Prior metrics include autocorrelation and permutation entropy (PE) as model-free indicators of complexity and predictability. The authors hypothesize and demonstrate that sampling systematically reduces predictability—decreasing autocorrelation and increasing PE—and that this loss cannot be fully recovered using informative external signals. They provide a theoretical framework and empirical validation across multiple real-world domains.

The study situates itself within time-series forecasting of complex phenomena using stochastic point processes, autoregressive models, and hidden Markov models applied to crime, social unrest, terrorism, epidemics, mobility, correspondence, online activity, and ecological systems. It highlights the pervasive issue of partial observability in social and techno-social data (e.g., Twitter API sampling, under-reporting in health data) and reviews statistical responses to missing data such as imputation, ensemble forecasts, and representativeness assessments, noting their limited applicability to temporal sequences. Predictability metrics include autocorrelation (widely used in finance) and permutation entropy (PE), a nonlinear, model-free complexity measure linked to Kolmogorov–Sinai entropy, used in ecology, epidemic forecasting, mobility, and paleoclimate anomaly detection. Prior work showed empirical correlation between model predictability and PE. The authors also discuss the risks of drawing conclusions from incomplete data and the need to quantify sampling-induced biases.

Modeling partial observation: The ground-truth process X = [X1, …, XT] is a time series of event counts. The observed time series Y = [Y1, …, YT] results from independent stochastic sampling of each event with probability p, i.e., Yt ~ Binomial(Xt, p). The expected observed count is E[Y] = p E[X]. An external signal S = [S1, …, ST] provides potentially informative covariates driving the system.

Theoretical characterization: Let ΣX and ΣY be covariance matrices of X and Y. The authors derive ΣY ≈ p² ΣX + p(1−p) E[X] I, where I is the identity matrix (Theorem 1). From this, for stationary X, the autocorrelation of Y at lag (i,j) is ρYiYj ≈ p² Cov(Xi, Xj) / (p² Var(X) + p(1−p) E[X]) (Corollary 1), which increases monotonically with p (Corollary 2). The covariance between Y and any external signal S scales linearly with p: Cov(Y, S) = p Cov(X, S) (Corollary 3).

Predictability measures: The study assesses predictability via (i) autocorrelation (Pearson correlation between time-lagged values), (ii) permutation entropy (PE), using weighted permutation entropy normalized by log2(d!) with parameters (embedding dimension d ∈ [2,5], delay τ ∈ [1,7]) selected by grid search minimizing normalized PE on the ground-truth series and then applied to sampled series, and (iii) mutual information (MI) between observed and external signals (computed with PyInform). For synthetic data, they also evaluate forecast error using autoregressive models.

Empirical design: They generate sampled series by stochastically downsampling ground-truth data at multiple rates p ∈ [0,1] and compute relative changes in PE and autocorrelation (normalized by the p=1 baseline). Datasets: (a) Epidemics—weekly state-level counts for eight diseases (Chlamydia, Gonorrhea, Hepatitis A, Influenza, Measles, Mumps, Polio, Whooping cough) from NNDSS/ILINet; external signal: Google Flu Trends for influenza. (b) Social media—Twitter data (June–November 2014) for >600k users, analyzing daily popularity of 100 most-used hashtags and activity of 150 most active users. (c) Software development and cryptocurrencies—GitHub repository popularity signals (watches, forks, create events) Jan–Mar 2015 for projects related to BTC, LTC, XMR, XRP; external signals: historical cryptocurrency prices from Kaggle. (d) Synthetic data—autoregressive processes with sampling to examine prediction error and heteroskedasticity effects.

• Sampling reduces predictability: As sampling rate decreases, autocorrelation of observed signals decreases and permutation entropy increases across domains. This loss is monotonic in p and matches theoretical predictions from the derived formulas.
• Analytical results: ΣY ≈ p² ΣX + p(1−p)E[X]I; ρYiYj ≈ p² Cov(Xi,Xj)/(p² Var(X)+p(1−p)E[X]); Cov(Y,S) = p Cov(X,S). Autocorrelation of Y increases with p; covariance with an external signal scales linearly with p.
• Epidemics: For eight diseases, PE over 1-year moving windows increases as p decreases, and autocorrelation decreases; empirical autocorrelation loss aligns with Eq. (3). For influenza with Google Flu Trends as S, covariance increases linearly with p; Pearson correlation shows little change due to Var(X) ≫ E[X], consistent with σY ≈ σX √(p² + p(1−p)E[X]/Var(X)) and the correlation ratio canceling. MI between Google Flu Trends and influenza activity also decreases as sampling increases.
• Social media: For the 100 most popular hashtags, relative autocorrelation declines with decreased p; empirical decay closely fits theoretical predictions. Weighted PE increases for hashtag popularity and user activity; for user activity, 63% of users show predictability loss. With common Twitter sampling (Decahose ~10%, streaming API ~1%), relative autocorrelation may be about half of that with full Firehose data, underscoring potential bias.
• Cryptocurrencies and GitHub: Covariance between repository popularity and coin prices decreases with p; for coins with lower variance relative to mean, Pearson correlation also declines with p. MI decreases for BTC and LTC; XMR and XRP appear independent of the external signal.
• Synthetic data: In sampled autoregressive processes, forecast error increases as p decreases; at low p, AR forecasts are no better than Poisson models assuming independent events. Sampling induces heteroskedasticity, degrading prediction performance.

The findings directly answer the research question: partial observation intrinsically degrades predictability of dynamic systems. Even with random sampling, the observed process exhibits reduced autocorrelation and increased complexity (higher permutation entropy), indicating a fundamental limit on what can be predicted from incomplete data. External signals, though informative, cannot fully recover predictability lost to sampling because the covariance with the observed series scales linearly with the sampling rate. These results are robust across epidemiological, social media, and software development domains and are supported by theory and experiments. The implications are broad: forecasts and inferences made from sampled data (e.g., Twitter APIs, partial health surveillance) may be systematically biased, leading to over-optimistic assessments of model performance or mistaken causal conclusions. The work emphasizes the need to account for sampling-induced changes to dynamics in forecasting and causal inference, including methods like interrupted time series that rely on temporal correlation structures.

The paper provides a theoretical and empirical framework establishing that temporal sampling imposes a fundamental, irreversible limit on predictability in partially observed systems. It shows analytically how sampling reshapes covariance structure and autocorrelation, and validates these effects across real-world domains and synthetic data. Practically, the results caution against uncritical use of sampled data for forecasting and causal inference, and highlight the necessity of adjusting for sampling-induced biases. Future research should develop statistical tools and sampling methodologies to correct or mitigate these biases, improve robustness of predictive models under partial observability, and ensure valid inference in studies relying on temporally sampled data.