Dilution of expertise in the rise and fall of collective innovation

The study investigates how the balance between expert-driven invention and imitator-driven diffusion shifts as cultural trends become popular. The core hypothesis is that during boom phases, growth in imitators outpaces the supply of experts, causing a dilution of expertise that tilts production toward copying rather than creating. This shift is expected to reduce diversity and increase redundancy in cultural products. The work situates the problem across cultural evolution, social-learning theory, diffusion of innovations, and complexity science, highlighting consistent conceptual distinctions between invention (experts) and imitation (adopters). The purpose is to connect mechanistic processes of invention versus imitation to measurable changes in multi-scale diversity and to explain boom-and-bust dynamics via endogenous dilution of expertise rather than solely exogenous shocks. The importance lies in understanding and potentially forecasting stagnation in popular culture, technology trends, and information ecosystems when imitation surpasses invention capacity.

The paper synthesizes concepts from several fields that differentiate invention from imitation: Schumpeter’s view of innovation as copying unique inventions, Bass’s split between innovation (invention) and adoption (copying), social-learning’s producer–scrounger models, and exploitation–exploration in complexity and management literature. Prior work shows that expert-to-imitator proportions shape turnover and persistence of ideas; fast-changing environments benefit from more inventors to avoid outdated ideas persisting via imitation. Popular culture is novelty driven, with marginal returns from repetition (e.g., video games, cryptocurrencies, memes). Earlier models of diffusion (Bass; Henrich; neutral models) often assume fixed ratios of innovators to imitators and focus on adoption counts of a single entity. Research on cumulative culture emphasizes recombination of components over wholly novel products, transmission fidelity, and effects of component diversity on technological complexity. The authors position their contribution as integrating these insights with a data-driven, structural view of cultural products, predicting how changing imitation-to-invention ratios alter lexical diversity, information density, and structural complexity.

Datasets: Six datasets were assembled: three target boom-and-bust cases and three matched controls. Targets: (1) Atari 2600 video games (1980s), (2) cryptocurrency white papers (2009–2020), and (3) Reddit r/PunPatrol posts/comments (2018 start). Controls: (1) Commodore Vic-20 video games (same machine language/chipset as Atari), (2) scientific publications in optics (1980–2020), and (3) Reddit r/Politics posts/comments. Video games: 738 ROM cartridges (Atari and Vic-20) were downloaded (atarimania.com; tosecdev.org). Custom reverse engineering recovered assembly source code from binaries (details in Supplementary Materials). Cryptocurrencies: 1,383 white papers collected (whitepaperdatabase.com, allcryptowhitepapers.com, whitepaper.io), dated by document production date or first transaction. Pdf text extracted via Textract. Control scientific papers (optics) were retrieved via PubMed and processed similarly. Reddit: 10,761 posts/comments collected via PRAW across the two subreddits over 330 days using a depth-first strategy. Text processing: English tokenization with NLTK v3.6.1; stop words removed; synonym normalization using WordNet synsets by replacing each non-stop-word with its most frequent synonym from the Brown corpus. This collapses synonym space to focus on conceptual components. Representation: Cultural products are modeled as hierarchical sequences of symbols forming sentences and products. This is captured as a three-layer structure (symbols, sentences, products) with bipartite networks between adjacent layers. For games, assembly code provides natural symbol and sentence boundaries (see Supplementary Materials). Modeling: A modified Pólya urn model represents invention and imitation. An urn U contains components (symbols). At each step, an agent selects a component: choosing an existing component reflects imitation and adds q copies (reinforcement) to the urn; choosing a new component reflects expert invention and adds the chosen new component plus 1+μ additional new components adjacent to it (novelty expansion). The urn parameters are dynamically linked to a population of creators: experts x(t) and imitators y(t), with total cultural production N(t)=x(t)+y(t) (each agent produces one artifact per time unit). In this linkage, μ equals the expert population size x(t) (invention capacity), and q equals the imitator population size y(t) (imitation reinforcement). Agents can switch strategies with probabilities δx (imitator→expert) and δy (expert→imitator). The total creator population grows or shrinks based on expected success, defined as the probability p(t) that a product contains at least one novel component. p(t) depends on the probability Ω(t) of drawing a new component and product size C (details in Supplementary Materials). Metrics of multi-scale diversity: (1) Lexical diversity via Zipf’s law exponent b, fitting the tail (last 1.5 orders of magnitude in rank) of the rank-frequency distribution; the model predicts f(r)∼r^−q/μ, so larger q/μ implies steeper slopes (higher b) and lower lexical richness. (2) Information density via Lempel–Ziv–Welch compression: define information density as L(x)/(L log2(a)), where L(x) is compressed length, L is product length (symbols), and a is alphabet size. With constant q and μ, information density decays toward an entropy-production rate; higher μ>q yields higher density (less redundancy), while q>μ yields lower density (more redundancy). (3) Structural complexity via normalized Block Decomposition Method (NBDM) applied to the bipartite network adjacency matrix B of symbols–sentences, using Coding Theorem estimates of algorithmic complexity (PyBDM). NBDM(B)=BDM(B)/(L log a), normalized by matrix size and alphabet. Shared symbols across sentences reduce structural complexity (increase redundancy). Simulations: Synthetic cultural histories (e.g., 10 runs of 1000 products per parameter pair) were generated varying q and μ to benchmark how rank-frequency slopes, information density trajectories, and NBDM respond to imitation/invention ratios. Empirical analyses: For each dataset, the team tracked over time the Zipf tail exponent, information density versus accumulated symbol counts, and log10 NBDM versus time, with comparisons to control datasets. Statistical significance of complexity drops was assessed where applicable (see Supplementary Materials).

- All three target domains exhibited boom-and-bust cultural productivity: rapid growth followed by rapid decline. The population-based urn model reproduces these trajectories and infers a shift toward imitators (higher q/μ), consistent with dilution of expertise. In the Atari case, the inferred abundance of imitators was greatest among the targets; even when experts later outnumbered imitators during decline, the earlier loss of diversity was not reversed. - Atari 2600 vs. Commodore Vic-20 (control): Early phases showed similar Zipf exponents, but starting around 1982 (onset of exponential growth) Atari’s tail exponent increased markedly, indicating reduced lexical richness and higher q/μ. Information density trends diverged around the boom, with Atari showing greater redundancy than Vic-20. Structural complexity (log10 NBDM) for Atari declined significantly during peak cultural diversity, consistent with imitation-driven redundancy, whereas Vic-20 showed no clear temporal trend and no significant complexity drops. - Cryptocurrencies vs. optics (control): Zipf tail exponents began departing from the control in 2017, aligning with an explosion of redundant products. Information density showed similar overall slopes to optics but with earlier reductions in crypto (by 2015) as new documents increasingly reused prior symbols, lowering per-symbol information. Structural complexity in crypto decreased significantly during the 2018 crash and the preceding year; optics remained relatively stable over more than a decade despite fluctuations in output. - Reddit r/PunPatrol vs. r/Politics (control): Starting similarly, by day ~120 or ~20,000 symbols, r/PunPatrol became less lexically rich than r/Politics. Information density in r/PunPatrol decreased sharply between days ~130–150, coinciding with peak productivity, indicating heightened redundancy; r/Politics maintained higher per-symbol information. Post-level structural complexity changes were not significant, likely due to short message length, but coarse-graining posts into 10-day bins revealed a marked complexity decrease during r/PunPatrol’s boom-and-bust, with r/Politics remaining fairly constant. - Model–data alignment: Simulations predict that increasing imitation steepens rank–frequency slopes, lowers information density, and simplifies symbol–sentence networks (lower NBDM). Empirical results across domains matched these predictions. The transition point corresponds to a dilution of expertise, when the expanding trend outstrips the expert community’s capacity to inject novelty.

The findings support the hypothesis that an endogenous shift from invention to imitation—driven by rapid growth in imitators relative to experts—produces observable declines in diversity, information density, and structural complexity. This mechanism explains boom-and-bust cycles without requiring solely exogenous shocks. By directly representing cultural products and their internal structure, the framework links measurable multi-scale diversity to underlying generative processes. The work advances cultural evolution theory by integrating recombination-based innovation and open-ended evolution concepts, emphasizing novelty generation and maintenance rather than only selection among fixed traits. Practically, recognizing dilution of expertise as trends scale may help interpret stagnation in popular culture, technology, and online communities and suggests monitoring strategies for early warning of excessive imitation that can precede declines in innovation and diversity.

This study introduces a data-driven, structural framework connecting invention–imitation dynamics to measurable diversity patterns across multiple cultural domains. By coupling a modified Pólya urn to populations of experts and imitators, the authors explain boom-and-bust cycles and demonstrate that rapid popularity growth can dilute expertise, leading to reduced diversity, lower information density, and simplified structural complexity. Contributions include: (1) a unifying mechanistic model for collective innovation and diffusion; (2) multi-scale diversity metrics applicable across codes, texts, and memes; and (3) empirical validation in video games, cryptocurrencies, and social media with matched controls. Future research directions include incorporating memory effects and richer population structures, refining estimates to reduce representation biases, and developing system-specific early warning indicators to monitor and mitigate excessive imitation and its economic or cultural impacts.

- Measurement constraints: For Reddit, short communications limit detection of structural-complexity changes at the post level; coarse-graining improved signal but precluded formal significance testing. - Data coverage: Cryptocurrency documentation was limited to parseable files; many projects may have existed without accessible or standardized white papers, potentially biasing samples. - Modeling simplifications: The model assumes one artifact per agent per time unit and directly ties invention/reinforcement rates to expert/imitator counts. Strategy-switching rates and success-driven population changes are simplified. More complex memory, learning biases, and population structures are not included. - Representation biases: Text processing choices (stop-word removal, synonym collapsing) and assembly-code parsing may influence measured diversity; authors note that reducing such biases would yield more accurate imitation/invention estimates. - External factors: Although the model explains boom-and-bust endogenously via expertise dilution, real systems may also involve exogenous shocks (e.g., market, regulatory, or technological shifts) not explicitly modeled.