Learning the dynamics of complex systems is challenging due to the exponential increase in possible transitions with system size. Traditional methods often involve coarse-graining or projecting degrees of freedom into a smaller subspace. This research proposes a novel approach using transformer neural networks, known for their success in natural language processing and computer vision, to learn dynamics without these limitations. The key hypothesis is that transformers, capable of representing numerous and nonlocal rules, can learn the underlying dynamics from a single trajectory, enabling predictions under novel conditions. This is important because it allows for the study of large, complex systems that were previously intractable due to computational limitations. The ability to learn dynamics directly from observation, without explicit enumeration or approximations, would revolutionize fields requiring analysis of complex systems dynamics. This paper aims to validate this hypothesis and demonstrate the methodology's efficacy using a specific model. The ability to predict emergent behavior from limited data is crucial across multiple scientific domains, including material science, fluid dynamics, and biological systems, highlighting the significance of this research.

Prior research has explored learning deterministic and stochastic dynamics. Maximum likelihood estimation has been used for small state spaces or deterministic systems like cellular automata. Methods exist for learning intermolecular potentials from particle trajectories, and machine learning has been applied to rediscover Newtonian gravity from solar system data. Physics-informed neural networks have successfully predicted fluid dynamics and turbulent flows. However, many approaches rely on dimensionality reduction or coarse-graining techniques, which can introduce biases or inaccuracies. This work differs by directly learning the high-dimensional dynamics using transformers, avoiding explicit reduction of the system’s degrees of freedom, addressing a crucial gap in existing methodology. Existing work either simplifies the system under study or requires substantial computational resources. This paper proposes a method that overcomes these limitations.

The study employs a lattice model of active matter simulated using continuous-time Monte Carlo dynamics. A transformer neural network is trained on a single trajectory of this model at a density where small, dispersed clusters are observed in the steady state. The transformer is provided with information about the types of possible moves (single particle rotations and translations), but not the specific rates. The training process maximizes the log-likelihood of the observed trajectory. Two training modes are used: Mode 1 allows the transformer to freely predict all transition rates, while Mode 2 classifies transitions into a fixed number of classes, and a second network determines the rates for each class. This approach allows for determining the number of distinct dynamical processes. The trained transformer can then be used to generate new trajectories, even at densities different from those observed during training. To capture the dynamics, the transformer uses a multi-head attention mechanism to learn which parts of the configuration are relevant for each transition. The embedding of particle positions and orientations are input into multiple layers of the transformer consisting of alternating processes of attention and application of fully-connected neural networks. The output of this network is used to calculate transition rates, which are further used in Monte Carlo simulations to generate new trajectories. In Mode 2, a classification step, implemented using a fully-connected network with softmax activation, preceeds the rate calculation. The backpropagation algorithm with the AdaBelief optimizer is utilized for training, enabling the efficient adjustment of network weights to maximize the log-likelihood. The number of epochs required for convergence depends on the size of the model and data.

The transformer successfully learns the dynamics of the active matter model. In Mode 2, the network identifies four distinct classes of moves, matching the underlying dynamics. The learned transition rates closely match those of the original model. Importantly, the transformer accurately predicts the existence and characteristics of a motility-induced phase separation (MIPS) at higher densities than those observed during training. This demonstrates the ability of the transformer to learn emergent behavior not explicitly present in the training data. Quantitative comparisons of time-averaged observables (fraction of particles with four occupied neighbors, variance, number of clusters, average cluster size) between the original and learned dynamics show strong agreement at different densities, further validating the accuracy of the learned dynamics. The method correctly identifies the translational invariance and local nature of the interactions. Analysis of forbidden processes reveals small deviations, highlighting the balance between model accuracy and complexity. The capability to extrapolate beyond training conditions, combined with the ability to determine the number of distinct dynamical processes (indicated by Nθ), makes this approach more efficient and informative than existing methods.

The findings demonstrate that transformer networks can effectively learn stochastic dynamics from limited observational data, overcoming the limitations of traditional methods that require explicit enumeration of rates or dimensionality reduction. The successful prediction of the MIPS transition highlights the ability to learn emergent behavior, which was not explicitly observed during training, showcasing the power of this approach. This methodology's applicability extends beyond the specific active matter model considered, suggesting a broader impact on fields dealing with complex systems. The ability to accurately capture not only the numerical values of transition rates but also the underlying structure of the dynamics offers valuable insights into the system's behavior. Future research can explore applications to diverse systems, including more complex interactions, higher dimensions, and different types of stochastic dynamics.

This paper presents a novel method for learning and predicting stochastic dynamics using transformer neural networks. The approach successfully learns the dynamics of an active matter model from a single trajectory, accurately predicting emergent behavior at conditions not seen during training. The ability to learn high-dimensional dynamics without explicit rate enumeration or coarse-graining opens up opportunities for studying complex systems previously intractable due to computational constraints. Future work will explore applications to other systems and expand the capabilities of the proposed method.

The current study focuses on a specific active matter model. While the results are promising, further research is needed to assess the generalizability of the method to other types of systems and more complex dynamics. The accuracy of the learned dynamics might depend on the length of the training trajectory and the choice of hyperparameters. Although the transformer can learn long-range interactions, computational limitations may arise when dealing with extremely large systems or very long trajectories. The assumption of time-independent rates might not be applicable to all systems. The introduction of volume exclusion constraints also introduces limitations. Therefore, more robust analysis of constraints is necessary in future iterations.