Machine learning the microscopic form of nematic order in twisted double-bilayer graphene

The study explores how machine learning can solve inverse problems in quantum materials where extracting microscopic models directly from experimental observations is challenging. Scanning tunneling microscopy (STM) yields rich, image-like data that encode many-body physics, and moiré superlattices offer tunability and large unit cells that enhance access to intra-unit-cell information. The research focuses on twisted double-bilayer graphene (TDBG), where electronic nematic order breaks rotational symmetry and has been observed experimentally. The central goal is to learn the microscopic form of nematicity, distinguishing between graphene-scale (GN) and moiré-scale (MN) contributions, and to disentangle them from heterostrain, by training convolutional neural networks (CNNs) on local density of states (LDOS) data across energies. This approach aims to reconstruct parameters of an effective Hamiltonian from STM data and to determine whether multi-energy correlations enable reliable identification of nematic order and strain.

Recent advances have applied machine learning to quantum many-body physics for tasks such as solving inverse problems and analyzing imaging experiments (STM, ARPES, electron microscopy). Prior STM-based ML studies often addressed classification or denoising rather than learning microscopic Hamiltonian parameters. In moiré graphene systems, numerous correlated phases have been reported, including insulating states, magnetism, superconductivity, and nematic order. Evidence for nematicity in TDBG has been observed, but a systematic extraction of its microscopic form (GN vs MN contributions) has been lacking. Previous ML works have identified nematicity’s presence or performed phenomenological analysis, while this study targets learning the underlying couplings that break symmetries, and distinguishes them from strain, a ubiquitous source of C3 symmetry breaking in moiré systems.

- Physical modeling: Start from a continuum model of TDBG with point group D3. Model electronic nematic order as a two-component order parameter transforming under the E representation, with two classes: moiré nematicity (MN), which modifies effective inter–moiré-unit-cell couplings, and graphene nematicity (GN), which acts locally on graphene-layer degrees of freedom. Allow for mixing of MN and GN terms. Include uniaxial heterostrain parameterized by intensity ε and angle θ_ε via a deformation of in-plane coordinates for each rotated Bernal bilayer. - Synthetic data generation: Compute LDOS maps D_ω(r) and point spectra D_r(ω) from the continuum model for randomly sampled parameter sets β, including Φ_MN, Φ_GN (in 0.001–0.1 eV), relevant phases/angles (mapped via arctan2 for periodic variables), and strain parameters (ε in 0–0.8% and θ_ε in 0–π/3 when strain is included). Add Gaussian noise to LDOS to reduce overfitting and emulate experimental noise. Generate 12,000 images split into training (60%), validation (20%), and test (20%). - Inputs to CNN: Use multi-channel inputs combining (i) LDOS maps D_ω(r) at several energies spanning valence/conduction flat bands and remote bands (e.g., ω0 = −35, −15, 1, 23 meV) and (ii) point spectra D_r(ω) at selected high-symmetry stacking positions (e.g., BAAC, ARAB, ABCA) converted to scaleogram images for compatibility with CNNs. - CNN architecture and training: Convolutional layers (3×3 kernels; filters 16–32–32–16) each followed by batch normalization and max pooling (2×2). After parallel Conv-Batch-MaxPool branches for each input channel, concatenate and pass through dense and dropout (20%) layers; final linear layer outputs target parameters β. Train with mean squared error loss, Adam optimizer (learning rate 0.001), batch size 64. Map angular targets (e.g., nematic director orientation) to two-component vectors (cos 2θ, sin 2θ) to address periodicity, then recover θ via arctan2 for evaluation. - Tasks: (1) Predict nematic director orientation θ from single-energy D_ω(r). (2) Learn microscopic form parameters β = {Φ_MN, Φ_GN, α} using multi-energy channels; show insufficiency of single-energy inputs. (3) Include strain and predict ε, θ_ε alongside Φ_MN, Φ_GN. (4) Apply trained networks to experimental STM data (D^exp) across fillings n_s, using same channel configuration, to infer β for real samples.

- Nematic director orientation: A CNN trained on single-energy LDOS maps can accurately predict the nematic director orientation θ when nematic intensity is non-negligible. Mapping labels to (cos 2θ, sin 2θ) avoids errors at 0 and π. - Necessity of multi-energy inputs: Single-energy LDOS images are insufficient to determine the microscopic form of nematicity (GN vs MN) and parameter α; qualitatively different nematic couplings can produce visually similar single-energy patterns. Including multiple D_ω(r) energies and D_r(ω) scaleograms enables accurate reconstruction of α, Φ_MN, and Φ_GN (high R^2 and low MAPE on test data). - Disentangling strain and nematicity: When uniaxial heterostrain is included, the CNN accurately predicts Φ_MN and Φ_GN and learns strain intensity ε. Prediction of strain angle θ_ε is accurate except when ε is very small, where θ_ε becomes effectively undefined; excluding small-ε samples restores accuracy. Mean absolute error in nematic intensities remains uniformly low even with variable strain. - Application to experiments: Applying the trained model to STM data across fillings n_s shows finite nematicity and strain at all fillings. For n_s ≥ 0.47, the network finds MN dominates over GN, consistent with prominent stripe patterns; simulated LDOS using predicted parameters reproduces experimental features across energies. Predicted strain is relatively constant across n_s and falls in the expected ε ≈ 0.003–0.1% range at higher fillings, while larger apparent ε at low fillings likely reflects model–experiment discrepancies the network compensates for.

The results demonstrate that multi-channel CNNs leveraging correlations across energies can perform Hamiltonian learning from STM data, extracting not only the presence but also the internal structure and strength of nematic order. Crucially, the approach differentiates electronic nematicity from heterostrain even when both break the same rotational symmetries, addressing a central challenge in moiré systems. The study clarifies which STM data are most informative: multi-energy LDOS maps combined with point spectra provide complementary constraints that single-energy images lack. The procedure is robust to noise and, per supplementary analysis, to disorder, indicating general applicability. Beyond TDBG, the framework can be extended to other moiré materials and instabilities (e.g., correlated insulators and superconductivity), offering a path to systematically infer effective microscopic parameters from experimental imaging data.

This work introduces a machine-learning-based framework that learns the microscopic form of electronic nematicity in TDBG from STM LDOS data. By combining synthetic training data from a continuum model with multi-channel inputs spanning several energies and positions, CNNs successfully recover nematic director orientation, distinguish between graphene-scale and moiré-scale nematic contributions, and disentangle them from uniaxial heterostrain. Application to experimental data reveals dominant moiré nematicity at higher fillings with finite strain, and reproduces key spectral features. The approach provides a general strategy for solid-state Hamiltonian learning from imaging experiments and can be extended to other correlated phases in moiré systems. Future work should refine models to better capture low-filling regimes, incorporate broader energy ranges and disorder effects, and apply the method to additional materials and orders such as superconductivity and correlated insulators.

- The continuum-model parameterization retains only leading terms in an infinite-dimensional coupling space; full reconstruction of all possible nematic couplings is not feasible with finite-resolution data. - Predictions of angular parameters (nematic director θ and strain angle θ_ε) are unreliable when the corresponding intensities are very small (weak nematicity or ε → 0), where angles have little or no effect on LDOS. - Some discrepancies between model and experiment at low fillings suggest model misspecification or missing physics; the network compensates by attributing larger strain, indicating sensitivity to modeling assumptions. - Results rely on quality and representativeness of synthetic training data and the chosen energy channels; single-energy inputs are insufficient to identify microscopic form. - The study assumes weak spin–orbit coupling and valley-diagonal nematicity; other interactions or orders not included could affect generalizability.