Dimensionality crossover to 2D vestigial nematicity from 3D zigzag antiferromagnetism in an XY-type honeycomb van der Waals magnet

The study investigates whether enhanced fluctuations from dimensionality reduction can melt a primary magnetic order while stabilizing a vestigial order in a 2D limit. Vestigial orders correspond to composite order parameters (η^Tη) that break a subset of symmetries of the primary order η. Between a fully ordered phase (region I) with both η and η^Tη nonzero and a disordered phase (region III) with both vanishing, theory predicts an intermediate vestigial phase (region II) where η = 0 but η^Tη ≠ 0. Prior work has emphasized Ising (Z2) nematicity emerging from spin or charge fluctuations in tetragonal systems; in hexagonal systems, a Z3 Potts-nematicity is expected, breaking 3-fold (or 6-fold) rotational symmetry while preserving translations. However, direct observation of intrinsic Z3 Potts-nematic order without coexisting primary order (region II) has been scarce. NiPS3 is a quasi-2D XY-type honeycomb antiferromagnet that in bulk hosts long-range zigzag AFM order (TAFM,3D = 155 K) breaking both rotational (BRS) and translational (BTS) symmetries. The authors hypothesize that enhanced fluctuations in few-layer NiPS3 suppress long-range zigzag AFM (η → 0) but retain a vestigial Z3 Potts-nematic order (η^Tη ≠ 0), leading to a dimensionality-driven crossover from region I to region II.

Vestigial orders have been proposed and observed across correlated systems, including nematicity in cuprate and iron-based superconductors, charge-4e superconductivity from pair-density-wave states, and intertwined orders in kagome metals. In tetragonal crystals, Z2 Ising nematicity is well established as a vestigial phase of spin or charge density waves, preserving translational symmetry while reducing rotational symmetry. For hexagonal systems, theory anticipates Z3 Potts-nematicity that breaks threefold (or sixfold) rotational symmetry without breaking translational symmetry. Recent experiments have reported Potts-nematic-like features in Fe1/3NbS2 and FePS3 but within phases where the primary magnetic order coexists (region I). NiPS3 is a layered honeycomb antiferromagnet with XY-type anisotropy; monolayers have D3d symmetry while bulk/few-layer stacks are monoclinic (C2h). Bulk NiPS3 exhibits zigzag AFM with wavevector at the M point, breaking rotational and translational symmetries, and prior Raman studies connected a splitting of an Eg doublet (~180 cm−1) to zigzag AFM. This work seeks intrinsic Potts-nematicity (region II) in the absence of long-range AFM by leveraging reduced dimensionality to enhance fluctuations in NiPS3.

- Sample preparation: High-quality NiPS3 single crystals were grown via chemical vapor transport. Few-layer flakes were mechanically exfoliated in an inert glovebox and transferred (some encapsulated with hBN for NV measurements). Thicknesses were first estimated by optical contrast and confirmed by AFM after measurements. - NV spin relaxometry (GHz scale): Wide-field NV relaxometry imaging on diamond membranes with proximal few-layer NiPS3 was used to map spin fluctuations via the NV spin relaxation rate, proportional to the magnetic noise spectral density and thus magnetic susceptibility χ. Measurements covered temperatures from ~4.5 K to 350 K, with an external ~70 G field. Area-averaged relaxation rates were converted to χ to compare thickness-dependent spin fluctuations (e.g., 4-layer vs ~10 nm). - Raman spectroscopy (THz scale): Backscattering Raman with 532 nm and 633 nm excitation was used to access quasi-elastic scattering (QES, 8–40 cm−1) and phonons. QES spectral weight (SWQES) was integrated over 8–40 cm−1 (excluding the breathing mode) to quantify spin fluctuations vs temperature for 2L, 3L, 4L, ~13.1 nm, and bulk. Two symmetry-sensitive Raman signatures were targeted: (i) PBRS, splitting of a doubly degenerate Eg(D3d) mode near ~180 cm−1 into Ag and Bg below TAFM,3D, indicative of broken rotational symmetry (BRS) and proportional to the composite (nematic) order parameter η^Tη; (ii) PBTS, a zone-folded phonon near ~30 cm−1 that appears below TAFM,3D due to the AFM wavevector Q = kM, indicative of broken translational symmetry (BTS) and scaling with the primary order parameter η. Magnetic-field dependence and comparison to phonon calculations were used to rule out magnon origin and assign PBTS to an M-point phonon folded to Γ by zigzag AFM. Polarization dependence confirmed single-Q anisotropy for PBTS. Multiple-Lorentzian fits extracted ΔωBRS (splitting), ΓBRS, ΓBTS, and intensity ratios IBTS/IBRS vs thickness at 7–10 K. - Linear dichroism (LD) microscopy: Polarization-resolved LD imaging at 635 nm mapped nematic domains in a large (~40 μm) 4-layer flake. The LD signal, sensitive to BRS and proportional to η^Tη, was measured as a polarization rotation Δθ. Angular dependence was recorded across domains; the crystallographic a-axis was calibrated via polarization-resolved PL. Polar plots of the lower-frequency PBRS mode intensity vs polarization corroborated domain orientations. - Monte Carlo simulations: Classical O(3) spin Monte Carlo for bilayer NiPS3 with Hamiltonian including intralayer Heisenberg exchanges J1, J2, J3, easy-plane anisotropy Dz, easy-axis anisotropy Dx, and interlayer exchange Jinter. Parameters {J1, J2, J3, Jinter, Dx, Dz} = {−1, 0, 2.22, −0.07, −0.002, 0.047} with |J1| as unit. System sizes L = 36–108, periodic in-plane, open along z. Updates included local, Wolff, and over-relaxation. The Z3 nematic order parameter m3 and its finite-size scaling with 2D Z3 Potts exponents (β = 1/9, ν = 5/6) determined the nematic transition temperature TN,2L. Correlation functions Cnematicity(r) and Cspin(r) were computed to contrast long-range nematic order with the absence of long-range spin order. Real-space snapshots of spin and nematic director textures were visualized.

- Fluctuation enhancement with reduced thickness: - NV relaxometry: Magnetic susceptibility χ (proportional to spin fluctuations) is significantly enhanced in 4-layer NiPS3 compared to ~10 nm below ~150 K; only slight enhancement above ~150 K. - Raman QES: For 2L, 3L, 4L, QES persists down to 10 K (TSQES < 10 K), indicating strong low-temperature fluctuations. Bulk exhibits a suppression of QES below TSQES ≈ 90 K (below TAFM,3D = 155 K), and a ~13.1 nm flake shows TSQES ≈ 50 K. - Symmetry analysis via Raman: - PBRS (~180 cm−1 splitting) indicates BRS (nematicity) and persists from bulk down to 2L. - PBTS (~30 cm−1 zone-folded phonon) indicates BTS (zigzag AFM) and disappears below a critical thickness tc ≈ 10.6 nm. - Thickness trends at 7–10 K: - ΔωBRS (splitting) is approximately thickness independent, supporting robust nematic order down to bilayer. - ΓBRS nearly constant across tc, with slight increase in 3L and more in 2L, reflecting nematic coherence changes. - ΓBTS shows divergent broadening as thickness approaches tc from above, signaling melting of zigzag AFM. - IBTS/IBRS decreases to zero as thickness falls below tc, consistent with loss of BTS while BRS survives. - Domain imaging and Z3 character: - LD microscopy visualizes three distinct nematic domains in 4L NiPS3, with orientations related by 120° rotations (Z3 Potts-nematicity). Polarization-dependent Raman PBRS corroborates the three domain orientations. - Monte Carlo corroboration (bilayer NiPS3 model): - Finite-size scaling of ⟨|m3|⟩ yields TN,2L = 1.285|J1| ≈ 67.1 K (same order as experimental ~120 K), confirming a 2D Z3 Potts-nematic transition. - At T = 1.25|J1| < TN,2L, Cnematicity(r) saturates to an L-independent plateau (long-range nematic order) while Cspin(r) decays with system size (no long-range spin order). Real-space snapshots show disordered spins but homogeneous nematic directors. - Overall: Thickness reduction drives a crossover from 3D zigzag AFM (BRS + BTS; region I) to a 2D vestigial Z3 nematic phase (BRS only; region II) at tc ≈ 10.6 nm.

The combined NV, Raman, LD, and Monte Carlo results demonstrate that enhanced fluctuations in reduced dimensionality suppress the primary zigzag AFM order (η → 0) while preserving a composite Z3 nematic order (η^Tη ≠ 0). The disappearance of PBTS below tc and persistence of PBRS down to 2L directly evidences rotational symmetry breaking without translational symmetry breaking—hallmarks of vestigial Potts-nematicity. Divergent ΓBTS near tc and thickness-independent ΔωBRS further separate the primary and vestigial order parameters. LD imaging reveals three domain orientations consistent with Z3 symmetry breaking. Simulations reproduce a finite-temperature Z3 transition and the decoupling between long-range nematic order and short-range spin correlations in bilayers, supporting the experimental phase identification. These findings validate the hypothesis that dimensionality-induced fluctuations can drive a crossover from a 3D ordered AFM phase to a 2D vestigial nematic phase, offering a new route to engineer exotic 2D magnetic states by embracing, rather than suppressing, fluctuations.

This work establishes a dimensionality-driven crossover in NiPS3 from 3D zigzag antiferromagnetism (with broken rotational and translational symmetries) to a 2D vestigial Z3 Potts-nematic phase (broken rotational symmetry only) below a critical thickness of ~10.6 nm. Evidence includes enhanced spin fluctuations in few layers (NV relaxometry and Raman QES), the disappearance of a zone-folded PBTS phonon while PBRS persists in Raman, direct LD imaging of three Z3 nematic domains, and Monte Carlo simulations confirming a 2D Potts-nematic transition without long-range spin order. The results highlight a strategy to discover fluctuation-stabilized 2D phases by leveraging reduced dimensionality. Future directions include: (i) exploiting enhanced 2D fluctuations to realize additional fluctuation-driven magnetic phases, (ii) determining the spin coherence length in few-layer NiPS3 with nanoscale probes to quantify partial melting of the primary order, (iii) exploring twisted/moiré NiPS3 structures where monoclinic stacking variants act as tuning fields for nematicity, and (iv) studying interplay between strongly fluctuating spins and correlated charge/exciton degrees of freedom in 2D NiPS3.

- The precise spin coherence length in few-layer NiPS3 remains undetermined; nanoscale spatially resolved probes are needed to quantify the extent of primary order melting in the vestigial phase. - Monoclinic interlayer stacking likely pins nematic domain orientations, introducing a weak structural coupling to nematicity; while deemed weak, it may influence domain configurations and measurements. - The Monte Carlo TN,2L differs quantitatively from experiment (same order but not identical), reflecting model parameter choices (classical spins, specific exchange and anisotropy values) and finite-size effects. - Thickness dependence was probed on selected flakes; systematic variability (e.g., disorder, strain, dielectric environment) could affect absolute intensity measures, mitigated here by using ratios (IBTS/IBRS) and linewidth analyses.