Dimensionality crossover to 2D vestigial nematicity from 3D zigzag antiferromagnetism in an XY-type honeycomb van der Waals magnet

Fluctuations and disorder effects are substantially enhanced in reduced dimensionalities. While they are mostly considered as the foe for long-range orders, fluctuations and disorders can also stimulate the emergence of novel phases of matter, for example, vestigial orders. Taking two-dimensional (2D) magnetism as a platform, existing efforts have been focused on maintaining 2D long-range magnetic orders by suppressing the effect of fluctuations, whereas the other side, exploiting fluctuations for realizing new 2D magnetic phases, remains as an uncharted territory. Here, using a combination of nitrogen-vacancy (NV) spin relaxometry, optical spectroscopy, and Monte Carlo simulations, we report, in an XY-type honeycomb magnet NiPS3, the phase transition from the zigzag antiferromagnetic (AFM) order in three-dimensional (3D) bulk to a new Z3 vestigial Potts-nematicity in 2D few layers. Spin fluctuations are shown to significantly enhance over the GHz-THz range as the layer number of NiPS3 reduces, using the NV spin relaxometry and the optical Raman quasi-elastic scattering. As a result, the Raman signatures of the zigzag AFM for bulk NiPS3, a zone-folded phonon at ~30 cm−1 from the broken translational symmetry (PBTS) and a degeneracy lift of two phonons at ~180 cm−1 for the broken 3-fold rotational symmetry (PBRS), evolve into the disappearance of PBTS and the survival of PBRS in few-layer NiPS3, with a critical thickness of ~10 nm. The optical linear dichroism microscopy images all three nematic domain states in a single few-layer NiPS3 flake. The large-scale Monte Carlo simulations for bilayer NiPS3 model confirms the absence of long-range zigzag AFM order but the formation of the Z3 vestigial Potts-nematic phase, which corroborates with the experimental finding of 3-fold rotational symmetry breaking, but translational symmetry restoring in <10 nm NiPS3. Our results demonstrate the positivity of strong quantum fluctuations in creating new phases of matter after destroying more conventional ones, and thus, offer an unprecedented pathway for developing novel 2D phases.

Zeliang Sun, Gaihua Ye, Mengqi Huang, Chengkang Zhou, Nan Huang, Qiuyang Li, Zhipeng Ye, Cynthia Nnokwe, Hui Deng, David Mandrus, Zi Yang Meng, Kai Sun, Chunhui Du, Rui He, Liuyan Zhao