This paper investigates the phase transition in the XY-type honeycomb magnet NiPS₃ from a 3D zigzag antiferromagnetic (AFM) order to a 2D Z₃ vestigial Potts-nematicity in few layers. Using NV spin relaxometry, optical spectroscopy, and Monte Carlo simulations, the study reveals enhanced spin fluctuations in thinner NiPS₃ samples, leading to the disappearance of broken translational symmetry (PBTS) and the survival of broken 3-fold rotational symmetry (PBRS) in few-layer NiPS₃. The experimental findings are supported by large-scale Monte Carlo simulations, confirming the absence of long-range zigzag AFM order but the presence of a Z₃ vestigial Potts-nematic phase in bilayer NiPS₃. The results highlight the role of strong quantum fluctuations in creating novel 2D magnetic phases.

Zeliang Sun, Gaihua Ye, Mengqi Huang, Chengkang Zhou, Nan Huang, Qiuyang Li, Zhipeng Ye, Cynthia Nnokwe, Hui Deng, David Mandrus, Zi Yang Meng, Kai Sun, Chunhui Du, Rui He, Liuyan Zhao