The interplay between dimensionality, fluctuations, and the emergence of novel phases of matter is a central theme in condensed matter physics. While reduced dimensionality typically enhances fluctuations that disrupt long-range order, it can also lead to the formation of vestigial orders—phases that retain some symmetry-breaking characteristics of a primary order even after the primary order itself has been suppressed. This paper focuses on exploring this phenomenon in two-dimensional (2D) magnetism, a field where maintaining 2D long-range magnetic orders has been the primary focus. In contrast, this research delves into the under-explored territory of leveraging fluctuations to create new 2D magnetic phases. The study uses NiPS₃, an XY-type honeycomb magnet, as a model system due to its inherent spin anisotropy and potential for strong spin fluctuations. NiPS₃'s transition from a 3D zigzag AFM order to a novel 2D phase upon reduction in dimensionality is investigated to demonstrate the positive role of quantum fluctuations in generating new phases of matter. The significance of this research lies in its potential to provide a novel pathway for developing 2D magnetic materials with unique properties and functionalities not readily achievable through conventional methods.

Vestigial orders, characterized by the partial melting of a primary order parameter while retaining aspects of its symmetry breaking, have been observed in various quantum material systems, including cuprate and iron-based superconductors. These orders, often nematicity—the breaking of rotational symmetry while preserving translational symmetry—arise from enhanced fluctuations or disorder. Experimental realizations have predominantly focused on nematicity in tetragonal or hexagonal systems, where Z₂ Ising-nematicity and Z₃ Potts-nematicity, respectively, have been observed. Recent studies have reported Potts-nematicity characteristics in the long-range ordered magnetic phases of some materials. However, the existence of intrinsic Potts-nematic order without the primary order has been less explored. The current research builds upon this background by investigating the dimensionality-driven transition in NiPS₃ as a means of promoting fluctuations and potentially accessing a novel vestigial nematic phase. Previous research on 2D magnetism predominantly focuses on techniques to maintain long-range magnetic orders. The presented study represents a departure from this trend, instead investigating the potential of using enhanced fluctuations to create new 2D phases.

The study employed a multi-pronged approach combining experimental techniques and theoretical modeling to comprehensively characterize the magnetic phase transition in NiPS₃. Nitrogen-vacancy (NV) spin relaxometry was used to spatially visualize and quantify spin fluctuations in exfoliated NiPS₃ flakes with varying thicknesses (from bulk to a few layers). This technique exploits the high sensitivity of NV centers in diamond to fluctuating magnetic fields, allowing for the measurement of magnetic susceptibility, a quantity directly related to the strength of spin fluctuations. The measurements were performed at varying temperatures to track the evolution of fluctuations with temperature. Complementary to NV relaxometry, quasi-elastic scattering (QES) in Raman spectroscopy was utilized to probe spin fluctuations over a wider frequency range (GHz to THz). The temperature dependence of the integrated QES intensity provided additional insights into the spin dynamics. Raman spectroscopy itself was also employed to distinguish between the broken rotational symmetry (BRS) and broken translational symmetry (BTS) of magnetic phases. The characteristic phonon modes, and their splitting with temperature and thickness dependence, serve as fingerprints of the different magnetic phases present in NiPS₃. Specifically, the mode at ~30 cm⁻¹ was attributed to BTS and the modes around 180 cm⁻¹ to BRS. Scanning optical linear dichroism (LD) microscopy was implemented to image nematic domains in few-layer NiPS₃, providing direct visualization of the symmetry breaking. The orientation of the nematic order parameter was calibrated through polarization-resolved photoluminescence (PL) measurements. To support the experimental findings, large-scale Monte Carlo simulations were carried out on a bilayer NiPS₃ model. These simulations provided insights into the spin configurations and order parameters in the different phases and confirmed the absence of long-range AFM order and the existence of a long-range Potts-nematic order. The Hamiltonian used in the simulations included intralayer Heisenberg exchange couplings, single-ion anisotropies, and an interlayer exchange coupling.

The experimental results demonstrate a clear enhancement of spin fluctuations as the thickness of NiPS₃ is reduced from 3D bulk to 2D few layers. Both NV relaxometry and Raman QES consistently reveal stronger fluctuations in thinner samples at low temperatures. The Raman spectroscopic analysis reveals a critical thickness of ~10 nm, below which the broken translational symmetry (BTS) signature disappears, while the broken rotational symmetry (BRS) signature persists. This indicates a phase transition from the 3D zigzag AFM order (characterized by both BRS and BTS) to a state with BRS but no BTS. LD microscopy images revealed three distinct nematic domain states in a single few-layer NiPS₃ flake, confirming the presence of Z₃ Potts-nematicity. Monte Carlo simulations of a bilayer NiPS₃ system corroborated the experimental findings. The simulations showed the absence of long-range zigzag AFM order but the presence of a long-range Z₃ vestigial Potts-nematic phase, consistent with the experimental observation of BRS but no BTS below the critical thickness. The simulated nematic order parameter exhibited a critical temperature in good agreement with the experimental results. The simulations also showed that while spin correlation decayed with increasing distance, indicating a disordered spin arrangement, the nematic correlation reached a plateau, demonstrating the long-range nature of the vestigial order.

The findings presented directly address the research question of whether enhanced fluctuations in 2D materials can lead to novel magnetic phases. The observed dimensionality-driven crossover from 3D zigzag AFM order to 2D Z₃ Potts-nematicity unequivocally demonstrates that strong fluctuations can indeed facilitate the emergence of a new phase by partially melting a more conventional order. The distinct behavior of the primary and composite order parameters across the critical thickness further supports this conclusion. The agreement between the experimental results and Monte Carlo simulations strengthens the validity of the proposed phase transition and the identification of the novel nematic phase. The research opens up new avenues in 2D magnetism research by highlighting the potential of exploiting, rather than suppressing, fluctuations to discover and design new magnetic phases. The study provides a clear example of how a change in dimensionality can trigger a fundamental alteration in the ground state of a magnetic system, leading to emergent properties that are not present in the higher-dimensional counterpart.

This study demonstrates a dimensionality-driven crossover from 3D zigzag antiferromagnetism to 2D Z₃ Potts-nematicity in the XY-type honeycomb magnet NiPS₃. The enhanced spin fluctuations in reduced dimensions play a crucial role in this transition, highlighting the potential of exploiting fluctuations to create novel magnetic phases. The experimental results and theoretical simulations strongly support the findings, opening exciting avenues for exploring spin-fluctuation-driven phenomena in 2D materials. Future research could focus on determining the spin coherence length in 2D NiPS₃, investigating the impact of moiré superlattices, and exploring the interplay between spin and charge degrees of freedom in this unique system.

The study primarily focuses on a specific material system, NiPS₃. The generalizability of the observed phase transition and the role of fluctuations to other 2D magnetic systems warrants further investigation. The Monte Carlo simulations, while extensive, rely on a specific Hamiltonian model, and slight variations in model parameters may impact the results. Furthermore, the experimental determination of spin coherence length in the 2D nematic phase remains an open question, requiring advanced techniques with nanoscale spatial resolution.