Geometrically frustrated quantum systems, particularly those based on triangular and kagomé lattices, are central to the study of quantum spin liquids (QSLs). These systems are characterized by long-range quantum entanglement, charge fractionalization, and emergent gauge structures. The realization of QSLs in materials has been a significant challenge, with few confirmed candidates. In recent years, Mo₃O₈ cluster magnets have emerged as potential QSL hosts. In these materials, Mo atoms form anisotropic kagomé layers, trimerizing into Mo₃O₁₃ clusters arranged on a triangular lattice. Each Mo₃O₁₃ cluster possesses six strongly bonded valence electrons and one unpaired electron in a symmetric a₁ state. LiZn₂Mo₃O₈ initially showed promise as a QSL material, exhibiting magnetic susceptibility that follows a Curie-Weiss law with distinct high- and low-temperature regimes and a Curie constant ratio of C₁≈C_H/3, attributed to valence bond condensation. Proposed explanations included the formation of an emergent honeycomb lattice and a plaquette charge order (PCO) hosting a U(1) QSL. However, the synthesis of Li₂InMo₃O₈ and Li₂ScMo₃O₈, exhibiting distinct magnetic properties despite similar crystal structures, challenged these explanations. Li₂InMo₃O₈ displays Néel 120° magnetic order, while Li₂ScMo₃O₈ shows short-range magnetic order and QSL-like excitations. This inconsistency highlights the need for a comprehensive theoretical understanding of the magnetic behavior in Mo₃O₈ systems. This work addresses this gap by utilizing first-principles calculations in conjunction with an extended Hubbard model to investigate the interplay of kinetic energy and intersite Coulomb interactions within these materials, exploring regimes overlooked in previous studies.

Extensive theoretical work has focused on abstract electronic and spin models to understand QSLs. However, the complexity of these systems often leads to hypothetical proposals disconnected from real materials. Several studies have focused on the Mo₃O₈ family. Flint and Lee (2013) proposed an emergent honeycomb lattice in LiZn₂Mo₃O₈ to explain its QSL behavior. Chen et al. (2016) suggested a PCO state on an anisotropic kagomé lattice at 1/6 filling that could host a U(1) QSL. However, these models did not fully capture the diverse magnetic properties observed across different Mo₃O₈ compounds. Previous work on the PCO state on the kagomé lattice for spinless fermions at 1/3 filling and spinful electrons at 1/6 and 5/6 fillings, primarily focused on scenarios where hopping parameters have the same sign, leading to different conclusions about the ground state compared to the case with opposite signs studied here. The current study addresses these limitations by considering the extended Hubbard model with material-specific parameters obtained from first-principles calculations, allowing for a more accurate representation of the complex interactions within Mo₃O₈ systems. The literature lacks a comprehensive model unifying the different magnetic behaviours in LiZn₂Mo₃O₈, Li₂InMo₃O₈, and Li₂ScMo₃O₈. This work aims to fill this gap.

The authors employ a single-orbital extended Hubbard model on the anisotropic kagomé lattice at 1/6 filling to explore the magnetism in Mo₃O₈ systems. This model considers the interplay between kinetic energy and intersite Coulomb interactions. Material-specific parameters are obtained through first-principles calculations using density functional theory (DFT) within the generalized gradient approximation (GGA) and the projector augmented-wave (PAW) method. The calculations are performed using Quantum ESPRESSO. The hopping parameters (t and t') are obtained by constructing maximally localized Wannier functions using Wannier90. The extended Hubbard model is then analyzed using different limiting cases to investigate the ground states. The strong interaction limit leads to a plaquette charge ordered (PCO) state. A quantum dimer model is used to study this PCO state, where ring tunnelling processes within hexagons are considered. The model is further developed to incorporate antiferromagnetic spin fluctuations, allowing the examination of ground-state properties with opposite signs for intracluster and intercluster hopping parameters. The weak interaction limit is described by a three-orbital extended Hubbard model on the triangular lattice of Mo₃O₁₃ clusters. This model considers the crystal field splitting of the electronic states into a₁ and e states. The on-site Coulomb interaction (U) and intersite interactions (V and V') are considered to investigate the electronic localization and magnetic ordering. A spin model on a triangular lattice is then derived to investigate magnetic ordering in the cluster Mott insulator phase. Finite geometry calculations with periodic boundary conditions were conducted to validate the theoretical predictions. Specific heat and magnetic susceptibility calculations were performed to compare theoretical predictions with experimental data, particularly the two paramagnetic regimes observed in LiZn₂Mo₃O₈. Spin-spin correlation functions and static structure factors were also calculated to examine the nature of the PCO state.

The study reveals two distinct ground states depending on the interplay between kinetic energy and Coulomb interactions, controlled by the trimerization of the kagomé lattice and the sign of hopping parameters. In the strong interaction limit, a plaquette charge order (PCO) with unpaired spins at resonating hexagons is observed, which is particularly realized in LiZn₂Mo₃O₈. This PCO is a consequence of opposite signs of intracluster and intercluster hopping parameters (t > 0 and t' < 0), leading to an antisymmetric configuration of plaquette states with one dangling spin per hexagon. The calculated transition temperature (T_c ≈ 92 K) between two paramagnetic regimes excellently matches experimental neutron powder diffraction data. The weak interaction limit leads to a cluster Mott insulator phase, where electrons are localized at Mo₃O₁₃ clusters. Li₂InMo₃O₈, with large t and t', exhibits long-range antiferromagnetic order. Li₂ScMo₃O₈, with weaker t and t', displays short-range magnetic order and QSL-like excitations. The exchange interaction (J_A) calculated from the derived spin model on the triangular lattice shows good agreement with experimental values for both Li₂InMo₃O₈ (J_A = 9.5 meV, 109.8 K; experimental 112 K) and Li₂ScMo₃O₈ (J_A = 4.0 meV, 46.7 K; experimental 67 K). The difference in magnetic ordering between Li₂InMo₃O₈ and Li₂ScMo₃O₈ is attributed to the strength of t and t', which determines the degree of electron localization and the susceptibility to quantum fluctuations. The opposite signs of t and t' are identified as a crucial characteristic of the trimerized kagomé lattice in Mo₃O₈ systems. Spin-spin correlation functions and structure factors support the distinct nature of the PCO states with same-sign and opposite-sign hopping parameters.

The results presented significantly advance the understanding of magnetism in Mo₃O₈ systems. The study successfully unifies the seemingly disparate magnetic behaviors of LiZn₂Mo₃O₈, Li₂InMo₃O₈, and Li₂ScMo₃O₈ within a single theoretical framework. The identification of two distinct regimes, PCO and cluster Mott insulator, based on the competition between kinetic energy and Coulomb interactions, provides a comprehensive explanation for the experimental observations. The key finding of the opposite signs of hopping parameters being responsible for the PCO in LiZn₂Mo₃O₈ resolves a long-standing ambiguity in the literature. This work emphasizes the importance of considering both kinetic energy and Coulomb interaction effects in understanding the complex magnetism of frustrated systems and provides a benchmark for future studies on similar quantum cluster magnets. The excellent agreement between theoretical predictions and experimental data, particularly the calculated T_c and J_A values, strongly supports the validity of the proposed model. The detailed analysis of spin-spin correlation functions provides insights into the nature of spin interactions and facilitates further experimental verification through techniques like inelastic neutron scattering.

This study provides a unified theoretical model explaining the diverse magnetic phenomena observed in Mo₃O₈ quantum magnets. The model highlights the critical role of the competition between kinetic energy and Coulomb interactions, mediated by lattice trimerization and the sign of hopping parameters, in determining the ground state. The identification of a PCO state with unpaired spins in LiZn₂Mo₃O₈ and a cluster Mott insulator phase in Li₂InMo₃O₈ and Li₂ScMo₃O₈ offers a comprehensive understanding of the experimentally observed magnetic properties. Future studies could explore other Mo₃O₈-based materials with different chemical compositions to further validate and extend this model. Investigating the effects of external pressure or magnetic fields on the system's phase transitions would also be valuable.

The model employs several approximations, including the use of an extended Hubbard model with a single orbital per Mo atom. While this simplification captures the essential physics, incorporating additional orbitals or electron correlations might provide even more accurate results. The first-principles calculations rely on DFT within the GGA approximation, which can have limitations in accurately describing strongly correlated systems. The analysis of the quantum dimer model is mainly focused on finite geometries. Scaling the model to the thermodynamic limit might reveal additional effects. The experimental uncertainties in the determination of hopping parameters and interaction strengths might influence the precision of the model's predictions. The current work has focused on explaining static properties of the system; a dynamic extension of the model would further enhance our understanding of the low-energy excitation spectrum in the QSL regime.