Quantum spin liquid and cluster Mott insulator phases in the Mo₃O₈ magnets

Geometrically frustrated quantum systems are central to the study of quantum spin liquids (QSLs), especially on triangular and kagome lattices where various QSL phases can emerge. The Mo3O8 cluster magnets, comprising anisotropic kagome layers of trimerised Mo atoms forming [Mo3O13]6− clusters on a triangular lattice, have recently been highlighted as candidate QSL materials. In LiZn2Mo3O8, magnetic susceptibility exhibits two Curie–Weiss regimes with a crossover near 96 K and an apparent loss of 2/3 of paramagnetic spins, originally attributed to valence-bond condensation. Competing proposals include an emergent honeycomb lattice decoupling orphan spins and a plaquette charge order (PCO) in a 1/6-filled anisotropic kagome Mott insulator that reconstructs a spinon Fermi surface. Newly synthesised Li2InMo3O8 and Li2ScMo3O8, however, display moments localised on Mo3O13 clusters, with Li2InMo3O8 showing 120° antiferromagnetic order and Li2ScMo3O8 lacking magnetic order down to 0.5 K but exhibiting short-range correlations and QSL-like excitations. This work revises a single-orbital extended Hubbard model at 1/6 filling on the anisotropic kagome lattice, uncovering two regimes—PCO and cluster Mott insulator—governed by the interplay between kinetic energy and intersite Coulomb interactions and the sign of hopping parameters. First-principles calculations are used to obtain material-specific parameters for LiZn2Mo3O8, Li2ScMo3O8, and Li2InMo3O8, showing that the opposite signs of intra- and intercluster hoppings are key to stabilising PCO with unpaired spins in LiZn2Mo3O8, while Li2InMo3O8 and Li2ScMo3O8 realise cluster Mott insulating behaviour with differing magnetic outcomes.

Prior interpretations of LiZn2Mo3O8 include (i) an emergent honeycomb lattice due to opposite rotations of Mo3O13 clusters, leaving an orphan spin per triangle (Flint and Lee, 2013), and (ii) a plaquette charge order (PCO) on an anisotropic kagome lattice at 1/6 filling conjectured to host a U(1) QSL with a reconstructed spinon Fermi surface (Chen, Kee, Kim, 2016). Experimental studies of Li2InMo3O8 and Li2ScMo3O8 report, respectively, 120° antiferromagnetic order (TN ~ 12 K) and absence of long-range order down to 0.5 K with short-range order and QSL-like excitations (Yu et al., 2015; Iida et al., 2019), challenging earlier mechanisms. Previous theoretical works on PCO and related models often assumed equal signs of hopping parameters, leading to different ground states (including ferromagnetism at certain fillings via Perron–Frobenius arguments) and did not account for two Curie–Weiss regimes without phenomenological modifications. The present study emphasizes that opposite signs of intra- and intercluster hoppings qualitatively alter the ring-exchange processes and stabilise a PCO with partial spin pairing and dangling spins, consistent with experimental thermodynamics.

- Low-energy model: Single-orbital extended Hubbard model on the anisotropic kagome lattice at 1/6 filling to describe Mo-based kagome layers trimerised into Mo3O13 clusters. The key parameters include intracluster hopping t, intercluster hopping t', on-site and intersite Coulomb interactions (U, V, V'). - Strong interaction (PCO) regime: In the limit where intersite Coulomb interactions dominate and charge configurations avoid nearest-neighbour repulsion, the model maps onto a quantum dimer model describing ring tunnelling on kagome hexagons. Effective ring-exchange amplitudes g1 and g2 (proportional to t^3 with geometry-dependent coefficients) govern resonance between plaquette states |A⟩ and |B⟩. Antiferromagnetic spin fluctuations between next-nearest neighbours within a hexagon are included via H_S with coupling J_nn ≈ 4 t_nn^2 / U. - Exact diagonalization: The single-hexagon Hamiltonian H = H0 + H_S is solved exactly for spinful electrons to obtain ground-state wavefunctions and degeneracies. Finite-geometry exact diagonalization is used to confirm PCO as the ground state and to compute spin-spin, density-density, and plaquette-plaquette correlations and structure factors. - Weak interaction (cluster Mott insulator) regime: When |t| becomes comparable to intersite Coulomb terms, electrons delocalise within kagome triangles (T), forming molecular orbitals split by crystal-field-like term H_CF with Δ = 3t. Molecular states |a1⟩ and doublet |e⟩ are constructed from site basis states. For t < 0, the a1 state is occupied, localising the unpaired electron on the Mo3 triangle (cluster Mott insulator). - Effective spin model: For large |t| and |t'| with t < 0 and t' > 0, double occupancy of T triangles is forbidden (U_eff = U + 2V'), and a nearest-neighbour Heisenberg model on the triangular lattice is derived to second order in t'/U and t'/Δ. The exchange coupling J_A (J_Δ) is obtained analytically (Eq. 4) and evaluated using parameters from first-principles. - First-principles calculations: Density functional theory and Wannier-function downfolding are used to construct maximally localised Wannier functions and extract t, t', and Coulomb parameters (U, V, V'). Constrained-RPA-like approaches inform screened interactions where applicable. These parameters determine g1, g2, J_nn, Δ, and J_A for LiZn2Mo3O8, Li2InMo3O8, and Li2ScMo3O8. - Thermodynamics: Using the single-hexagon spectrum with fitted g1, g2, and J_nn, specific heat and spin susceptibility are computed to identify two Curie–Weiss regimes and estimate the crossover temperature Tc. Structure factors are computed to generate predictions for inelastic neutron scattering.

- Two regimes identified: (i) Plaquette charge order (PCO) with resonating hexagons that host valence-bond configurations leaving one dangling (unpaired) spin per hexagon; (ii) Cluster Mott insulator with electrons localised on Mo3 triangles (Mo3O13 clusters). - Opposite sign hoppings as key mechanism: When intra- and intercluster hoppings have opposite signs (t and t' of opposite sign), ring tunnelling amplitudes g1 and g2 acquire opposite signs, stabilising an antisymmetric superposition of plaquette states with partial spin pairing and a dangling spin in each resonating hexagon. This naturally yields two Curie–Weiss regimes without invoking phenomenological modifications. - LiZn2Mo3O8 (PCO regime): First-principles yield small splitting between a1 and e1 molecular levels, preventing full localisation; extracted parameters support PCO with unpaired spins. Using g1 = 13.5 meV, g2 = −40.1 meV, and J_nn = 1.4 meV, the calculated crossover temperature between two paramagnetic regimes is Tc ≈ 92 K, matching experiments (~96 K). The Curie constants follow CL ≈ CH/3 at low vs high temperatures. Spin-spin correlations for g1 and g2 of opposite signs predict structure factor maxima at the K points of the original Brillouin zone. - Li2InMo3O8 (cluster Mott insulator): Large Δ = 3t and ratios t/V' and t'/V favour electron localisation on Mo3O13 clusters. The derived antiferromagnetic exchange coupling is J_A ≈ 9.5 meV (~109.8 K), consistent with experimental ~112 K, explaining the observed 120° Néel order. - Li2ScMo3O8 (near instability within cluster Mott regime): Calculated J_A ≈ 4.0 meV (~46.7 K), close to an instability where exchange is small, making long-range order susceptible to fluctuations. Consistent with absence of magnetic order down to low T and presence of short-range order with QSL-like excitations. - Prior ferromagnetism scenarios (equal sign hoppings) invalid here: For t and t' of opposite signs, Perron–Frobenius arguments do not apply; exact diagonalization finds antisymmetric minimal-spin configurations favoured. - Neutron-scattering fingerprints: For g1 and g2 with the same sign (no spin pairing), structure factor maxima are at Γ and K of the extended Brillouin zone; for opposite signs (partial pairing with dangling spins), maxima are at K of the original Brillouin zone, providing an experimental discriminator.

The extended Hubbard model on the anisotropic kagome lattice at 1/6 filling captures two distinct physical limits relevant to Mo3O8 magnets. In LiZn2Mo3O8, opposite-sign intra- and intercluster hoppings drive a plaquette charge order of resonating hexagons in which valence-bond formation coexists with a delocalised dangling spin per hexagon. This directly explains two Curie–Weiss regimes and the reduction of the effective moment by a factor of three without invoking phenomenological modifications. The calculated Tc (~92 K) quantitatively matches neutron data. The predicted spin structure factor with K-point maxima provides a testable signature in inelastic neutron scattering. In Li2InMo3O8 and Li2ScMo3O8, larger |t| and crystal-field splitting Δ favour a cluster Mott insulator in which electrons occupy molecular a1 states on Mo3 triangles. The resulting triangular-lattice Heisenberg model yields antiferromagnetic exchange consistent with 120° order in Li2InMo3O8, whereas Li2ScMo3O8 lies near a fluctuation-driven instability with only short-range order and QSL-like excitations. The study clarifies that trimerisation controls the balance between kinetic energy and intersite Coulomb repulsion, as well as the sign structure of hoppings, thereby determining whether a material realises PCO or a cluster Mott state. These insights reconcile disparate experimental observations across the Mo3O8 family and offer clear neutron-scattering and thermodynamic fingerprints to further validate the theory.

This work identifies and microscopically grounds two competing phases in Mo3O8 cluster magnets: a plaquette charge ordered state with dangling spins (realised in LiZn2Mo3O8) driven by opposite-sign intra- and intercluster hoppings, and a cluster Mott insulator with electrons localised on Mo3 triangles (realised in Li2InMo3O8 and Li2ScMo3O8). First-principles-derived parameters feed low-energy models that quantitatively reproduce key experimental observations, including the two Curie–Weiss regimes and Tc in LiZn2Mo3O8 and the antiferromagnetic exchange scale in Li2InMo3O8, while explaining the lack of long-range order in Li2ScMo3O8. The results emphasise the role of trimerisation and sign of hoppings in determining ground states and provide experimentally verifiable predictions (e.g., spin structure factors). These findings offer a framework to interpret and design properties in other quantum cluster magnets and related trimerised kagome systems.

- The PCO analysis relies on mapping to a quantum dimer model and on exact solutions of isolated hexagons (assuming effective decoupling in the PCO phase); inter-hexagon couplings and long-wavelength fluctuations are not fully treated. - Exact diagonalization is performed on finite clusters, which may not capture all aspects of the thermodynamic limit. - The stability of PCO may be affected by strong spin fluctuations; while discussed, a full treatment beyond single-hexagon and finite-size calculations is not provided. - First-principles-derived parameters (t, t', U, V, V') depend on downfolding and screening approximations; uncertainties propagate to effective couplings (g1, g2, J_nn, J_A). - The publication does not report a direct experimental determination of the sign of hoppings; conclusions about sign structure are inferred from electronic-structure calculations and phenomenology.