Effect of the social environment on olfaction and social skills in wild-type and a mouse model of autism

Autism spectrum disorder (ASD) is a complex neurodevelopmental condition diagnosed primarily based on impairments in social interaction and communication and the presence of repetitive, restricted, and stereotyped behaviors. Co-occurring traits can include anxiety, epilepsy, motor and cognitive deficits, and sleep disturbances. ASD prevalence has reached about 1% of the global population. Genetic studies have identified over a thousand susceptibility genes, with SHANK3 being a high-confidence gene located within the 22q13 chromosomal region implicated in Phelan-McDermid syndrome. Shank3 knockout (KO) mice mirror core ASD-like features, including deficits in social novelty preference and increased motor stereotypies. However, genetic variants alone do not account for the variability in ASD severity, underscoring environmental influences and gene dosage effects. Pharmacological treatments for core ASD features remain lacking, with current drugs addressing only co-occurring symptoms and showing limited efficacy and side effects. Numerous targeted compounds have failed in clinical trials, potentially due to placebo effects and ASD heterogeneity; the oxytocin receptor remains a promising target for subgroup-tailored therapies. Behavioral interventions such as ABA can help core social and cognitive impairments but are resource-intensive, require early initiation, and often improve only targeted behaviors. Alternatives such as inclusive classroom models show promise but face implementation challenges, and poor implementation can lead to behavioral difficulties and program discontinuation. The COVID-19 pandemic introduced widespread social isolation during critical developmental periods, with reports of increased aggression and depression. Rodent studies show that isolation effects on sociability depend on duration, with acute isolation often enhancing social seeking and chronic isolation potentially impairing sociability and increasing aggression; rapid social memory impairments have also been documented. The effects of early social environments and social isolation on ASD mouse models and on WT mice remain underexplored. This study aims to determine how early social environments, including isolation and varying group housing, affect social skills and olfactory processing in WT and Shank3 KO mice.

Prior work emphasizes ASD’s genetic heterogeneity with SHANK3 as a high-confidence gene associated with ASD and Phelan-McDermid syndrome. Pharmacological efforts targeting receptors and signaling pathways have largely failed in clinical trials, possibly due to high placebo response and population heterogeneity; oxytocin receptor signaling is a potential therapeutic avenue. ABA-based behavioral strategies show some efficacy but are costly, intensive, and often limited to targeted behaviors; early initiation is recommended, and translation to animal models has shown specific improvements in targeted domains. Inclusive education can yield positive outcomes with neurodiversity and teacher engagement but suffers from implementation and sustainability challenges; poor implementation may exacerbate behavioral issues. Social isolation’s impact is duration-dependent in rodents; acute or short-term isolation may enhance social seeking but can impair social memory and alter gene expression in the medial amygdala, with lingering molecular changes after behavioral recovery. Semi-naturalistic enriched environments can improve ASD-like traits in certain mouse models (e.g., BTBR). Olfactory cues, mediated by the main olfactory epithelium and the vomeronasal organ (VNO), are central to mouse social behaviors; oxytocin modulates VNO activity and social behaviors. Prior reports in Shank3 models show striatal and prefrontal circuit dysfunction, reduced social motivation, and olfactory deficits in complex odor backgrounds. Other ASD models (e.g., Nlgn3 R451C, Cx3cr1) show that social isolation or housing conditions can modify social and sensorimotor phenotypes, and co-housing mixed genotypes can influence behaviors.

Ethics and housing: All procedures complied with European and French directives and were approved by CEEA Val de Loire N°19 and the French ministry (APAFIS #18035-2018121213436249). Sexually naïve 2-month-old Shank3 KO (JAX #017688) and WT males and females on a mixed 50% C57BL/6J × 50% 129S2 background were used. To mitigate early social environment effects, F2 offspring from multiple F1 couples were tested. WT and KO mice were group-housed (2–4 per cage) unless subjected to 1- or 4-week chronic social isolation prior to phenotyping. Cages were randomly allocated to housing conditions. All mice were maintained in the same room on a 12-h light/dark cycle with food/water ad libitum, temperature ~21 °C and humidity ~50%. Behavioral assays: Tests were conducted in the morning under dim light in standardized setups to minimize anxiety. Manual scoring was performed by a trained, blinded experimenter unless stated otherwise. Social interaction was assessed over 10 min using: (a) reciprocal social interaction with a sex-, housing-, and genotype-matched conspecific; and (b) the three-chambered test (habituation, sociability, social novelty phases). Group social interaction was quantified using the Live Mouse Tracker (LMT), which automatically detects multiple behaviors among four sex- and age-matched animals over 10-min or 30-min sessions. For WT-only experiments, three interactions on separate days involved four unknown, sex- and age-matched animals. For mixed-genotype experiments: Trial 1 with cage mates; Trial 2 with a mix of unknown WT and KO (WT+KO); Trial 3 with unknown WT or unknown KO separately. Cognitive flexibility was measured in the spatial Y-maze (automatic scoring). Repetitive/stereotyped behaviors were scored in a motor stereotypy test (manual). Locomotion and anxiety-like behaviors were recorded in the open field (automatic). Olfaction was assessed with a two-choice preference test (manual), including simultaneous recording of 4–6 mice to quantify synchronized grooming. Molecular assays: Total RNA from main olfactory epithelium (MOE), vomeronasal organ (VNO), and olfactory bulb (OB) was extracted (Zymo Direct-zol RNA Microprep). cDNA synthesis used 450 ng (MOE), 320 ng (VNO), and 115 ng (OB) total RNA (SuperScript III). qPCRs were run in triplicate (384-well plates) with 1 µL cDNA (MOE/VNO 1:50 dilution; OB 1:25), 1 µM primers (Actb, Gapdh, Oxt, Oxtr), and ONEGreen Fast qPCR premix. Thermocycling: 95 °C 3 min; 40 cycles 95 °C 5 s; 60 °C 15 s; 60 °C 30 s. Oxytocin concentrations in plasma and urine were measured by EIA (Enzo ADI-901-153) following C18 solid-phase extraction (Sep-Pak C18). Plasma (120 µL) or urine (150 µL) extracts were reconstituted equivalently; assays were run in duplicate and normalized by animal number; sensitivity 7.8 pg/mL. RNAscope FISH: Mice were anesthetized, perfused, and VNO tissue was fixed in 4% PFA, cryoprotected, embedded, and sectioned (16 µm). RNAscope Fluorescent Multiplex V2 with tyramide amplification was used with mm-Oxtr probe, followed by streptavidin-Cy2 detection and Hoechst counterstaining. Images were acquired by confocal microscopy (Zeiss LSM-780). Subcellular puncta quantification was done in QuPath. Calcium imaging: Freshly dissociated vomeronasal sensory neurons (VSNs) from 5–6 males per genotype and housing condition were stimulated in random order with HBSS-Hepes control, 1 µM oxytocin, and male mouse urine (1:100 dilution; pooled from ≥3 adult males). Responses were recorded and analyzed as previously described. Statistics: Analyses were conducted in R 4.4.0. Linear models included mouse line, housing condition, number of mice per cage, trial, and interactions; model selection used AIC. Post hoc contrasts employed estimated marginal means with Tukey correction (emmeans). Assumptions were checked with DHARMa. qPCR expression ratios were computed via Pfaffl’s method, then analyzed with Kruskal–Wallis and Dunn’s post hoc tests (rstatix); one outlier was removed by MAD criterion. Calcium imaging used Fisher’s exact tests (GraphPad Prism 10.0.2). Data are reported as mean ± sd; raw data and statistics are in Tables S2–S4; sex effects in supplementary figures.

- Social isolation enhances social interaction: - 4-week isolation increased nose-to-nose contact time in the reciprocal interaction test in WT and Shank3 KO mice versus group-housed controls (multiple backgrounds and both sexes). - Shank3 KO mice showed impaired social novelty preference when group-housed; 4-week isolation increased nose contact in both sociability and social novelty phases and restored preference for the novel mouse. - Isolated Shank3 KO mice exhibited increased huddling, indicating heightened social comfort seeking; WT isolation did not alter huddling. - 1-week isolation produced intermediate effects in Shank3 KO mice (enhanced reciprocal interaction without social novelty restoration). - Motor stereotypies and synchronized grooming: - Group-housed Shank3 KO mice displayed increased self-grooming time and episodes, more head shakes, and reduced digging compared to WT. - Isolation normalized self-grooming (time, number, mean duration) and partially restored digging in Shank3 KO mice; head shakes remained elevated. These normalizations were also observed after 1-week isolation. - A novel phenomenon—synchronized self-grooming—was observed in group-housed Shank3 KO mice across adjacent cages, minimal in WT, and nearly absent in isolated WT and KO, suggesting social housing induces this behavior. - No effects of isolation on cognitive flexibility, locomotion, or anxiety in WT. In Shank3 KO, isolation increased locomotion and normalized spontaneous alternations (Y-maze), reduced time in open-field periphery, and did not induce aggression. - Early social environment richness (Live Mouse Tracker): - WT housed in groups of 4 maintained high and stable social behaviors across three trials with unknown conspecifics (nose contact, huddling, social approach, move-in-contact; less isolation). WT housed in groups of 2 or 3 showed declines across trials. Time in periphery increased across trials regardless of housing. - In mixed-genotype interaction series: interactions were higher with unknown mice than with cage mates. Shank3 KO mice made less nose contact with cage mates than WT. When interacting only with Shank3 KO mice (Trial 3), KO mice increased nose contact and huddling relative to the mixed group (Trial 2). Social approach (motivation) was comparable across genotypes and trials. - Housing-size effects differed by genotype: in Trial 3, Shank3 KO raised in groups of 2 showed higher nose contact/huddling versus WT groups of 2; Shank3 KO raised in groups of 4 showed reduced nose contact and social approach versus WT groups of 4. Sex effects indicated KO females had greater social exploration than KO males. - Olfactory system and oxytocin: - Isolation increased Oxtr mRNA in the VNO of WT (qPCR), with no changes in MOE or OB; RNAscope showed a non-significant trend toward elevated Oxtr puncta in VNE. - VSN calcium imaging: In group-housed conditions, Shank3 KO VSNs responded less to urine (from ~3% to ~1.6%; ~1.9-fold lower) and to oxytocin (from ~3% to ~1.9%; ~1.6-fold lower) compared to WT, indicating sensory dysfunction. - Isolation increased the fraction of urine-responsive VSNs in both WT (~1.7-fold) and Shank3 KO (~5-fold), with no housing effect on oxytocin responsiveness (WT: ~3% to ~3.7%; KO: ~1.9% to ~2.4%). - Olfactory two-choice tests: WT (grouped and isolated) preferred oxytocin (60 µg) over saline (both presented in a saline:urine mix) and preferred opposite-sex urine over saline; Shank3 KO lacked opposite-sex urine preference. WT preferred urine from isolated donors over group-housed donors. Urinary oxytocin concentrations did not differ by housing, whereas plasma oxytocin was 2.8-fold higher in isolated WT than grouped WT. Overall, 4-week isolation enhanced social interaction in WT and improved social interaction, social novelty preference, and certain stereotypies in Shank3 KO, alongside increased VNO Oxtr expression and urine-evoked VSN responsiveness. Richer social environments benefited WT but could diminish social exploration/motivation in Shank3 KO, highlighting genotype-specific effects of social context.

The data show that a 4-week period of social isolation can increase social interaction in WT mice, consistent with a social “craving” upon reunion and aligning with some prior mouse studies, though differing from rat studies reporting social novelty impairments after isolation. Environmental conditions and testing paradigms likely modulate these effects more than duration, sex, or background. Sex-specific patterns emerged, with WT females generally showing enhanced sociability compared to males and WT males exhibiting a potential U-shaped relation between group size and sociability, possibly reflecting dominance dynamics. In Shank3 KO mice, isolation unexpectedly ameliorated deficits in social novelty preference and increased social interaction and huddling, suggesting restored social motivation and altered circuit function. Isolation normalized self-grooming and reduced synchronized grooming, indicating that certain stereotypies may be socially induced or maintained in group contexts, potentially via communication (e.g., ultrasonic vocalizations) and/or aversive social touch processing. The social environment’s richness differentially affected genotypes: it sustained or improved WT social behaviors but reduced social exploration/motivation in Shank3 KO when group size increased, with evidence that KO females may prefer fewer conspecifics. These findings have welfare implications for optimizing housing conditions (3Rs) in ASD models and provide a framework for back-translating social-context interventions. Olfactory processing via the VNO emerged as a key contributor: Shank3 KO VSNs showed reduced responsiveness to social cues (urine) and oxytocin, consistent with VNO-mediated social behavior deficits. Isolation increased Oxtr expression in the VNO and enhanced urine-evoked VSN responses in both WT and KO, more strongly in KO, aligning with improved social phenotypes. The demonstration that mice can detect oxytocin in a complex odor context is novel, although the mechanistic link between Oxtr expression and olfactory processing requires further study. The circuitry underlying isolation’s effects may involve PVN oxytocin neurons and mesocorticolimbic pathways (VTA dopamine–mPFC), previously implicated in both WT and Shank3 models. Comparative literature in other ASD models (Nlgn3 R451C, Cx3cr1) supports the notion that social isolation and housing conditions can modulate social phenotypes. Collectively, the results underscore that social context robustly shapes social behavior, stereotypies, and olfactory processing in WT and Shank3 KO mice, with translational implications for tailoring social environments (e.g., smaller peer groups) in inclusive educational settings for ASD, acknowledging that direct extrapolation to humans is limited.

This study demonstrates that early social environments, including chronic social isolation and varying group sizes, strongly modulate social behavior, stereotyped behaviors, and olfactory processing in WT and Shank3 KO mice. In WT, isolation enhanced social interaction; richer social environments maintained social engagement across repeated exposures. In Shank3 KO, isolation improved social interaction, restored social novelty preference, and normalized self-grooming, coinciding with increased VNO Oxtr expression and augmented urine-evoked VSN responses. Conversely, larger group housing diminished social exploration/motivation in Shank3 KO, revealing genotype-specific sensitivities to social context. These findings position Shank3 KO mice as a useful model to back-translate behavioral interventions and to emulate elements of inclusive classroom strategies, suggesting that controlled, smaller-group exposures may be more beneficial than large-group contexts for some individuals. Future work should: (1) dissect mechanisms linking isolation to circuit and VNO functional changes (e.g., PVN oxytocin–VTA–mPFC pathways); (2) vary isolation duration and reunion schedules; (3) test long-term outcomes of mixed-genotype co-housing; (4) evaluate broader sensory modalities, including social touch; and (5) extend analyses to other ASD models (e.g., Fmr1 KO) to generalize findings and refine translational strategies.

- Translational scope: Findings in mice cannot be directly applied to autistic individuals; human social dynamics and environments are more complex. - Duration and context: Behavioral assessments focused on short-term interactions upon reunion; long-term social outcomes after different isolation durations were not tested. - Statistical power and variability: Some olfactory preference results (e.g., oxytocin preference) were not statistically significant due to interindividual variability and limited sample sizes in subgroups. - Sex and housing confounds: Certain assays (e.g., VSN calcium imaging) were performed in males only; observed housing-size effects may be influenced by sex-specific differences (dominance in males). - Environmental factors: Differences in animal facility conditions and testing environments can modulate isolation effects, potentially limiting generalizability. - Mechanistic uncertainty: The pathway by which increased Oxtr expression in the VNO influences olfactory processing and social behavior remains to be elucidated. - Synchronized grooming mechanism: The communicative cues (e.g., ultrasonic vocalizations) and neural basis underlying synchronized self-grooming in group-housed Shank3 KO mice were not directly tested.