High-dimensional single-cell RNA sequencing (scRNA-seq) profiles cellular changes in response to perturbations like disease or drug treatment. Existing methods, such as differential expression and differential abundance analyses, identify changes in gene expression or cell type proportions but are limited when condition labels inaccurately represent the cell-level effects of perturbations. Many perturbations affect only a subset of cells within a cell type. Condition-agnostic methods aim to identify perturbed cell groups in latent space but struggle with confounding variations. Detecting subtle transcriptional changes across conditions is crucial. Current methods assume condition labels accurately represent cell-level effects, that the perturbation is the dominant signal, or that confounding factors are known and removable. These assumptions often fail because perturbation effects are small relative to biological heterogeneity and technical noise, or the affected cell proportion is low, leading to incorrect labels. This necessitates a new approach that explicitly addresses these limitations.

Several methods exist to characterize perturbation effects in single-cell data. The standard workflow involves label-agnostic dimensionality reduction, clustering, and comparing cell attributes across conditions. CNA offers a cluster-free approach to identifying regions of uneven condition label mixing. MELD provides a continuous measure of perturbation effects. Milo performs differential abundance testing among conditions with continuous trajectories. Mixscape removes known confounding variations and separates successfully and unsuccessfully perturbed cells in gene knockout screens. These methods rely on assumptions about label accuracy, signal dominance, and known confounders, which are often violated in real-world datasets.

HIDDEN, a statistical framework, refines individual cell labels to accurately reflect their affected/unaffected status. It uses a two-step process: First, gene expression profiles are summarized using dimensionality reduction (e.g., PCA, NMF, or autoencoders). A prediction model (logistic regression is used in this study, although other classifiers are possible) then takes the reduced expression profiles and sample-level binary labels to produce per-cell continuous perturbation scores. These scores reflect the likelihood of a cell being affected by the perturbation. Finally, the continuous scores from case samples are binarized (e.g., using k-means clustering) into HIDDEN-refined binary labels. The number of features in the prediction model (a single parameter) can be automatically chosen using heuristics based on the number of differentially expressed genes or the Kolmogorov-Smirnov test statistic comparing perturbation scores between affected and unaffected cells. The performance of various dimensionality reduction techniques (PCA, NMF, autoencoders) was assessed. Comparisons were made against CNA, MELD, Milo, and Mixscape to evaluate HIDDEN’s performance in refining sample-level labels into perturbed/unperturbed cell labels. HIDDEN's ability to handle batch effects was evaluated by using batch-sensitive and batch-agnostic training strategies.

Simulations using Naive B and Memory B cells demonstrated HIDDEN's superior ability to identify perturbation signals, especially when the affected cell subset was small (e.g., 5%) or the perturbation was weak. Standard analysis failed to recover marker genes in datasets with fewer than 20% perturbed cells, while HIDDEN successfully identified them. In human multiple myeloma data, HIDDEN recapitulated manual annotation of neoplastic cells, discovered malignancy in early-stage samples missed by manual annotation, and produced sample purity estimates that outperformed manual annotation. HIDDEN also identified 8208 differentially expressed genes, with 2400 overlapping with known malignancy markers and 5808 unique to precursor samples. In a mouse demyelination model, HIDDEN identified a previously unrecognized endothelial cell subpopulation (LPC1) affected in early stages of demyelination. This LPC1 subpopulation showed unique markers consistent with inflammation, blood-brain barrier breakdown, and extracellular matrix remodeling. Analysis of cellular communication revealed altered pathways, including decreased anti-angiogenic interactions and increased expression of Vcam1, supporting increased angiogenesis and immune cell recruitment. These findings highlight HIDDEN's ability to identify subtle transcriptional changes, even in small datasets.

HIDDEN offers a sensitive approach to detecting perturbation effects otherwise missed by existing methods, especially when affected cells are rare or the effect is subtle. It outperforms existing methods in producing accurate perturbation labels and can be used to improve the performance of other methods that rely on accurate cell-level labels. The method is efficient and scalable to large datasets. Future directions include extending HIDDEN to analyze spatial, multi-omics data, and modeling more complex perturbation responses (e.g., multi-stage disease progressions).

HIDDEN provides a powerful tool for identifying subtle perturbation effects in single-cell data. Its superior performance in detecting disease-relevant cell populations is demonstrated in simulations and real-world datasets. The method's efficiency and scalability make it a valuable asset in single-cell analysis, particularly for studies involving complex biological systems and subtle transcriptional changes. Future work could focus on extending the framework to analyze more complex experimental designs and integrate additional data modalities.

HIDDEN requires cell type-specific analysis, does not explicitly model continuous perturbation strength, and does not provide a statistical test to distinguish between binary and continuous perturbation effects. The interpretation of continuous perturbation scores requires careful consideration by the researcher.