Single-cell RNA sequencing (scRNA-seq) is revolutionizing biological and clinical research by providing a comprehensive view of cellular heterogeneity within complex tissues. However, the massive datasets generated and the high noise levels pose significant challenges to analysis. Existing methods for scRNA-seq analysis, including those for cell segregation (clustering), transcriptome landscape visualization, cell classification, and pseudo-time inference, often struggle with scalability and accuracy, particularly with large, noisy datasets. This paper addresses these challenges by proposing scDHA, a novel analysis framework that leverages the power of hierarchical autoencoders to extract meaningful biological signals from noisy single-cell data. The development of robust and efficient methods for scRNA-seq analysis is crucial for unlocking the full potential of this technology, enabling a deeper understanding of complex biological processes and informing clinical applications.

Numerous computational methods exist for analyzing scRNA-seq data. For cell segregation (clustering), popular approaches include SC3, Seurat, SINCERA, CIDR, and Scanpy. Visualization of the transcriptome landscape often employs non-linear dimensionality reduction techniques like Isomap, Diffusion Map, t-SNE, and UMAP. Cell classification frequently utilizes machine learning algorithms such as XGBoost, Random Forest, Deep Learning, and Gradient Boosting Machine. Pseudo-time inference, which orders cells along developmental trajectories, relies on methods like Monocle, TSCAN, Slingshot, and Scanpy. While these methods have proven useful, they face limitations in terms of scalability and robustness, particularly when dealing with the increasing volume and complexity of scRNA-seq data. This paper positions scDHA as a superior alternative that addresses these shortcomings.

The scDHA pipeline consists of two core modules. The first module is a non-negative kernel autoencoder that provides a part-based representation of the data and removes genes or components with insignificant contributions. The input expression matrix is first normalized using min-max scaling to reduce the influence of outliers. Then, a one-layer autoencoder is used to filter out insignificant genes, using non-negative weights to facilitate feature selection. Genes with high weight variances are retained, forming a subset considered sufficient to represent the data. The second module is a stacked Bayesian autoencoder built upon a variational autoencoder (VAE). This module projects the filtered data onto a low-dimensional space. To enhance robustness and prevent overfitting, multiple realizations of the latent space are generated using a re-parameterization trick. The model is trained using AdamW optimizer with a two-stage training scheme (warm-up and VAE stage) and uses a scaled exponential linear unit as the activation function. The compressed data generated by the second module serves as input for downstream analyses. Cell segregation is performed using k-nearest neighbor spectral clustering, with the number of clusters predicted using two indices: the ratio of between sum of squares to total sum of squares, and the change in within sum of squares as the number of clusters increases. For visualization, a neural network projects the compressed data onto a 2D space minimizing the Kullback-Leibler divergence between the probability distributions in the high-dimensional and low-dimensional spaces. Cell classification uses k-nearest neighbor classification on concatenated and compressed training and testing data. Pseudo-time inference employs a minimum spanning tree algorithm on the similarity matrix derived from the compressed data, defining pseudo-time as the distance from a designated starting point. A voting procedure is also integrated for faster analysis of large datasets, where only a subset of data points is clustered, and the rest are assigned using k-nearest neighbor classification.

scDHA was rigorously evaluated on 34 publicly available scRNA-seq datasets encompassing diverse tissues and protocols. In cell segregation, scDHA significantly outperformed SC3, Seurat, SINCERA, CIDR, Scanpy, and k-means, achieving a significantly higher average adjusted Rand index (ARI) of 0.81 compared to the next best method (CIDR, ARI=0.5). scDHA also demonstrated superior speed. In dimensionality reduction and visualization, scDHA exceeded t-SNE, UMAP, Scanpy, and PCA based on silhouette index. In classification, scDHA's accuracy (average 0.96) surpassed XGBoost, Random Forest, Deep Learning, and Gradient Boosting Machine across five human pancreatic datasets. Finally, in pseudo-time inference, scDHA accurately reconstructed developmental trajectories in three mouse embryo datasets, outperforming Monocle, TSCAN, Slingshot, and Scanpy based on R-squared values. These findings consistently demonstrate scDHA's superior performance across various scRNA-seq analysis tasks.

The results demonstrate scDHA's significant advantages over existing methods in terms of accuracy, speed, and robustness. Its hierarchical autoencoder architecture effectively removes noise and selects informative features, improving both the quality of the input for downstream analysis and the overall efficiency of the process. The use of a stacked Bayesian autoencoder with multiple latent space realizations enhances the model's generalization ability and prevents overfitting. The integration of several downstream analysis methods within a single, user-friendly package further enhances its practical utility. This framework is not limited to single-cell RNA sequencing and has potential applications across various research areas employing high-throughput data. The superior performance of scDHA, particularly in handling large and noisy datasets, addresses a critical need in the field, enabling more comprehensive and accurate analyses of complex biological systems.

scDHA provides a fast, precise, and user-friendly framework for the comprehensive analysis of scRNA-seq data. Its superior performance across multiple analysis tasks demonstrates its potential to advance various biological and clinical research areas. Future work could explore applications to other types of single-cell omics data and improvements to the autoencoder architecture, including the number of hidden layers and the type of activation functions used.

While scDHA demonstrates excellent performance, potential limitations include the computational resource requirements for very large datasets. Though the voting procedure alleviates this to some extent, extremely large datasets might still necessitate significant computational resources. Furthermore, the optimal parameters for the autoencoder architecture may vary depending on the specific dataset and application, requiring some level of parameter tuning. Finally, the choice of distance metric for clustering and pseudo-time inference (e.g., Pearson correlation) could influence the results and might be explored using alternative metrics in future research.