The pursuit of Artificial General Intelligence (AGI), aiming to replicate human-like intelligence, faces challenges inherent in scenarios lacking sufficient data, clearly defined problems, complete knowledge, static states, and reliance on a single system. Combining computer science-based ANNs and neuroscience-inspired SNNs is a promising pathway towards AGI. ANNs excel in processing static data and leveraging large datasets for complex relationships, while SNNs efficiently handle event-driven, sparse, or temporal data and offer increased parallelism and energy efficiency. However, significant differences between ANNs and SNNs—including coding schemes, synchronization, and neuronal dynamics—hinder seamless integration. Recent advancements in neuromorphic computing hardware provide platforms that support both ANNs and SNNs, but a general framework for versatile task integration that fully exploits the strengths of both remains absent. This paper addresses this gap by introducing a novel framework for the design and computation of HNNs. Unlike approaches that narrowly focus on specific task solutions, this framework aims for general-purpose computation by decoupling ANNs and SNNs and using HUs as linkage interfaces. This decoupled approach inherits the key characteristics of both paradigms while enhancing flexibility and model design. The HUs, acting as information transformers with intermediate representations, bridge the gap between synchronous real-valued ANN representations and asynchronous spike-based SNN representations.

Existing research explores hybrid neuromorphic computing platforms and model combinations. Intel's Loihi, IBM's in-memory computing, and the University of Manchester's Spinnaker are examples of hybrid designs. Previous work on combining SNNs and ANNs focuses on aspects like information processing, computational efficiency, and incorporating biological attributes, but these efforts are typically task-specific. This study addresses the need for a general framework that comprehensively leverages the advantages of both ANNs and SNNs for diverse applications. The authors refer to several papers that showcase specific applications of hybrid neural network approaches, highlighting the current limitations of narrow, task-specific designs and setting the stage for a more generalized approach.

The proposed framework centers on the concept of Hybrid Units (HUs). Unlike conventional signal converters, HUs employ a generalized asynchronous real-valued intermediate representation to bridge the gap between ANNs and SNNs. They coordinate synchronization, time scales, and coding schemes for hybrid information transformation. The HU model formalizes this transformation as Y = HU[X] = Q F H W(X), where X is the input, W is a window function for time-scale synchronization, H is a kernel function for spatiotemporal information extraction, F is a nonlinear operation for domain transformation, and Q is an optional signal operation (e.g., thresholding or discretization) to support diverse representation characteristics. HUs can be configured manually (designable) using prior knowledge when the relationship between heterogeneous representations is known, or automatically learned (learnable) using various approaches—joint training with connected networks, independent optimization goals, or training with the complete model—when the relationship is unknown or complex. The paper details the mathematical formulation of HUs, explaining the functions of each component (W, H, F, Q) and their role in bridging the gap between ANN and SNN representations. Three learning approaches for learnable HUs are proposed: joint training with frontend or backend networks, separate modeling with independent optimization goals, and training with the complete model. The authors explain how both designable and learnable HUs can be used to create accurate and approximate models. The paper further details the methodologies employed for each of the three case studies: the Hybrid Sensing Network (HSN), the Hybrid Modulation Network (HMN), and the Hybrid Reasoning Network (HRN). For the HSN, a divide-and-conquer strategy is used, separating visual information into static and transient signals processed by distinct pathways (ANNs and SNNs) and combined through learnable HUs. The HMN employs a hierarchical structure with an ANN-based backbone network generating modulation signals (through HUs) to control the dynamics of an SNN-based branch network, enabling meta-continual learning. Finally, the HRN integrates multimodal information processing with an ANN-based frontend and a symbolic reasoning SNN-based backend, using both designable and learnable HUs for information transformation. Specific details on network architectures, training algorithms (including spatiotemporal backpropagation for SNNs), and datasets are provided.

The paper presents three case studies demonstrating the framework's effectiveness: 1. **Hybrid Sensing Network (HSN):** The HSN, applied to a visual tracking task, effectively combines the high precision of ANNs and high efficiency of SNNs. Experimental results show significant improvements over single-paradigm approaches, achieving a high tracking accuracy (0.679 mIoU) even under real-world constraints (using Tianjic chips), which is over 100% higher than a real-world ANN implementation. The HSN achieves a high tracking speed (5952 FPS) and high power efficiency (130 µJ/inference), demonstrating its superiority in real-time visual perception. 2. **Hybrid Modulation Network (HMN):** The HMN tackles the meta-continual learning (MCL) problem by using hierarchical information abstraction. An ANN-based backbone network generates modulation signals to control an SNN-based branch network. Results on the permuted N-MNIST dataset show that the HMN significantly outperforms single SNNs and SNNs with other continual learning methods (like EWC), effectively mitigating catastrophic forgetting. The HMN demonstrates improved accuracy on unlearned tasks with high similarity, showcasing its capacity for generalization. 3. **Hybrid Reasoning Network (HRN):** The HRN addresses multimodal reasoning in visual question answering (VQA). It uses an ANN-based frontend for multimodal information processing and an SNN-based backend for interpretable logical reasoning. Evaluated on the CLEVRER dataset, the HRN achieves high accuracy across different question types (descriptive, explanatory, predictive, counterfactual), surpassing existing state-of-the-art methods. It further displays remarkable parallelism and robustness to abnormal commands, highlighting its efficiency and resilience in complex reasoning tasks. The HRN demonstrates high parallelism and robustness to abnormal data. Latency remains almost constant even with increased numbers of objects and events. The model’s robustness stems from its graph structure that embeds prior knowledge, reducing the answering space.

The results demonstrate the versatility and effectiveness of the proposed HNN framework. The three case studies highlight the ability to combine the strengths of ANNs and SNNs for diverse applications. The HSN demonstrates record-breaking performance in high-speed visual tracking, the HMN shows the potential for efficient and robust continual learning, and the HRN showcases capabilities in interpretable and parallel logical reasoning. The framework addresses the need for a flexible and general-purpose approach to integrating ANNs and SNNs. Learnable HUs are crucial when dealing with complex environments and uncertainties, leading to better adaptation and performance. The authors further draw parallels between the presented HNN architectures and biological neural systems, suggesting the framework's relevance to neuroscience. They discuss potential applications in functional neuroscience and brain simulations, emphasizing the potential to bridge the gap between theoretical models and practical biological understanding.

This research presents a novel framework for designing and computing hybrid neural networks that successfully integrates the strengths of ANNs and SNNs. The framework, based on the use of hybrid units (HUs), has been validated through three diverse applications, demonstrating significant performance improvements over single-paradigm approaches. Future research could explore the incorporation of heterogeneous dynamics and connectivity within homogenous networks to further enhance HNN capabilities. Additionally, exploring more sophisticated learning algorithms and expanding the range of applications are potential avenues for future investigation.

While the framework demonstrates significant potential, limitations exist. The design and learning of HUs can be complex and require careful consideration. The performance gains observed depend on the specific application and dataset, and the generalizability of the framework needs further investigation across broader application domains. The current implementations primarily use homogenous ANN and SNN components; exploring heterogeneous network components could lead to further improvements.