Multi-compartment Neuron and Population Encoding improved Spiking Neural Network for Deep Distributional Reinforcement Learning

Spiking neural networks (SNNs), inspired by the brain's use of binary spikes, offer advantages in energy efficiency and the incorporation of multi-scale biological characteristics. However, most SNNs simplify neurons, often using the Leaky Integrate-and-Fire (LIF) point neuron model, neglecting the multi-compartmental structure of biological neurons. This simplification limits the computational and learning capabilities of SNNs. Neurobiological research reveals that neuron dendrites play a crucial role in integrating synaptic inputs, enhancing information processing capacity. Multi-compartment neuron (MCN) models, incorporating apical and basal dendrites and soma, have shown promise in various tasks but lack exploration in complex decision-making scenarios such as video game playing. Population encoding, a biologically plausible mechanism for representing continuous values, is also underutilized in SNN-based reinforcement learning. Current SNN approaches to reinforcement learning (RL) often use simple network structures or rely on ANN-SNN conversion methods, which hinders optimization and limits performance. This work addresses these limitations by introducing a novel SNN-based deep distributional reinforcement learning model.

The paper reviews existing SNN applications in various fields, including image classification, object detection, speech recognition, decision-making, and robot control, highlighting their comparable performance to traditional artificial neural networks (ANNs) and the energy efficiency enabled by neuromorphic hardware. It discusses limitations of existing spiking neuron models (LIF, Izhikevich, Hodgkin-Huxley) that don't fully leverage the structural properties of biological neurons. The literature review also covers existing MCN models focusing on simpler tasks and existing SNN-based reinforcement learning approaches, including Spiking DQN, PopSAN, and methods using knowledge distillation or spatio-temporal backpropagation (STBP). The authors highlight the lack of SNN applications in deep distributional reinforcement learning, which is more biologically plausible and robust compared to traditional reinforcement learning methods.

The paper introduces a multi-compartment spiking fully parameterized quantile function network (MCS-FQF) model, the first SNN application in deep distributional reinforcement learning. The model consists of a spiking convolutional neural network (SCNN) for state embedding, a population encoding method for quantile fraction representation, a multi-compartment neuron (MCN) model integrating state and quantile fraction information, and a fully connected SNN for quantile value generation. The MCN model comprises basal and apical dendrite modules, receiving inputs from different sources, and a somatic module integrating the dendritic potentials. The population encoding method uses a population of neurons with Gaussian receptive fields to represent quantile fractions implicitly using spike trains. The model is trained end-to-end using a modified STBP algorithm with a surrogate function to approximate the spike gradient, minimizing the Huber quantile regression loss. The quantile fractions are calculated from the state embeddings using a learnable weight matrix and a softmax function. The paper provides equations for the LIF neuron, MCN compartments (basal dendrite, apical dendrite, soma), population encoding, and the training process, along with detailed explanations and a theorem regarding MCN somatic potential integration. The training algorithm is explained using equations for the surrogate gradient, the Wasserstein loss, and weight updates for different components of the network.

Experiments on 19 Atari games show that the MCS-FQF model outperforms both the vanilla ANN-based FQF model and an ANN-SNN conversion-based Spiking-FQF model. The performance improvement is more significant on complex games with elaborate enemy behaviors and rapidly changing scenarios. Ablation studies demonstrate the crucial roles of both the MCN and population encoding methods. Replacing the MCN with Leaky-Integrate (LI) neurons (S-FQF-POP model) reduces performance, indicating the advantages of the multi-compartmental structure for information integration. Similarly, replacing the population encoding with cosine embedding (S-FQF model) significantly hinders performance, suggesting that this encoding scheme is more effective for spiking neural networks. Analysis of MCN spiking activity reveals distinct patterns and a mechanism of mutual inhibition and cooperation between dendrites and soma, enhancing the neuron's information processing capacity. The model learns faster and more stably compared to the baseline FQF model. Specific game examples are presented, highlighting the significant performance boosts (e.g., 1.7x and 2.5x higher scores than FQF and converted Spiking-FQF respectively in VideoPinball).

The superior performance of the MCS-FQF model is attributed to several factors: the more complex dynamics of spiking neurons compared to ReLU neurons, the effective integration of information from different sources using the MCN, the direct training method, and the improved representation of quantile fractions using population encoding. The results validate the effectiveness of integrating biologically plausible mechanisms (MCN and population encoding) into SNN-based reinforcement learning. The findings suggest that more complex SNN structures capable of handling information integration are necessary for complex decision-making tasks. The study demonstrates the potential of brain-inspired SNNs for surpassing traditional ANN-based approaches in reinforcement learning.

This paper successfully applies SNNs to deep distributional reinforcement learning, introducing a novel MCS-FQF model. The model incorporates a biologically inspired MCN and population encoding method, showing superior performance in Atari games. Future research could focus on more sophisticated MCN models, exploring mutual interactions between dendrites and soma and developing efficient training methods for such models. Investigating alternative population encoding schemes with diverse receptive fields is another promising avenue.

The current MCN model simplifies the biological neuron by omitting mutual interactions between dendrites and soma, which could lead to oscillations or instability. The lack of this mutual interaction limits the model's capacity to model complex biological processes. Furthermore, the population neurons all share the same receptive field which restricts the richness of the neural representation and the model's capacity to handle highly complex tasks. Future work could address these aspects.