Human-centred physical neuromorphics with visual brain-computer interfaces

Brain-computer interfaces (BCIs) aim to establish direct communication pathways between the human brain and external devices. Steady-state visual evoked potentials (SSVEPs), neural responses generated in the visual cortex to periodic visual stimuli, are a promising method for BCIs due to their consistent frequency components. Previous SSVEP-based BCIs utilized low-density FDM, employing only a few light-modulation frequencies, resulting in low information transfer rates (~1 bit/s, recently improved to 5 bits/s). However, the human visual system's inherent capacity for information transmission is significantly higher (~10 Mbits/s), suggesting untapped potential for BCIs beyond simple tasks. This research explores the use of high-density FDM to leverage this potential, aiming to significantly enhance the information transfer rate and computational capabilities of SSVEP-based BCIs. Recent work demonstrated that the human visual system can perform 'computational imaging' tasks like ghost imaging, indicating the potential for SSVEP-BCIs in more complex image processing and pattern recognition. This study extends this by investigating high-density FDM for image transmission and simple classification tasks using SSVEPs to build a physical neural network (PNN). The PNN utilizes the intrinsic nonlinearity of the visual system to mix input data and control parameters, performing computation across both the biological visual system and a silicon-based computer. This hybrid approach offers the possibility of scalable, human-centered neuromorphic computing.

The use of VEPs, particularly SSVEPs, in BCIs has been an active area of research. Studies have focused on low-density FDM techniques, using one or two frequencies to encode information. These systems achieved information rates up to 5 bits/s. However, this is far below the processing capacity of the human visual system (~10 Mbits/s), leaving substantial room for improvement. Researchers have also observed harmonic and intermodulation frequencies generated in SSVEPs due to the intrinsic nonlinearity of neurons. These nonlinearities arise from the nonlinear response of individual neurons and are linked to neuron ion channels. The understanding of the physical origins of these intermodulation frequencies is not yet complete, but their presence suggests opportunities for more complex computation. Recent work has explored the use of SSVEPs for computational imaging tasks, demonstrating the principle of using SSVEPs for image reconstruction. This study builds on these advancements, aiming to improve the information transfer rate and computational capacity of SSVEP-based BCIs.

The study employed a high-density FDM SSVEP system. A red LED illuminated a white screen, and its intensity was modulated using hundreds of frequencies simultaneously encoded according to the information being transmitted. This signal was observed by a participant wearing an EEG device. The EEG recorded the SSVEP response from the occipital cortex (Oz electrode). The system was tested in two ways: 1) Image transmission: 14x14 pixel handwritten digits were used as input. Each pixel was assigned a unique frequency, and the amplitude of the frequency component corresponded to the pixel's gray-scale value. The reconstructed image was obtained by analyzing the normalized power spectral distribution (NPSD) of the SSVEP. 2) Physical neural network (PNN): The handwritten digits "0" and "1" (8x8 pixels) were used for a classification task. The input image information was encoded into one frequency band, while control parameters were encoded into another non-overlapping frequency band. The intermodulation frequencies generated by the brain's nonlinearity were then used for classification. A numerical model, based on the observation of harmonic and intermodulation frequencies in the SSVEP spectrum, was used to simulate the PNN. This model includes a weighting function (exp(-f)) to account for the nonlinear response and was used to train the control parameters using a genetic algorithm. A multi-layer PNN was tested by using the output of one participant's PNN as input for another participant, thus establishing a brain-to-brain connection. The Iris flower dataset was used to test the multi-layer PNN. The classification probabilities were calculated based on the power fractions in different frequency segments of the intermodulation frequencies. In addition, a test assessing the impact of human attention was conducted. Participants were instructed to either focus their attention on the stimulus or perform mental calculations ('disrupt' condition) to see if attention affects the classification accuracy and the strength of the intermodulation frequency.

The high-density FDM SSVEP system successfully transmitted images of handwritten digits. The quality of the reconstructed image improved with increasing bandwidth and longer acquisition time. A trade-off was observed between bandwidth, acquisition time, and signal-to-noise ratio. A shorter acquisition time with a higher bandwidth could yield similar results as a longer acquisition time with a lower bandwidth. The SSVEP-based PNN successfully classified handwritten digits "0" and "1". The numerical model accurately predicted the measured SSVEP spectrum. The multi-layer PNN, connecting two brains, significantly improved the classification accuracy for three classes, compared to a single-layer PNN. This suggests scalability of the approach. The effect of human attention on the PNN was demonstrated. During the 'disrupt' condition (mental calculations), the classification accuracy and the strength of the intermodulation frequencies were both significantly reduced compared to the 'focus' condition. The results indicate that human attention can modulate the performance of the PNN.

The results demonstrate the potential of high-density FDM SSVEPs for enhancing information transfer rates and computational capabilities of BCIs. The successful image transmission and classification tasks highlight the feasibility of using SSVEPs for complex information processing. The multi-layer PNN shows that the system is scalable and holds promise for more complex computations. The finding that attention modulates PNN performance suggests a potential application in the diagnostic assessment of attention and cognitive fatigue. The hybrid computational approach, combining biological and silicon-based computation, is a novel paradigm for neuromorphic computing. The relatively good performance of the numerical model implies that extensive experimental measurements may not be necessary for training the PNN, thus potentially reducing the experimental burden for future applications.

This research successfully demonstrated high-density FDM SSVEPs for image transmission and PNN implementation, highlighting the potential for substantially improved information transfer rates and computational power in BCIs. The multi-layer PNN approach using connected brains showcases the system's scalability. Future research should explore even higher-density FDM, the integration of different sensory modalities (e.g., auditory stimuli), and the development of more sophisticated PNN architectures for addressing complex tasks. The role of attention in modulating PNN performance warrants further investigation for applications in cognitive assessment and enhancement.

The study focused on relatively simple tasks (handwritten digit classification and a three-class Iris dataset). More complex tasks and larger datasets should be tested to fully evaluate the capabilities of the system. The current sample size was limited, potentially affecting the generalizability of the findings. Future studies should involve a more diverse population. The numerical model, while effective, is phenomenological, and a deeper understanding of the underlying neural mechanisms is needed for further improvement and refinement.