Human-centred physical neuromorphics with visual brain-computer interfaces

Visual evoked potentials (VEPs), and particularly steady-state VEPs (SSVEPs), enable direct communication between the visual cortex and external devices by eliciting neural responses at the stimulation frequencies. Traditional SSVEP BCIs primarily encode information by mapping light flicker frequency to the same EEG frequency component, and by spatially separating flickering lights to infer user focus. Beyond fundamental responses, simultaneous stimulation at multiple frequencies produces harmonics and intermodulation components at sums of integer multiples (e.g., f1+f2, 2f1+f2), arising from neuronal nonlinearities and linked to ion channel dynamics. Prior SSVEP systems have used low-density frequency-division multiplexing (FDM) with few frequencies, achieving information rates on the order of ~1 bit/s, with recent advances up to ~5 bits/s. In contrast, the human visual system can receive information at ~10 Mbit/s, suggesting capacity for transmitting richer information to the brain than used in standard BCI control tasks. This work explores high-density FDM to encode and transmit large amounts of information simultaneously, including entire images, and to exploit nonlinear intermodulation for computation in a physical neural network (PNN).

The study builds on extensive work using SSVEPs for BCIs, where stimulation frequencies produce consistent EEG spectral components used to decode intent. Researchers have observed robust harmonic and intermodulation components under multifrequency stimulation, with consistent frequencies across individuals, attributed to intrinsic neuronal nonlinearities and potentially to ion-channel mechanisms. Historically, SSVEP BCIs have relied on low-density FDM (one or two frequencies), limiting effective information transfer for control, though some systems have reached up to 5 bits/s. However, the visual system’s higher input capacity suggests untapped potential for richer brain-directed communication. Related efforts in computational imaging with the human visual system (e.g., ghost imaging) indicate that sufficient information for grayscale image reconstruction can be transmitted via SSVEP-based BCI. In parallel, physical neural networks and reservoir computing approaches in photonic and other physical media demonstrate that combining input data with trainable parameters in a nonlinear physical system can enable efficient classification—this work adapts that paradigm to the brain’s visual processing via SSVEPs.

High-density FDM SSVEP encoding: Input data (e.g., image pixels or control parameters) are mapped to a set of light modulation frequencies. Each element m is assigned a frequency f_m = f0 + m·δf and amplitude A_m proportional to its value. The composite signal x(t)=Σ A_m cos(2π f_m t) is used to modulate a red LED (640 nm) that illuminates a white screen. A participant observes the screen while EEG is recorded with a 3-pole system: active electrode at Oz (primary visual cortex), reference at M1 (above the left ear), ground at M2 (above the right ear). The resulting EEG is converted to a normalized power spectral density (NPSD). Operating regime: Experiments use the fundamental input frequency band and the second-order intermodulation band (sum frequencies f_m+f_n), neglecting higher-order terms. A narrowband constraint ensures the highest input frequency is well below 2f0, so intermodulation around 2(f0+m·δf) remains separable from the fundamentals. For visual perception, with M=200 frequencies, the stimulus appears as random flicker repeating with period T=1/δf.

Image transmission: 28×28 MNIST digits are downsampled to 14×14 (M+1=196 elements), flattened, and encoded with binary amplitudes (0/1). Frequencies are centered at f0=12 Hz with varying δf to produce total bandwidths of 1, 2, 4, 8, 12, and 16 Hz. Maximum screen luminance is 8 cd. Acquisition times: 196 s for 1–8 Hz bandwidth cases (corresponding to one period for 1 Hz bandwidth) and 16.3 s for 12 Hz bandwidth in a shorter measurement; a blindfolded control shows only the alpha peak (~10 Hz). Reconstruction: The amplitude at each encoded frequency in the NPSD is reassigned to its original pixel to form the grayscale image; similarity to ground truth is quantified via SSIM.

Physical neural network (PNN) for classification: The input image vector X and a set of control parameters α are frequency-encoded into two non-overlapping narrow bands to avoid overlap of fundamentals, second harmonics, and intermodulation. Example configuration: 64 image frequencies in [15.0,15.5] Hz and 64 control frequencies in [20.0,20.5] Hz (each 0.5 Hz bandwidth); the intermodulation band is [35,36] Hz (sum frequencies). Intermodulation often exhibits stronger contrast than second harmonics. Control parameters are optimized using a genetic algorithm trained on a phenomenological numerical model of the SSVEP that reproduces fundamental, harmonic, and intermodulation components, with second-order mixing weighted by a universal χ² profile ~exp(-f), independent of user. For classification, the intermodulation band is divided into segments equal to the number of classes; after normalizing the maximum to 1, the power fraction in each segment yields class probabilities.

Multi-layer PNN (connected brains): To improve multi-class performance without increasing per-layer bandwidth or measurement time, outputs from the first SSVEP PNN layer are rigidly re-encoded into a new frequency set with new control parameters and presented to a second participant, forming a two-layer network. Control parameters for each layer are retrained using the numerical model adapted for multilayer structure. The approach was tested on the Iris dataset: five input features were resampled to 2-bit depth and encoded into ten 1-bit frequency components.

Attention manipulation: In a two-layer PNN setting, participants (layer 2) were instructed either to focus on the flicker (focus) or to disrupt attention by performing mental arithmetic (disrupt) during 200 s stimulation blocks; block order was counterbalanced. Classification probabilities and intermodulation power were compared across conditions.

Experimental setup and controls: Participants sat 50–70 cm from a 10×10 cm white screen illuminated by the modulated red LED at a comfortable angle to fixate the screen reflection. Total light intensity was renormalized to maintain comparable average luminance across experiments (0–8 cd for imaging and PNN classification; 0–1.6 cd for attention tests). EEG signals were routed via an amplifier to a computer microphone port for recording. Ethics approval no. 300210003 (University of Glasgow); informed consent obtained. A blindfold control confirmed that observed spectral features beyond alpha rhythms are visually evoked.

Modeling framework: A phenomenological SFG-based model relates the measured SFG profile to |X|^2 within a narrowband and the mixed term to |X∗α| for two-band inputs, capturing second-order intermodulation generation consistent with experimental NPSD. The model, validated by agreement between measured and simulated intermodulation spectra, enables offline training of α via a genetic algorithm, then applied to unseen experimental EEG data.

• High-density FDM SSVEP encoding with hundreds of frequencies enables parallel transmission of rich information. Using 196 frequencies to encode 14×14 images, the system reconstructs images from EEG-derived NPSD amplitudes with increasing quality as bandwidth increases (SSIM rises with more periods). A 16.3 s acquisition at 12 Hz bandwidth yields a reconstruction comparable to a single-period 196 s acquisition at 1 Hz bandwidth, indicating a favorable time–bandwidth trade-off.
• Intermodulation components (sum frequencies) present higher contrast than second harmonics, supporting their use as computational readout channels for the PNN.
• Single-layer SSVEP PNN correctly classifies two classes (e.g., handwritten digits “0” vs “1”) by encoding image and trainable control parameters in separate bands (15–15.5 Hz and 20–20.5 Hz) and reading out the 35–36 Hz intermodulation band. Simulated and measured intermodulation spectra agree, validating the model-driven training of control parameters.
• Multi-layer (two-brain) PNN substantially improves multi-class performance without increasing per-layer bandwidth/time. On the Iris dataset (features resampled and encoded as ten 1-bit frequencies), single-layer performance was near chance (~50% or lower), whereas two-layer performance improved classification probabilities up to 80% and higher across participants; similar improvements were observed in 13 individuals.
• Human attention modulates computational efficacy: in two-layer PNN experiments with six participants as layer 2, classification accuracy and intermodulation power were significantly reduced in the disrupt versus focus condition (classification: t(5)=6.29, p=0.00006; intermodulation power: t(5)=4.18, p=0.002), demonstrating the human element directly affects PNN performance.

The study demonstrates that SSVEPs can be driven by high-density frequency-division multiplexing to communicate substantially more information to the brain than traditional low-density BCIs and to perform computation via inherent neural nonlinearities. Image transmission tests confirm that parallel frequency encoding across a broad EEG band can recover structured visual information, with improved SSIM as the number of acquisition periods increases and with comparable image quality achievable in shorter times using broader bandwidths. This verifies feasibility for higher-throughput brain-directed communication.

By encoding inputs and trainable parameters in separate bands and exploiting sum-frequency intermodulation as a nonlinear mixing readout, the SSVEP PNN performs classification tasks. Agreement between measured and simulated intermodulation spectra enables efficient offline training of control parameters that transfer to experiments, indicating user-independent model generality. Extending to multiple layers by serially connecting participants markedly improves multi-class classification without increasing per-layer complexity, highlighting a practical scalability pathway when bandwidth and time constraints limit per-layer node counts.

Attention’s significant impact on classification accuracy and intermodulation power underscores the uniquely human component of this neuromorphic platform: cognitive states can modulate computational performance. Together, these results address the central question of whether high-density FDM SSVEPs can encode and process complex information and show that leveraging intermodulation enables neuromorphic computation with potential for scalable, multi-brain systems. The approach opens avenues for assistive technologies and cognitive enhancement interfaces where richer information transfer and adaptive, brain-in-the-loop computation are advantageous.

This work introduces a human-centred physical neuromorphic platform that uses high-density FDM SSVEPs to (i) transmit entire images from computer to brain/EEG readout, (ii) implement a physical neural network that performs classification through nonlinear intermodulation, and (iii) scale computation by stacking layers across connected brains, markedly improving multi-class performance. Intermodulation bands provide robust computational readouts, and a simple, user-independent model supports efficient offline training of control parameters that generalize to experiments. The demonstrated sensitivity of performance to attention highlights the system’s human-in-the-loop nature.

Future directions include: optimizing time–bandwidth trade-offs and node density; developing demodulation strategies when operating bandwidths overlap with intermodulation regions; expanding to more complex, multi-class datasets and real-time operation; exploring other sensory modalities (e.g., auditory) and broader participant demographics; and integrating attention-aware protocols to enhance robustness and usability in assistive and diagnostic applications.

• Time–bandwidth–noise trade-off: Broader bandwidths enable shorter acquisitions but increase risk of overlap between fundamentals, harmonics, and intermodulation when bandwidth exceeds ~2×f0; careful band planning is required.
• Limited task complexity: Demonstrations focus on binary classification (digits) and a three-class dataset (Iris) with discretized features; performance is acceptable but not exceptional for single-layer networks.
• Measurement duration: Some experiments require relatively long acquisitions (e.g., 196 s), which may limit practicality; though broader bandwidths reduce time, this risks spectral overlap.
• Phenomenological modeling: The training model abstracts neural nonlinearities and uses a universal weighting profile; while effective, it may not capture all inter-individual or context-specific dynamics.
• Participant scope and generalizability: Although tested across multiple individuals, broader demographic and clinical validation is needed; attention strongly influences performance, introducing variability that must be managed in real-world deployments.