The Metaverse is creating a parallel digital reality mirroring our physical world. To enable realistic interactions within this virtual space, including virtual games like basketball (Fig. 1), accurate rendering of visual, auditory, and haptic sensations is crucial. While computer vision (CV) and computer audition (CA) technologies utilize cameras and microphones (Fig. 1b) to capture and render visual and auditory data effectively, a comparable generic tactile data capturing device for computer haptics (CH) is currently lacking. Haptic rendering in CH is essential for immersive experiences across various Metaverse applications, including entertainment, education, telemedicine, and manufacturing. This bidirectional interaction involves tactile sensing (acquiring information from a physical object), haptic rendering (computing and generating tactile sensations), and haptic feedback (transmitting these sensations to the user). Analogous to cameras in CV and microphones in CA, a "tactile scanner" (Fig. 1b) is needed to acquire neuron-like signals for tactile perception and rendering. Previous research has explored various artificial electronic skin-like devices and tactile sensors, but a widely adopted solution for mass production remains unavailable. This paper addresses this gap by designing and deploying a bioinspired tactile scanner that processes tactile stimuli information with naturalistic tactile features.

Researchers have explored various artificial electronic skin-like devices and tactile sensors using materials like carbon nanotubes, nanowires, graphene, and nanocomposite materials, along with micro-nano structures such as pyramids, micro-domes, microcones, and interlocking microstructures. While some devices demonstrate sensitivity comparable to human skin, mass production remains a challenge. Neuromorphic devices like synaptic electrolyte-gated transistors and memristors have been used in artificial neuromorphic somatosensory systems, showcasing tactile perception and feedback, although they are still in early stages. Bioinspired afferent nervous systems integrated with pressure sensors and synaptic devices have demonstrated pressure and vibration detection, along with sensation and feedback in a closed-loop human-machine interaction. Artificial neural tactile sensation of naturalistic textures has seen notable advancements in hand prostheses, incorporating sensorimotor loops. However, a generic tactile data-capturing device, a "tactile scanner," is still missing for CH.

The design of the bioinspired tactile scanner draws inspiration from the human skin's neural processes (Fig. 2a). Slow adaptation (SA) receptors like Merkel disks and Ruffini endings detect static force, while fast adaptation (FA) receptors like Meissner and Pacinian corpuscles respond to dynamic force. Upon mechanical stimuli, action potentials (APs) are generated and transmitted through synapses. The proposed artificial neuromorphic tactile system (Fig. 2b, 2c) mimics this process, using thin-film transistor (TFT) technology for large-area, mass-producible implementation. The tactile scanner comprises three stages: sensing, processing, and receiving. A piezoelectric transducer (PVDF) acts as the artificial axon neuron, converting mechanical stimuli into electrical potentials (mimicking the mechano-gating model of mechanoreceptors, Fig. 3b). A dual-gate TFT (TFT N1) functions as the artificial synapse, while a TFT-based circuit (TFT N2) serves as the dendrite. The PVDF transducer's piezoelectric properties generate charges proportional to the mechanical stimulus (Fig. 4b). The synaptic TFT's conductance increases exponentially with the input voltage (Fig. 4c), mimicking the LIF model. A 10x10 array of piezoelectric transducers and NPUs was fabricated (Fig. 3e, 3f). The synaptic TFT's performance was evaluated through measurement of excitatory post-synaptic currents (EPSCs), showing exponential relaxation after stimuli (Fig. 4f, 4g). The NPU, integrating the synapse and dendrite functions (Fig. 5a), performs the integration and firing processes, generating spike trains encoding tactile information (Fig. 5b, 5c, 5d, 5e, 5f). For tactile rendering, five materials (cotton, fleece, knit, metal, wood) were scanned (Fig. 6e). Fast Fourier Transform (FFT) was used to analyze the tactile data, classifying textures with 93% accuracy (Fig. 6b, 6c). A commercial actuator rendered the tactile information, showing high correlation (68%) with the original signals (Fig. 6d, 6e, 6f). The detailed modeling of biological and artificial sensory receptors and neurons is described in the methods section (Equations 2-10). Fabrication processes for the axon sensor array and neuromorphic circuit array are also detailed. The characterization of synaptic TFT and NPU, along with the evaluation of the tactile scanner, including the methods for surface texture recognition and material classification are provided.

The bioinspired tactile scanner successfully acquired tactile information from various surfaces, achieving over 85% accuracy in texture recognition and classification. This was accomplished through the use of piezoelectric transducers and neuromorphic circuits, which effectively mimicked the biological processes of the human tactile system. The system’s ability to accurately capture and process tactile information, such as texture, was verified through a series of experiments. Frequency-domain analysis of tactile data (Fig. 6b) revealed characteristic frequency clusters (CFCs) specific to different materials. These CFCs were used to train a machine-learning model, resulting in high accuracy in texture recognition and classification (Fig. 6c). The tactile stimulus information was further rendered using a commercial actuator (Fig. 6a, 6e). The rendered tactile signals were quantitatively and qualitatively assessed through a correlation study (Fig. 6f) and a psychophysical-perceptual experiment (Supplementary Fig. 25). Results indicated a high degree of correlation (68%) between acquired and rendered tactile signals, demonstrating the effectiveness of the tactile rendering process. The NPU, the core of the neuromorphic processing, exhibited low power consumption (~0.39 nJ/spike), significantly outperforming some existing LIF circuits. The proposed tactile scanner displayed repeatable tactile responses under various stimulus conditions (Supplementary Fig. 13), further validating its reliability and robustness.

This research successfully developed a bioinspired tactile scanner for computer haptics (CH), addressing the lack of a generic tactile data acquisition device. The scanner's design, based on the human tactile somatosensory system, enables real-time acquisition of tactile stimuli. The use of piezoelectric transducers and TFT-based neuromorphic circuits allows for efficient conversion and processing of mechanical stimuli into neuron-like signals. The high accuracy in texture recognition and classification, coupled with effective tactile rendering, demonstrates the potential of this system for creating immersive and realistic haptic experiences in the Metaverse. The low power consumption of the NPU highlights the system's energy efficiency. The results contribute significantly to the advancement of CH, paving the way for more realistic and interactive virtual environments.

This study presented a bioinspired tactile scanner for computer haptics, successfully achieving real-time acquisition and rendering of tactile information. The integration of piezoelectric transducers and neuromorphic circuits provides a biomimetic approach to capturing and processing tactile stimuli, with high accuracy in texture classification and rendering. Future research could focus on improving the scanner's resolution, expanding its range of detectable tactile stimuli, and integrating it with more advanced haptic rendering systems for a broader range of applications.

The current prototype of the tactile scanner has a limited spatial resolution (10x10 array). The frequency range of detectable tactile stimuli is also restricted (up to 200 Hz), lower than the human range. The study focused primarily on texture recognition; future work should investigate the system's ability to capture other tactile properties such as shape, weight, and hardness. The psychophysical-perceptual experiment involved a limited number of participants.