A star-nose-like tactile-olfactory bionic sensing array for robust object recognition in non-visual environments

The study addresses the challenge of robust object recognition in environments where visual sensing is unreliable due to occlusions, darkness, or poor lighting. Drawing inspiration from the star-nosed mole, which relies on tactile and olfactory senses for perception, the authors propose fusing tactile and olfactory modalities to recognize objects and identify humans during rescue missions. The purpose is to design a non-visual sensing system that captures local topography, stiffness, and odor, and to process these signals via a bioinspired machine-learning architecture that mimics multisensory integration in biological systems. The importance lies in improving recognition accuracy, environmental robustness, efficiency, and power consumption compared to vision-dominant approaches, particularly in hazardous or buried scenarios.

Prior work in object recognition has been largely vision-based, with performance degraded under occlusions or poor lighting. Multimodal approaches combining vision with somatosensory or auditory signals have made progress. Tactile and olfactory perception are key animal capabilities for object recognition; notably, the star-nosed mole’s neural organization prioritizes tactile and olfactory fusion over vision, illustrating advantages such as compact sensing, environmental suitability, and efficiency. Existing tactile recognition often uses flexible non-silicon strain sensors, while olfactory sensing has been underutilized due to temporal and dimensional mismatches with tactile data and susceptibility to ambient interference. The authors position their work to overcome these mismatches through preprocessing and complementary gas sensor selection, enabling effective tactile-olfactory fusion.

Hardware: A bioinspired system was built on a mechanical hand with five fingertip tactile arrays (14 silicon-based force sensors per fingertip; 70 total, 0.5 × 0.5 mm² each) and one palm olfactory array (six silicon-based resistive gas sensors, 3 × 3 mm² each) functionalized for ethanol, acetone, ammonia, carbon monoxide, hydrogen sulfide, and methane. Sensors were fabricated via MEMS processes. Force sensors use piezoresistive Wheatstone bridges on Si beams with protective encapsulation; sensitivity is 0.375 mV kPa−1 over 0–400 kPa, stable from 20–60 °C. Gas sensors use micro-heaters and diverse sensitive materials (CNTs with MgO or Pt, graphene with CuO, Pt-doped SnO2, Pt-doped WO3, ZnO/SnO2 composite), showing ~10 s response and ~10 s recovery and stability over 60 days. Integration and electronics: Sensors were integrated on flexible printed circuits, wire-bonded, reinforced with silver paste and vinyl, and encapsulated with silica gel. Data acquisition used instrumentation amplifiers (gain ×40) and NI DAQ for tactile voltages and a portable resistance unit for olfactory responses. A LabVIEW system sampled multi-channel data at 50 kS/s. Dataset: A custom dataset of 55,000 samples across 11 object types in five categories (Human: arm, leg; Olfactory interference: worn clothes, hair; Tactile interference: mouse; Soft objects: orange, towel; Rigid objects: stone, can, mug, carton). Each sample contains 70-channel tactile voltage time series during a standard interaction (reach, load, hold, release) and 6-channel olfactory resistance signals. Preprocessing normalized olfactory data to mitigate device variation and hysteresis. Machine learning (BOT architecture): Multistage, bioinspired fusion. Early processing uses a VGG-like CNN (two convolutions) to extract tactile features capturing local topography and stiffness, producing a 512D vector. Olfactory data are mapped to a 512D representation via a fully connected network. A scenario-dependent feedback mechanism adjusts relative weights of modalities through proportionality coefficients, resizing feature lengths (D_T, D_O) and scaling inputs (k_T, k_O). Final fusion employs a three-layer fully connected network with 0.5 dropout. The multimodal fusion employs Multimodal Compact Bilinear-like operation using FFT-based feature combination to yield a fusion vector of length d, improving robustness to interference. Training/testing split used 4:1 ratio from the 55,000-sample dataset. Variants evaluated: BOT (baseline fusion), BOT-R (random point extraction), BOT-F (feature-point selection), BOT-M (feature-point selection + multiplicative output fusion). Baselines: Unimodal tactile-only CNN and olfactory-only feedforward network. Robustness tests: Gaussian white noise added to testing data to simulate sensor noise. Scenarios included gas interference (acetone and ammonia at 50–200 ppm), partial burial/occlusion (blocking one to four fingers), and random sensor damage (0–60% failed sensors). A simulated rescue pipeline integrated with a UR5 robotic arm and Allegro hand, with OTG-based trajectory planning, force-threshold safety, and decision logic to assess burial degree by classifying debris vs human and removing debris iteratively.

- Multimodal accuracy: BOT-M achieved 96.9% overall accuracy on 11-class recognition, outperforming BOT (91.2%), BOT-R (93.8%), BOT-F (94.7%), tactile-only (81.9%), and olfactory-only (66.7%). - Convergence: Training/testing accuracy stabilized after ~20 training cycles; loss decreased accordingly. - Noise robustness: With increasing Gaussian noise (σ up to 0.2), BOT-M maintained substantially higher recognition accuracy than unimodal models, which degraded sharply. - Gas interference: Under acetone leak (50–200 ppm), olfactory-only accuracy dropped dramatically, while BOT-M maintained >99% human (arm/leg) recognition. Under ammonia (50–200 ppm, odor similar to human), BOT-M maintained >80% accuracy versus a rapid decline for olfactory-only. - Partial burial/occlusion: With increasing burial level (blocking 1–4 fingers), tactile-only accuracy decreased markedly, whereas BOT-M maintained >99% for arm recognition by reweighting toward olfaction when tactile input was impaired. - Sensor damage tolerance: With random failure of 0–60% of sensors, BOT-M preserved high recognition accuracy compared to tactile-only, demonstrating compensation via the complementary modality and scenario-dependent weighting. - Sensing performance: Force sensors showed sensitivity of 0.375 mV kPa−1 over 0–400 kPa, discriminated objects of varying stiffness (e.g., skin, cotton, stone), and captured key contact phases. Gas sensors exhibited ~10 s response and recovery, stability over 60 days, and discrimination of different gases and concentrations. The system recognized 11 typical objects and a simulated rescue scenario at a fire department site achieved 96.9% classification accuracy without visual input.

The results demonstrate that fusing tactile and olfactory modalities—mirroring the star-nosed mole’s neural integration—enables robust object and human recognition in conditions where vision fails. The BOT architecture effectively handles the temporal and dimensional disparities between modalities, leveraging scenario-dependent weighting and FFT-based fusion to maintain performance under gas interference, occlusion/burial, and partial sensor failure. Compared to vision-based systems, the proposed approach uses smaller input data, reducing computational requirements and potentially enabling faster decisions—important for time-critical rescue tasks. Silicon-based MEMS force and gas sensors contributed high sensitivity, stability, and compactness, while the olfactory modality provided rich, distinctive chemical fingerprints that complemented tactile features like local topography and stiffness. The robustness to environmental noise and sensor defects highlights the system’s suitability for hazardous, non-visual environments.

The study introduces a star-nose-inspired tactile-olfactory sensing array combined with a bioinspired olfactory-tactile (BOT) learning architecture that achieves robust, non-visual object recognition. Using high-performance silicon-based force and gas sensors and a multimodal fusion network with scenario-dependent weighting, the system accurately classifies 11 objects (96.9% with BOT-M) and excels in human identification under hazardous conditions (gas interference, burial, sensor damage). This tactile-olfactory strategy offers an efficient, compact, and environmentally resilient alternative or complement to vision in dark or obstructed settings, with demonstrated applicability to rescue scenarios through robotic integration for debris removal and assessment of burial degree. Future work could expand object classes and environmental conditions, integrate additional modalities or adaptive control policies, and advance real-world deployments in emergency response.

- The evaluation was conducted on a curated set of 11 object types and simulated rescue scenarios; broader generalization to more diverse objects and real-world disaster environments requires further validation. - Gas sensors are inherently susceptible to ambient interference and exhibit slower dynamics than force sensors; while preprocessing and complementary sensor selection mitigate this, extreme or mixed gas environments may still challenge performance. - The system relies on physical contact for tactile sensing and near-field exposure for olfaction, which may not be feasible for all targets or safe in certain hazardous contexts. - Reported robustness tests (noise, burial, sensor damage) are controlled experiments; long-term field reliability under mechanical shocks, contamination, and environmental extremes was not exhaustively characterized.