Shape-position perceptive fusion electronic skin with autonomous learning for gesture interaction

Hand gestures are considered more natural and user-friendly interactions for virtual reality (VR), augmented reality (AR) and other human-machine interaction (HMI) systems. Wearable devices consisting of electronic skins (e-skin) can perceive human instructions, behaviors and even emotions by tracking hand motion with the help of deep learning; in contrast, visual images/videos and brain-computer interfaces are affected by environmental interference, such as blocked objects and physical conditions. Standalone bendable and stretchable sensing platforms powered by human motion, biofuel or radio frequency (RF) can be integrated into wearable devices, clothing or accessories to unobtrusively and accurately monitor human movement. Multiple fabrication methods, including printing, magnetron sputtering and electrodeposition, have been widely explored to integrate micro/nanoscale film devices prepared on planar growth substrates onto the surface of the target substrate for the preparation of small functional devices. They can be combined with transfer print or assembly methods to allow precise deposition of electronic components and interconnections on curved or irregular surfaces for device integration on 3D surfaces conforming to the human body structure. However, the above sensors are typically limited in their ability to measure the shape of the user's finger joints, resulting in a lack of position information and creating an intention gap between gestures and truth labels. For example, the shape of the hand in which the thumb is stretched out in a one-handed fist, with the thumb pointing up (Like) and down (No-like) having opposite meanings, is indistinguishable to these sensors that detect only finger movements. Meanwhile, limited computing power in wearable applications restricts multimodal fusion and deployment of deep models. The proposed perceptive fusion electronic skin (PFES) aims to overcome these issues by simultaneously sensing curvature (shape) and magnetic field (position) and by integrating an autonomous learning framework for efficient on-device deployment.

Materials and film preparation: A magnetostrictive Co70Fe30 alloy film was selected for the magnetostrictive (MS) layer due to its large magnetostrictive coefficient and soft magnetic properties. Copper or titanium foils served as cathode/electrodeposition substrates, cleaned with acetone and deionized water. A platinum strip anode was positioned parallel to and centered on the cathodes at 3 cm separation. The electrolyte bath contained 0.3 mol/L H3BO3 (buffer), 0.3 mol/L NaCl (conductivity enhancer), 0.4 mol/L ascorbic acid (antioxidant), 0.4 mol/L glycine (complexing agent), and 0.3 mol/L sodium citrate (complexing agent). pH was adjusted to 2.5 with 10% H2SO4. Electrodeposition was conducted at 50 °C with stirring to enhance mass transport. Co70Fe30 was achieved using 0.08 mol/L FeSO4 and 0.3 mol/L CoSO4. Constant current density of 0.035 A/cm2 for 40 min yielded ~20 µm film thickness. Post-deposition, samples were rinsed, dried, and released from Ti substrates mechanically. Composition and structure were confirmed by EDS and XRD (BCC, strong (110) peak). Magnetization hysteresis loops were measured at room temperature; films showed good magnetostrictive response and flexibility (no surface chapping; smooth fracture). Device design and simulation: PFES architecture comprises a magnetoexcitation (ME) layer, a magnetoinduction (MI) pickup layer, and the MS layer. COMSOL Multiphysics simulated magnetization distributions under bending strain and external magnetic fields to optimize coil placement; locating coils at the film center maximizes vertical magnetization change collection. The ME and MI layers are flexible printed circuits (FPC) of ~40 µm thickness each, on polyimide (PI) substrates. Substrate and copper thicknesses are 12.5 µm and 18 µm, respectively. The ME layer includes one pair of planar exciting coils; the MI layer includes one pair of pickup coils connected in subtractive series to cancel transformer effects, producing a net flux proportional to external curvature and magnetic field. Coil turn ratio (exciting:pickup) is 1:1; routing width and clearance are 70 µm. The MS layer is the electrodeposited Co70Fe30 film (~20 µm) on the copper layer. Due to small impedance angle, no matching circuit is needed. Excitation: 10 mA at 5 kHz; calculated average power <25 mW. Electronics and signal processing: The MI output is processed via a half-wave amplifier, then a peak detection (PKD) amplifier, and digitized by a 12-bit ADC. A dual-output 3 A MOSFET driver provides a ±10 V square-wave excitation to the ME coils. A linear resonance actuator (LRA) for haptic feedback is driven by PWM via a haptic driver (100% duty = full vibration; 0–50% = braking/no vibration). Processed digital signals from multiple PFES units are sent via Bluetooth to a PC for analysis. Experimental characterization: Static tests applied controlled bending (3-point bend fixture) over curvature radii ρ = 10–65 mm and magnetic fields H = 1–11 kA/m using NdFeB N35 permanent magnets (Ø20×5 mm, Ø30×5 mm, Ø40×5 mm) at varying distances. The MI output was amplified 200× and analyzed with a lock-in amplifier. Sign conventions: front side (MS layer) positive for applied force and magnetic field; back side negative. Geometry effects were assessed with PFES lengths of 30, 35, and 40 mm; 35 mm was selected to match finger/joint size. MS layer thickness effects were studied with films of 10, 20, and 30 µm. Thermal reliability was evaluated from 25–100 °C using an integrated furnace and control software. Dynamic performance was measured under periodic force and magnetic stimuli at 1–4 Hz. Hysteresis and response/recovery times were measured at ρ = 65 mm and H = 11 kA/m. Autonomous learning framework: For on-device gesture interaction, sensor data from PFES units installed at hand joints were used to train deep models for motion classification (gesture recognition) and position regression (for haptic feedback). A Deep Reinforcement Learning-based Knowledge Distillation (DRL-KD) scheme was developed: multiple pretrained teacher models are weighted dynamically by a loss-driven policy so the student model preferentially distills knowledge aligned to a user’s motion characteristics, enabling compression and rapid deployment with few-shot user data while mitigating effects of training sample complexity and teacher capability differences.

- PFES provides dual-mode sensing, with output amplitude positively related to curvature (bend) and negatively related to magnetic field (position), enabling comprehensive biperceptive information from a single sensor and recognition of motion features from low-dimensional data. - Geometry effects: Increasing sensor length (30→40 mm) increases curvature sensitivity; 35 mm chosen to match finger/joint size. Larger magnets (Ø20, Ø30, Ø40×5 mm) increase detectable magnetic field at longer distances, aiding position detection. - MS layer thickness optimization: 20 µm Co70Fe30 yields maximal sensitivity, balancing increased magneto-mechanical coupling with increased brittleness seen at thicker films (10 vs 20 vs 30 µm comparisons). - Temperature robustness: Measurement error <4.2% across 25–100 °C for both curvature radius and magnetic field sensing, indicating high thermal reliability. - Static characteristics: Symmetrical response for opposite bending/magnetic directions. Maximal sensitivities reported: 22 mV/mm over curvature radius 10–65 mm; 1.78 mV/(kA/m) over magnetic field 1–11 kA/m. - Dynamic performance: Response time ~17 ms; recovery time ~39 ms. Sensitivity variation <1.6% under 1–4 Hz dynamic bending and magnetic field, indicating good dynamic stability. - Power and electronics: PFES can be directly excited without a matching circuit; average device power <25 mW at 10 mA, 5 kHz excitation. Integrated electronics (amplification, PKD, 12-bit ADC, Bluetooth) and LRA-based haptics demonstrated. - Autonomous learning: DRL-KD enables a student model to dynamically weight multiple teacher models based on loss, selecting knowledge aligned to user motion characteristics and allowing few-shot, rapid deployment for gesture recognition and haptic feedback.

The PFES addresses a key gap in wearable gesture interfaces: conventional bend/stretch sensors capture finger shape but not joint position, leading to ambiguity in intent (e.g., Like vs No-like thumbs). By fusing curvature-induced strain and magnetic-field-based position in a single magnetostrictive device, PFES yields low-dimensional yet rich features that disambiguate gestures. The device exhibits high sensitivity to bending and sufficient magnetic responsiveness, with symmetric bidirectional behavior, enabling flexible placement on hand joints. Its robustness across temperatures (25–100 °C) and under dynamic conditions (1–4 Hz) supports practical, real-world HMI scenarios. Low power operation (<25 mW) and straightforward excitation simplify integration into wearable systems. On the algorithmic side, DRL-KD allows on-device models to be compressed while adapting to user-specific motion patterns. By dynamically weighting multiple teachers via a loss-based policy, the student model distills more relevant knowledge, mitigating variability in training sets and teacher capacities. This reduces data collection burdens and accelerates deployment with few-shot samples. Together, the hardware biperception and autonomous learning enable reliable gesture recognition and closed-loop haptic feedback for cross-space interaction between real and virtual environments.

This work introduces a perceptive fusion electronic skin (PFES) that integrates a magnetostrictive Co70Fe30 film with flexible excitation/pickup coils to simultaneously sense curvature and magnetic field, providing joint shape and position information from a single device. The PFES demonstrates high sensitivity, bidirectional symmetry, thermal and dynamic stability, and low power consumption, with design optimizations for sensor length, magnet size, and MS layer thickness (optimal at 20 µm). An integrated electronics platform supports real-time processing and haptic feedback. Complementing the hardware, a DRL-based knowledge distillation approach enables autonomous selection of teacher knowledge and compression to deployable student models using few-shot user data, facilitating rapid, personalized gesture interaction and haptics across real and virtual spaces.

- The PFES exhibits strong nonlinearity under large strains due to magnetostrictive hysteresis, which may affect accuracy in high-curvature regimes; hysteresis is evidenced by distinct loading/unloading behavior despite relatively fast response (17 ms) and recovery (39 ms). - Magnetic-field response is characterized and near-linear within a limited low-field range (1–11 kA/m); performance at higher fields/distances was not detailed in the provided text. - Dynamic testing was limited to 1–4 Hz; behavior at higher frequencies or under complex, multi-axis motions was not reported in the provided text. - While DRL-KD is proposed for few-shot deployment, quantitative comparisons and on-device benchmarks were not included in the provided excerpt.