Behavioral biometric optical tactile sensor for instantaneous decoupling of dynamic touch signals in real time

Behavioral biometric tactile systems authenticate individuals by analyzing interaction patterns such as writing speed and applied pressure, which are difficult to replicate and thus enhance security against fraud. Optical tactile sensors are attractive for interactive devices due to low cost and ease of integration, but conventional designs mainly measure normal force and infer in-plane motion (e.g., velocity) by tracking position across multiple frames. Such time-averaged estimates can be inaccurate for rapid directional changes limited by camera frame rates. Because instantaneous slip velocity correlates with shear force at solid-solid interfaces, an optical tactile sensor capable of instantaneous measurement of both normal and shear forces from a single frame is needed for accurate behavioral biometrics. Prior optical tactile approaches using polymers, photonic crystals, light guides, quantum dots, and dyes typically have slow response, relatively high detection limits, and modest spatiotemporal resolution, preventing single-frame decomposition of dynamic forces. The present work addresses these limitations with a UCN-based optical tactile sensor that decouples normal and shear forces in real time from a single image, targeting applications in surface profiling, fingerprinting, braille recognition, and handwriting-based behavioral biometrics.

Existing optical tactile sensors have used optical polymers, photonic crystals, light guides, quantum dots, and organic dyes to visualize pressure distributions. While effective for static or quasi-static normal force mapping, these platforms generally cannot decompose dynamic force components due to slow response times, higher limits of detection, and limited spatiotemporal resolution. Conventional optical tactile sensors infer in-plane motion through sequential image analysis, resulting in time-averaged velocities that may not reflect instantaneous motion during rapid direction changes. Mechanoluminescence-based sensors have enabled dynamic pressure mapping but still face challenges in sensitivity and response time relative to upconversion-based approaches. Recent e-skin designs can detect pressure direction but typically require multi-frame coordination. The current study builds on these advances by employing lanthanide-doped UCNs, whose narrow emission, large anti-Stokes shifts, stability, and non-linear excitation properties reduce background and enhance sensitivity, enabling single-frame estimation of normal and shear forces with high temporal resolution.

System architecture: The tactile sensor system comprises a tactile sensor pad, a total internal reflection (TIR) dove prism, and a camera (CCD/CMOS). The pad includes (top-to-bottom) a stress concentration layer, a 20 nm anti-reflective Pt layer, and a UCN-embedded PDMS micro-hemisphere (UM) tactile layer. The stress concentration layer mimics the human dermal–epidermal junction using a stiff TPU (≈52.5 MPa) interlocked with soft PDMS (≈1.76 MPa) via an undulated 2D micro-hemisphere array, enhancing force transmission and localization. The tactile layer contains PDMS micro-hemispheres embedded with hexagonal-phase β-NaYF4:Yb3+/Er3+/Gd3+ (30/2/30 mol%) UCNs synthesized hydrothermally. UCN synthesis: A hydrothermal method produced rod-like hexagonal UCNs by mixing NaOH, ethanol, oleic acid, lanthanide chloride solution (RECl3; Gd, Y, Yb, Er, Tm), and NH4F, followed by reaction at 200 °C for 3 h, washing, and dispersion. Emission color and spectra were characterized; UCNs retained luminescence after embedding in PDMS micro-hemispheres. Signal generation principle: A 980 nm NIR laser illuminates the TIR prism, forming evanescent fields at the UM/prism interface. Under applied force, local contact area increases, frustrating TIR and converting evanescent fields into propagating light that excites UCNs within compressed UMs. Due to UCN non-linear luminescence, background from minimal contact is suppressed, enabling high-contrast imaging of pressure distribution. Axisymmetric luminescence corresponds to static normal force; non-axisymmetric profiles arise from shear-induced asymmetric deformation under friction. Device fabrication: UCNs were dispersed in PDMS (Sylgard 184), cast into a micro-hemisphere mold (diameter/pitch: 90/100 µm), cured, and coated with 20 nm Pt. The stress concentration layer was made by forming an undulating PDMS microstructure (20/40 µm diameter/pitch), plasma treating, then casting TPU from DMF onto the PDMS to form the interlocked interface. Imaging and measurements: Luminescence images were acquired at 10 fps (CCD) and 30 fps (CMOS). Spectra were measured with a modular spectrometer. Normal force was applied via a spherical indenter with a vertical force gauge (resolution 0.01 N); shear force was applied with a right-angle configuration measuring lateral force as velocity and acceleration varied. Luminescence images were converted to pressure contour maps (Origin/MATLAB); centerline profiles were analyzed (ImageJ). A thin 200 µm pad without the stress concentration layer was also fabricated for high-resolution surface texture recognition. Friction characterization: A surface forces apparatus (SFA) with a piezoelectric 3D sensor and biomorph slider quantified friction forces between the PDMS pad and a hemispherical brass counterface at driving velocities of 11.6 and 58.0 µm/s, extracting kinetic friction coefficients from friction–load slopes. Finite element analysis (FEA): 2D models with and without TPU–PDMS undulation were constructed to compute local stress distributions and UM–substrate contact width under prescribed normal (Fn) and shear (Fs) forces. Skewness of the simulated contact-width distribution was compared with experimental luminescence intensity profiles. Machine learning (force decoupling): Feature engineering extracted RGB-based features from single-frame luminescence images. Support vector regression (SVR, rbf kernel; C=10, degree=3, epsilon=0.1) estimated normal force from RGB tendencies, then predicted a standard threshold to delineate the normal-force region contour; half-threshold defined the shear region. Direction and magnitude of shear force were inferred via statistical skewness (γ) of the intensity profile, empirically calibrated to measured shear forces. Models were trained on 70% of data and evaluated on the remainder using r², RMSE, and MAPE. Applications: A refined thin pad (200 µm, no stress layer) enabled high-resolution surface profiling (e.g., golf ball dimples, screw threads), fingerprint imaging, and a Braille-to-speech (BTS) pipeline converting luminescence dot patterns to audio via ML. For behavioral biometrics, decoupled normal/shear features and velocity-derived features from handwriting videos (letter 'e', 3 subjects) were used in linear discriminant analysis (LDA) for clustering and classification performance assessment (Silhouette coefficient, Calinski–Harabasz index).

- Single-frame force decoupling: The sensor instantaneously decomposes normal and shear forces from a single luminescence image, providing direction and magnitude of shear without multi-frame tracking. - Sensitivity and detection limit: Optimized pad (1000 µm total thickness with 200 µm stress concentration layer) detects normal forces as low as 0.05 N; without the stress concentration layer, detection down to 0.02 N, outperforming prior optical tactile sensors by ~an order of magnitude. - Linear correlation: Luminescence intensity (notably at 530 and 660 nm) and cumulative grayscale correlate with applied normal force, enabling quantitative pressure mapping. - Temporal performance and stability: Fast response time of 9.12 ms (Gaussian fit). Stable luminescence over 5000 loading cycles at 5.0 N; negligible hysteresis during loading/unloading; stable performance even at elevated temperatures. - Shear characterization: Under lateral motion (preload Fn=1.0 N), circular profiles transform to non-axisymmetric ovals with skewness increasing as shear force grows; higher velocities produce more pronounced non-axisymmetry. Adding silicone oil (reduced friction) eliminated non-axisymmetry, confirming friction-driven origin. - Quantitative shear correlation: Statistically defined skewness (γ) of the intensity profile matches measured shear force; for a given γ, increasing Fn lowers γ due to larger normal-force contribution. - Friction behavior (SFA): Kinetic friction coefficient increased from 0.148 to 0.531 as driving velocity rose from 11.6 to 58.0 µm/s, indicating velocity-dependent friction typical of soft polymer interfaces. - Stress concentration layer effect: Undulated TPU–PDMS interface amplifies shear transmission to the UM layer. Max skewness increased from ~0.1 (no undulation) to ~0.37 (with undulation) at 1.0 mm/s; FEA shows higher asymmetric σyx and larger contact-width skewness with undulation and increasing Fs. Regions beneath the undulated interface had on average ~50% higher σyx. - Spatial resolution and applications: A thin (200 µm) pad without stress layer achieved 100 µm spatial resolution for surface texture recognition (e.g., golf ball dimples, screw threads), high-resolution fingerprint imaging, and real-time BTS translation of Braille patterns. - Machine learning performance: SVR-based decoupling achieved high reliability with r² ≈ 0.965 (normal) and 0.947 (shear). Shear direction was determined from single-frame non-axisymmetric profiles. Single-frame decoupling remained consistent across frame rates; 10 fps sufficed. - Behavioral biometric handwriting: LDA clustering of features (normal + shear) best separated three individuals’ handwritten 'e' compared with normal-only or normal+velocity features. Achieved Silhouette coefficient ~0.867 and Calinski–Harabasz index ~5.76×10^2; adding velocity slightly reduced these metrics (0.858 and 4.52×10^2), highlighting the discriminative power of shear features.

The study addresses the central challenge of capturing instantaneous dynamic touch information by enabling single-frame decoupling of normal and shear forces, overcoming the limitations of multi-image averaging in conventional optical tactile sensors. By combining a human skin-mimetic stress concentration layer with a UCN-embedded micro-hemisphere tactile layer and TIR-based excitation, the system translates localized mechanical contact into high-contrast luminescence with distinct axisymmetric (normal) and non-axisymmetric (shear) signatures. The statistical skewness of the luminescence intensity profile serves as a robust and immediately interpretable metric for shear magnitude and direction, validated against direct shear measurements and friction behavior characterized by SFA. FEA elucidates how the undulated TPU–PDMS interface enhances shear transmission, producing stronger non-axisymmetry and improving sensitivity to dynamic forces. Practically, the sensor’s low detection limits, millisecond response, and cycling stability support use in real-time applications. The ML pipeline reliably separates force components and, in behavioral biometrics, demonstrates that shear-force features significantly improve handwriting discrimination over normal-force or velocity features, reflecting richer dynamic information intrinsic to writing style. The high spatial resolution achieved by the thin-pad variant extends utility to fine surface profiling, fingerprint recognition, and braille translation, further underscoring the platform’s versatility.

This work presents a UCN-based optical tactile sensor that instantaneously and quantitatively decouples normal and shear forces from a single image, enabled by a human skin-inspired stress concentration layer and TIR-frustrated excitation. The sensor delivers high sensitivity (down to 0.02–0.05 N), rapid response (~9 ms), strong durability (≥5000 cycles), and fine spatial resolution (100 µm with a thin pad). Quantitative shear estimation via skewness and an ML pipeline (SVR + thresholding + LDA) enables real-time force mapping, high-resolution surface and fingerprint recognition, braille-to-speech translation, and enhanced behavioral biometric handwriting classification, where shear features provide pronounced discriminative power. Future research directions include expanding dynamic range and force bandwidth, integrating higher-frame-rate or on-sensor imaging for improved temporal fidelity, exploring diverse materials and geometries to tailor friction and sensitivity, developing calibration-free or adaptive models robust to environmental variations, and incorporating flexible, wearable form factors for pervasive behavioral biometric sensing.

- Dependence on frictional interaction: Shear-force-induced non-axisymmetric profiles rely on interfacial friction; lubrication (e.g., silicone oil) suppresses the effect, indicating performance depends on surface conditions and may require calibration across substrates or environments. - Shear force attenuation: Reported shear force can be attenuated by friction at the sensor–prism interface; although accounted for via skewness calibration and ML, this introduces a dependency on calibration quality. - Tip/indenter shape effects: Stress concentration can complicate precise determination of tip shape; while deep learning aids shape identification, this adds algorithmic complexity and may require additional training data. - Frame-rate constraints: Although performance is consistent at 10 fps for decoupling, very rapid transients might benefit from higher frame rates to capture ultra-fast dynamics. - Specific optical setup: Operation requires 980 nm NIR excitation and a TIR prism configuration; alignment and optical components may limit portability until integrated designs are developed. - Generalizability of ML models: SVR/LDA models trained on specific datasets and conditions may need retraining for different sensors, materials, or environmental conditions to maintain accuracy.