Learning and navigating digitally rendered haptic spatial layouts

The study investigates whether ultrasonic friction-reduction based digital haptics can convey spatial information about complex scenes well enough to support learning of layouts and navigation without vision. Prior work showed that digitally simulated haptics can support object recognition and mental rotation, and that spatial cognition is supported by multisensory processes with functional overlap across modalities. Traditional tactile maps are static and face design, cost, and resolution constraints. The authors hypothesized that blindfolded sighted participants could learn a 2D haptic rendering of an apartment layout, reconstruct it, and navigate trajectories within the real space. They expected high similarity in LEGO reconstructions and better performance on trained versus untrained trajectories, examining generalization across trajectories of differing difficulty.

The paper reviews evidence that vision and touch are closely linked and can yield functionally equivalent spatial images. Neural substrates of spatial learning and navigation include the hippocampal-entorhinal system and parietal cortex, supporting allocentric and egocentric representations, replay, and route planning. Prior studies demonstrate the utility of vibrotactile cues for object recognition, orientation detection, cognitive map formation, and wayfinding, including touchscreen-based multimodal maps. However, tactile cartography faces challenges such as limited resolution, bespoke design requirements, and expense. Digital haptic technologies on mobile devices offer dynamic, multimodal, and portable alternatives. Earlier work by the authors showed that ultrasonic digital haptics can support mental imagery and manipulation of simple 2D objects in both sighted and visually impaired users. It remained unknown if ultrasonic haptics alone could support learning of complex scenes and navigation within them. The review also situates the work within multisensory spatial cognition and sensory substitution research and highlights potential rehabilitation applications for visually impaired populations and other clinical groups.

Participants: 25 healthy adults (15 women, 10 men; age 18–39 years; mean 27.08 ± 4.04), normal or corrected vision, no neurological or psychiatric history. One left-handed, others right-handed. Ethics approval obtained; informed consent provided.

Apparatus: A 7-inch TFT capacitive touchscreen tablet (1024×600) controlled by Raspberry Pi 3 (Raspbian), Broadcom ARMv7 quad-core 1.2 GHz, 1 GB RAM. Haptics rendered via ultrasonic friction reduction using pre-installed Texture Editor; figures in JPEG recoded to haptic format using C++. Ultrasonic waveforms: square (walls, obstacles, outside areas) with 4660 µm period; sinusoidal (trajectory) with 2550 µm period; amplitude approx. 2 µm. Two GoPro cameras recorded navigation and tablet exploration.

Stimuli: Three JPEG images (1023×574 px) derived from a real apartment floor plan: one base layout and two depicting different trajectories to Room1 and Room2. Four textures mapped features: walls/doors, furniture obstacles, outside areas, and the trajectory. White background carried no texture.

Design and procedure: Participants were blindfolded and wore noise-cancelling headphones throughout. Training occurred outside the apartment; testing occurred inside. Training stages: (1) Familiarization with haptic textures and exploration strategy (lateral finger sweeps) while keeping non-dominant hand as spatial reference on tablet edge; (2) Learning the base layout via tablet exploration; (3) Verbal description check of main layout, room number and placement, and corridor shape; (4) LEGO reconstruction of the apartment layout on a standardized board while blindfolded; (5) Introduction to the trajectory texture and training on one trajectory leading to either Room1 or Room2. Participants were randomly and counterbalanced assigned to Group 1 (trained on the harder trajectory) or Group 2 (trained on the easier trajectory) based on pilot difficulty differences.

Sightless navigation familiarization: Inside the apartment, participants attempted the trained trajectory while blindfolded to learn navigation technique and error zones; no feedback on correctness was given.

Testing: Ten trials per participant, five for each of the two trajectories (trained and untrained), random order and target room not cued. On each trial, participants freely explored the tablet until confident, then navigated the real space as quickly and accurately as possible. Measures: Accuracy (reaching correct room), Reaction time (RT, seconds for accurate trials), Tablet exploration time (ET), Errors (deviations into predefined error zones with graded points), and Time off track (Toff, seconds spent off the intended path). Missed trials were scored only for accuracy; errors not assigned on missed trials. After each trial, participants were returned to the entrance.

Inter-rater reliability: Three independent raters scored LEGO reconstruction quality (1–5). Two independent raters scored behavioural videos for Accuracy, Errors, and Toff. Cohen’s kappa for Accuracy; ICCs for Errors and Toff. LEGO reconstructions were also quantified with a Jaccard similarity index computed by binarizing reconstruction photos, preprocessing for alignment, and comparing to an ideal reconstruction using morphological operations and Jaccard formula. Mean rater scores and Jaccard indices were correlated after normality checks.

Statistical analysis: Behavioural data analyzed with repeated-measures permutation mixed-design ANOVAs (5000 permutations) with within-subject factor Training (trained vs untrained) and between-subject factor Group (Group 1 vs Group 2). Post hoc paired permutation t tests (5000 permutations) or Wilcoxon tests as appropriate. Inverse Efficiency (IE) scores computed to combine RT and Accuracy. Correlations between Jaccard indices and significant behavioural measures were assessed with Kendall’s tau and FDR correction.

• Inter-rater reliability:
  • LEGO reconstruction ratings: ICCs for agreement/consistency were excellent (ICC 0.863 and 0.826).
  • Behavioural scoring: Accuracy kappa k = 1, p < 0.01 (perfect agreement). Errors ICC = 0.64 (good). Time off track ICC = 0.76 (excellent).
• LEGO reconstruction performance:
  • All Jaccard similarity indices exceeded 0.7, indicating high similarity to the ideal layout.
  • Jaccard index correlated with mean subjective scores: r = 0.53, t(23) = 2.99, p = 0.006; 95% CI [0.17, 0.77].
• Trajectory navigation performance (Training × Group interactions):
  • Accuracy: Significant interaction p = 0.004, ω2 = 0.14. Group 2 showed higher accuracy on trained vs untrained trajectories (0.88 vs 0.57), t = -3.44, p = 0.01, d = -0.95. No significant trained vs untrained difference for Group 1.
  • Time off track: Significant interaction p = 0.02, ω2 = 0.06. Group 2 spent more time off track on untrained vs trained trajectories (2.7 s vs 0.1 s), t = 2.12, p = 0.003, d = -0.59. No significant difference for Group 1.
  • Reaction times: Significant interaction p = 0.01, ω2 = 0.11. Group 1 took longer on trained vs untrained trajectories (18.93 vs 13.71 s), t = -2.95, p = 0.01, d = -0.81. No significant RT difference for Group 2.
  • Inverse Efficiency (IE): Significant interaction p = 0.008, ω2 = 0.17. Group 1 showed worse IE on trained vs untrained trajectories (means 29.6 vs 17.7), Wilcoxon t = -5.7015, p = 0.0008, d = -1.58; no significant difference for Group 2.
• Correlations between layout learning and navigation:
  • Group 2 untrained trajectories showed a negative association between Jaccard and IE (tau reported as r = -0.63, p = 0.04), but not significant after FDR correction (p = 0.14).
• Overall, participants could learn spatial layouts from ultrasonic digital haptics and translate this knowledge to blindfolded navigation in a real apartment, with better performance on trained than untrained paths, especially when initially trained on easier trajectories.

The findings demonstrate that ultrasonic friction reduction based digital haptics can convey complex spatial layouts and support blindfolded navigation within real spaces. Participants rapidly formed mental images of a novel apartment layout from haptic rendering alone and reconstructed these layouts with high similarity, validating the capacity of digital haptics to support topographical mapping and externalization of mental representations. Navigation results revealed asymmetries in generalization: participants trained on the easier trajectory (Group 2) showed marked drops in accuracy and increased time off track on the untrained harder trajectory, whereas participants trained on the harder trajectory generalized better to the easy trajectory, albeit with increased traversal time on trained paths. This pattern suggests that training difficulty modulates navigation strategies and generalization, with harder training potentially fostering more robust spatial representations or cautious navigation strategies reflected in longer RTs. Moderate associations between reconstruction quality and efficiency on untrained paths in Group 2 hint that stronger layout encoding benefits transfer to harder routes. The results align with multisensory theories and the functional equivalence hypothesis by showing that spatial images derived from touch can guide behavior similarly to visually derived representations. These findings also support the broader adoption of digital haptics for spatial learning, navigation assistance, and rehabilitation, while indicating the need to optimize training regimens and trajectory sets to enhance generalization.

Ultrasonic digital haptics enabled blindfolded sighted participants to learn a complex apartment layout, reconstruct it with high fidelity, and navigate trained and untrained trajectories in the real space. Training on harder trajectories facilitated better generalization to easier paths, whereas training on easier trajectories led to degraded performance on untrained harder routes. The study provides proof of concept that digital haptics can effectively convey scene-level spatial information for navigation without vision, with implications for sensory substitution, neurorehabilitation of spatial functions, mobility training in visual impairment, and applications in virtual reality and education. Future work should expand the diversity and number of trajectories, refine haptic rendering parameters and interaction strategies, examine multimodal combinations, and include visually impaired participants to tailor and assess translational impact.

• Potential influence of verbal guidance during training on participants spatial representations, despite efforts to avoid quantitative feedback.
• Limited set of two target trajectories and focus on two rooms to contain session duration, constraining assessment of generalization across diverse routes.
• Only sighted participants tested under blindfold; results may not generalize to visually impaired or congenitally blind populations with different haptic expertise.
• No analysis of exploration strategy videos included in the current report.
• The tablet haptic rendering used limited waveform options and fixed amplitude; broader parameter exploration may improve performance.
• Small sample size for subgroup analyses may limit power to detect finer effects and correlations (e.g., FDR-corrected correlations became nonsignificant).