Lip language, a method of silent communication using only lip movements, offers a valuable alternative for individuals with vocal impairments. However, accurately decoding lip language presents a significant challenge. Existing methods, such as vision-based systems, suffer from limitations like sensitivity to lighting, facial angles, and obstructions. Sign language, while widely used, requires extensive learning and restricts hand movements. This research proposes a novel lip-language decoding system (LLDS) to address these limitations. The system leverages the advantages of flexible, low-cost, and self-powered triboelectric sensors (TENGs) to capture subtle lip muscle movements. These sensors avoid the drawbacks of visual methods and provide a non-invasive, contact-based approach. The collected signals are then processed using a deep learning model, specifically a dilated recurrent neural network (RNN) with prototype learning, to achieve accurate lip language recognition. The focus is on developing a robust and accurate system that can translate lip movements into understandable speech or text, thereby facilitating barrier-free communication for individuals who lack the ability to speak.

Previous research on lip language recognition has explored various techniques. Vision-based approaches, while prevalent, are susceptible to environmental factors. Other methods, like those employing magnetic implants or surface electromyography (sEMG), have shown promise but may be invasive or expensive. Recent advancements in machine learning, particularly deep learning, have improved the accuracy of lip reading systems. However, these often require large datasets, which can be difficult to obtain for lip language. Triboelectric nanogenerators (TENGs) have emerged as a promising technology for self-powered sensing applications due to their low cost and flexibility. Previous work has demonstrated their use in human motion detection, human-computer interaction, and other areas. This study builds upon these advancements by combining the advantages of TENG-based sensors and deep learning algorithms to create a novel and improved lip-language decoding system.

This study developed a lip-language decoding system (LLDS) comprising flexible triboelectric sensors, a fixing mask, readout electronics, and a deep learning classifier. The triboelectric sensors, based on a contact-separation mode, were fabricated using low-cost materials like PVC, nylon films, copper electrodes, and a polyurethane sponge. The flexible design ensures comfortable and effective signal acquisition from lip muscles. The mask aids in sensor placement and fixation, providing a consistent signal acquisition while ensuring user privacy. The electrical characteristics of the sensors were thoroughly investigated to understand the impact of force, frequency, size, and connection type (series/parallel) on the generated signals. These characteristics, including open-circuit voltage, short-circuit current, and maximum output power, were measured and analysed. Lip motion data was collected synchronously with audio recordings during the pronunciation of vowels, words, and phrases. The data captured the dynamic nature of lip movements, allowing for analysis of the impact of speech speed and individual speaking habits. A dilated RNN model with prototype learning was employed for signal classification. The prototype learning approach addresses the challenge of limited data samples by learning representative prototypes for each class, improving classification accuracy. Model training and testing were performed using a dataset of lip-motion signals from multiple individuals. The performance of the model was evaluated using standard metrics, such as accuracy and confusion matrices. Finally, the developed system was applied to various scenarios, including identity verification, toy car control, and lip motion to speech conversion, demonstrating the system’s versatility and potential.

The fabricated triboelectric sensors demonstrated good sensitivity and stability, showing a sensitivity of 0.376 V/N. The sensor's output voltage and current increased with contact area and thickness up to an optimal point, after which diminishing returns were observed. The sensor exhibited high durability, with minimal signal attenuation even after 2000 cycles of continuous pressing. Analysis of lip-motion signals revealed distinct patterns for different vowels and words. Importantly, the lip-motion signals for silent and voiced speech showed strong consistency, suggesting that this system could also be applied for silent speech applications. Variations in speech speed affected signal amplitude but not the overall waveform shape. Significant individual variations in lip-motion signals were observed across participants, highlighting the potential of this system for personal identification. The dilated RNN model with prototype learning achieved a test accuracy of 94.5%, significantly exceeding the accuracy of a standard softmax-based classifier (91.75%). This improvement was particularly pronounced when using limited training samples. The confusion matrix revealed that while the model achieved high overall accuracy (95%), certain words with similar lip motion patterns showed higher confusion probabilities. In practical applications, the system successfully enabled identity verification by recognizing unique lip-motion patterns for unlocking a gate and directional control of a toy car based on lip commands. The conversion of lip motion to speech was demonstrated, illustrating the system's potential for aiding individuals with vocal impairments.

The results demonstrate the effectiveness of the proposed lip-language decoding system. The use of triboelectric sensors provides a non-invasive, low-cost, and self-powered solution for capturing lip movements. The high accuracy achieved by the dilated RNN model with prototype learning addresses the challenge of limited training data, making the system practical for real-world applications. The system’s ability to distinguish between individuals based on their lip-motion patterns offers a unique avenue for personal identification. Furthermore, the ability to translate lip movements into speech provides a powerful tool for aiding individuals with communication difficulties. These findings highlight the potential for significant advancements in human-computer interaction and assistive technology. The system's potential extends beyond the scenarios tested here; the principles could be applied to other areas such as robotics, VR systems, and silent communication in challenging environments.

This study successfully demonstrated a novel lip-language decoding system using triboelectric sensors and a deep learning model. The system achieved high accuracy, was robust to variations in speech speed and individual differences, and successfully performed in various application scenarios. Future work could explore improving the model's ability to handle more complex linguistic units such as sentences, and enhancing its robustness across diverse populations. Expanding the vocabulary and integrating the system with other assistive technologies, such as speech synthesizers, would enhance its practical utility. Further investigation into miniaturization and integration of the sensors for more comfortable and inconspicuous use is also warranted.

While the system demonstrated high accuracy, it was tested on a relatively limited dataset of 20 words. The generalizability of the findings to larger vocabularies and diverse populations needs further investigation. The system's performance might be affected by factors such as variations in skin moisture and external noise. Furthermore, the current prototype uses a relatively bulky mask for sensor placement; future work should explore more comfortable and discreet sensor integration methods.