The Impact of Gamified Auditory-Verbal Training for Hearing-Challenged Children at Intermediate and Advanced Rehabilitation Stages

Auditory-verbal therapy is important for children who are hard of hearing. With the increasing attention to hearing-challenged children's health and education, the possible rehabilitation training methods and supportive services have been widely studied. Moreover, the technology advancements in hearing aids and cochlear implants have enabled hearing-challenged children to hear sound in some degree, thus the rehabilitation training with auditory and verbal tasks for more effective training effect and enjoyable training experience is better expected.

Looking into the existing rehabilitation training services, they put more focus on the comprehensiveness of training contents and tasks, but neglecting the varying hearing states and training needs of children at different rehabilitation stages. Therefore, the training tasks are not fully tailored to fit different children characteristics. Moreover, the current training process is often repetitive and lacks the consideration of attractive and enjoyable task settings, resulting in the weak user engagement and training compliance. To tackle the issues above, diverse training theories and modes have emerged. For example, Auditory-Verbal Therapy (AVT) focuses on teaching children to listen and speak using their residual hearing. Language Experience Approach (LEA) is to create a language-rich environment through a gamified way with books, stories, and activities. For individuals with severe hearing loss or those who may not benefit from traditional speech training, Augmentative and Alternative Communication (AAC) methods, such as communication boards, speech-generating devices, and gamified mobile apps, can facilitate effective rehabilitation. Amongst them, the gamification theory has been demonstrated effective in strengthening children's learning performance and improving their training experience, and has been widely applied in children's education and training products. Therefore, the integration of gamification approach in auditory-verbal training has become a promising direction for hearing rehabilitation.

Therefore, this research is aimed to investigate the training effect of gamified auditory-verbal rehabilitation. For the purpose, a gamified auditory-verbal training system was developed, in which training tasks are divided into different challenge levels and tailored to match the training needs of children at different rehabilitation stages. Considering that the children at initial rehabilitation stage often have more severe hearing loss and are too young to make independent game interactions, the intermediate and advanced rehabilitation stages are more emphasized. Based on the gamified training system, the training evaluation experiments are organized. Particularly, three research questions are examined: 1) Do the training performances of children at advanced rehabilitation stage differ before and after using the gamified training system? 2) Do the training performances of children at intermediate rehabilitation stage differ before and after using the gamified training system? and 3) Do children enjoy the gamified training approach? The findings from this study are expected to contribute: i) the empirical examination of the training effect of gamified auditory-verbal rehabilitation approach; ii) the detailed analysis of the influence of different gamified task settings to intermediate and advanced rehabilitation stages; and iii) the in-depth understanding of training experience of the gamified hearing rehabilitation approach.

Related works synthesize rehabilitation training theories and treatment approaches for children with hearing impairment and the role of gamification in education and therapy. Rehabilitation commonly proceeds through staged development, from early auditory awareness to auditory discrimination and speech production, requiring individualized, multimodal interventions (e.g., auditory-verbal therapy, auditory-based phonological training, technology-supported tools like cochlear implants, VR, and robot-assisted platforms). Challenges in current tools include insufficient personalization to rehabilitation stage, repetitive content, poor child-centered UX, and compliance issues.

Gamification is grounded in learning theories (Constructivism, Social Development Theory, Social Cognitive Theory, Operant Conditioning) and applies game elements (points, levels, feedback, narrative) to enhance motivation, engagement, and performance across domains, especially in children’s education. Existing gamified auditory-language products (e.g., Splingo, LiSN & Learn, AngelSound, Hear Coach, Hearoes; VR/AR games) indicate potential but lack rigorous empirical evaluation of effectiveness across rehabilitation stages. Prior studies in related contexts (e.g., math performance, sign language apps) show positive impacts of gamification but do not directly assess staged auditory-verbal rehabilitation outcomes. This gap motivates an empirical examination of gamified auditory-verbal training effects, particularly differentiating intermediate versus advanced rehabilitation stages.

Design and system: A digital auditory-verbal training game, Hear to See, was developed for children aged 3–6 to train sound volume perception, auditory memory, auditory description, articulation, sentence expression, and cognitive abilities. It integrates gamification elements (narrative storytelling, rewards/feedback, vivid visuals, multimodal interactions) and tailors training modes to intermediate and advanced rehabilitation stages.

Game modules/tasks:

• Warming up: Sound Leaping (voice-controlled platformer). Phonation volume and duration control a kitten’s movement and jump; used to engage; no data analyzed.
• Auditory memory: Scenario Matching (pair animals and fruits per spoken instructions) and Scenario Understanding Test (identify items from descriptive audio cues). Measures include completion time, plays/repeats of prompts, errors/scores.
• Articulation and expression: Lip-syncing Game (imitate mouth positions; OpenCV facial recognition compares to standard patterns) and Dubbing Game (repeat longer sentences and describe scenes; Baidu AI speech recognition evaluates correctness). Measures include completion time, number of prompt replays, pronunciation errors, and specific mouth/sound errors.
• Cognitive (advanced stage only): Direction Game (pronounce up/down/left/right to navigate a kitten). Measures include completion time and pronunciation errors.

Participants: 31 preschool children (3–6 years) from Ren’ai Rehabilitation Center (Hangzhou): 15 intermediate-stage and 16 advanced-stage; 17 boys, 14 girls; otherwise healthy. Six therapists assisted. Informed consent obtained; IRB approval granted.

Setting and apparatus: Experiments conducted in familiar rehab rooms with an experimenter, participant, and a center teacher present. Guardians observed externally. Device: 13.3" MacBook (2560×1600, macOS 13.0). Additional cameras recorded facial expressions and interactions; screen recordings captured gameplay.

Procedure: Two-week study with weekly sessions (matching center’s schedule). Each session assigned stage-appropriate modules: intermediate stage completed Warming up, Auditory memory, and Articulation/Expression; advanced completed all modules including Direction. Each game had 5 minutes of practice (total 20–25 minutes per round). Task orders were randomized; content variants used between sessions to mitigate repetition effects.

Assessments and measures:

• Pre/post professional assessments: Brief Evaluation Form for Auditory Rehabilitation of Hearing-Impaired Children; Brief Assessment Form for Articulation and Phonological Rehabilitation; Speech Intelligibility Grading Questionnaire; Brief Evaluation Form for Language Abilities of Hearing-Impaired Children.
• In-game performance metrics: completion time, number of prompt plays, errors, and specific error types per game; scenario tests include scores.
• User experience: UEQ (dimensions: attractiveness, perspicuity, efficiency, stimulation, novelty) post-training.
• Interviews: children/guardians and therapists on usability, engagement, and suggestions.

Analysis: Normality tested; paired t-tests compared pre vs post or first vs second training rounds. SPSS Statistics v29 used. Performance evaluation synthesized metrics for auditory memory (time and error rate); articulation/expression (time, prompt plays, error rate); direction cognition (completion time); training engagement (language abilities assessment).

Advanced rehabilitation stage:

• Significant improvements in articulation performance (20.88±3.98 to 22.38±2.68; t=-3.674; p=0.002) and language for expression (t=-2.611; p=0.020). No significant changes in auditory memory (p=0.083), auditory descriptions with/without hints (p=0.083; p=0.164), speech intelligibility (p=0.083), or language as game tool/for coordination (ns).
• Lip-syncing game: Significant reductions in Task A completion time (p=0.024) and errors (p=0.016); Task B errors decreased (p=0.012). Other time metrics showed non-significant trends (p≈0.10).
• Dubbing game: Significant reductions in completion time for Task A (p=0.010) and Task C (p=0.003); pronunciation errors unchanged.
• Direction game: Notable improvement in completion time across the two rounds, indicating better directional recognition and pronunciation (table narrative; exact values not provided).
• Scenario Understanding Test (extended analysis): Largely no significant improvement; one task (D) showed slower completion post-training (p=0.003), suggesting ceiling effects or task mismatch for advanced users.

Intermediate rehabilitation stage:

• Significant gains in auditory memory (3.67±0.49 to 4.00±0.53; t=-2.646; p=0.019), articulation performance (16.20±1.57 to 18.93±4.28; t=-4.771; p<0.001), auditory description with hints (p=0.019), and language for expression (p=0.007). No significant changes in auditory description without hints (p=0.164), speech intelligibility (p=0.334), language as game tool, or language for coordination (ns).
• Lip-syncing game: Significant reductions in completion times for Tasks A (p=0.008), B (p=0.041), and C (p=0.049); errors decreased for Task A (p=0.045), other error changes NS.
• Dubbing game: Significant reductions in completion time for Tasks B (p=0.012) and C (p=0.037); errors NS.
• Scenario Understanding Test: Significant reductions in completion time across Tasks A–E (p values 0.028, 0.008, 0.014, 0.027, 0.009), indicating improved auditory memory/comprehension.

User experience (UEQ; n=31):

• Attractiveness 2.17 (Excellent; top 10%); Novelty 2.15 (Excellent; top 10%).
• Perspicuity 1.79, Efficiency 1.60, Stimulation 1.68 (all Good; 10% better, 75% worse vs benchmark). Interviews confirmed higher interest and preference over classroom teaching.

Qualitative insights:

• Intermediate-stage children favored Sound Leaping and auditory memory games; less favored Lip-syncing/Dubbing due to pronunciation difficulty.
• Advanced-stage children favored Direction and Dubbing; found auditory memory tasks basic/repetitive.
• Therapists observed improved motivation/compliance; recommended better adaptation to colloquial pronunciations and lip-reading recognition, balancing visual elements to avoid distraction, and finer-grained difficulty levels.

Overall pattern: Gamification produced measurable improvements in short-term game performance metrics, particularly for intermediate-stage children’s basic auditory and articulation/expressive skills. Advanced-stage children benefited in specific areas (articulation performance, direction cognition), but broad auditory measures showed limited short-term change; more precise/professional content and longer training may be required.

The study addressed three questions: (1) whether advanced-stage children improve after gamified training; (2) whether intermediate-stage children improve; and (3) whether the approach is enjoyable. Results show that gamification improves selected outcomes for advanced-stage children (notably articulation performance and expression; gains in direction cognition), but broader auditory abilities show minimal short-term change. Intermediate-stage children exhibit consistent improvements in basic auditory memory and articulation/expressive functions, including faster and more accurate performance on scenario-based tasks.

These findings suggest that gamification is most effective for capabilities amenable to short-term change and practice effects—e.g., response speed, simple recall, and guided articulation—while comprehensive language competence (speech intelligibility, functional language use) likely requires longer, more intensive training. The Direction game indicates cognitive training meshes well with gamification at advanced stages.

Importantly, improvements were more apparent in directly logged in-game metrics than in professional clinical assessments, highlighting a gap between performance improvements and underlying, generalized abilities. This underscores the need to align gamified tasks with professional goals, calibrate task complexity to rehabilitation stage, and plan for sustained interventions to translate engagement-driven gains into clinically meaningful outcomes.

Feasibility is high: digital content allows customization to different hearing states and rapid iteration of tasks; multimodal interactions foster engagement; and the system aligns with professional materials. However, designers must balance engaging visuals with cognitive load, adapt recognition algorithms to colloquial speech and diverse mouth positions, and offer finer-grained difficulty tiers to match stage-specific needs.

This study developed and evaluated a gamified auditory-verbal training system for hearing-challenged children and empirically examined its effects at intermediate and advanced rehabilitation stages. Among 31 children over two weeks, gamified training generally improved basic auditory memory, articulation and expression performance, and delivered highly positive user experience. Intermediate-stage children benefited more in basic capabilities, with broad improvements in scenario-based auditory memory and faster articulation tasks. Advanced-stage children showed significant gains in articulation performance and direction cognition, but limited short-term changes in broader auditory measures, indicating the need for precise, professional content and potentially longer training.

Contributions include: (i) empirical evidence on stage-differentiated effectiveness of gamified auditory-verbal training; (ii) an in-depth evaluation of training experience and practical feasibility; and (iii) design insights to tailor gamification (content difficulty, narrative, feedback) to rehabilitation stage. Future work should involve larger samples, longer intervention periods, refined speech/lip recognition adapted to colloquial variability, and expanded, stage-calibrated content to reduce distraction and enhance transfer to clinically assessed abilities.

• Limited sample size (31 children; 6 therapists) restricts generalizability and statistical power.
• Short intervention (two weeks; weekly sessions) may be insufficient to affect comprehensive language abilities and clinically assessed outcomes.
• Some task-content mismatches for advanced-stage participants (e.g., basic/repetitive auditory memory tasks) may have constrained observable improvements.
• Speech and lip-position recognition may not fully capture colloquial variations; visual gamification elements risk distraction if not carefully balanced.
• Results emphasize short-term, in-game performance changes; longer-term retention and transfer to real-world communication were not assessed.