Usability Comparison Among Healthy Participants of an Anthropomorphic Digital Human and a Text-Based Chatbot as a Responder to Questions on Mental Health: Randomized Controlled Trial

Background: The use of chatbots in mental health support has increased exponentially in recent years, and more recently, visual avatars (“digital humans”) capable of facial expressions have been introduced. It is important to study differences in emotional response and usability preferences between text-based chatbots and digital humans in mental health services. Objective: To compare usability between a digital human interface and a text-only chatbot interface using BETSY (Behavior, Emotion, Therapy System, and You), and to explore how chatbot-generated conversations affect self-reported feelings and biometrics. Methods: Healthy participants (n=45) were randomized to use either a digital human with anthropomorphic features (n=25) or a text-only chatbot (n=20). Usability was measured with the System Usability Scale (SUS-10). Emotional reactions were assessed via electroencephalography (EEG) and self-reported feelings (eg, closeness, nervousness). Groups were compared using linear regression analyses and t tests. Results: No demographic differences were observed between groups. Mean SUS-10 was 75.34 (SD 10.01; range 57-90) for the text-only chatbot versus 64.80 (SD 14.14; range 40-90) for the digital human interface, indicating significantly higher perceived usability for the text-only interface. Both interfaces were rated average or above average in usability. Women were more likely to report feeling annoyed by BETSY. Conclusions: The text-only chatbot was perceived as significantly more user-friendly than the digital human, although EEG measurements showed no significant differences. Male participants reported lower levels of annoyance than female participants, contrary to previous findings.

Almira Osmanovic Thunström, Hanne Krage Carlsen, Lilas Ali, Tomas Larson, Andreas Hellström, Steinn Steingrimsson