Recent advancements in Natural Language Processing (NLP), particularly the emergence of large language models like OpenAI's GPT-3, have generated significant interest in creating more human-like conversational agents. GPT-3's ability to generate surprisingly coherent and contextually relevant text responses opens up exciting possibilities for Human-Computer Interaction (HCI) designers. However, applying GPT-3 effectively to specific tasks, like creating a helpful chatbot, presents several challenges. Evaluating the chatbot's feasibility and designing prompts that optimize GPT-3's performance for a given task is complex. This study focuses on these challenges, using a case study that explores the design and application of a GPT-3-based chatbot aimed at improving users' moods and mental well-being within a brief, five-minute interaction. The research aims to address critical questions regarding the representativeness of initial conversations, scalable testing methodologies, and prompt engineering techniques for HCI researchers lacking NLP expertise. The inherent complexities of GPT-3 and the open-ended nature of natural language pose significant barriers to predicting chatbot behavior and the impact of prompt modifications. This study's focus on a controlled, five-minute interaction on a platform like Mechanical Turk allows for the collection of a large dataset to understand the dynamics of GPT-3 chatbot behavior and the influence of prompt designs. This controlled environment also mitigates potential risks associated with using LLMs in sensitive contexts like mental health support, while still addressing the crucial need for understanding GPT-3's capabilities and limitations in such applications. The concept of "prompt engineering" is central to this research, involving the systematic design and testing of prompts to guide the LLM's output. This paper provides a case study in prompt engineering specifically tailored for the development of GPT-3-based chatbots, offering insights and methodologies that can be applied across a range of applications.

The paper reviews existing literature on GPT-3, highlighting its capabilities and limitations, particularly in sensitive contexts like eHealth. The authors discuss the challenges of ensuring safety, trust, and efficacy when deploying GPT-3 for mental well-being applications. They also examine previous research on chatbots designed for mental health support, noting the prevalence of rule-based approaches and the exploration of generative pre-trained models in this domain. The literature review highlights the successful use of chatbots for encouraging self-disclosure and self-compassion, along with their role in emotion regulation and digital counseling. The study incorporates established best practices and design guidelines for mental health chatbots, adapting them to the context of GPT-3. The existing work on prompt engineering, particularly in the context of text-to-image generation, is examined and contrasted with the challenges of evaluating prompt designs in the context of chatbot interactions. The recursive nature of chatbot conversations, where user input influences the LLM's response, and the LLM's response, in turn, influences the user, complicates the evaluation process significantly.

The study employed a randomized factorial experiment to investigate the design space of prompts for a GPT-3-based chatbot. Three dimensions of prompt modification were tested: identity (Coach vs. Friend), intent (open-ended reflection, Cognitive Behavioral Therapy (CBT), and problem-solving), and behavior (strong interpersonal skills, specific positive attributes, and a detailed professional/client relationship). These three dimensions created a 2 x 3 x 3 factorial design, resulting in 18 distinct chatbot variations. The experiment recruited 945 participants from Amazon Mechanical Turk (MTurk), a platform for online crowdsourcing. Participants engaged in a five-minute conversation with the chatbot, after which they completed a survey. The survey gathered demographic information, measured their initial mood and energy levels, assessed their prior experience seeking mental health support, and gauged their comfort with technology. A key element of the methodology was ensuring transparency about the AI nature of the chatbot to manage ethical concerns and set appropriate expectations. The chat interface explicitly stated that the interaction was with an AI agent designed to help manage mood and mental health. The survey incorporated attention checks using reverse-formulated questions, and participants with suspiciously similar responses were excluded (114 participants, 12.1%). The data collected included user ratings on perceived risk, trust, expertise, and willingness to interact with the chatbot again, alongside logs of the actual conversations. The study used a factorial structure in the experimental design, offering advantages in testing multiple components independently and assessing their combined effects, including potential interactions. This approach allowed for the efficient investigation of various prompt modifiers, identifying which combinations might yield the most positive user outcomes.

The study collected 945 valid survey responses. Quantitative findings indicated a moderately high perception of risk, moderate trust, high evaluation of expertise, and moderate willingness to interact again with the chatbot. No significant differences were found across the 18 different experimental conditions. However, analysis revealed significant differences based on participants' prior history of seeking professional mental health help. Individuals with such history perceived higher risks, slightly lower trust, but significantly higher willingness to interact again. Their perception of expertise remained consistently high. Participants' technology affinity also showed significant associations with their ratings. A more favorable attitude towards technology correlated with higher perceived risk, lower trust, but higher assessed expertise and willingness to interact again. Qualitative analysis, using thematic analysis of user comments on their comfort level, revealed predominantly positive interactions (approximately 70%), with many participants expressing comfort and a sense of being heard. However, approximately 30% of responses expressed concerns regarding data privacy and information handling. Analysis of conversation logs, guided by a psychologist, revealed several patterns. No significant differences were found based on identity (Friend vs. Coach). However, users interacting with the "Friend" identity used more words on average. The "Intent" of the prompt had an impact on conversation length, with the open-ended reflection intent resulting in shorter conversations. Qualitative analysis highlighted how the chatbot's behavior varied across different intents and facilitated aspects like restating concerns in alternative terms, breaking down problems, encouraging elaboration on thoughts and feelings, creating a sense of being heard, maintaining context across conversations, providing rationales for suggestions, offering both short-term and long-term solutions, and acknowledging its limitations. The word count of user responses varied across different prompt modifiers, with subtle differences in average word count observed for different identities and intents.

The findings suggest that designing effective prompts for GPT-3-based chatbots requires a nuanced approach. While the quantitative analysis didn't reveal strong effects of the manipulated variables, the qualitative analysis offers valuable insights into the dynamics of chatbot conversations and user experiences. The study highlights the importance of considering user characteristics like prior mental health experiences and technology affinity when designing and evaluating chatbots. The chatbot's ability to restate concerns, break down problems, and offer rationales for suggestions appears beneficial. The identification of privacy concerns underscores the need for transparent data handling practices. The study contributes to the understanding of how prompt design can influence the interaction flow and overall user experience in a controlled setting. The lack of significant quantitative differences across the various prompt combinations might indicate the need for further refinement of the prompt engineering process or the exploration of other prompt dimensions. The finding that individuals with prior experience seeking professional mental health support showed a higher willingness to interact with the chatbot is unexpected and worthy of further investigation.

This study provides a valuable case study demonstrating the application of GPT-3 in creating a mood-management chatbot. The research showcases a methodology for prompt engineering and evaluation, which is crucial for HCI designers and researchers aiming to develop similar applications. The findings, both quantitative and qualitative, highlight the importance of considering various factors, including user background and technology comfort levels, during the design process. Further research should focus on exploring additional prompt dimensions, optimizing the design of the chatbot's responses, and investigating the long-term effects of such interactions. Addressing user privacy concerns and incorporating automated risk detection mechanisms are vital for enhancing the safety and ethical considerations of these applications.

The study's limitations include the use of MTurk participants, which might not fully represent the broader population. The five-minute interaction time limits the depth of conversation and the assessment of long-term impact. The specific prompt designs tested might not exhaust all possibilities, and the focus was on a particular mental health-related task. Furthermore, the absence of strong quantitative effects in the primary analysis requires further investigation and potential refinements in the experimental design. Finally, the reliance on self-reported data in the surveys necessitates careful consideration of potential biases.