Response-act Guided Reinforced Dialogue Generation for Mental Health Counseling

The study addresses the challenge of generating contextually appropriate, semantically rich responses in virtual mental health assistants (VMHAs), which exhibit a hybrid conversational flow blending open-domain and task-oriented characteristics. Existing open-domain systems (e.g., GPT-based) often fail to control dialogue acts and maintain therapeutic direction, while task-oriented systems rely on slot-filling unsuitable for counseling’s nuanced dynamics. The research question is whether explicitly leveraging future dialogue-acts (response-acts) can guide response generation to maintain conversational flow and therapeutic intent in counseling dialogues. The purpose is to harmonize VMHA responses with intended dialogue-acts, improving engagement and clinical usefulness. This is important given the rapid adoption of VMHAs and the need for systems that understand and respond appropriately to clients’ directives in sensitive mental health contexts.

Related work spans (i) dialogue-act classification and (ii) response generation. Dialogue-act classification evolved from sparse one-hot act representations (domain-action-slot) to structured and hierarchical encodings and multi-decoder fusion approaches; large-scale pretraining improves recognition but typically requires fine-tuning. For response generation, transformer LMs (GPT-2, XLNet) and fine-tuned conversational models (DialoGPT) have advanced fluency and coherence; controllable generation has been pursued via meta-words, distributional constraints, belief states (SimpleTOD), and prototype-based paraphrasing. Reinforcement learning has been used for dialogue optimization, but prior work rarely integrates predicted next dialogue-acts as guidance for generation. The gap: existing methods either lack act-level control, rely on templates, or require fine-grained slots unsuitable for counseling’s hybrid nature; hence, an act-guided, semantically-aware RL approach is needed.

Problem: Given a counseling dialogue with utterances and their dialogue-acts up to time t, jointly (a) predict the next dialogue-act (response-act) at t+1 and (b) generate the next utterance aligned with that response-act. Model: READER is a transformer-reinforcement-learning (TRL) framework built on GPT-2 as the foundation language model with three heads: (1) LM-Head for text generation; (2) RAC-Head (Response-Act Classification Head), a transformer-based encoder-only module that uses the last hidden states from the base LM, passes them through a GRU and linear projections, followed by multi-head attention (GRU as query, projections as key/value) and softmax to predict the next dialogue-act; (3) V-Head (Value Head) to compute and aggregate rewards guiding PPO optimization. Training via PPO: The model generates a response and predicts a response-act given context; outcomes are compared to reference models to compute rewards; PPO optimizes the LM policy with a clipped objective and an entropy-like constraint to avoid diverging from a strong reference. Reference models and reward: For LM-Head, pre-trained GPT-2 serves as the reference; for RAC-Head, SPARTA (state-of-the-art dialogue-act classifier on HOPE) provides logits. Rewards combine: (i) ROUGE, (ii) BERTScore (semantic similarity), (iii) relative entropy (KL divergence) between READER and the reference LM (penalizes divergence), and (iv) RAC-Head’s logit for the predicted act (to steer generation towards the desired response-act). The overall reward is a weighted sum: w1*ROUGE + w2*BERTScore + w3*RAC logit − w4*KL, with hyperparameters tuned for stability and performance. Training procedure: (1) Initialize RAC-Head and LM-Head to produce response-act and response; (2) evaluate log-probabilities vs reference models; (3) compute ROUGE, BERTScore, KL, and RAC logits; (4) V-Head aggregates rewards and PPO updates the LM policy. Implementation details include careful reward scaling (e.g., scaling KL), learning-rate and batch-size tuning (e.g., lr around 2e-6; large batch size ~128; PPO epochs ~4), Adam optimizer, and training for up to 50 epochs on NVIDIA A6000 GPU. Dataset: HOPE counseling dataset with 12.8k utterances from 212 dyadic sessions, annotated with 12 dialogue-acts (ID, IRQ, YNQ, CRQ, ORQ, CD, PA, NA, OD, GT, ACK, GC). The model also evaluates on Switchboard Dialogue Act (SWDA) for generalizability. Ablations: (a) swapping foundation LM (DialoGPT vs GPT-2), (b) removing RAC-Head, and (c) reward-component ablations (e.g., removing ROUGE, BERTScore, RAC logits, or KL) to assess component contributions.

- On HOPE, READER outperforms strong baselines (DialoGPT, GPT-2, DialogVED, ProphetNet-Dialog, VHCR, HRED variants) across ROUGE, METEOR, and BERTScore, with improvements ranging from 0.82% to 11.53% over the best baselines. Notable results include: ROUGE-2 recall 13.67 (+15.50% vs DialoGPT’s 11.83), ROUGE-1 recall 43.93, ROUGE-L recall 40.82; BERTScore 0.7666 (+4.05% over DialoGPT), METEOR 0.2103. - RAC-Head is critical: removing it reduces ROUGE-1/2/L recall (e.g., −2.45, −1.84, −2.18 vs full model). - Foundation LM choice: READER with GPT-2 slightly outperforms READER with DialoGPT on recall metrics and BERTScore. - Reward design matters: the best performance comes from combining ROUGE, BERTScore, RAC-Head logits, and relative entropy. Using only ROUGE risks repetitive text (reward collapse); using only BERTScore leads to arbitrarily long, unstable outputs. Incorporating SPARTA (RAC reference) substantially improves quality and metric scores. - Human evaluation (50 instances, 10 expert raters) on relevance, consistency, fluency, and coherence shows READER’s outputs slightly but consistently better than DialoGPT and GPT-2. - Generalizability: On Switchboard, READER improves over DialoGPT/GPT-2 on 10/11 metrics, including a +9.4% absolute gain in BERTScore, indicating richer semantics; other metrics show modest gains (0.1–0.6% absolute in ROUGE/METEOR).

The findings support the hypothesis that explicitly guiding response generation with predicted response-acts improves the flow and intent alignment in counseling dialogues. By integrating RAC-Head signals and semantic metrics into a PPO-based reward, READER produces responses that are both semantically rich (higher BERTScore) and structurally appropriate (maintain intended dialogue-acts), addressing the limitations of purely open-domain generators in therapeutic contexts. The model’s superior performance across automatic metrics and human judgments, along with improvements on an out-of-domain dataset (SWDA), indicates better context handling and act-aware control, essential for the hybrid nature of counseling conversations. The approach demonstrates practical promise for VMHAs, potentially enhancing engagement and clinical relevance while maintaining stability via KL regularization against a strong reference LM.

The paper introduces READER, a response-act guided dialogue generation model that jointly predicts the next dialogue-act and generates the next utterance, optimized via PPO with a reward combining ROUGE, BERTScore, relative entropy, and response-act logits. Across the HOPE dataset, READER outperforms multiple baselines on ROUGE, METEOR, and BERTScore, with quantitative gains and better human evaluation results; it also generalizes to Switchboard with notable semantic improvements. The work underscores the importance of act-aware control in counseling dialogue generation and provides a principled TRL framework to enforce it. Future directions include expanding VMHA modules (e.g., empathetic understanding), scaling datasets and deployment, and further refining reward formulations and stability for real-world use.

- Dependence on dialogue-act annotations and an external reference classifier (SPARTA) for reward signals may limit applicability where such labels/models are unavailable. - Training stability is sensitive to reward design and hyperparameters: using only ROUGE can cause repetitive outputs (reward collapse); using only BERTScore can produce overly long, unstable responses; higher learning rates destabilize PPO. - Human evaluation covers 50 instances with 10 raters—useful but relatively small-scale. - Evaluation is limited to HOPE and Switchboard; broader domains and multilingual settings are not explored. - The authors emphasize retaining a human-in-the-loop for safety due to the sensitivity of mental health counseling; full autonomy is not claimed. - Some deployment results (A/B testing) are not disclosed due to privacy, limiting external validation.