SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue in Multiple Domains

Task-oriented dialogue (TOD) models have made significant progress on written datasets, but there remains a gap to realistic spoken conversations. Prior spoken TOD datasets are small-scale and often lack human-to-human audio or focus mainly on ASR robustness, overlooking unique spoken challenges. The authors introduce SpokenWOZ, a large-scale speech-text dataset for spoken TOD with 8 domains, 203k turns, 5.7k dialogues, and 249 hours of human-to-human audio. SpokenWOZ captures spoken characteristics such as incremental word-by-word processing and commonsense reasoning, and defines new challenges: cross-turn slot and reasoning slot detection. Comprehensive experiments with text-only baselines, new dual-modal models, and LLMs show substantial headroom for improvement in spoken settings, including for fine-tuned models and LLMs (e.g., ChatGPT).