Addressing the Blind Spots in Spoken Language Processing

Human speech is typically accompanied by a dynamic combination of co-speech gestures and facial expressions, together forming an integral part of human communication. These non-verbal cues provide additional layers of meaning, clarify intention, emphasize points, regulate conversation flow, and facilitate emotional connection, conveying complex or nuanced information that words alone may not capture.

Co-speech gestures supplement verbal communication by offering additional information (e.g., object size or shape), emphasizing or concretizing abstract concepts (e.g., gesturing upwards to signify increase), controlling conversation flow, and compensating for limitations of speech in high-stakes or noisy settings.

Facial expressions indicate emotion and stance, emphasize important aspects (e.g., raised eyebrows), provide social cues, clarify verbal meaning in ambiguity (e.g., confused expression), and enhance rapport and engagement.

While NLP excels at understanding text, understanding speech remains more complex and current text-based models ignore rich non-verbal layers. Existing work often treats gestures as accessories rather than integral components. The paper proposes leveraging advances in sign language processing to implement universal automatic gesture segmentation and transcription, transcribing co-speech gestures into text to bridge blind spots in spoken language understanding and move toward a holistic, multi-modal understanding of communication.

The paper notes that despite progress in NLP, non-verbal cues are often ignored. Prior work has explored generating co-speech gestures from audio (e.g., Ginosar et al., 2019; Bhattacharya et al., 2021; Liu et al., 2022), generally treating gestures as ancillary rather than integral to meaning. Other multimodal modeling efforts incorporate images, videos, or audio using approaches like VQ-VAE (van den Oord et al., 2017; Razavi et al., 2019; Yan et al., 2021), but these can increase context size, require original high-bandwidth signals, and limit transferability. The proposed direction contrasts by transcribing non-verbal cues into discrete textual tokens, aligning with sign language transcription traditions (e.g., SignWriting; Sutton, 1990).

The paper proposes a text-centric integration of non-verbal information by transcribing gestures and facial expressions into a discrete, textual representation that can be ingested by existing NLP models.

Rationale: Text is abundant, compressible, semi-anonymous, and easy to edit, whereas raw audio/video adds significant bandwidth and computational costs. Prior multimodal approaches often require sending original signals and expand context windows. A textual transcription of non-verbal cues aims to be flexible, universal, efficient, and compatible with current pipelines.

Proposal and advantages:

• Universal transcription system for body language, analogous to orthographies for spoken languages, converting gestures, facial expressions, and other non-verbal cues into text.
• Flexibility: Localized transcription conventions can reflect cultural/contextual variation.
• Computational efficiency: Lower resource requirements versus image/video processing.
• Compatibility: Discrete tokens fit seamlessly into existing LLMs without modification.
• Anonymity: Removes biometric information by avoiding raw media sharing.
• Explainability: Textualized non-verbal input is transparent and auditable.

Seamless integration: The transcription layer augments current NLP inputs without requiring architectural changes; systems can include or omit it as needed.

Implementation steps:

1. Capture both video and audio during speech.
2. Use sign language segmentation models to identify boundaries of individual gestures.
3. Transcribe these gestures into a textual notation system (e.g., SignWriting).
4. Use speech-to-text models to transcribe the spoken language and identify word boundaries.
5. If word boundaries are unavailable, use re-alignment to approximate boundaries.
6. Combine speech and gesture transcriptions into a single text stream, where gestures add contextual tags to the spoken words.

Training and inference: Similar to context-injection in MT, models trained predominantly on unmarked text can learn biases; adding smaller datasets with non-verbal transcription helps learn correlations between language and context. At inference, users can provide only text for generalized outputs or include non-verbal tags for more accurate, context-aware outputs.

• Text-only NLP misses critical layers of meaning conveyed by non-verbal cues, leading to misunderstandings of intent, emphasis, discourse regulation, and emotion.
• Motivating examples show congruent cues (e.g., nodding with “yes”) reinforce meaning, while incongruent cues (e.g., rolling eyes while saying “OK”) reveal contradiction.
• A sentiment analysis vignette illustrates that text or audio alone may rate neutral/positive while body language signals negative affect; thus, current models can misinterpret emotionally charged interactions.
• Cultural variation in non-verbal communication underscores the need for flexible, universal yet locally adaptable transcription systems.
• Transcribing non-verbal cues into discrete text tokens offers computational efficiency, anonymization, explainability, and seamless compatibility with existing NLP models, avoiding the overhead of raw video/audio processing.

The paper argues that integrating transcribed non-verbal cues directly into NLP inputs addresses blind spots in spoken language understanding by explicitly representing gesture and facial expression information that modulates or contradicts verbal content. This approach enables more accurate interpretation of intent, emotion, emphasis, and turn-taking, improving tasks such as sentiment analysis, dialogue understanding, and machine translation. By leveraging sign language processing advances (segmentation and transcription), the proposal offers a practical path that maintains the efficiency and transparency of text-based pipelines while enriching them with multimodal context. The cultural variability of non-verbal cues is accommodated by flexible transcription conventions, supporting broader applicability across diverse settings.

The paper underscores the fundamental role of non-verbal cues in human communication and highlights gaps in current NLP systems that focus on text alone. It proposes adopting universal automatic gesture segmentation and transcription, inspired by sign language processing, to convert co-speech gestures and facial expressions into textual tokens that can be seamlessly integrated with existing models. This holistic approach promises richer, more context-aware spoken language understanding. The authors call on the community to develop universal gesture transcription methods and to create challenge sets to validate their utility, moving toward robust, real-world multimodal interactions.

• Cultural variability: Non-verbal cues vary widely across cultures and contexts, complicating universal transcription; the paper illustrates this with stereotypical regional differences.
• Lack of empirical validation: The work is a proposal with motivating examples rather than a large-scale empirical evaluation; it calls for community-built challenge sets and validation.
• Data and tooling needs: Effective deployment depends on reliable gesture segmentation, transcription standards (e.g., SignWriting usage), and aligned multimodal datasets.
• Alignment challenges: Accurate temporal alignment between speech word boundaries and gesture units may require re-alignment models when boundaries are not directly accessible.