Intersemiotic interpretation of demonstrative *that* in modern TV series

The paper examines how demonstrative that functions in multimodal contexts, focusing on deictic proximity and reference resolution when linguistic cues alone are insufficient. Prior linguistic studies distinguish this vs. that via spatial, temporal, and egocentric orientation, often privileging the speaker’s perspective and textual retrieval. With the rise of multimodal discourse analysis, visual elements (e.g., gaze, arrows, reappearance) are recognized as independent deictic resources. The study addresses three research questions: (1) how visual deictics eliminate the deictic indeterminacy of that in modern TV series; (2) how explicit vs. implicit uses of that reveal divergences between visual and linguistic proximities when identifying referents; and (3) how visual deictics can force a transformation of that into this. Using ELAN 6.3 to annotate audiovisual data, the study aims to reveal how verbal and visual deictic relations map onto cognition, perception, and memory in real-time discourse.

In linguistic discourse, demonstrative that typically marks distal reference and contrasts with this (near), often requiring backward textual retrieval and aligning with speaker-oriented deixis. In visual semiotics, arrows, gaze, pointing, and reappearance function as deictic vectors with flexible, non-sequential spatial orientation, reflecting the iconic nature of images. Multimodal studies show that visual deictics can clarify referents when language is ambiguous or underspecified, pushing communication forward without backward retrieval. However, most visual-deixis work targets concrete, one-to-one references; fewer studies address abstract or compressed referents, indicating a gap this paper engages with.

A qualitative, multimodal analysis was conducted on scenes from modern TV series: Milo Murphy's Law (Animated Series, S1E4), The Big Bang Theory (S10E9), and Grey's Anatomy (S7E1). Data were annotated with ELAN 6.3, creating tiers for words, gestures, visual deictics (arrows, curve lines, crosses, gaze), and other relevant elements, with precise temporal alignment. Analysis proceeded in three steps: (1) show how visual deictics correct/modify and disambiguate the referent of that to clarify causal chains; (2) classify that into explicit (forming a distal contrast with a proximal deictic, linguistic or visual) vs. implicit (neutral, non-contrastive) to account for divergence between visual and linguistic proximities; (3) demonstrate how visual deictic reappearance and episode shifts can break spatial/temporal constraints and cause that→this transformation. The approach centers on intersemiotic complementarity to model forward-moving communication without reliance on retrospective textual retrieval.

• Visual deictics eliminate ambiguity of that: In Milo Murphy's Law (S1E4), red arrows, a yellow curve line, and crosses clarify a causal chain where multiple uses of that (backdrop, ax, rat, lamp, rope) would otherwise be ambiguous due to shifting perspectives between speakers; visual cues disambiguate and keep the narrative moving forward from the audience’s perspective. - Explicit vs. implicit that: Implicit that is an unmarked, neutral deictic not forming a contrast with this and can refer to proximal content (e.g., Sheldon in The Big Bang Theory S10E9 says “Oh, that’s … why” while gazing at a nearby page). Explicit that forms a distal-proximal pair with a proximal visual deictic even if no linguistic this occurs (e.g., “Not that lamp” contrasted with an on-screen arrow indicating a proximal lamp). - Visual vectors can trigger that→this transformation: In Grey’s Anatomy (S7E1), reappearing memories (episode shifts) serve as visual deictic vectors that foreground immediacy and emotional proximity, justifying Bailey’s use of this for a past event, overriding expectations based on tense or discursive proximity. - Dynamics and perspective shifts: Visual deictics (gaze, arrows, reappearance) coordinate with linguistic that to reflect cognitive processing and shifting viewpoints, allowing forward progression without textual backtracking. - Abstract referents: Visual deictics indirectly construct abstract entities (e.g., the day), complementing language’s symbolic directness and demonstrating multimodal means of deixis beyond concrete objects.

The findings address the research questions by showing that visual deictic resources resolve the indeterminacy of linguistic that, especially under perspective shifts and underspecified verbal references. Classifying that as explicit vs. implicit accounts for divergences between visual and linguistic proximities, explaining otherwise non-canonical uses (e.g., using that for proximal items). The analyses further demonstrate that visual episode shifts and reappearance can reframe temporal/spatial relations and drive a transformation from that to this, highlighting how multimodal cues can neutralize or override default linguistic deictic contrasts. These results underscore the complementarity of visual and verbal systems in real-time meaning-making, enabling forward narrative flow and enhancing audience comprehension. The study expands deictic theory to multimodal contexts, indicating that deixis is not solely speaker-anchored nor strictly tied to linguistic tense or spatial metrics, but dynamically co-constructed by multiple semiotic vectors.

When linguistic that cannot adequately identify a referent, visual deictic vectors (arrows, curve lines, crosses, gaze, reappearing images) can eliminate deictic indeterminacy in multimodal discourse. The explicit/implicit distinction for that clarifies divergences between visual and linguistic proximities and explains artistic, narrative effects. Visual deictics can transcend spatio-temporal constraints and, via reappearance or episode shifts, force a transformation from that to this. These insights affirm visual images as legitimate, independent semiotic resources that can influence linguistic deictic choice, enriching our understanding of multimodal communication and the dynamic, multi-dimensional world it represents. Future research can extend intersemiotic analyses to other deictics and genres such as advertisements, news reports, and commercials.

The study does not exhaust all visual and linguistic deictics and focuses on selected instances from modern TV series, leaving room for further intersemiotic investigations across other deictics and genres.