Intersemiotic interpretation of demonstrative *that* in modern TV series

The demonstrative pronoun "that" is a fundamental element in English grammar, typically used to indicate spatial distance from the speaker. Previous research in pragmatics, semantics, and cognitive linguistics has largely focused on its spatial and temporal aspects, distinguishing it from "this" based on proximity and speaker orientation. While prior work acknowledged the interplay between linguistic deixis and visual pointing, the visual aspect wasn't treated as an independent semiotic system. Multimodality, however, emphasizes visual images as equal meaning-making resources alongside language. This study addresses three key questions regarding the demonstrative "that" in modern TV series: 1) How do visual deictics resolve the deictic indeterminacy of "that" regarding proximity? 2) How do explicit and implicit uses of "that" highlight the discrepancy between visual and linguistic proximity? 3) How do visual deictics force the transformation of "that" into "this"? These questions aim to reveal how visual and verbal deictic relations connect the semiotic world to reality. The analysis utilizes ELAN 6.3 to annotate audiovisual data from modern TV series, providing a multimodal perspective on the processing, perception, and memory involved in interpreting deictic expressions.

Linguistic studies have established the contrastive pair of "this" and "that," with "that" typically indicating a distal entity. This distinction rests on spatial proximity, speaker location, and egocentric orientation. "That" often retrieves preceding discourse, interrupting the linear flow of communication. Early research considered visual pointing alongside linguistic deixis, but the visual element lacked independent semiotic status. Multimodal discourse analysis, however, recognizes visual elements – gestures, gaze, arrows – as independent deictic systems. Existing research on visual deixis often focuses on concrete referents, neglecting abstract or compressed ones. This study builds upon this foundation, examining how visual deixis interacts with the demonstrative "that" to specify referents, particularly in situations where verbal context alone is insufficient.

This study employs a qualitative method to analyze how visual deictics resolve the ambiguity of "that" in identifying elements within multimodal discourse. Data were collected from modern TV series, including *Milo Murphy's Law*, *The Big Bang Theory*, and *Gray's Anatomy*, focusing on scenes where visual and linguistic deictics interact. The ELAN 6.3 annotation tool was used to meticulously tag the audiovisual recordings. ELAN's multimodal annotation capabilities allowed for the simultaneous analysis of verbal and visual elements, enabling a detailed examination of the interplay between linguistic and visual deictic systems. The analysis proceeded in three stages: 1) Demonstrating how "that" may require visual deictics for adequate interpretation, particularly regarding cause-and-effect relationships. 2) Categorizing "that" as explicit or implicit based on its proximal-distal function within the multimodal context, explaining divergences between visual and linguistic proximity. 3) Illustrating how visual deictics, particularly reappearance, can override spatial and temporal limitations, prompting the transformation of "that" into "this." The combined visual and verbal analysis aimed to show how the intersemiotic interpretation of "that" provides a richer understanding of multimodal meaning-making.

The analysis revealed several key findings: 1) Visual deictics significantly clarify the referents of "that" when linguistic expressions are insufficient, eliminating deictic ambiguity. In *Milo Murphy's Law*, visual elements such as arrows and lines guided the audience through a complex cause-and-effect scenario that was difficult to follow solely from the verbal dialogue. The shifts in perspective between characters were made clear visually, enabling the audience to interpret the ambiguous use of "that." 2) The study distinguished between explicit and implicit uses of "that." Explicit "that" forms a contrastive pair with "this" based on spatial or temporal distance, but in multimodal contexts, this can be relative to both linguistic and visual aspects. Implicit "that," on the other hand, functions neutrally without a clear proximal-distal contrast. In *The Big Bang Theory*, Sheldon's use of "that's...why" showed how the implicit "that" can be used in close proximity (spatial) despite implying a past-time referent (temporal). 3) Visual deictics were shown to influence the choice between "that" and "this." The reappearance of visual elements, as demonstrated in *Gray's Anatomy*, could cause a shift from "that" to "this," even when the traditional rules of proximity based on time and speaker-orientation would dictate otherwise. In this case, the visual elements of memory effectively brought the past events into the present conversational context, influencing the demonstrative's usage. This shows that the visual components can directly influence linguistic choices within a multimodal context. The study found that visual deictics work synergistically with linguistic ones to create a more complete and dynamic interpretation of meaning in television discourse.

These findings demonstrate that a purely linguistic interpretation of "that" is insufficient for understanding its function in multimodal discourse. The integration of visual deictics dramatically alters its usage and interpretation. The distinction between explicit and implicit "that" provides a more nuanced framework for analyzing deictic choices. The capacity for visual deictics to overcome spatial and temporal limitations further expands the understanding of how multimodal communication operates. The results underscore the vital role of intersemiotic interpretation in accurately understanding the meaning created in film and television. The dynamic interplay between visual and linguistic elements highlights the limitations of analyzing these forms of media from solely a linguistic perspective.

This study demonstrates the crucial role of visual deictics in resolving the deictic ambiguity of "that" in modern TV series. The findings reveal a more complex and dynamic interplay between visual and linguistic elements than previously understood. Future research could extend this analysis to other genres, exploring the influence of visual deictics on other deictic expressions and the potential for cross-cultural variations in interpreting these multimodal signals.

While the study provides a rich analysis of "that's" function in selected TV series, generalizing findings to all multimodal contexts requires further research. The selection of TV series may not fully represent all forms of multimodal discourse, and cultural differences in deictic usage are not examined in detail. Further research could address these limitations by broadening the scope of genres and cultures examined.