Eye contact is a fundamental aspect of human social communication, crucial for relationship building, expressing interest, and facilitating joint attention. Abnormal gaze patterns are associated with several neurological and psychiatric conditions, particularly Autism Spectrum Disorder (ASD), where reduced eye contact is a diagnostic criterion. While existing eye-tracking technologies offer gaze measurement, they are often expensive, cumbersome, and unsuitable for naturalistic face-to-face interactions, especially with young children or individuals with high support needs. Manual coding of eye contact from wearable point-of-view (PoV) camera footage is time-consuming and subjective. This research proposes a novel approach using deep learning to automate eye contact detection from PoV videos. The researchers hypothesize that a deep convolutional neural network, trained on a large dataset of human-annotated eye contact events, can achieve accuracy comparable to human experts. They also explore the potential of transfer learning to improve model generalization and address the relatively small number of unique subjects in their dataset (around 100). The study aims to develop a scalable, objective, and accessible tool for clinicians and researchers to analyze gaze behavior.

The paper reviews existing technologies for measuring gaze behavior, highlighting the limitations of conventional monitor-based eye tracking and the challenges associated with using wearable eye trackers, especially in populations like young children or individuals with ASD. It discusses the need for a scalable and less burdensome method for analyzing eye contact in naturalistic settings. The authors cite prior work on automated gaze detection, noting that existing methods fall short of the accuracy achieved by human raters. Previous studies on automated analysis of social behaviors in clinical contexts are mentioned, emphasizing the lack of methods achieving expert-level performance in naturalistic face-to-face interactions, particularly regarding eye contact. The authors also refer to studies using deep learning for biomedical data analysis, but highlight the relative scarcity of such techniques applied to the automated analysis of social behaviors.

The study uses a dataset of 4,339,879 annotated images collected from 103 subjects (57 with ASD) using wearable glasses with a PoV camera. The images were annotated by human raters, indicating the presence or absence of eye contact in each frame. A deep convolutional neural network (CNN) with a ResNet-50 backbone architecture was used. A two-stage training process incorporating transfer learning was employed. The first stage trained the model on public datasets to learn the relationship between pose and eye gaze direction. The second stage transferred this learned model and further trained it on the study's dataset of eye contact events. The model's performance was evaluated using precision, recall, F1 score, and average precision. Inter-rater reliability was assessed using Cohen's kappa, comparing the model's output to the ratings of 10 human coders. The researchers also conducted reproducibility studies, replicating analyses from two previous studies using both manual and automated eye contact coding to assess the congruence of results.

The deep learning model achieved a precision of 0.936 and a recall of 0.943 on the validation set, on par with the performance of 10 human coders (mean precision 0.918, recall 0.946). The model's F1 score (0.940 after smoothing) was comparable to the mean human rater F1 score (0.932). Inter-rater reliability analysis showed high agreement between the model and human coders (average human-detector kappa of 0.891), comparable to human-human reliability (0.888). Statistical tests confirmed the equivalence of the model's reliability to that of human raters. Reproducibility studies demonstrated that findings from two previously published studies on eye contact in autism remained consistent when using the automated coding results instead of human ratings. Correlation analyses showed a negative correlation between the severity of ASD symptoms and eye contact frequency and duration, aligning with expectations. Transfer learning was found to be superior to multi-task learning for this specific task.

The findings demonstrate that deep learning can accurately detect eye contact from PoV video, achieving performance equivalent to human experts. This has significant implications for research and clinical practice, offering a scalable, objective, and cost-effective alternative to manual coding. The study's success in replicating findings from previous studies using automated coding reinforces the validity and reliability of the method. The negative correlation between ASD symptom severity and eye contact provides further support for the model's accuracy and its potential for clinical application. The superior performance of transfer learning over multi-task learning highlights the importance of leveraging pre-trained models for this type of task.

This study presents a novel deep learning model for automated eye contact detection that achieves accuracy comparable to human experts. The model's scalability, objectivity, and reliability make it a valuable tool for researchers and clinicians working with diverse populations. Future research could focus on expanding the dataset to include even greater diversity, exploring the model's performance in different interaction contexts, and investigating its application to other social behavioral analyses.

The study's dataset, while large, is still limited in the number of unique subjects. The model's performance might vary depending on factors such as video quality, lighting conditions, and the presence of occlusions. Further validation across different populations and interaction settings is needed to ensure generalizability. The reliance on human-annotated data for training might introduce some degree of human bias into the model.