Blinded, randomized trial of sonographer versus AI cardiac function assessment

Accurate quantification of cardiac function, particularly left ventricular ejection fraction (LVEF), is crucial for disease diagnosis, risk stratification, and treatment response assessment. LVEF is routinely used to guide clinical decisions regarding various therapies and interventions. However, conventional LVEF measurement methods rely on manual and subjective human ratings, leading to heterogeneity and variance. Clinical practice guidelines recommend repeated measurements to account for intra- and inter-observer variability; however, this is rarely feasible due to practical limitations. Visual estimation is often used as a pragmatic alternative, but this is suboptimal for detecting subtle changes critical for therapeutic decisions. Recent advancements in artificial intelligence (AI) have led to the development of algorithms aimed at automating LVEF assessment. While these algorithms have shown promise in retrospective datasets, they lack validation in blinded, randomized clinical trials. This study aimed to prospectively assess the impact of initial AI versus conventional sonographer assessment on final cardiologist interpretation of LVEF through a blinded, randomized, non-inferiority clinical trial.

The literature highlights the limitations of traditional methods for assessing LVEF, emphasizing the subjectivity and variability inherent in manual measurements. The need for repeated measurements to improve accuracy is acknowledged, but the practical challenges of implementing this in clinical settings are also discussed. The introduction of AI algorithms for LVEF assessment offers a potential solution to improve precision and reduce variability. However, a gap in the literature existed regarding the validation of these AI technologies in blinded, randomized clinical trials, prompting the authors to conduct their study to fill this gap.

This study was a blinded, randomized, non-inferiority clinical trial (ClinicalTrials.gov ID: NCT01406412) involving 3,769 transthoracic echocardiogram studies from an academic medical center. After excluding 274 studies due to poor image quality, 3,495 studies were randomized (1:1) to AI or sonographer initial LVEF assessment. 25 cardiac sonographers (mean 14.1 years of practice) and 10 cardiologists (mean 12.7 years of practice) participated. Baseline characteristics were well-balanced between the two groups. Cardiologists then performed a blinded assessment of LVEF using the full echocardiogram study and initial annotations. The primary outcome was the proportion of studies showing a substantial change ( >5%) in LVEF between initial and final cardiologist assessment. Secondary outcomes included the mean absolute difference between initial and final assessments, and the difference between the final cardiologist-adjudicated LVEF and a previously reported LVEF. Subgroup analyses were performed based on various factors including demographics and image quality. Blinding of sonographers was assessed using a blinding index.

The primary outcome showed that a substantially changed LVEF occurred in 16.8% of studies in the AI group and 27.2% in the sonographer group (difference: -10.4%, 95% CI: -13.2% to -7.7%, P<0.001 for both non-inferiority and superiority). The mean absolute difference between initial and final LVEF assessments was 2.79% in the AI group and 3.77% in the sonographer group (difference: -0.97%, 95% CI: -1.33% to -0.54%, P<0.001 for superiority). The secondary safety outcome showed that a substantial difference between the final and previously reported LVEF occurred in 50.1% of studies in the AI group and 54.5% in the sonographer group (difference: -4.5%, 95% CI: -7.8% to -1.2%, P=0.008). The mean absolute difference between previous and final cardiologist assessments was 6.29% in the AI group and 7.23% in the sonographer group (difference: -0.94%, 95% CI: -1.34% to -0.54%, P<0.001 for superiority). The reduction in the primary endpoint with the AI group was consistent across major subgroups. Cardiologists spent significantly less time reviewing AI-guided assessments (median 54s vs. 64s, P<0.001). Only 22 of 1740 (1.3%) studies in the AI group crossed the 35% LVEF threshold.

This study demonstrated that AI-guided initial evaluation of LVEF is non-inferior and superior to sonographer-guided initial evaluation in a blinded, randomized trial involving board-certified cardiologists. Cardiologists were less likely to substantially change LVEF assessments when using AI-guided initial assessments. AI-guided assessment also resulted in faster review times for cardiologists and was more consistent with previously reported LVEF values. This study represents the first blinded, randomized trial evaluating AI technology for LVEF assessment, contributing significantly to the field of cardiovascular AI. The results support the potential for AI to improve efficiency and consistency in LVEF assessment in clinical practice.

This study provides strong evidence that AI-based initial assessment of LVEF is non-inferior and even superior to sonographer-based assessment. The AI system reduced the likelihood of substantial changes in LVEF during cardiologist review, demonstrating potential for improved workflow efficiency and diagnostic accuracy. Future research could explore the integration of AI into wider clinical workflows and evaluate its impact on patient outcomes.

The study was conducted at a single academic medical center, potentially limiting the generalizability of the findings. The specific AI algorithm used may not be representative of all AI technologies for LVEF assessment. Although the blinding was successful for sonographers and cardiologists, there remains a possibility for bias in assessment based on AI generated visualizations.