Machine learning (ML) is transforming medical research and practice, particularly in diagnosis and outcome prediction. Its applications span image analysis, public health, clinical trials, and operational organization. Factors like increased data availability, computational power, and research activity are driving wider ML adoption in medicine. However, the complexity of developing, implementing, and validating ML models makes them inaccessible to most clinicians and researchers, limiting their use to those with expertise in both medicine and data science. Automated machine learning (AutoML) aims to address this by making ML accessible to non-technical experts. Existing AutoML platforms automate algorithm training and fine-tuning, but often require specialized knowledge. Large language models (LLMs), such as GPT-4, offer a more accessible AutoML option. LLMs can reason, perform logical deduction, and generate code, making them potentially valuable for analyzing data and creating ML models. This study aimed to evaluate the validity and reliability of ChatGPT ADA in autonomously developing and implementing ML models for clinical data analysis.

The paper reviews existing AutoML platforms like MATLAB's Classification Learner, Vertex AI, and Azure, highlighting their ability to enable non-technical experts to create ML models. However, these platforms typically require specific instructions or code. The paper then discusses the emergence of powerful LLMs like GPT-4 and their potential to revolutionize AutoML through natural language interaction. The ability of LLMs to analyze data, write and execute code, and provide feedback is emphasized. While the ChatGPT Code interpreter has shown promise in data analysis, its validity and reliability in advanced healthcare data processing hadn't been thoroughly evaluated before this study.

The study used real-world datasets from four large clinical trials across various medical specialties. These datasets included information on metastatic disease in endocrinologic oncology, esophageal cancer, hereditary hearing loss, and cardiac amyloidosis. The researchers provided ChatGPT ADA with the datasets and study details without specific guidance on data preprocessing or ML methodology. ChatGPT ADA autonomously selected and implemented appropriate ML models, generating predictions. A seasoned data scientist then re-implemented and optimized the best-performing models from the original studies for comparison. The performance of ChatGPT ADA-generated models was compared to the original study results (benchmark publication) and the re-implemented models (benchmark validation re-implementation) using metrics such as AUROC, accuracy, F1-score, sensitivity, and specificity. A Shapley Additive exPlanations (SHAP) analysis was also conducted to assess model explainability. The study also investigated the consistency of ChatGPT ADA's behavior by prompting it multiple times with the same datasets and instructions. Data pre-processing steps such as imputation of missing values and handling of categorical variables are detailed for each clinical trial dataset. Specific ML model types (e.g., AdaBoost, Gradient Boosting Machine, LightGBM, Random Forest, Support Vector Machine) and their parameter settings are described for both ChatGPT ADA's choices and the re-implemented models.

Across the four clinical datasets, ChatGPT ADA autonomously formulated and executed advanced ML techniques for disease screening and prediction. Its performance consistently matched or exceeded the benchmark and custom ML methods re-implemented based on the original studies. In the metastatic disease prediction task, ChatGPT ADA achieved a slightly improved AUROC (0.949 vs. 0.942), accuracy (0.922 vs. 0.907), and F1-score (0.806 vs. 0.755) compared to the best-performing published model. Similarly, in esophageal cancer prediction, ChatGPT ADA's model achieved a higher AUROC (0.981 vs. 0.960). For hereditary hearing loss and cardiac amyloidosis, ChatGPT ADA's performance was comparable to the original models. Head-to-head comparisons between ChatGPT ADA-generated models and re-implemented models revealed no significant differences in performance metrics (p ≥ 0.072). The SHAP analysis successfully identified and quantified the importance of various features contributing to the model's predictions, enhancing transparency and trust in the model's outputs. The study also demonstrated that ChatGPT ADA consistently selected the same ML model and parameters when provided with identical inputs, showcasing the tool's reliability.

The study demonstrates the potential of advanced LLMs like ChatGPT ADA to simplify complex ML methods, making them accessible to clinicians and researchers with varying levels of ML expertise. This has implications for accelerating medical research, validating or refuting prior studies, and ultimately improving patient care. The tool's ability to automate data processing and model selection could significantly reduce the time and resources required for clinical data analysis. However, the study acknowledges limitations such as the tool's black-box nature, potential for commercial bias, and the risk of perpetuating algorithmic bias if the training data is not representative. The importance of external validation, especially in the absence of benchmark publications, is stressed. The ease of use and natural language interaction offered by ChatGPT ADA is a significant advantage, although careful consideration of data privacy and security is crucial.

Advanced LLMs like ChatGPT ADA represent a significant advancement in data-driven medicine, simplifying complex ML methods and democratizing access to these powerful tools. This study shows its potential to streamline data analysis for researchers, reducing the burden of pre-processing and model optimization. While limitations exist, and enhancements to specialized training are still needed, these tools are poised to bridge the gap between complex ML methods and their practical application in medical research and practice.

The study acknowledges several limitations. First, the effectiveness of ChatGPT ADA with less curated real-world clinical data containing quality issues (missing and irregular values) remains to be assessed. Second, the study's reliance on existing publications might introduce bias, as different random forest classifiers were implemented in the original studies and the ChatGPT ADA models. Third, the model's performance might be influenced by different prompting strategies. Finally, external validation was absent in one of the datasets (Cardiac Amyloidosis), limiting the generalizability of the findings.