부산대학교 도서관

Introduction: ChatGPT is an artificial intelligence (AI) chatbot released in November 2022. It is a large scale language model that has gained widespread popularity for its fine-tuned conversational abilities, attracting over 1.8 billion monthly visitors. A noted drawback to the AI chatbot is its tendency to confidently present users with inaccurate information. Goals: To evaluate the quality of ChatGPT responses to questions pertaining to atrial fibrillation for patient education. This includes the accuracy of answers, estimated grade level, and references of answers.Methods: ChatGPT was prompted four times then was asked 16 questions derived from the American Heart Associationʼs frequently asked questions on atrial fibrillation. Prompts included: no prompt (Form 1), patient-friendly prompt (Form 2), physician-level prompt (Form 3), and prompting for statistics/references (Form 4). Responses were scored as incorrect, partially correct, correct, or correct with references (perfect). Flesch-Kincaid grade level and response lengths were recorded. Proportions of responses at differing scores were compared using chi-squared analysis. The relationship between form and grade level was assessed using ANOVA.Results: Across all forms scoring frequencies were: 1(1.6%) incorrect, 5 (7.8%) partially correct, 55 (85.9%) correct, and 3 (4.7%) perfect. Proportions of responses that were at least correct did not differ by form (p=0.350); responses that were perfect did (p<0.001). Form 2 answers had a lower mean grade level (12.81 ± 3.38) than Forms 1 (14.23 ± 2.34), 3 (16.73 ± 2.65), and 4 (14.85 ± 2.76) (p<0.01).Conclusions: ChatGPT provides accurate and comprehensive answers to most questions about atrial fibrillation regardless of prompting. Interestingly, when prompted as a patient, ChatGPT will provide lower grade level responses. Given ChatGPTs rapid popularity and usage, cardiologists may seek to further investigate its accuracy and utility for patients.