Comparative analysis of AI chatbot (ChatGPT-4.0 and Microsoft Copilot) and expert responses to common orthodontic questions: patient and orthodontist evaluations

Abstract

Objective The aim of this study was to evaluate the adequacy of responses provided by experts and artificial intelligence-based chatbots (ChatGPT-4.0 and Microsoft Copilot) to frequently asked orthodontic questions, utilizing scores assigned by patients and orthodontists. Methods Fifteen questions were randomly selected from the FAQ section of the American Association of Orthodontists (AAO) website, addressing common concerns related to orthodontic treatments, patient care, and post-treatment guidelines. Expert responses, along with those from ChatGPT-4.0 and Microsoft Copilot, were presented in a survey format via Google Forms. Fifty-two orthodontists and 102 patients rated the three responses for each question on a scale from 1 (least adequate) to 10 (most adequate). The findings were analyzed comparatively within and between groups. Results Expert responses consistently received the highest scores from both patients and orthodontists, particularly in critical areas such as Questions 1, 2, 4, 9, and 11, where they significantly outperformed chatbots (P < 0.05). Patients generally rated expert responses higher than those of chatbots, underscoring the reliability of clinical expertise. However, ChatGPT-4.0 showed competitive performance in some questions, achieving its highest score in Question 14 (8.16 +/- 1.24), but scored significantly lower than experts in several key areas (P < 0.05). Microsoft Copilot generally received the lowest scores, although it demonstrated statistically comparable performance to other groups in certain questions, such as Questions 3 and 12 (P > 0.05). Conclusions Overall, the scores for ChatGPT-4.0 and Microsoft Copilot were deemed acceptable (6.0 and above). However, both patients and orthodontists generally rated the expert responses as more adequate. This suggests that current current chatbots does not yet match the theoretical adequacy of expert opinions.