The Evaluation Of Responses Provided By ChatGPT 4.0 to Different Educational Levels Regarding Orthognathic Surgery

Abstract

Background: In recent years, as ChatGPT has becomes more prevalent in healthcare, understanding its potential to support patient education and professional communication is crucial. This study aims to evaluate ChatGPT's effectiveness in providing accurate and level-appropriate information on orthognathic surgery across different educational levels. The focus is on assessing the model's performance in addressing questions about postoperative care, surgical risks, and treatment procedures. Methods: In this study, ChatGPT 4.0 was employed to answer the most frequently asked questions related to orthognathic surgery. The questions were obtained from alsoasked.com and reviewed by researchers for relevance. Two distinct prompts were used to simulate responses at bachelor's and academic levels. The accuracy of the responses was evaluated using a 5-point Likert scale by an orthodontist and an oral and maxillofacial surgeon. Additionally, the appropriateness of the responses to the educational level was assessed using a 5-point Likert scale by two bachelor's degree graduates and two academics. Statistical analyses, including Cohen's Kappa, Kolmogorov-Smirnov test, and Mann-Whitney U test, were conducted to determine inter-rater agreement and data normality. Furthermore, cosine similarity was calculated to compare the similarity of responses provided for different educational levels. Results: ChatGPT 4.0 provided highly accurate responses for both educational levels, with average accuracy scores of 4.58±0.7980 for bachelor's and 4.4107±0.8745 for academic levels (p=0,123). However, the appropriateness of responses varied, with bachelor's graduates rating the responses at 4.64±0.10 and academics at 3.05±0.319. Statistical tests confirmed a significant difference between the groups (p<0.001). These findings indicate that while ChatGPT's responses are accurate, they do not always align well with the educational level of the prompts, highlighting the importance of tailoring AI-generated content to the target audience. Conclusions: ChatGPT is effective in supporting patient education in orthognathic surgery procedure but may be less suitable for providing detailed academic-level insights. It is most effective when used alongside expert guidance. © 2025, Selcuk University. All rights reserved.