Accuracy and validity of common dental terms using chat GPT model:  a cross-sectional study

Keerthana Sree V.; Kumara Raja Balasubramanian; Madan Kumar Parangimalai Diwakar; Menaka Sathish Vasudevan

doi:10.18203/2394-6040.ijcmph20240624

Authors

Keerthana Sree V. Department of Public Health Dentistry, Ragas Dental College and Hospital, Uthandi, Chennai, Tamil Nadu, India
Kumara Raja Balasubramanian Department of Public Health Dentistry, Ragas Dental College and Hospital, Uthandi, Chennai, Tamil Nadu, India
Madan Kumar Parangimalai Diwakar Department of Public Health Dentistry, Ragas Dental College and Hospital, Uthandi, Chennai, Tamil Nadu, India
Menaka Sathish Vasudevan Department of Periodontics, Ragas Dental College and Hospital, Uthandi, Chennai, Tamil Nadu, India

DOI:

https://doi.org/10.18203/2394-6040.ijcmph20240624

Keywords:

Accuracy, Artificial intelligence chatbot, Chat generative pre-trained transformer, Chat GPT

Abstract

Background: Chat generative pre-trained transformer, an artificial intelligence chatbot can generate text-based content for information purpose. This study aims to find the accuracy and reliability of the chat GPT generated definitions for 30 common dental terms.

Methods: A 15 current dental teaching staffs grading from Professors and Readers of various specialities participated in this study. They graded the chat GPT generated terms on a 5-point Likert scale (1- Strongly disagree, 2- Disagree, 3- Neutral, 4- Agree, 5- Strongly disagree). Scores were obtained and descriptive statistics was done and compared using Mann-Whitney U test.

Results: Among 30 dental terms, 13 terms which were generated from the chat GPT model were found to be more appropriate when compared to text book definition. On comparison of reviewers’ perceptions for accuracy of definitions generated from chat GPT compared with text book definitions in which among the 30 dental terms, 9 terms were found to be statistically significant (p<0.05*).

Conclusions: Chat GPT is a potential tool for answering knowledge based questions with equal vigor in the field of dentistry. Moreover, the accuracy of Chat GPT to solve questions in dentistry has a relational level of accuracy.

Metrics

Metrics Loading ...

References

Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Research Square. 2023;28:2023.

Gao CA, Howard FM, Markov NS, Dyer EC, Ramesh S, Luo Y, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv. 2022:2022-12.

Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, Moy L. ChatGPT and other large language models are double-edged swords. Radiol. 2023;307(2):e230163.

Stokel-Walker C. Chat GPT listed as author on research papers: many scientists disapprove. Nature. 2023;613(7945):620-1.

Iftikhar L. Docgpt: Impact of chatgpt-3 on health services as a virtual doctor. EC Paediatrics. 2023;12(1):45-55.

Khanagar SB, Al-ehaideb A, Maganur PC, Vishwanathaiah S, Patil S, Baeshen HA, et al. Developments, application, and performance of artificial intelligence in dentistry-a systematic review. J Dent Sci. 2021;16(1):508-22.

Khanagar SB, Al-Ehaideb A, Vishwanathaiah S, Maganur PC, Patil S, Naik S, et al. Scope and performance of artificial intelligence technology in orthodontic diagnosis, treatment planning, and clinical decision-making-a systematic review. J Dent Sci. 2021;16(1):482-92.

Mahmood H, Shaban M, Indave BI, Santos-Silva AR, Rajpoot N, Khurram SA. Use of artificial intelligence in diagnosis of head and neck precancerous and cancerous lesions: a systematic review. Oral Oncol. 2020;110:104885.

Kalla D, Smith N. Study and analysis of Chat GPT and its impact on different fields of study. Inter J Innovat Sci Res Technol. 2023;8(3).

Hosseini M, Rasmussen LM, Resnik DB. Using AI to write scholarly publications. Accountability Res. 2023:1-9.

Thorp HH. ChatGPT is fun, but not an author. Sci. 2023;379(6630):313.

Sinha RK, Roy AD, Kumar N, Mondal H, Sinha R. Applicability of ChatGPT in assisting to solve higher order problems in pathology. Cureus. 2023;15(2).

Das D, Kumar N, Longjam LA, Sinha R, Roy AD, Mondal H, et al. Assessing the capability of ChatGPT in answering first-and second-order knowledge questions on microbiology as per competency-based medical education curriculum. Cureus. 2023;15(3).

Lemons PP, Lemons JD. Questions for assessing higher-order cognitive skills: It's not just Bloom’s. CBE-Life Sci Educat. 2013;12(1):47-58.

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educat. 2023;9(1):e45312.

Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digital Health. 2023;2(2):e0000198.

Huh S. Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study. J Educ Eval Health Prof. 2023;20(1).

Juhi A, Pipil N, Santra S, Mondal S, Behera JK, Mondal H, et al. The capability of ChatGPT in predicting and explaining common drug-drug interactions. Cureus. 2023;15(3).

Patterson C, Crooks D, Lunyk-Child O. A new perspective on competencies for self-directed learning. J Nurs Educat. 2002;41(1):25-31.

Bhandari B, Chopra D, Singh K. Self-directed learning: assessment of students' abilities and their perspective. Adv Physiol Educ. 2020;44(3):383-6.

Goisauf M, Cano Abadía M. Ethics of AI in radiology: a review of ethical and societal implications. Frontiers in Big Data. 2022;5:850383.

Karn A, Priyadarshi A, Roy AD. A review on digitalization of healthcare with SWOC analysis of digital pathology in the backdrop of COVID-19. Global J Res Analysis. 2022;11(7):1-2.