ChatGPT as a Clinical Decision Maker for Urolithiasis: Compliance with the Current European Association of Urology Guidelines

Ali Talyshinskii, Patrick Juliebø‐Jones, Belthangady M. Zeeshan Hameed, Nithesh Naik, Kinju Adhikari, Ulanbek Zhanbyrbekuly, Lazaros I. Tzelves, Bhaskar Kumar Somani

European Urology Open Science, 2024

Full text Open access: Gold Metadata (CSL-JSON)
DOI 10.1016/j.euros.2024.08.015
Data confidence 85 high
How this score is calculated

It is built from checkable properties of the record itself: whether the article has a DOI, how many independent sources confirmed its data, whether the required fields are filled in, and how recent the latest confirmation is. Disagreement between sources and a missing identifier lower it. A value of 80 or above means the record was published without further checking; 60 to 79 means it is published at reduced confidence, with a manual check advisable. A record verified by an editor by hand is never pushed below 90 by a recalculation. This scores our data about the article, not the article itself.

Abstract

Background and objective: Generative artificial intelligence models are among the most promising and widely used tools used in health care. This review investigates GPT-4 answers to decision-making questions regarding the diagnosis and treatment of urolithiasis across several clinical settings and their correspondence to the current European Association of Urology (EAU) guidelines. Methods: In March 2024, the GPT-4 model was asked 11 questions, containing a brief description of a patient with urolithiasis. All questions were grouped according to urolithiasis care step: diagnosis, urgent care, scheduled intervention, and metaphylaxis. When responses were received, compliance with the current EAU guidelines was assessed by experienced urologists. Key findings and limitations: Although all responses were provided with information that corresponded to EAU guidelines, six of the 11 answers were associated with missed guideline-provided parts, and incorrect data were given in eight of the 11 answers. GPT-4 is relatively safe in the initial diagnostic flow of patients suspected of having stones within the urinary tract and during treatment planning; however, its understanding of all the nuances of metaphylaxis leaves much to be desired and is far from the dogma given in the EAU guidelines. Moreover, GPT-4 knowledge of strategy and algorithm is not always aligned with the EAU guidelines. Conclusions and clinical implications: Despite the fact that from the perspective of patients with urolithiasis, GPT-4 is capable of answering their questions well, the specificity of questions from urologists is labor intensive for its current version, and necessitates the ability to interpret it correctly and further attempts to improve it. While some aspects of diagnostics are more accurate, these struggle with surgical planning and algorithms in line with the EAU guidelines. Patient summary: The generative artificial intelligence (AI) model GPT-4 is capable of answering urology-related questions, but lacks detailed responses. Although some aspects of the diagnostics are accurate, knowledge of surgical planning is not in line with the European Association of Urology guidelines. Future improvements should focus on efforts to enhance the accuracy, reliability, and clinical relevance of AI tools in urology.