Multicenter Retrospective Study Evaluating AI Models ChatGPT, Gemini, and DeepSeek for Diagnosis and Treatment of Urologic Diseases Using Inpatient Medical Records

What this Trial Is About

Researchers are evaluating the diagnostic and therapeutic abilities of three large language modelsChatGPT, Gemini, and Deepseekusing 800 archived inpatient medical records from urology departments across four tertiary hospitals. The study aims to assess how accurately these models recognize diseases, make preliminary diagnoses, and recommend treatments to explore their usefulness and limitations in supporting real-world clinical decisions. The study uses de-identified medical records that include standardized clinical details such as demographics, history of present illness, past medical history, physical exams, lab and imaging results, discharge diagnoses, and treatment plans. The diagnostic conclusions and treatment plans were removed before inputting the data to simulate a first-visit physician scenario. Each model then generated primary and secondary diagnoses, differential diagnosis lists with reasoning, and preliminary treatment suggestions. Participants are not directly involved as this is a retrospective observational study using archived records. Researchers will assess outcome measures like diagnostic accuracy Top-1 and Top-3, diagnostic completeness, differential diagnosis quality, treatment plan quality, and analysis time over an average of three months through study completion. The study is sponsored by the First Affiliated Hospital of Fujian Medical University and started in January 2026 with completion expected by June 2026.

Evaluation of AI Large Models for Diagnosis and Treatment in Real-World Cases: Multicenter Retrospective Study