Evaluating Diagnostic Accuracy of GPT-4o and Claude 4.6 Sonnet AI Models Using Turkish Emergency Department Patient Notes

What this Trial Is About

Researchers are evaluating the diagnostic accuracy of two large language models LLMs, GPT-4o and Claude 4.6 Sonnet, in interpreting anonymized Turkish-language emergency department ED anamnesis notes. This retrospective observational study compares these AI models diagnoses with those made by the treating emergency physicians and a consensus gold standard established by three independent emergency medicine specialists. The study aims to benchmark LLM performance in an emergency medicine setting using ICD-10 codes. The study uses a single-center, cohort design analyzing existing anonymized electronic health records of adult ED patients. Both LLMs are accessed via API in stateless sessions without additional training or fine-tuning, using a zero-shot prompting approach. The gold standard diagnosis is determined by majority vote among three board-certified specialists who independently review each note. All data is de-identified and stored securely in compliance with Turkish data protection laws. Participants are adults aged 18 and older who presented to the ED with detailed anamnesis notes and definitive ICD-10 diagnoses. Researchers assess diagnostic accuracy at the ICD-10 chapter level, inter-model agreement, and compare AI and physician performance. Statistical analyses include Cohens kappa and Fleiss kappa for agreement measures. Data privacy is strictly maintained, and no patient involvement occurs due to the retrospective nature. The study period extends from June 2026 to October 2026.

Diagnostic Accuracy of GPT-4o and Claude 4.6 Sonnet in Turkish ED Anamnesis Notes