Benchmark Development for AI Interpretation of Breast Ultrasound Images Evaluating Multimodal AI Model Performance Using ACR BI-RADS v2025 Criteria

What this Trial Is About

Researchers are conducting a single-center, retrospective observational study to develop a standardized benchmark system for evaluating intelligent breast ultrasound image interpretation. The study focuses on assessing the diagnostic accuracy of current mainstream multimodal artificial intelligence AI models in classifying breast ultrasound images according to the American College of Radiology ACR BI-RADS v2025 criteria. This research aims to address variability in ultrasound interpretation, especially for certain lesion categories, and to systematically evaluate AI performance using expert-annotated images. The study uses approximately 1,380 de-identified B-mode breast ultrasound images collected from an institutional archive and open-access datasets, covering normal breast tissue, benign lesions, and malignant lesions. Expert radiologists with varying experience levels will annotate all images independently. Baseline deep learning models ResNet-50 and USFM will establish performance baselines, and multiple multimodal large language models MLLMs will be evaluated using standardized chain-of-thought prompts through API calls. Safety assessments include out-of-distribution rejection testing and temperature-stability experiments. Participants are not directly involved as the study retrospectively analyzes existing images. Researchers will evaluate diagnostic accuracy, BI-RADS classification accuracy, agreement with expert consensus, and other performance metrics at study completion, approximately 12 months after starting. The study also monitors model robustness and safety through specific tests. The total study duration extends from March 2026 to March 2027.

Construction of a Benchmark for Breast Ultrasound AI Interpretation and Performance Evaluation of Multimodal AI Models