Abstract
To assess the clinical accuracy of nine artificial intelligence (AI) devices in the detection of pneumothorax (PTX) on a unified data set of chest X-rays (CXR), in addition to evaluating the impact of different demographic, clinical and imaging factors on AI performance.We conducted a retrospective evaluation of nine CE-marked AI tools capable of PTX detection using a curated data set of 394 CXRs obtained from an NHS hospital.Ground truth for PTX presence/absence on each CXR was established by two radiologists, with disagreement arbitrated by a third.Sensitivity, specificity and accuracy were calculated for each vendor.Receiver operating characteristic (ROC) curve was calculated where vendors were able to provide variable threshold scores.A data set of 394 CXRs (47.2% positive for PTX, 39.6% female, mean age 56 years) was inferenced by nine AI devices.Accuracy values ranged from 76.1% (95% confidence interval 71.6-80.3) to 93.9% (91.1-96.1).All specificity values were greater than 79.5%, with the highest at 97.1% (93.8-98.9).Overall, sensitivity was lower across the group and had a higher spread of results ranging from 54.3% (46.9-61.6) to 92.5% (87.7-95.8).Significant variation in performance is demonstrated for PTX between different commercially available AI devices.Future in-depth comparative analyses are critical for healthcare providers and developers alike to guide the development, adoption and deployment of CE-approved devices.