Assessment of the change in accuracy of an artificial intelligence algorithm for the detection of skin cancer in camera images following diversification and training
Abstract Background The US FDA recently stated in its Proposed Regulatory Framework for software as a medical device (SaMD) that “One of the greatest benefits of AI/ML in software resides in its ability to learn from real-world use and experience, and its capability to improve its performance.” This study follows two previous publications which addressed the accuracy of a machine learning algorithm for the detection of malignant melanoma. The aim of this study was to quantify the change in the accuracy following modifications to the algorithm (DERM) for the detection of non-melanoma skin cancers and potential precursors of skin cancer. A secondary aim was to assess any improvement in accuracy associated with continued training of the algorithm.Methods A total of 16,550 images of skin lesions with histopathology based assessment were available for assessment. The primary indicator of diagnostic accuracy was the area under the ROC curve with 95% confidence intervals. Sensitivity and specificity at the most efficient cut-point was also estimated together with the numbers of false negative and false positive results.Results The inclusion of squamous cell cancer, basal cell cancer and intra-epidermal carcinoma in addition to melanoma results in an improvement in the scope of the algorithm. For the most recent version of the algorithm all skin cancers show an area under the ROC curve greater than 95%. For melanoma sensitivity=91% and specificity = 89%; for all non-melanoma skin cancers sensitivity=97% and specificity=94%. Continued training of the algorithm results in a statistically significant (p<0.01) improvement in accuracy which diminishes as the ROC area approaches 100%. Conclusions The results indicate that as the algorithm is used in clinical practice it will become more accurate with continued training but the rate of improvement will diminish as the ROC area approaches 100%. A smartphone or other camera fitted with a dermoscopic lens and with internet access to the algorithm can provide an accurate additional assessment of a suspected skin cancer lesion or precursor for primary care physicians and dermatologists.