Evaluation Metrics and Evaluation

BACKGROUND Skin cancer is the most common cancer type affecting humans. Traditional skin cancer diagnosis methods are costly, require a professional physician, and take time. Hence, to aid in diagnosing skin cancer, Artificial Intelligence (AI) tools are being used, including shallow and deep machine learning-based techniques that are trained to detect and classify skin cancer using computer algorithms and deep neural networks. OBJECTIVE The aim of this study is to identify and group the different types of AI-based technologies used to detect and classify skin cancer. The study also examines the reliability of the selected papers by studying the correlation between the dataset size and number of diagnostic classes with the performance metrics used to evaluate the models. METHODS We conducted a systematic search for articles using IEEE Xplore, ACM DL, and Ovid MEDLINE databases following the PRISMA Extension for Scoping Reviews (PRISMA-ScR) guidelines. The study included in this scoping review had to fulfill several selection criteria; to be specifically about skin cancer, detecting or classifying skin cancer, and using AI technologies. Study selection and data extraction were conducted by two reviewers independently. Extracted data were synthesized narratively, where studies were grouped based on the diagnostic AI techniques and their evaluation metrics. RESULTS We retrieved 906 papers from the 3 databases, but 53 studies were eligible for this review. While shallow techniques were used in 14 studies, deep techniques were utilized in 39 studies. The studies used accuracy (n=43/53), the area under receiver operating characteristic curve (n=5/53), sensitivity (n=3/53), and F1-score (n=2/53) to assess the proposed models. Studies that use smaller datasets and fewer diagnostic classes tend to have higher reported accuracy scores. CONCLUSIONS The adaptation of AI in the medical field facilitates the diagnosis process of skin cancer. However, the reliability of most AI tools is questionable since small datasets or low numbers of diagnostic classes are used. In addition, a direct comparison between methods is hindered by a varied use of different evaluation metrics and image types.

Download Full-text

An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning Applied to Gastrointestinal Tract Abnormality Classification

ACM Transactions on Computing for Healthcare ◽

10.1145/3386295 ◽

2020 ◽

Vol 1 (3) ◽

pp. 1-29 ◽

Cited By ~ 3

Author(s):

Vajira Thambawita ◽

Debesh Jha ◽

Hugo Lewi Hammer ◽

Håvard D. Johansen ◽

Dag Johansen ◽

...

Keyword(s):

Machine Learning ◽

Gastrointestinal Tract ◽

Extensive Study ◽

Evaluation Metrics ◽

Tract Abnormality

Download Full-text

Query/Task Satisfaction and Grid-based Evaluation Metrics Under Different Image Search Intents

Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval ◽

10.1145/3397271.3401295 ◽

2020 ◽

Author(s):

Kosetsu Tsukuda ◽

Masataka Goto

Keyword(s):

Evaluation Metrics ◽

Image Search ◽

Task Satisfaction ◽

Grid Based

Download Full-text

An empirical evaluation of cost-based federated SPARQL query processing engines

Semantic Web ◽

10.3233/sw-200420 ◽

2021 ◽

pp. 1-26

Author(s):

Umair Qudus ◽

Muhammad Saleem ◽

Axel-Cyrille Ngonga Ngomo ◽

Young-Koo Lee

Keyword(s):

Query Processing ◽

Detailed Analysis ◽

Performance Metrics ◽

Empirical Evaluation ◽

Sparql Query ◽

Evaluation Metrics ◽

Future Cost ◽

Query Plan ◽

Fine Grained ◽

Runtime Performance

Finding a good query plan is key to the optimization of query runtime. This holds in particular for cost-based federation engines, which make use of cardinality estimations to achieve this goal. A number of studies compare SPARQL federation engines across different performance metrics, including query runtime, result set completeness and correctness, number of sources selected and number of requests sent. Albeit informative, these metrics are generic and unable to quantify and evaluate the accuracy of the cardinality estimators of cost-based federation engines. To thoroughly evaluate cost-based federation engines, the effect of estimated cardinality errors on the overall query runtime performance must be measured. In this paper, we address this challenge by presenting novel evaluation metrics targeted at a fine-grained benchmarking of cost-based federated SPARQL query engines. We evaluate five cost-based federated SPARQL query engines using existing as well as novel evaluation metrics by using LargeRDFBench queries. Our results provide a detailed analysis of the experimental outcomes that reveal novel insights, useful for the development of future cost-based federated SPARQL query processing engines.

Download Full-text