Discrete Protein Metric (DPM): A new image similarity metric to calculate accuracy of deep learning-generated cell focal adhesion predictions

Understanding cell behaviors can provide new knowledge on the development of different pathologies. Focal adhesion (FA) sites are important sub-cellular structures that are involved in these processes. To better facilitate the study of FA sites, deep learning (DL) can be used to predict FA site morphology based on limited datasets (e.g., cell membrane images). However, calculating the accuracy score of these predictions can be challenging due to the discrete/point pattern like nature of FA sites. In the present work, a new image similarity metric, discrete protein metric (DPM), was developed to calculate FA prediction accuracy. This metric measures differences in distribution (d), shape/size (s), and angle (a) of FA sites between the predicted image and its ground truth image. Performance of the DPM was evaluated by comparing it to three other commonly used image similarity metrics: Pearson correlation coefficient (PCC), feature similarity index (FSIM), and Intersection over Union (IoU). A sensitivity analysis was performed by comparing changes in each metric value due to quantifiable changes in FA site location, number, aspect ratio, area, or orientation. Furthermore, accuracy score of DL-generated predictions was calculated using all four metrics to compare their ability to capture variation across samples. Results showed better sensitivity and range of variation for DPM compared to the other metrics tested. Most importantly, DPM had the ability to determine which FA predictions were quantitatively more accurate and consistent with qualitative assessments. The proposed DPM hence provides a method to validate DL-generated FA predictions and can be extended to evaluating other predicted or segmented discrete structures of biomedical relevance.

On Comparing Cross-Validated Forecasting Models with a Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study

Time series cross-validation is a technique to select forecasting models. Despite the sophistication of cross-validation over single test/training splits, traditional and independent metrics, such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), are commonly used to assess the model’s accuracy. However, what if decision-makers have different models fitting expectations to each moment of a time series? What if the precision of the forecasted values is also important? This is the case of predicting COVID-19 in Amapá, a Brazilian state in the Amazon rainforest. Due to the lack of hospital capacities, a model that promptly and precisely responds to notable ups and downs in the number of cases may be more desired than average models that only have good performances in more frequent and calm circumstances. In line with this, this paper proposes a hybridization of the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and fuzzy sets to create a similarity metric, the closeness coefficient (CC), that enables relative comparisons of forecasting models under heterogeneous fitting expectations and also considers volatility in the predictions. We present a case study using three parametric and three machine learning models commonly used to forecast COVID-19 numbers. The results indicate that the introduced fuzzy similarity metric is a more informative performance assessment metric, especially when using time series cross-validation.