scholarly journals An evaluation of performance measures for arterial brain vessel segmentation

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Orhun Utku Aydin ◽  
Abdel Aziz Taha ◽  
Adam Hilbert ◽  
Ahmed A. Khalil ◽  
Ivana Galinovic ◽  
...  
Keyword(s):  
Performance Measures ◽  
Hausdorff Distance ◽  
Ground Truth ◽  
Relevant Information ◽  
Vessel Segmentation ◽  
Performance Measure ◽  
Medical Image Segmentation ◽  
Cerebral Vessel ◽  
Angiography Cerebral ◽  
Brain Vessel

Abstract Background Arterial brain vessel segmentation allows utilising clinically relevant information contained within the cerebral vascular tree. Currently, however, no standardised performance measure is available to evaluate the quality of cerebral vessel segmentations. Thus, we developed a performance measure selection framework based on manual visual scoring of simulated segmentation variations to find the most suitable measure for cerebral vessel segmentation. Methods To simulate segmentation variations, we manually created non-overlapping segmentation errors common in magnetic resonance angiography cerebral vessel segmentation. In 10 patients, we generated a set of approximately 300 simulated segmentation variations for each ground truth image. Each segmentation was visually scored based on a predefined scoring system and segmentations were ranked based on 22 performance measures common in the literature. The correlation of visual scores with performance measure rankings was calculated using the Spearman correlation coefficient. Results The distance-based performance measures balanced average Hausdorff distance (rank = 1) and average Hausdorff distance (rank = 2) provided the segmentation rankings with the highest average correlation with manual rankings. They were followed by overlap-based measures such as Dice coefficient (rank = 7), a standard performance measure in medical image segmentation. Conclusions Average Hausdorff distance-based measures should be used as a standard performance measure in evaluating cerebral vessel segmentation quality. They can identify more relevant segmentation errors, especially in high-quality segmentations. Our findings have the potential to accelerate the validation and development of novel vessel segmentation approaches.

scholarly journals On the usage of average Hausdorff distance for segmentation performance assessment: hidden error when used for ranking

2021 ◽  
Vol 5 (1) ◽  
Author(s):  
Orhun Utku Aydin ◽  
Abdel Aziz Taha ◽  
Adam Hilbert ◽  
Ahmed A. Khalil ◽  
Ivana Galinovic ◽  
...  
Keyword(s):  
Performance Assessment ◽  
Hausdorff Distance ◽  
Rank Correlation ◽  
Ground Truth ◽  
Performance Measure ◽  
Medical Image Segmentation ◽  
Cerebral Vessel ◽  
Angiography Cerebral ◽  
Median Correlation ◽  
Kendall Rank Correlation

AbstractAverage Hausdorff distance is a widely used performance measure to calculate the distance between two point sets. In medical image segmentation, it is used to compare ground truth images with segmentations allowing their ranking. We identified, however, ranking errors of average Hausdorff distance making it less suitable for applications in segmentation performance assessment. To mitigate this error, we present a modified calculation of this performance measure that we have coined “balanced average Hausdorff distance”. To simulate segmentations for ranking, we manually created non-overlapping segmentation errors common in magnetic resonance angiography cerebral vessel segmentation as our use-case. Adding the created errors consecutively and randomly to the ground truth, we created sets of simulated segmentations with increasing number of errors. Each set of simulated segmentations was ranked using both performance measures. We calculated the Kendall rank correlation coefficient between the segmentation ranking and the number of errors in each simulated segmentation. The rankings produced by balanced average Hausdorff distance had a significantly higher median correlation (1.00) than those by average Hausdorff distance (0.89). In 200 total rankings, the former misranked 52 whilst the latter misranked 179 segmentations. Balanced average Hausdorff distance is more suitable for rankings and quality assessment of segmentations than average Hausdorff distance.

Expanding the Capabilities of Radar-Based Vehicle Detection Systems: Noise Characterization and Removal Procedures

2019 ◽  
Vol 2673 (11) ◽  
pp. 150-160
Author(s):  
Kelvin R. Santiago-Chaparro ◽  
David A. Noyce
Keyword(s):  
Performance Measures ◽  
Vehicle Detection ◽  
Ground Truth ◽  
Noise Removal ◽  
Performance Measure ◽  
Signalized Intersections ◽  
Volume Data ◽  
Trajectory Data ◽  
Potential Accuracy ◽  
Detection Systems

The capabilities of radar-based vehicle detection (RVD) systems used at signalized intersections for stop bar and advanced detection are arguably underutilized. Underutilization happens because RVD systems can monitor the position and speed (i.e., trajectory) of multiple vehicles at the same time but these trajectories are only used to emulate the behavior of legacy detection systems such as inductive loop detectors. When full vehicle trajectories tracked by an RVD system are collected, detailed traffic operations and safety performance measures can be calculated for signalized intersections. Unfortunately, trajectory datasets obtained from RVD systems often contain significant noise which makes the computation of performance measures difficult. In this paper, a description of the type of trajectory datasets that can be obtained from RVD systems is presented along with a characterization of the noise expected in these datasets. Guidance on the noise removal procedures that can be applied to these datasets is also presented. This guidance can be applied to the use of data from commercially-available RVD systems to obtain advanced performance measures. To demonstrate the potential accuracy of the noise removal procedures, the procedures were applied to trajectory data obtained from an existing intersection, and data on a basic performance measure (vehicle volume) were extracted from the dataset. Volume data derived from the de-noised trajectory dataset was compared with ground truth volume and an absolute average difference of approximately one vehicle every 5 min was found, thus highlighting the potential accuracy of the noise removal procedures introduced.

Phantom-based ground-truth generation for cerebral vessel segmentation and pulsatile deformation analysis

2016 ◽  
Author(s):  
Daniel Schetelig ◽  
Dennis Säring ◽  
Till Illies ◽  
Jan Sedlacik ◽  
Fabian Kording ◽  
...  
Keyword(s):  
Ground Truth ◽  
Vessel Segmentation ◽  
Deformation Analysis ◽  
Cerebral Vessel ◽  
Ground Truth Generation

scholarly journals Predicting Hospital Overall Quality Star Ratings in the USA

Healthcare ◽  
2021 ◽  
Vol 9 (4) ◽  
pp. 486
Author(s):  
Nisha Kurian ◽  
Jyotsna Maid ◽  
Sharoni Mitra ◽  
Lance Rhyne ◽  
Michael Korvink ◽  
...  
Keyword(s):  
Performance Measures ◽  
Relevant Information ◽  
Performance Measure ◽  
Recent Criticism ◽  
Double Counting ◽  
Ordered Logistic Regression ◽  
Regression Approach ◽  
The Usa ◽  
Highly Correlated ◽  
The U.S

The U.S. Centers for Medicare and Medicaid Services (CMS) assigns quality star ratings to hospitals upon assessing their performance across 57 measures. Ratings can be used by healthcare consumers for hospital selection and hospitals for quality improvement. We provide a simpler, more intuitive modeling approach, aligned with recent criticism by stakeholders. An ordered logistic regression approach is proposed to assess associations between performance measures and ratings across eligible (n = 4519) U.S. hospitals. Covariate selection reduces the double counting of information from highly correlated measures. Multiple imputation allows for inference of star ratings when information on all measures is not available. Twenty performance measures were found to contain all the relevant information to formulate star rating predictions upon accounting for performance measure correlation. Hospitals can focus their efforts on a subset of model-identified measures, while healthcare consumers can predict quality star ratings for hospitals ineligible under CMS criteria.

scholarly journals Real-Time Cerebral Vessel Segmentation in Laser Speckle Contrast Image Based on Unsupervised Domain Adaptation

2021 ◽  
Vol 15 ◽  
Author(s):  
Heping Chen ◽  
Yan Shi ◽  
Bin Bo ◽  
Denghui Zhao ◽  
Peng Miao ◽  
...  
Keyword(s):  
Deep Learning ◽  
Real Time ◽  
Ground Truth ◽  
Vessel Segmentation ◽  
Laser Speckle ◽  
Cerebral Vessels ◽  
Retinal Images ◽  
Spatiotemporal Resolution ◽  
Cerebral Vessel ◽  
Speckle Contrast

Laser speckle contrast imaging (LSCI) is a full-field, high spatiotemporal resolution and low-cost optical technique for measuring blood flow, which has been successfully used for neurovascular imaging. However, due to the low signal–noise ratio and the relatively small sizes, segmenting the cerebral vessels in LSCI has always been a technical challenge. Recently, deep learning has shown its advantages in vascular segmentation. Nonetheless, ground truth by manual labeling is usually required for training the network, which makes it difficult to implement in practice. In this manuscript, we proposed a deep learning-based method for real-time cerebral vessel segmentation of LSCI without ground truth labels, which could be further integrated into intraoperative blood vessel imaging system. Synthetic LSCI images were obtained with a synthesis network from LSCI images and public labeled dataset of Digital Retinal Images for Vessel Extraction, which were then used to train the segmentation network. Using matching strategies to reduce the size discrepancy between retinal images and laser speckle contrast images, we could further significantly improve image synthesis and segmentation performance. In the testing LSCI images of rodent cerebral vessels, the proposed method resulted in a dice similarity coefficient of over 75%.

scholarly journals BRAVE-NET: Fully Automated Arterial Brain Vessel Segmentation In Patients with Cerebrovascular Disease

2020 ◽  
Author(s):  
Adam Hilbert ◽  
Vince I. Madai ◽  
Ela M. Akay ◽  
Orhun U. Aydin ◽  
Jonas Behland ◽  
...  
Keyword(s):  
Cerebrovascular Disease ◽  
Hausdorff Distance ◽  
Visual Analysis ◽  
Cerebrovascular Diseases ◽  
Vessel Segmentation ◽  
Diagnostic Process ◽  
Clinical Workflow ◽  
Clinical Routine ◽  
Large Variability ◽  
Brain Vessel

AbstractIntroductionArterial brain vessel assessment is crucial for the diagnostic process in patients with cerebrovascular disease. Noninvasive neuroimaging techniques such as time-of-flight (TOF) magnetic resonance angiography (MRA) imaging are applied in the clinical routine to depict arteries. They are, however, only visually assessed. Fully automated vessel segmentation integrated into the clinical routine could facilitate the time-critical diagnosis of vessel abnormalities and might facilitate the identification of valuable biomarkers for cerebrovascular events. In the present work, we developed and validated a new deep learning model for vessel segmentation, coined BRAVE-NET, on a large aggregated dataset of patients with cerebrovascular diseases.MethodsBRAVE-NET is a multiscale 3-D convolutional neural network (CNN) model developed on a dataset of 264 patients from 3 different studies enrolling patients with cerebrovascular diseases. A context path, dually capturing high- and low-resolution volumes, and deep supervision were implemented. The BRAVE-NET model was compared to a baseline Unet model and variants with only context paths and deep supervision, respectively. The models were developed and validated using high-quality manual labels as ground truth. Next to precision and recall, the performance was assessed quantitatively by Dice coefficient (DSC); average Hausdorff distance (AVD); 95- percentile Hausdorff distance (95HD) and via visual qualitative rating.ResultsThe BRAVE-NET performance surpassed the other models for arterial brain vessel segmentation with a DSC = 0.931, AVD = 0.165 and 95HD = 29.153. The BRAVE-NET model was also the most resistant towards false labelings as revealed by the visual analysis. The performance improvement is primarily attributed to the integration of the multiscaling context path into the 3-D Unet and to a lesser extent to the deep supervision architectural component.DiscussionWe present a new state-of-the-art of arterial brain vessel segmentation tailored to cerebrovascular pathology. We provide an extensive experimental validation of the model using a large aggregated dataset encompassing a large variability of cerebrovascular disease. The framework provides the technological foundation for improving the clinical workflow and can serve as a biomarker extraction tool in cerebrovascular diseases.

Effects of Accounting-Method Choices on Subjective Performance-Measure Weighting Decisions: Experimental Evidence on Precision and Error Covariance

2005 ◽  
Vol 80 (4) ◽  
pp. 1163-1192 ◽  
Author(s):  
Ranjani Krishnan ◽  
Joan L. Luft ◽  
Michael D. Shields
Keyword(s):  
Performance Measures ◽  
Error Variance ◽  
Spillover Effects ◽  
Performance Measure ◽  
Incentive System ◽  
Experimental Setting ◽  
Error Covariance ◽  
Psychology Theory ◽  
Optimal Weighting ◽  
Similarity And Difference

Performance-measure weights for incentive compensation are often determined subjectively. Determining these weights is a cognitively difficult task, and archival research shows that observed performance-measure weights are only partially consistent with the predictions of agency theory. Ittner et al. (2003) have concluded that psychology theory can help to explain such inconsistencies. In an experimental setting based on Feltham and Xie (1994), we use psychology theories of reasoning to predict distinctive patterns of similarity and difference between optimal and actual subjective performance-measure weights. The following predictions are supported. First, in contrast to a number of prior studies, most individuals' decisions are significantly influenced by the performance measures' error variance (precision) and error covariance. Second, directional errors in the use of these measurement attributes are relatively frequent, resulting in a mean underreaction to an accounting change that alters performance measurement error. Third, individuals seem insufficiently aware that a change in the accounting for one measure has spillover effects on the optimal weighting of the other measure in a two-measure incentive system. In consequence, they make performance-measure weighting decisions that are likely to result in misallocations of agent effort.

scholarly journals Sentiment Analysis Techniques Applied to Raw-Text Data from a Csq-8 Questionnaire about Mindfulness in Times of COVID-19 to Improve Strategy Generation

2021 ◽  
Vol 18 (12) ◽  
pp. 6408
Author(s):  
Mario Jojoa Acosta ◽  
Gema Castillo-Sánchez ◽  
Begonya Garcia-Zapirain ◽  
Isabel de la Torre Díez ◽  
Manuel Franco-Martín
Keyword(s):  
Health Care ◽  
Natural Language Processing ◽  
Natural Language ◽  
Sentiment Analysis ◽  
Transfer Learning ◽  
Language Processing ◽  
Health Care Professionals ◽  
Ground Truth ◽  
Relevant Information ◽  
Free Text

The use of artificial intelligence in health care has grown quickly. In this sense, we present our work related to the application of Natural Language Processing techniques, as a tool to analyze the sentiment perception of users who answered two questions from the CSQ-8 questionnaires with raw Spanish free-text. Their responses are related to mindfulness, which is a novel technique used to control stress and anxiety caused by different factors in daily life. As such, we proposed an online course where this method was applied in order to improve the quality of life of health care professionals in COVID 19 pandemic times. We also carried out an evaluation of the satisfaction level of the participants involved, with a view to establishing strategies to improve future experiences. To automatically perform this task, we used Natural Language Processing (NLP) models such as swivel embedding, neural networks, and transfer learning, so as to classify the inputs into the following three categories: negative, neutral, and positive. Due to the limited amount of data available—86 registers for the first and 68 for the second—transfer learning techniques were required. The length of the text had no limit from the user’s standpoint, and our approach attained a maximum accuracy of 93.02% and 90.53%, respectively, based on ground truth labeled by three experts. Finally, we proposed a complementary analysis, using computer graphic text representation based on word frequency, to help researchers identify relevant information about the opinions with an objective approach to sentiment. The main conclusion drawn from this work is that the application of NLP techniques in small amounts of data using transfer learning is able to obtain enough accuracy in sentiment analysis and text classification stages.

Abstract 3429: Implementation of Electronic Medical Record Stroke Templates Increases Compliance with Stroke Performance Measures

Stroke ◽  
2012 ◽  
Vol 43 (suppl_1) ◽  
Author(s):  
Crismely A Perdomo ◽  
Vepuka E Kauari ◽  
Elizabeth Suarez ◽  
Olajide Williams ◽  
Joshua Stillman ◽  
...  
Keyword(s):  
Medical Record ◽  
Electronic Medical Record ◽  
Performance Measures ◽  
New York State ◽  
Stroke Care ◽  
Performance Measure ◽  
Performance Comparison ◽  
York State Department ◽  
Free Text ◽  
And Performance

Background and Purpose The literature demonstrates how utilizing evidence-based, standardized stroke care can improve patient outcomes; however, the contribution of electronic medical record (EMR) systems may also impact outcomes by ensuring utilization and compliance with established stroke performance measures, facilitating and improving documentation requirements, and standardizing approach to care. In 2008, documentation in patients’ medical records was done in combination of paper and a template free EMR. Originally, the EMR was used for order entry, then transitioned to full electronic documentation in 2009. At that time we implemented our stroke templates and performance measures based on regulatory standards. We hypothesized that the stroke template implementation would help us achieve performance measure criteria above state benchmarks as set out by the New York State Department of Health (NYS DOH). Methods Implementation was phased in [over 18 months], initially using a template that only included neurological assessment and free text fields for stroke measures. By July 2010, existing templates were modified and additional stroke templates were implemented to meet new regulatory requirements and meaningful use criteria. Retrospective data review was conducted for performance comparison between 2008 -- one year prior to EMR/template implementation -- and 2010. In Quarter 1 of 2011 EMR was also implemented in the Emergency Department (ED). Data was reviewed for compliance with stroke measures. Results Documentation compliance substantially improved between 2008 and Quarter 1 2011: Compliance for these measures has been maintained ≥ 85% since November 2010, ≥ 90% Q1 2011 Conclusions The EMR implementation of stroke templates and performance measures can produce substantial improvement in performance measure compliance. Future steps will include automated documentation alerts to retrieve information and real time discovery of missing documentation for concurrent quality review and improvement

A Tutorial on the Use of ROC Analysis for Computer-Aided Diagnostic Systems

2005 ◽  
Vol 27 (3) ◽  
pp. 181-198 ◽  
Author(s):  
Ulrich Scheipers ◽  
Christian Perrey ◽  
Stefan Siebers ◽  
Christian Hansen ◽  
Helmut Ermert
Keyword(s):  
Performance Measures ◽  
Roc Curve ◽  
Roc Analysis ◽  
Real Life ◽  
Performance Measure ◽  
Diagnostic Systems ◽  
Life Data ◽  
Separation Threshold ◽  
Real Life Data ◽  
Computer Aided

The application of the receiver operating characteristic (ROC) curve for computer-aided diagnostic systems is reviewed. A statistical framework is presented and different methods of evaluating the classification performance of computer-aided diagnostic systems, and, in particular, systems for ultrasonic tissue characterization, are derived. Most classifiers that are used today are dependent on a separation threshold, which can be chosen freely in many cases. The separation threshold separates the range of output values of the classification system into different target groups, thus conducting the actual classification process. In the first part of this paper, threshold specific performance measures, e.g., sensitivity and specificity; are presented. In the second part, a threshold-independent performance measure, the area under the ROC curve, is reviewed. Only the use of separation threshold-independent performance measures provides classification results that are overall representative for computer-aided diagnostic systems. The following text was motivated by the lack of a complete and definite discussion of the underlying subject in available textbooks, references and publications. Most manuscripts published so far address the theme of performance evaluation using ROC analysis in a manner too general to be practical for everyday use in the development of computer-aided diagnostic systems. Nowadays, the user of computer-aided diagnostic systems typically handles huge amounts of numerical data, not always distributed normally. Many assumptions made in more or less theoretical works on ROC analysis are no longer valid for real-life data. The paper aims at closing the gap between theoretical works and real-life data. The review provides the interested scientist with information needed to conduct ROC analysis and to integrate algorithms performing ROC analysis into classification systems while understanding the basic principles of classification.

Sign in / Sign up

Export Citation Format

Share Document