Agreement and Reliability Analysis of Machine Learning Scaling in the Assessment of Acute Proximal Weakness by Experts and Non-Experts: Clinical Feasibility Study (Preprint)

2021 ◽  
Author(s):  
Eunjeong Park ◽  
Kijeong Lee ◽  
Taehwa Han ◽  
Hyo Suk Nam
Keyword(s):  
Machine Learning ◽  
Reliability Analysis ◽  
Gold Standard ◽  
Assessment Tool ◽  
Expert Knowledge ◽  
Hybrid Approach ◽  
Assessment Model ◽  
Lower Limbs ◽  
Neurological Deficits ◽  
Percent Agreement

BACKGROUND Assessing the symptoms of proximal weakness caused by neurological deficits requires expert knowledge and experienced neurologists. Recent advances in artificial intelligence and the Internet of Things have resulted in the development of automated systems that emulate physicians’ assessments. OBJECTIVE This study provides an agreement and reliability analysis of using an automated scoring system to evaluate proximal weakness by experts and non-experts. METHODS We collected 144 observations from acute stroke patients in a neurological intensive care unit to measure the symptom of proximal weakness of upper and lower limbs. A neurologist performed a gold standard assessment and two medical students performed identical tests as non-expert assessments for manual and machine learning-based scaling of Medical Research Council (MRC) proximal scores. The system collects signals from sensors attached on patients’ limbs and trains a machine learning assessment model using the hybrid approach of data-level and algorithm-level methods for the ordinal and imbalanced classification in multiple classes. For the agreement analysis, we investigated the percent agreement of MRC proximal scores and Bland-Altman plots of kinematic features between the expert- and non-expert scaling. In the reliability analysis, we analysed the intra-class correlation coefficients (ICCs) of kinematic features and Krippendorff’s alpha of the three observers’ scaling. RESULTS The mean percent agreement between the gold standard and the non-expert scaling was 0.542 for manual scaling and 0.708 for IoT-assisted machine learning scaling, with 30.63% enhancement. The ICCs of kinematic features measured using sensors ranged from 0.742 to 0.850, whereas the Krippendorff’s alpha of manual scaling for the three observers was 0.275. The Krippendorff’s alpha of machine learning scaling increased to 0.445, with 61.82% improvement. CONCLUSIONS Automated scaling using sensors and machine learning provided higher inter-rater agreement and reliability in assessing acute proximal weakness. The enhanced assessment supported by the proposed system can be utilized as a reliable assessment tool for non-experts in various emergent environments.

scholarly journals Agreement and Reliability Analysis of Machine Learning Scaling and Wireless Monitoring in the Assessment of Acute Proximal Weakness by Experts and Non-Experts: A Feasibility Study

2022 ◽  
Vol 12 (1) ◽  
pp. 20
Author(s):  
Eunjeong Park ◽  
Kijeong Lee ◽  
Taehwa Han ◽  
Hyo Suk Nam
Keyword(s):  
Machine Learning ◽  
Reliability Analysis ◽  
Health Management ◽  
Point Of Care ◽  
Correlation Coefficients ◽  
Assessment System ◽  
Neurological Deficits ◽  
Neurological Intensive Care Unit ◽  
Neurological Intensive Care ◽  
Percent Agreement

Assessing the symptoms of proximal weakness caused by neurological deficits requires the knowledge and experience of neurologists. Recent advances in machine learning and the Internet of Things have resulted in the development of automated systems that emulate physicians’ assessments. The application of those systems requires not only accuracy in the classification but also reliability regardless of users’ proficiency in the real environment for the clinical point-of-care and the personalized health management. This study provides an agreement and reliability analysis of using a machine learning-based scaling of Medical Research Council (MRC) proximal scores to evaluate proximal weakness by experts and non-experts. The system trains an ensemble learning model using the signals from sensors attached to the limbs of patients in a neurological intensive care unit. For the agreement analysis, we investigated the percent agreement of MRC proximal scores and Bland-Altman plots of kinematic features between the expert- and non-expert scaling. We also analyzed the intra-class correlation coefficients (ICCs) of kinematic features and Krippendorff’s alpha of the observers’ scaling for the reliability analysis. The mean percent agreement between the expert- and the non-expert scaling was 0.542 for manual scaling and 0.708 for autonomous scaling. The ICCs of kinematic features measured using sensors ranged from 0.742 to 0.850, whereas the Krippendorff’s alpha of manual scaling for the three observers was 0.275. The autonomous assessment system can be utilized by the caregivers, paramedics, or other observers during an emergency to evaluate acute stroke patients.

Hybrid Approach to Sentiment Analysis based on Syntactic Analy- sis and Machine Learning

2010 ◽  
Vol 14 (2) ◽  
pp. 159-181
Author(s):  
MUNPYO HONG ◽  
MIYOUNG SHIN ◽  
Shinhye Park ◽  
Hyungmin Lee
Keyword(s):  
Machine Learning ◽  
Sentiment Analysis ◽  
Hybrid Approach

Utilizing Blockchain Technology in Social Media Bot Identification

2020 ◽  
Author(s):  
Shreya Reddy ◽  
Lisa Ewen ◽  
Pankti Patel ◽  
Prerak Patel ◽  
Ankit Kundal ◽  
...  
Keyword(s):  
Machine Learning ◽  
Social Media ◽  
Gold Standard ◽  
The Internet ◽  
Learning Models ◽  
Current Time ◽  
Machine Learning Methods ◽  
Blockchain Technology ◽  
Modern Age ◽  
Machine Learning Models

<p>As bots become more prevalent and smarter in the modern age of the internet, it becomes ever more important that they be identified and removed. Recent research has dictated that machine learning methods are accurate and the gold standard of bot identification on social media. Unfortunately, machine learning models do not come without their negative aspects such as lengthy training times, difficult feature selection, and overwhelming pre-processing tasks. To overcome these difficulties, we are proposing a blockchain framework for bot identification. At the current time, it is unknown how this method will perform, but it serves to prove the existence of an overwhelming gap of research under this area.<i></i></p>

scholarly journals Recommendation Systems for Education: Systematic Review

Electronics ◽  
2021 ◽  
Vol 10 (14) ◽  
pp. 1611
Author(s):  
María Cora Urdaneta-Ponte ◽  
Amaia Mendez-Zorrilla ◽  
Ibon Oleagordia-Ruiz
Keyword(s):  
Machine Learning ◽  
Systematic Review ◽  
Formal Education ◽  
Research Work ◽  
Hybrid Approach ◽  
Recommendation Systems ◽  
Educational Resources ◽  
Future Research ◽  
Collaborative Approach ◽  
Developmental Approach

Recommendation systems have emerged as a response to overload in terms of increased amounts of information online, which has become a problem for users regarding the time spent on their search and the amount of information retrieved by it. In the field of recommendation systems in education, the relevance of recommended educational resources will improve the student’s learning process, and hence the importance of being able to suitably and reliably ensure relevant, useful information. The purpose of this systematic review is to analyze the work undertaken on recommendation systems that support educational practices with a view to acquiring information related to the type of education and areas dealt with, the developmental approach used, and the elements recommended, as well as being able to detect any gaps in this area for future research work. A systematic review was carried out that included 98 articles from a total of 2937 found in main databases (IEEE, ACM, Scopus and WoS), about which it was able to be established that most are geared towards recommending educational resources for users of formal education, in which the main approaches used in recommendation systems are the collaborative approach, the content-based approach, and the hybrid approach, with a tendency to use machine learning in the last two years. Finally, possible future areas of research and development in this field are presented.

scholarly journals Quantum chemical benchmark databases of gold-standard dimer interaction energies

2021 ◽  
Vol 8 (1) ◽  
Author(s):  
Alexander G. Donchev ◽  
Andrew G. Taube ◽  
Elizabeth Decolvenaere ◽  
Cory Hargus ◽  
Robert T. McGibbon ◽  
...  
Keyword(s):  
Machine Learning ◽  
Gold Standard ◽  
Training Data ◽  
Structure Theory ◽  
Cluster Method ◽  
Coupled Cluster ◽  
Interaction Energies ◽  
Gold Standard Method ◽  
Representative Subset ◽  
Semi Empirical

AbstractAdvances in computational chemistry create an ongoing need for larger and higher-quality datasets that characterize noncovalent molecular interactions. We present three benchmark collections of quantum mechanical data, covering approximately 3,700 distinct types of interacting molecule pairs. The first collection, which we refer to as DES370K, contains interaction energies for more than 370,000 dimer geometries. These were computed using the coupled-cluster method with single, double, and perturbative triple excitations [CCSD(T)], which is widely regarded as the gold-standard method in electronic structure theory. Our second benchmark collection, a core representative subset of DES370K called DES15K, is intended for more computationally demanding applications of the data. Finally, DES5M, our third collection, comprises interaction energies for nearly 5,000,000 dimer geometries; these were calculated using SNS-MP2, a machine learning approach that provides results with accuracy comparable to that of our coupled-cluster training data. These datasets may prove useful in the development of density functionals, empirically corrected wavefunction-based approaches, semi-empirical methods, force fields, and models trained using machine learning methods.

scholarly journals New Hybrid Approach for Developing Automated Machine Learning Workflows: A Real Case Application in Evaluation of Marcellus Shale Gas Production

Fuels ◽  
2021 ◽  
Vol 2 (3) ◽  
pp. 286-303
Author(s):  
Vuong Van Pham ◽  
Ebrahim Fathi ◽  
Fatemeh Belyadi
Keyword(s):  
Artificial Intelligence ◽  
Machine Learning ◽  
Shale Gas ◽  
Field Data ◽  
Marcellus Shale ◽  
Hybrid Approach ◽  
Gas Production ◽  
Bayesian Optimization ◽  
Real Field ◽  
Engineering Problems

The success of machine learning (ML) techniques implemented in different industries heavily rely on operator expertise and domain knowledge, which is used in manually choosing an algorithm and setting up the specific algorithm parameters for a problem. Due to the manual nature of model selection and parameter tuning, it is impossible to quantify or evaluate the quality of this manual process, which in turn limits the ability to perform comparison studies between different algorithms. In this study, we propose a new hybrid approach for developing machine learning workflows to help automated algorithm selection and hyperparameter optimization. The proposed approach provides a robust, reproducible, and unbiased workflow that can be quantified and validated using different scoring metrics. We have used the most common workflows implemented in the application of artificial intelligence (AI) and ML in engineering problems including grid/random search, Bayesian search and optimization, genetic programming, and compared that with our new hybrid approach that includes the integration of Tree-based Pipeline Optimization Tool (TPOT) and Bayesian optimization. The performance of each workflow is quantified using different scoring metrics such as Pearson correlation (i.e., R2 correlation) and Mean Square Error (i.e., MSE). For this purpose, actual field data obtained from 1567 gas wells in Marcellus Shale, with 121 features from reservoir, drilling, completion, stimulation, and operation is tested using different proposed workflows. A proposed new hybrid workflow is then used to evaluate the type well used for evaluation of Marcellus shale gas production. In conclusion, our automated hybrid approach showed significant improvement in comparison to other proposed workflows using both scoring matrices. The new hybrid approach provides a practical tool that supports the automated model and hyperparameter selection, which is tested using real field data that can be implemented in solving different engineering problems using artificial intelligence and machine learning. The new hybrid model is tested in a real field and compared with conventional type wells developed by field engineers. It is found that the type well of the field is very close to P50 predictions of the field, which shows great success in the completion design of the field performed by field engineers. It also shows that the field average production could have been improved by 8% if shorter cluster spacing and higher proppant loading per cluster were used during the frac jobs.

Machine learning assisted stochastic-XFEM for stochastic crack propagation and reliability analysis

2021 ◽  
Vol 112 ◽  
pp. 102882
Author(s):  
Edel R. Martínez ◽  
Souvik Chakraborty ◽  
Solomon Tesfamariam
Keyword(s):  
Machine Learning ◽  
Crack Propagation ◽  
Reliability Analysis

scholarly journals SARS-CoV-2 Epidemics in Retirement and Nursing Homes in Italy: A New Preparedness Assessment Model after the First Epidemic Wave

2021 ◽  
Vol 18 (11) ◽  
pp. 5712
Author(s):  
Carmelo Gugliotta ◽  
Davide Gentili ◽  
Silvia Marras ◽  
Marco Dettori ◽  
Pietro Paolo Muglia ◽  
...  
Keyword(s):  
Nursing Homes ◽  
Perceived Risk ◽  
Assessment Tool ◽  
Resident Training ◽  
Assessment Model ◽  
Infection Prevention And Control ◽  
Field Assessment ◽  
Italian Context ◽  
And Control ◽  
The City

The aim of the study is to evaluate the preparedness of retirement and nursing homes in the city of Sassari at the end of the first wave of the severe acute respiratory syndrome coronavirus 2 epidemic, first by investigating the risk perception of epidemic outbreaks by the facility managers and subsequently by carrying out a field assessment of these facilities. To perform the field assessment, a checklist developed by the CDC (Infection Prevention and Control Assessment Tool for Nursing Homes Preparing for COVID-19) and adapted to the Italian context was used. Fourteen facilities took part in the survey (87.5%). The application of good practices for each survey area was expressed as a percentage with the following median values: restriction policies (87.5%), staff training (53.8%), resident training (67.6%), availability of personal protective equipment (41.7%), infection control practices (73.5%) and communication (80%). Among the facilities, considerable variability was observed in these evaluation fields: only the restriction policies and communication activities were applied uniformly. A discrepancy was found between perceived risk and real danger in the facilities, requiring targeted communication actions. At present, it is necessary to promote a new approach based on the prediction of critical events, thereby providing the means to effectively address them.

Sign in / Sign up

Export Citation Format

Share Document