Human Versus Machine: How Do We Know Who Is Winning? ROC Analysis for Comparing Human and Machine Performance under Varying Cost-Prevalence Assumptions

Background Receiver operating characteristic (ROC) analysis is commonly used for comparing models and humans; however, the exact analytical techniques vary and some are flawed. Objectives The aim of the study is to identify common flaws in ROC analysis for human versus model performance, and address them. Methods We review current use and identify common errors. We also review the ROC analysis literature for more appropriate techniques. Results We identify concerns in three techniques: (1) using mean human sensitivity and specificity; (2) assuming humans can be approximated by ROCs; and (3) matching sensitivity and specificity. We identify a technique from Provost et al using dominance tables and cost-prevalence gradients that can be adapted to address these concerns. Conclusion Dominance tables and cost-prevalence gradients provide far greater detail when comparing performances of models and humans, and address common failings in other approaches. This should be the standard method for such analyses moving forward.

Reliabilitas Prediksi Curah Hujan Dasarian Pada Kejadian Curah Hujan Ekstrim Pemicu Banjir 26 Oktober 2020 di Kebumen: Model Statistik (HyBMG) versus Model Dinamik (ECMWF)

Kejadian banjir di Kabupaten Kebumen tanggal 26 Oktober 2020 dipicu oleh hujan dengan intensitas sangat lebat hingga ekstrim yang berlangsung sejak Minggu (25 Oktober 2020) sore hingga Senin (26 Oktober 2020). Beberapa pos pengamatan hujan (kerjasama) menunjukkan curah hujan >150mm/hari (kategori ekstrim) dalam rentang waktu hujan tanggal 24-26 Oktober 2020. Analisis curah hujan kumulatif dasarian ke-III bulan Oktober 2020 di wilayah Kabupaten Kebumen menunjukkan curah hujan >300mm/dasarian (kriteria sangat tinggi). Sebelumnya, pada dasarian ini BMKG Stasiun Klimatologi Semarang memprakirakan sebagian besar wilayah Kabupaten Kebumen diprakirakan dalam kriteria menengah dengan curah hujan berkisar antara 101-150mm/dasarian. Berdasarkan laporan yang masuk ke BPBD Kebumen sedikitnya 25 desa di 7 kecamatan terendam banjir karena beberapa sungai yang ada di Kebumen meluap. Paper ini bertujuan menguji keandalan prakiraan curah hujan dasarian operasional dengan membandingkan luaran prakiraan dengan data observasi pada kondisi ekstrem tersebut. Uji sensitivitas model univariat HyBMG dan ECMWF dilakukan dengan metode visual kesesuaian spasial, korelasi sederhana dan RMSE. Hasil analisis menunjukkan nilai luaran prakiraan ECMWF memiliki nilai RMSE terkecil namun dengan nilai korelasi yang negatif. Korelasi kuat diperoleh dari metode ANFIS dengan nilai RMSE sebesar 556,5. Dapat disimpulkan bahwa luaran model ANFIS memiliki tingkat sensitivitas luaran prakiraan yang lebih handal untuk kejadian hujan ekstrim pada hari Minggu (25 Oktober 2020) di Kabupaten Kebumen. Metode HyBMG memerlukan penambahan input data series lebih banyak lagi sehingga informasi yang terkumpul lebih banyak dan data grid luaran ECMWF menjadi lebih rapat, diharapkan dapat menghasilkan nilai prediksi yang lebih baik lagi.

Machine Learning Driven Contouring of High-Frequency Four-Dimensional Cardiac Ultrasound Data

Automatic boundary detection of 4D ultrasound (4DUS) cardiac data is a promising yet challenging application at the intersection of machine learning and medicine. Using recently developed murine 4DUS cardiac imaging data, we demonstrate here a set of three machine learning models that predict left ventricular wall kinematics along both the endo- and epi-cardial boundaries. Each model is fundamentally built on three key features: (1) the projection of raw US data to a lower dimensional subspace, (2) a smoothing spline basis across time, and (3) a strategic parameterization of the left ventricular boundaries. Model 1 is constructed such that boundary predictions are based on individual short-axis images, regardless of their relative position in the ventricle. Model 2 simultaneously incorporates parallel short-axis image data into their predictions. Model 3 builds on the multi-slice approach of model 2, but assists predictions with a single ground-truth position at end-diastole. To assess the performance of each model, Monte Carlo cross validation was used to assess the performance of each model on unseen data. For predicting the radial distance of the endocardium, models 1, 2, and 3 yielded average R2 values of 0.41, 0.49, and 0.71, respectively. Monte Carlo simulations of the endocardial wall showed significantly closer predictions when using model 2 versus model 1 at a rate of 48.67%, and using model 3 versus model 2 at a rate of 83.50%. These finding suggest that a machine learning approach where multi-slice data are simultaneously used as input and predictions are aided by a single user input yields the most robust performance. Subsequently, we explore the how metrics of cardiac kinematics compare between ground-truth contours and predicted boundaries. We observed negligible deviations from ground-truth when using predicted boundaries alone, except in the case of early diastolic strain rate, providing confidence for the use of such machine learning models for rapid and reliable assessments of murine cardiac function. To our knowledge, this is the first application of machine learning to murine left ventricular 4DUS data. Future work will be needed to strengthen both model performance and applicability to different cardiac disease models.

When will's wont wants wanting

We use neural reinforcement learning concepts including Pavlovian versus instrumental control, liking versus wanting, model-based versus model-free control, online versus offline learning and planning, and internal versus external actions and control to reflect on putative conflicts between short-term temptations and long-term goals.