Nowcasting heavy precipitation over the Netherlands using a 13-year radar archive: a machine learning approach

2021 ◽  
Author(s):  
Eva van der Kooij ◽  
Marc Schleiss ◽  
Riccardo Taormina ◽  
Francesco Fioranelli ◽  
Dorien Lugt ◽  
...  
Keyword(s):  
Machine Learning ◽  
The Netherlands ◽  
Heavy Rainfall ◽  
Predictive Performance ◽  
Heavy Precipitation ◽  
Early Warning Systems ◽  
Training Data ◽  
Short Term ◽  
Data Set ◽  
Radar Images

<p>Accurate short-term forecasts, also known as nowcasts, of heavy precipitation are desirable for creating early warning systems for extreme weather and its consequences, e.g. urban flooding. In this research, we explore the use of machine learning for short-term prediction of heavy rainfall showers in the Netherlands.</p><p>We assess the performance of a recurrent, convolutional neural network (TrajGRU) with lead times of 0 to 2 hours. The network is trained on a 13-year archive of radar images with 5-min temporal and 1-km spatial resolution from the precipitation radars of the Royal Netherlands Meteorological Institute (KNMI). We aim to train the model to predict the formation and dissipation of dynamic, heavy, localized rain events, a task for which traditional Lagrangian nowcasting methods still come up short.</p><p>We report on different ways to optimize predictive performance for heavy rainfall intensities through several experiments. The large dataset available provides many possible configurations for training. To focus on heavy rainfall intensities, we use different subsets of this dataset through using different conditions for event selection and varying the ratio of light and heavy precipitation events present in the training data set and change the loss function used to train the model.</p><p>To assess the performance of the model, we compare our method to current state-of-the-art Lagrangian nowcasting system from the pySTEPS library, like S-PROG, a deterministic approximation of an ensemble mean forecast. The results of the experiments are used to discuss the pros and cons of machine-learning based methods for precipitation nowcasting and possible ways to further increase performance.</p>

Short-term risk forecasts of heavy rainfall

2002 ◽  
Vol 45 (2) ◽  
pp. 121-125 ◽  
Author(s):  
W. Schmid ◽  
S. Mecklenburg ◽  
J. Joss
Keyword(s):  
Probability Distribution ◽  
Density Function ◽  
Heavy Rainfall ◽  
Heavy Precipitation ◽  
Severe Weather ◽  
Short Term ◽  
Radar Images ◽  
Error Density ◽  
Estimated Error ◽  
Near Future

Methodologies for risk forecasts of severe weather hardly exist on the scale of nowcasting (0–3 hours). Here we discuss short-term risk forecasts of heavy precipitation associated with local thunderstorms. We use COTREC/RainCast: a procedure to extrapolate radar images into the near future. An error density function is defined using the estimated error of location of the extrapolated radar patterns. The radar forecast is folded (“smeared”) with the density function, leading to a probability distribution of radar intensities. An algorithm to convert the radar intensities into values of precipitation intensity provides the desired probability (or risk) of heavy rainfall at any position within the considered window in space and time. We discuss, as an example, a flood event from summer 2000.

scholarly journals Comparison of intracranial injury predictability between machine learning algorithms and the nomogram in pediatric traumatic brain injury

2021 ◽  
Vol 51 (5) ◽  
pp. E7
Author(s):  
Thara Tunthanathip ◽  
Jarunee Duangsuwan ◽  
Niwan Wattanakitrungroj ◽  
Sasiporn Tongman ◽  
Nakornchai Phuenpathom
Keyword(s):  
Machine Learning ◽  
Traumatic Brain Injury ◽  
Brain Injury ◽  
Predictive Value ◽  
Predictive Performance ◽  
Training Data ◽  
Intracranial Injury ◽  
Data Set ◽  
Head Ct ◽  
Cranial Ct

OBJECTIVE The overuse of head CT examinations has been much discussed, especially those for minor traumatic brain injury (TBI). In the disruptive era, machine learning (ML) is one of the prediction tools that has been used and applied in various fields of neurosurgery. The objective of this study was to compare the predictive performance between ML and a nomogram, which is the other prediction tool for intracranial injury following cranial CT in children with TBI. METHODS Data from 964 pediatric patients with TBI were randomly divided into a training data set (75%) for hyperparameter tuning and supervised learning from 14 clinical parameters, while the remaining data (25%) were used for validation purposes. Moreover, a nomogram was developed from the training data set with similar parameters. Therefore, models from various ML algorithms and the nomogram were built and deployed via web-based application. RESULTS A random forest classifier (RFC) algorithm established the best performance for predicting intracranial injury following cranial CT of the brain. The area under the receiver operating characteristic curve for the performance of RFC algorithms was 0.80, with 0.34 sensitivity, 0.95 specificity, 0.73 positive predictive value, 0.80 negative predictive value, and 0.79 accuracy. CONCLUSIONS The ML algorithms, particularly the RFC, indicated relatively excellent predictive performance that would have the ability to support physicians in balancing the overuse of head CT scans and reducing the treatment costs of pediatric TBI in general practice.

Comparative Analysis of Machine Learning Techniques Using Predictive Modeling

2020 ◽  
Vol 13 ◽  
Author(s):  
Ritu Khandelwal ◽  
Hemlata Goyal ◽  
Rajveer Singh Shekhawat
Keyword(s):  
Machine Learning ◽  
Comparative Analysis ◽  
Data Science ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Future Trends ◽  
Data Set ◽  
Learning Stage ◽  
Learning Techniques ◽  
Different Types

Introduction: Machine learning is an intelligent technology that works as a bridge between businesses and data science. With the involvement of data science, the business goal focuses on findings to get valuable insights on available data. The large part of Indian Cinema is Bollywood which is a multi-million dollar industry. This paper attempts to predict whether the upcoming Bollywood Movie would be Blockbuster, Superhit, Hit, Average or Flop. For this Machine Learning techniques (classification and prediction) will be applied. To make classifier or prediction model first step is the learning stage in which we need to give the training data set to train the model by applying some technique or algorithm and after that different rules are generated which helps to make a model and predict future trends in different types of organizations. Methods: All the techniques related to classification and Prediction such as Support Vector Machine(SVM), Random Forest, Decision Tree, Naïve Bayes, Logistic Regression, Adaboost, and KNN will be applied and try to find out efficient and effective results. All these functionalities can be applied with GUI Based workflows available with various categories such as data, Visualize, Model, and Evaluate. Result: To make classifier or prediction model first step is learning stage in which we need to give the training data set to train the model by applying some technique or algorithm and after that different rules are generated which helps to make a model and predict future trends in different types of organizations Conclusion: This paper focuses on Comparative Analysis that would be performed based on different parameters such as Accuracy, Confusion Matrix to identify the best possible model for predicting the movie Success. By using Advertisement Propaganda, they can plan for the best time to release the movie according to the predicted success rate to gain higher benefits. Discussion: Data Mining is the process of discovering different patterns from large data sets and from that various relationships are also discovered to solve various problems that come in business and helps to predict the forthcoming trends. This Prediction can help Production Houses for Advertisement Propaganda and also they can plan their costs and by assuring these factors they can make the movie more profitable.

scholarly journals Building Damage Detection from Post-Event Aerial Imagery Using Single Shot Multibox Detector

2019 ◽  
Vol 9 (6) ◽  
pp. 1128 ◽  
Author(s):  
Yundong Li ◽  
Wei Hu ◽  
Han Dong ◽  
Xueyan Zhang
Keyword(s):  
Machine Learning ◽  
Data Augmentation ◽  
Hurricane Sandy ◽  
Training Data ◽  
Aerial Images ◽  
Detection Methods ◽  
Single Shot ◽  
Data Set ◽  
Augmentation Strategies ◽  
Post Disaster

Using aerial cameras, satellite remote sensing or unmanned aerial vehicles (UAV) equipped with cameras can facilitate search and rescue tasks after disasters. The traditional manual interpretation of huge aerial images is inefficient and could be replaced by machine learning-based methods combined with image processing techniques. Given the development of machine learning, researchers find that convolutional neural networks can effectively extract features from images. Some target detection methods based on deep learning, such as the single-shot multibox detector (SSD) algorithm, can achieve better results than traditional methods. However, the impressive performance of machine learning-based methods results from the numerous labeled samples. Given the complexity of post-disaster scenarios, obtaining many samples in the aftermath of disasters is difficult. To address this issue, a damaged building assessment method using SSD with pretraining and data augmentation is proposed in the current study and highlights the following aspects. (1) Objects can be detected and classified into undamaged buildings, damaged buildings, and ruins. (2) A convolution auto-encoder (CAE) that consists of VGG16 is constructed and trained using unlabeled post-disaster images. As a transfer learning strategy, the weights of the SSD model are initialized using the weights of the CAE counterpart. (3) Data augmentation strategies, such as image mirroring, rotation, Gaussian blur, and Gaussian noise processing, are utilized to augment the training data set. As a case study, aerial images of Hurricane Sandy in 2012 were maximized to validate the proposed method’s effectiveness. Experiments show that the pretraining strategy can improve of 10% in terms of overall accuracy compared with the SSD trained from scratch. These experiments also demonstrate that using data augmentation strategies can improve mAP and mF1 by 72% and 20%, respectively. Finally, the experiment is further verified by another dataset of Hurricane Irma, and it is concluded that the paper method is feasible.

Improved Training for Machine Learning: The Additional Potential of Innovative Algorithmic Approaches.

2021 ◽  
Author(s):  
Octavian Dumitru ◽  
Gottfried Schwarz ◽  
Mihai Datcu ◽  
Dongyang Ao ◽  
Zhongling Huang ◽  
...  
Keyword(s):  
Machine Learning ◽  
Remote Sensing ◽  
Test Data ◽  
Satellite Images ◽  
Training Data ◽  
Data Selection ◽  
Generative Adversarial Networks ◽  
Radar Images ◽  
Basic Work ◽  
Selection Of

<p>During the last years, much progress has been reached with machine learning algorithms. Among the typical application fields of machine learning are many technical and commercial applications as well as Earth science analyses, where most often indirect and distorted detector data have to be converted to well-calibrated scientific data that are a prerequisite for a correct understanding of the desired physical quantities and their relationships.</p><p>However, the provision of sufficient calibrated data is not enough for the testing, training, and routine processing of most machine learning applications. In principle, one also needs a clear strategy for the selection of necessary and useful training data and an easily understandable quality control of the finally desired parameters.</p><p>At a first glance, one could guess that this problem could be solved by a careful selection of representative test data covering many typical cases as well as some counterexamples. Then these test data can be used for the training of the internal parameters of a machine learning application. At a second glance, however, many researchers found out that a simple stacking up of plain examples is not the best choice for many scientific applications.</p><p>To get improved machine learning results, we concentrated on the analysis of satellite images depicting the Earth’s surface under various conditions such as the selected instrument type, spectral bands, and spatial resolution. In our case, such data are routinely provided by the freely accessible European Sentinel satellite products (e.g., Sentinel-1, and Sentinel-2). Our basic work then included investigations of how some additional processing steps – to be linked with the selected training data – can provide better machine learning results.</p><p>To this end, we analysed and compared three different approaches to find out machine learning strategies for the joint selection and processing of training data for our Earth observation images:</p><ul><li>One can optimize the training data selection by adapting the data selection to the specific instrument, target, and application characteristics [1].</li> <li>As an alternative, one can dynamically generate new training parameters by Generative Adversarial Networks. This is comparable to the role of a sparring partner in boxing [2].</li> <li>One can also use a hybrid semi-supervised approach for Synthetic Aperture Radar images with limited labelled data. The method is split in: polarimetric scattering classification, topic modelling for scattering labels, unsupervised constraint learning, and supervised label prediction with constraints [3].</li> </ul><p>We applied these strategies in the ExtremeEarth sea-ice monitoring project (http://earthanalytics.eu/). As a result, we can demonstrate for which application cases these three strategies will provide a promising alternative to a simple conventional selection of available training data.</p><p>[1] C.O. Dumitru et. al, “Understanding Satellite Images: A Data Mining Module for Sentinel Images”, Big Earth Data, 2020, 4(4), pp. 367-408.</p><p>[2] D. Ao et. al., “Dialectical GAN for SAR Image Translation: From Sentinel-1 to TerraSAR-X”, Remote Sensing, 2018, 10(10), pp. 1-23.</p><p>[3] Z. Huang, et. al., "HDEC-TFA: An Unsupervised Learning Approach for Discovering Physical Scattering Properties of Single-Polarized SAR Images", IEEE Transactions on Geoscience and Remote Sensing, 2020, pp.1-18.</p>

Synthetic Sonic Log Generation With Machine Learning: A Contest Summary From Five Methods

2021 ◽  
Vol 62 (4) ◽  
pp. 393-406
Author(s):  
Yanxiang Yu ◽  
◽  
Chicheng Xu ◽  
Siddharth Misra ◽  
Weichang Li ◽  
...  
Keyword(s):  
Machine Learning ◽  
Test Data ◽  
Short Term Memory ◽  
Rock Physics ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Blind Test ◽  
Data Set ◽  
Benchmark Model ◽  
Sonic Log

Compressional and shear sonic traveltime logs (DTC and DTS, respectively) are crucial for subsurface characterization and seismic-well tie. However, these two logs are often missing or incomplete in many oil and gas wells. Therefore, many petrophysical and geophysical workflows include sonic log synthetization or pseudo-log generation based on multivariate regression or rock physics relations. Started on March 1, 2020, and concluded on May 7, 2020, the SPWLA PDDA SIG hosted a contest aiming to predict the DTC and DTS logs from seven “easy-to-acquire” conventional logs using machine-learning methods (GitHub, 2020). In the contest, a total number of 20,525 data points with half-foot resolution from three wells was collected to train regression models using machine-learning techniques. Each data point had seven features, consisting of the conventional “easy-to-acquire” logs: caliper, neutron porosity, gamma ray (GR), deep resistivity, medium resistivity, photoelectric factor, and bulk density, respectively, as well as two sonic logs (DTC and DTS) as the target. The separate data set of 11,089 samples from a fourth well was then used as the blind test data set. The prediction performance of the model was evaluated using root mean square error (RMSE) as the metric, shown in the equation below: RMSE=sqrt(1/2*1/m* [∑_(i=1)^m▒〖(〖DTC〗_pred^i-〖DTC〗_true^i)〗^2 + 〖(〖DTS〗_pred^i-〖DTS〗_true^i)〗^2 ] In the benchmark model, (Yu et al., 2020), we used a Random Forest regressor and conducted minimal preprocessing to the training data set; an RMSE score of 17.93 was achieved on the test data set. The top five models from the contest, on average, beat the performance of our benchmark model by 27% in the RMSE score. In the paper, we will review these five solutions, including preprocess techniques and different machine-learning models, including neural network, long short-term memory (LSTM), and ensemble trees. We found that data cleaning and clustering were critical for improving the performance in all models.

A sentiment analysis system for social media using machine learning techniques: Social enablement

2018 ◽  
Vol 34 (3) ◽  
pp. 569-581 ◽  
Author(s):  
Sujata Rani ◽  
Parteek Kumar
Keyword(s):  
Machine Learning ◽  
Social Media ◽  
Sentiment Analysis ◽  
Media Analysis ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Support Vector ◽  
Analysis Tool ◽  
Data Set ◽  
Learning Techniques

Abstract In this article, an innovative approach to perform the sentiment analysis (SA) has been presented. The proposed system handles the issues of Romanized or abbreviated text and spelling variations in the text to perform the sentiment analysis. The training data set of 3,000 movie reviews and tweets has been manually labeled by native speakers of Hindi in three classes, i.e. positive, negative, and neutral. The system uses WEKA (Waikato Environment for Knowledge Analysis) tool to convert these string data into numerical matrices and applies three machine learning techniques, i.e. Naive Bayes (NB), J48, and support vector machine (SVM). The proposed system has been tested on 100 movie reviews and tweets, and it has been observed that SVM has performed best in comparison to other classifiers, and it has an accuracy of 68% for movie reviews and 82% in case of tweets. The results of the proposed system are very promising and can be used in emerging applications like SA of product reviews and social media analysis. Additionally, the proposed system can be used in other cultural/social benefits like predicting/fighting human riots.

scholarly journals TermPicks: A century of Greenland glacier terminus data for use in machine learning applications

2021 ◽  
Author(s):  
Sophie Goliber ◽  
Taryn Black ◽  
Ginny Catania ◽  
James M. Lea ◽  
Helene Olsen ◽  
...  
Keyword(s):  
Machine Learning ◽  
Greenland Ice Sheet ◽  
Median Number ◽  
Google Earth ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Data Set ◽  
Outlet Glacier ◽  
Glacier Terminus ◽  
Median Error

Abstract. Marine-terminating outlet glacier terminus traces, mapped from satellite and aerial imagery, have been used extensively in understanding how outlet glaciers adjust to climate change variability over a range of time scales. Numerous studies have digitized termini manually, but this process is labor-intensive, and no consistent approach exists. A lack of coordination leads to duplication of efforts, particularly for Greenland, which is a major scientific research focus. At the same time, machine learning techniques are rapidly making progress in their ability to automate accurate extraction of glacier termini, with promising developments across a number of optical and SAR satellite sensors. These techniques rely on high quality, manually digitized terminus traces to be used as training data for robust automatic traces. Here we present a database of manually digitized terminus traces for machine learning and scientific applications. These data have been collected, cleaned, assigned with appropriate metadata including image scenes, and compiled so they can be easily accessed by scientists. The TermPicks data set includes 39,060 individual terminus traces for 278 glaciers with a mean and median number of traces per glacier of 136 ± 190 and 93, respectively. Across all glaciers, 32,567 dates have been picked, of which 4,467 have traces from more than one author (duplication of 14 %). We find a median error of ∼100 m among manually-traced termini. Most traces are obtained after 1999, when Landsat 7 was launched. We also provide an overview of an updated version of The Google Earth Engine Digitization Tool (GEEDiT), which has been developed specifically for future manual picking of the Greenland Ice Sheet.

Assessment of Alternative Ways to Integrate Weather Predictions in Photovoltaic Generation Forecasting. 

2021 ◽  
Author(s):  
Kevin Bellinguer ◽  
Robin Girard ◽  
Guillaume Bontron ◽  
Georges Kariniotakis
Keyword(s):  
Linear Model ◽  
Relevant Information ◽  
Training Data ◽  
Short Term ◽  
Data Set ◽  
Photovoltaic Generation ◽  
Weather Information ◽  
Weather Situation ◽  
Conditioned Learning ◽  
Pv Generation

<div> <p>In recent years, the share of photovoltaic (PV) power in Europe has grown: the installed capacity increased from around 10 GW in 2008 to nearly 119 GW in 2018 [1]. Due to the intermittent nature of PV generation, new challenges arise regarding economic profitability and the safe operation of the power network. To overcome these issues, a special effort is made to develop efficient PV generation forecasting tools.</p> <p> </p> <p>For short-term PV production forecasting, past production observations are typically the main drivers. In addition, spatio-temporal (ST) inputs such as Satellite-Derived Surface Irradiance (SDSI) provide relevant information regarding the weather situation in the vicinity of the farm. Moreover, the literature shows us that Numerical Weather Predictions (NWPs) provide relevant information regarding weather trends.</p> <p> </p> <p>NWPs can be integrated in the forecasting process in two different ways. The most straightforward approach considers NWPs as explanatory input variables to the forecasting models. Thus, the atmosphere dynamics are directly carried by the NWPs. The alternative considers NWPs as state variables: weather information is used to filter the training data set to obtain a coherent subset of PV production observations measured under similar weather conditions as the PV production to be predicted. This approach is based on analog methods and makes the weather dynamics to be implicitly contained in the PV production observations. This conditioned learning approach permits to perform local regressions and is adaptive in the sense that the model training is conditioned to the weather situation.</p> <p>The specialized literature focuses on spot NWPs which permits to find situations that evolve in the same way but does not preserve ST patterns. In this context, the addition of SDSI features cannot make the most of the conditioning process. Ref. [3] proposes to use geopotential fields, which are wind drivers, as analog predictors.</p> <p> </p> <p>In this work, we propose the following contributions to the state of the art:</p> <p>We investigate the influence of spot NWPs on the performances of an auto-regressive (AR) and a random forest models according to the two above-mentioned approaches: either as additional explanatory features and/or as analog features. The analogy score proposed by [2] is used to find similar weather situations, then the model is trained over the associated PV production observations. The results highlight that the linear model performs better with the conditioned approach while the non-linear model obtains better performances when fed with explanatory features.</p> <p>Then, the similarity score is extended to gridded NWPs data through the use of a principal component analysis. This method allows to condition the learning to large-scale weather information. A comparison between spot and gridded NWPs conditioned approaches applied with AR model highlights that gridded NWPs improves the influence of SDSI over forecasting performances.</p> <p> </p> <p>The proposed approaches are evaluated using 9 PV plants in France and for a testing period of 12 months.</p> <p> </p> <strong>References</strong> <p>[1]      IRENA - https://www.irena.org/Statistics/Download-Data</p> <p>[2]      Alessandrini, Delle Monache, et al. An analog ensemble for short-term probabilistic solar power forecast. Applied Energy, 2015. https://doi.org/10.1016/j.apenergy.2015.08.011</p> <p>[3]      Bellinguer, Girard, Bontron, Kariniotakis. Short-term Forecasting of Photovoltaic Generation based on Conditioned Learning of Geopotential Fields. 2020, UPEC. https://doi.org/10.1109/UPEC49904.2020.9209858</p> </div>

Water Quality Prediction Using Statistical Tool and Machine Learning Algorithm

2020 ◽  
pp. 609-623
Author(s):  
Arun Kumar Beerala ◽  
Gobinath R. ◽  
Shyamala G. ◽  
Siribommala Manvitha
Keyword(s):  
Machine Learning ◽  
Learning Algorithm ◽  
Training Data ◽  
Machine Learning Techniques ◽  
Statistical Tool ◽  
Data Set ◽  
Water Quality Prediction ◽  
Living Things ◽  
Sampling Locations ◽  
Different Seasons

Water is the most valuable natural resource for all living things and the ecosystem. The quality of groundwater is changed due to change in ecosystem, industrialisation, and urbanisation, etc. In the study, 60 samples were taken and analysed for various physio-chemical parameters. The sampling locations were located using global positioning system (GPS) and were taken for two consecutive years for two different seasons, monsoon (Nov-Dec) and post-monsoon (Jan-Mar). In 2016-2017 and 2017-2018 pH, EC, and TDS were obtained in the field. Hardness and Chloride are determined using titration method. Nitrate and Sulphate were determined using Spectrophotometer. Machine learning techniques were used to train the data set and to predict the unknown values. The dominant elements of groundwater are as follows: Ca2, Mg2 for cation and Cl-, SO42, NO3− for anions. The regression value for the training data set was found to be 0.90596, and for the entire network, it was found to be 0.81729. The best performance was observed as 0.0022605 at epoch 223.

Sign in / Sign up

Export Citation Format

Share Document