Exploring multi-modalities in weather prediction using a univariate graph based on machine learning techniques

2021 ◽  
Author(s):  
Natacha Galmiche ◽  
Nello Blaser ◽  
Morten Brun ◽  
Helwig Hauser ◽  
Thomas Spengler ◽  
...  
Keyword(s):  
Machine Learning ◽  
Standard Deviation ◽  
Probability Distributions ◽  
Weather Prediction ◽  
A Priori ◽  
Clustering Algorithms ◽  
Quantitative Information ◽  
Machine Learning Techniques ◽  
Topological Data Analysis ◽  
Learning Techniques

<p>Probability distributions based on ensemble forecasts are commonly used to assess uncertainty in weather prediction. However, interpreting these distributions is not trivial, especially in the case of multimodality with distinct likely outcomes. The conventional summary employs mean and standard deviation across ensemble members, which works well for unimodal, Gaussian-like distributions. In the case of multimodality this misleads, discarding crucial information. </p><p>We aim at combining previously developed clustering algorithms in machine learning and topological data analysis to extract useful information such as the number of clusters in an ensemble. Given the chaotic behaviour of the atmosphere, machine learning techniques can provide relevant results even if no, or very little, a priori information about the data is available. In addition, topological methods that analyse the shape of the data can make results explainable.</p><p>Given an ensemble of univariate time series, a graph is generated whose edges and vertices represent clusters of members, including additional information for each cluster such as the members belonging to them, their uncertainty, and their relevance according to the graph. In the case of multimodality, this approach provides relevant and quantitative information beyond the commonly used mean and standard deviation approach that helps to further characterise the predictability.</p>

scholarly journals Using Machine-Learning Techniques to Identify Responders vs. Non-responders in Randomized Clinical Trials.

2020 ◽  
Author(s):  
Vasiliki Nikolodimou ◽  
Paul Agapow
Keyword(s):  
Machine Learning ◽  
Randomized Clinical Trials ◽  
Clustering Algorithms ◽  
Human Monoclonal Antibody ◽  
Clinical History ◽  
Differential Response ◽  
Unsupervised Clustering ◽  
Machine Learning Techniques ◽  
Genetic Characteristics ◽  
Learning Techniques

Despite the expectation of heterogeneity in therapy outcomes, especially for complex diseases like cancer, analyzing differential response to experimental therapies in a randomized clinical trial (RCT) setting is typically done by dividing patients into responders and non-responders, usually based on a single endpoint. Given the existence of biological and patho-physiological differences among metastatic colorectal cancer (mCRC) patients, we hypothesized that a data-driven analysis of an RCT population outcomes can identify sub-types of patients founded on differential response to Panitumumab - a fully human monoclonal antibody directed against the epidermal growth factor receptor. Outcome and response data of the RCT population were mined with heuristic, distance-based and model-based unsupervised clustering algorithms. The population sub-groups obtained by the best performing clustering approach were then examined in terms of molecular and clinical characteristics. The utility of this characterization was compared against that of the sub-groups obtained by the conventional responders' analysis and then contrasted with aetiological evidence around mCRC heterogeneity and biological functioning. The Partition around Medoids clustering method results into the identification of seven sub-types of patients, statistically distinct from each other in survival outcomes, prognostic biomarkers and genetic characteristics. Conventional responders analysis was proven inferior in uncovering relationships between physical, clinical history, genetic attributes and differential treatment resistance mechanisms. Combined with improved characterization of the molecular subtypes of CRC, applying Machine Learning techniques, like unsupervised clustering, onto the wealth of data already collected by previous RCTs can support the design of further targeted, more efficient RCTs and better identification of patient groups who will respond to a given intervention.

scholarly journals Can machine learning improve the model representation of TKE dissipation rate in the boundary layer for complex terrain?

2020 ◽  
Author(s):  
Nicola Bodini ◽  
Julie K. Lundquist ◽  
Mike Optis
Keyword(s):  
Machine Learning ◽  
Numerical Weather Prediction ◽  
Complex Terrain ◽  
Dissipation Rate ◽  
Prediction Models ◽  
Weather Prediction ◽  
Machine Learning Techniques ◽  
Learning Techniques ◽  
Tke Dissipation Rate ◽  
Numerical Weather Prediction Models

Abstract. Current turbulence parameterizations in numerical weather prediction models at the mesoscale assume a local equilibrium between production and dissipation of turbulence. As this assumption does not hold at fine horizontal resolutions, improved ways to represent turbulent kinetic energy (TKE) dissipation rate (ε) are needed. Here, we use a 6-week data set of turbulence measurements from 184 sonic anemometers in complex terrain at the Perdigão field campaign to suggest improved representations of dissipation rate. First, we demonstrate that a widely used Mellor, Yamada, Nakanishi, and Niino (MYNN) parameterization of TKE dissipation rate leads to a large inaccuracy and bias in the representation of ε. Next, we assess the potential of machine-learning techniques to predict TKE dissipation rate from a set of atmospheric and terrain-related features. We train and test several machine-learning algorithms using the data at Perdigão, and we find that multivariate polynomial regressions and random forests can eliminate the bias MYNN currently shows in representing ε, while also reducing the average error by up to 30 %. Of all the variables included in the algorithms, TKE is the variable responsible for most of the variability of ε, and a strong positive correlation exists between the two. These results suggest further consideration of machine-learning techniques to enhance parameterizations of turbulence in numerical weather prediction models.

scholarly journals Can machine learning improve the model representation of turbulent kinetic energy dissipation rate in the boundary layer for complex terrain?

2020 ◽  
Vol 13 (9) ◽  
pp. 4271-4285
Author(s):  
Nicola Bodini ◽  
Julie K. Lundquist ◽  
Mike Optis
Keyword(s):  
Machine Learning ◽  
Kinetic Energy ◽  
Complex Terrain ◽  
Dissipation Rate ◽  
Prediction Models ◽  
Weather Prediction ◽  
Machine Learning Techniques ◽  
Learning Techniques ◽  
Tke Dissipation Rate ◽  
Numerical Weather Prediction Models

Abstract. Current turbulence parameterizations in numerical weather prediction models at the mesoscale assume a local equilibrium between production and dissipation of turbulence. As this assumption does not hold at fine horizontal resolutions, improved ways to represent turbulent kinetic energy (TKE) dissipation rate (ϵ) are needed. Here, we use a 6-week data set of turbulence measurements from 184 sonic anemometers in complex terrain at the Perdigão field campaign to suggest improved representations of dissipation rate. First, we demonstrate that the widely used Mellor, Yamada, Nakanishi, and Niino (MYNN) parameterization of TKE dissipation rate leads to a large inaccuracy and bias in the representation of ϵ. Next, we assess the potential of machine-learning techniques to predict TKE dissipation rate from a set of atmospheric and terrain-related features. We train and test several machine-learning algorithms using the data at Perdigão, and we find that the models eliminate the bias MYNN currently shows in representing ϵ, while also reducing the average error by up to almost 40 %. Of all the variables included in the algorithms, TKE is the variable responsible for most of the variability of ϵ, and a strong positive correlation exists between the two. These results suggest further consideration of machine-learning techniques to enhance parameterizations of turbulence in numerical weather prediction models.

scholarly journals Using Real-Time Data and Unsupervised Machine Learning Techniques to Study Large-Scale Spatio–Temporal Characteristics of Wastewater Discharges and their Influence on Surface Water Quality in the Yangtze River Basin

Water ◽  
2019 ◽  
Vol 11 (6) ◽  
pp. 1268 ◽  
Author(s):  
Zhenzhen Di ◽  
Miao Chang ◽  
Peikun Guo ◽  
Yang Li ◽  
Yin Chang
Keyword(s):  
Machine Learning ◽  
Surface Water ◽  
Real Time ◽  
Yangtze River Basin ◽  
Clustering Algorithms ◽  
Machine Learning Techniques ◽  
Unsupervised Machine Learning ◽  
Learning Techniques ◽  
The Yangtze River Basin ◽  
Spatio Temporal

Most worldwide industrial wastewater, including in China, is still directly discharged to aquatic environments without adequate treatment. Because of a lack of data and few methods, the relationships between pollutants discharged in wastewater and those in surface water have not been fully revealed and unsupervised machine learning techniques, such as clustering algorithms, have been neglected in related research fields. In this study, real-time monitoring data for chemical oxygen demand (COD), ammonia nitrogen (NH3-N), pH, and dissolved oxygen in the wastewater discharged from 2213 factories and in the surface water at 18 monitoring sections (sites) in 7 administrative regions in the Yangtze River Basin from 2016 to 2017 were collected and analyzed by the partitioning around medoids (PAM) and expectation–maximization (EM) clustering algorithms, Welch t-test, Wilcoxon test, and Spearman correlation. The results showed that compared with the spatial cluster comprising unpolluted sites, the spatial cluster comprised heavily polluted sites where more wastewater was discharged had relatively high COD (>100 mg L−1) and NH3-N (>6 mg L−1) concentrations and relatively low pH (<6) from 15 industrial classes that respected the different discharge limits outlined in the pollutant discharge standards. The results also showed that the economic activities generating wastewater and the geographical distribution of the heavily polluted wastewater changed from 2016 to 2017, such that the concentration ranges of pollutants in discharges widened and the contributions from some emerging enterprises became more important. The correlations between the quality of the wastewater and the surface water strengthened as the whole-year data sets were reduced to the heavily polluted periods by the EM clustering and water quality evaluation. This study demonstrates how unsupervised machine learning algorithms play an objective and effective role in data mining real-time monitoring information and highlighting spatio–temporal relationships between pollutants in wastewater discharges and surface water to support scientific water resource management.

scholarly journals Analysis of Cattle Social Transitional Behaviour: Attraction and Repulsion

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5340
Author(s):  
Haocheng Xu ◽  
Shenghong Li ◽  
Caroline Lee ◽  
Wei Ni ◽  
David Abbott ◽  
...  
Keyword(s):  
Machine Learning ◽  
Social Interactions ◽  
Management Practices ◽  
Clustering Algorithms ◽  
Physical Distance ◽  
Machine Learning Techniques ◽  
Learning Approaches ◽  
Video Recordings ◽  
Learning Techniques ◽  
The Social

Understanding social interactions in livestock groups could improve management practices, but this can be difficult and time-consuming using traditional methods of live observations and video recordings. Sensor technologies and machine learning techniques could provide insight not previously possible. In this study, based on the animals’ location information acquired by a new cooperative wireless localisation system, unsupervised machine learning approaches were performed to identify the social structure of a small group of cattle yearlings (n=10) and the social behaviour of an individual. The paper first defined the affinity between an animal pair based on the ranks of their distance. Unsupervised clustering algorithms were then performed, including K-means clustering and agglomerative hierarchical clustering. In particular, K-means clustering was applied based on logical and physical distance. By comparing the clustering result based on logical distance and physical distance, the leader animals and the influence of an individual in a herd of cattle were identified, which provides valuable information for studying the behaviour of animal herds. Improvements in device robustness and replication of this work would confirm the practical application of this technology and analysis methodologies.

scholarly journals Multi-Step Ahead Wind Power Generation Prediction Based on Hybrid Machine Learning Techniques

Energies ◽  
2018 ◽  
Vol 11 (8) ◽  
pp. 1975 ◽  
Author(s):  
Wei Dong ◽  
Qiang Yang ◽  
Xinli Fang
Keyword(s):  
Machine Learning ◽  
Phase Space ◽  
Fuzzy Inference ◽  
Weather Prediction ◽  
Wind Farms ◽  
Machine Learning Techniques ◽  
Multiple Time ◽  
Inference System ◽  
Learning Techniques ◽  
Benchmark Solutions

Accurate generation prediction at multiple time-steps is of paramount importance for reliable and economical operation of wind farms. This study proposed a novel algorithmic solution using various forms of machine learning techniques in a hybrid manner, including phase space reconstruction (PSR), input variable selection (IVS), K-means clustering and adaptive neuro-fuzzy inference system (ANFIS). The PSR technique transforms the historical time series into a set of phase-space variables combining with the numerical weather prediction (NWP) data to prepare candidate inputs. A minimal redundancy maximal relevance (mRMR) criterion based filtering approach is used to automatically select the optimal input variables for the multi-step ahead prediction. Then, the input instances are divided into a set of subsets using the K-means clustering to train the ANFIS. The ANFIS parameters are further optimized to improve the prediction performance by the use of particle swarm optimization (PSO) algorithm. The proposed solution is extensively evaluated through case studies of two realistic wind farms and the numerical results clearly confirm its effectiveness and improved prediction accuracy compared to benchmark solutions.

scholarly journals Hierarchical Sparse Subspace Clustering (HESSC): An Automatic Approach for Hyperspectral Image Analysis

2020 ◽  
Vol 12 (15) ◽  
pp. 2421
Author(s):  
Kasra Rafiezadeh Shahi ◽  
Mahdi Khodadadzadeh ◽  
Laura Tusa ◽  
Pedram Ghamisi ◽  
Raimon Tolosana-Delgado ◽  
...  
Keyword(s):  
Machine Learning ◽  
Clustering Algorithm ◽  
Hyperspectral Image ◽  
Imaging Techniques ◽  
Clustering Algorithms ◽  
Machine Learning Techniques ◽  
Mixed Data ◽  
Number Of Clusters ◽  
Hyperspectral Image Analysis ◽  
Learning Techniques

Hyperspectral imaging techniques are becoming one of the most important tools to remotely acquire fine spectral information on different objects. However, hyperspectral images (HSIs) require dedicated processing for most applications. Therefore, several machine learning techniques were proposed in the last decades. Among the proposed machine learning techniques, unsupervised learning techniques have become popular as they do not need any prior knowledge. Specifically, sparse subspace-based clustering algorithms have drawn special attention to cluster the HSI into meaningful groups since such algorithms are able to handle high dimensional and highly mixed data, as is the case in real-world applications. Nonetheless, sparse subspace-based clustering algorithms usually tend to demand high computational power and can be time-consuming. In addition, the number of clusters is usually predefined. In this paper, we propose a new hierarchical sparse subspace-based clustering algorithm (HESSC), which handles the aforementioned problems in a robust and fast manner and estimates the number of clusters automatically. In the experiment, HESSC is applied to three real drill-core samples and one well-known rural benchmark (i.e., Trento) HSI datasets. In order to evaluate the performance of HESSC, the performance of the new proposed algorithm is quantitatively and qualitatively compared to the state-of-the-art sparse subspace-based algorithms. In addition, in order to have a comparison with conventional clustering algorithms, HESSC’s performance is compared with K-means and FCM. The obtained clustering results demonstrate that HESSC performs well when clustering HSIs compared to the other applied clustering algorithms.

scholarly journals Combining Ensemble Kalman Filter and Reservoir Computing to predict spatio-temporal chaotic systems from imperfect observations and models

2020 ◽  
Author(s):  
Futo Tomizawa ◽  
Yohei Sawada
Keyword(s):  
Machine Learning ◽  
Kalman Filter ◽  
Chaotic Systems ◽  
Weather Prediction ◽  
Machine Learning Techniques ◽  
Reservoir Computing ◽  
Learning Techniques ◽  
Spatio Temporal ◽  
Ensemble Transform Kalman Filter ◽  
Lorenz 96

Abstract. Prediction of spatio-temporal chaotic systems is important in various fields, such as Numerical Weather Prediction (NWP). While data assimilation methods have been applied in NWP, machine learning techniques, such as Reservoir Computing (RC), are recently recognized as promising tools to predict spatio-temporal chaotic systems. However, the sensitivity of the skill of the machine learning based prediction to the imperfectness of observations is unclear. In this study, we evaluate the skill of RC with noisy and sparsely distributed observations. We intensively compare the performances of RC and Local Ensemble Transform Kalman Filter (LETKF) by applying them to the prediction of the Lorenz 96 system. Although RC can successfully predict the Lorenz 96 system if the system is perfectly observed, we find that RC is vulnerable to observation sparsity compared with LETKF. To overcome this limitation of RC, we propose to combine LETKF and RC. In our proposed method, the system is predicted by RC that learned the analysis time series estimated by LETKF. Our proposed method can successfully predict the Lorenz 96 system using noisy and sparsely distributed observations. Most importantly, our method can predict better than LETKF when the process-based model is imperfect.

Leaf Disease Detection Using AI

2021 ◽  
pp. 110-136
Author(s):  
Praveen Kumar Maduri ◽  
Tushar Biswas ◽  
Preeti Dhiman ◽  
Apurva Soni ◽  
Kushagra Singh
Keyword(s):  
Machine Learning ◽  
Clustering Algorithms ◽  
Essential Elements ◽  
Plant Diseases ◽  
Machine Learning Techniques ◽  
Learning Techniques ◽  
Initial Stage ◽  
Large Farm ◽  
Time And Energy ◽  
Financial Losses

Plants play a significant role in everyone's life. They provide us essential elements like food, oxygen, and shelter, so plants must be supervised and nurtured properly. During cultivation, crops are prone to different kinds of diseases which can severely damage the whole yield leading to financial losses for farmers. In last 10 years, researchers have used different machine learning techniques to detect the disease on plants, but either the methods were not efficient enough to be implemented or were not able to cover the wide area in which plant diseases can be detected. So, the author has introduced a method which is efficient enough to easily detect plant disease and can be implemented in large fields. The author has used a combination of CNN and k-means clustering algorithms. By using this method, crops disease is detected by analyzing the leaves, which notifies users for action in the initial stage. Thus, the proposed method prevents whole crops from getting damaged and saves time and energy of farmers as disease will be identified way before a human eye can detect it on a large farm.

Sign in / Sign up

Export Citation Format

Share Document