scholarly journals Coastal change patterns from time series clustering of permanent laser scan data

2021 ◽  
Vol 9 (1) ◽  
pp. 89-103
Author(s):  
Mieke Kuschnerus ◽  
Roderik Lindenbergh ◽  
Sander Vos
Keyword(s):  
Time Series ◽  
Laser Scanning ◽  
Clustering Algorithms ◽  
Coastal Areas ◽  
Spatiotemporal Data ◽  
Small Scale ◽  
Clustering Methods ◽  
Agglomerative Clustering ◽  
Data Set ◽  
Deformation Processes

Abstract. Sandy coasts are constantly changing environments governed by complex, interacting processes. Permanent laser scanning is a promising technique to monitor such coastal areas and to support analysis of geomorphological deformation processes. This novel technique delivers 3-D representations of the coast at hourly temporal and centimetre spatial resolution and allows us to observe small-scale changes in elevation over extended periods of time. These observations have the potential to improve understanding and modelling of coastal deformation processes. However, to be of use to coastal researchers and coastal management, an efficient way to find and extract deformation processes from the large spatiotemporal data set is needed. To enable automated data mining, we extract time series of surface elevation and use unsupervised learning algorithms to derive a partitioning of the observed area according to change patterns. We compare three well-known clustering algorithms (k-means clustering, agglomerative clustering and density-based spatial clustering of applications with noise; DBSCAN), apply them on the set of time series and identify areas that undergo similar evolution during 1 month. We test if these algorithms fulfil our criteria for suitable clustering on our exemplary data set. The three clustering methods are applied to time series over 30 d extracted from a data set of daily scans covering about 2 km of coast in Kijkduin, the Netherlands. A small section of the beach, where a pile of sand was accumulated by a bulldozer, is used to evaluate the performance of the algorithms against a ground truth. The k-means algorithm and agglomerative clustering deliver similar clusters, and both allow us to identify a fixed number of dominant deformation processes in sandy coastal areas, such as sand accumulation by a bulldozer or erosion in the intertidal area. The level of detail found with these algorithms depends on the choice of the number of clusters k. The DBSCAN algorithm finds clusters for only about 44 % of the area and turns out to be more suitable for the detection of outliers, caused, for example, by temporary objects on the beach. Our study provides a methodology to efficiently mine a spatiotemporal data set for predominant deformation patterns with the associated regions where they occur.

scholarly journals Coastal Change Patterns from Time Series Clustering of Permanent Laser Scan Data

2020 ◽  
Author(s):  
Mieke Kuschnerus ◽  
Roderik Lindenbergh ◽  
Sander Vos
Keyword(s):  
Time Series ◽  
Laser Scanning ◽  
Clustering Algorithm ◽  
Coastal Areas ◽  
Small Scale ◽  
Temporal Data ◽  
Agglomerative Clustering ◽  
Data Set ◽  
Deformation Processes ◽  
Spatio Temporal

Abstract. Sandy coasts are constantly changing environments governed by complex interacting processes. Permanent laser scanning is a promising technique to monitor such coastal areas and support analysis of geomorphological deformation processes. This novel technique delivers 3D representations of a part of the coast at hourly temporal and centimetre spatial resolution and allows to observe small scale changes in elevation over extended periods of time. These observations have the potential to improve understanding and modelling of coastal deformation processes. However, to be of use to coastal researchers and coastal management, an efficient way to find and extract deformation processes from the large spatio-temporal data set is needed. In order to allow data mining in an automated way, we extract time series in elevation or range and use unsupervised learning algorithms to derive a partitioning of the observed area according to change patterns. We compare three well known clustering algorithms, k-means, agglomerative clustering and DBSCAN, and identify areas that undergo similar evolution during one month. We test if they fulfil our criteria for a suitable clustering algorithm on our exemplary data set. The three clustering methods are applied to time series of 30 epochs (during one month) extracted from a data set of daily scans covering a part of the coast at Kijkduin, the Netherlands. A small section of the beach, where a pile of sand was accumulated by a bulldozer is used to evaluate the performance of the algorithms against a ground truth. The k-means algorithm and agglomerative clustering deliver similar clusters, and both allow to identify a fixed number of dominant deformation processes in sandy coastal areas, such as sand accumulation by a bulldozer or erosion in the intertidal area. The DBSCAN algorithm finds clusters for only about 44 % of the area and turns out to be more suitable for the detection of outliers, caused for example by temporary objects on the beach. Our study provides a methodology to efficiently mine a spatio-temporal data set for predominant deformation patterns with the associated regions, where they occur.

scholarly journals Cluster analysis of the results of intraoperative optical spectroscopic diagnostics In brain glioma neurosurgery

2019 ◽  
Vol 7 (4) ◽  
pp. 23-34
Author(s):  
I. A. Osmakov ◽  
T. A. Savelieva ◽  
V. B. Loschenov ◽  
S. A. Goryajnov ◽  
A. A. Potapov
Keyword(s):  
Cluster Analysis ◽  
Optical Spectroscopy ◽  
Protoporphyrin Ix ◽  
Spectroscopy Data ◽  
Clustering Methods ◽  
Agglomerative Clustering ◽  
Data Set ◽  
Broadband Radiation ◽  
Degree Of Malignancy ◽  
Normal White Matter

The paper presents the results of a comparative study of methods of cluster analysis of optical intraoperative spectroscopy data during surgery of glial tumors with varying degree of malignancy. The analysis was carried out both for individual patients and for the entire dataset. The data were obtained using combined optical spectroscopy technique, which allowed simultaneous registration of diffuse reflectance spectra of broadband radiation in the 500–600 nm spectral range (for the analysis of tissue blood supply and the degree of hemoglobin oxygenation), fluorescence spectra of 5‑ALA induced protoporphyrin IX (Pp IX) (for analysis of the malignancy degree) and signal of diffusely reflected laser light used to excite Pp IX fluorescence (to take into account the scattering properties of tissues). To determine the threshold values of these parameters for the tumor, the infltration zone and the normal white matter, we searched for the natural clusters in the available intraoperative optical spectroscopy data and compared them with the results of the pathomorphology. It was shown that, among the considered clustering methods, EM‑algorithm and k‑means methods are optimal for the considered data set and can be used to build a decision support system (DSS) for spectroscopic intraoperative navigation in neurosurgery. Results of clustering relevant to thepathological studies were also obtained using the methods of spectral and agglomerative clustering. These methods can be used to postprocess combined spectroscopy data.

scholarly journals Time series event correlation with DTW and Hierarchical Clustering methods

2019 ◽  
Author(s):  
Srishti Mishra ◽  
Zohair Shafi ◽  
Santanu Pathak
Keyword(s):  
Time Series ◽  
Hierarchical Clustering ◽  
Time Series Data ◽  
Series Data ◽  
Data Sets ◽  
Multiple Time ◽  
Clustering Methods ◽  
Event Correlation ◽  
Data Set ◽  
Causation Analysis

Data driven decision making is becoming increasingly an important aspect for successful business execution. More and more organizations are moving towards taking informed decisions based on the data that they are generating. Most of this data are in temporal format - time series data. Effective analysis across time series data sets, in an efficient and quick manner is a challenge. The most interesting and valuable part of such analysis is to generate insights on correlation and causation across multiple time series data sets. This paper looks at methods that can be used to analyze such data sets and gain useful insights from it, primarily in the form of correlation and causation analysis. This paper focuses on two methods for doing so, Two Sample Test with Dynamic Time Warping and Hierarchical Clustering and looks at how the results returned from both can be used to gain a better understanding of the data. Moreover, the methods used are meant to work with any data set, regardless of the subject domain and idiosyncrasies of the data set, primarily, a data agnostic approach.

A Nonparametric Clustering Algorithm with a Quantile-Based Likelihood Estimator

2014 ◽  
Vol 26 (9) ◽  
pp. 2074-2101 ◽  
Author(s):  
Hideitsu Hino ◽  
Noboru Murata
Keyword(s):  
Clustering Algorithm ◽  
Clustering Algorithms ◽  
Conditional Entropy ◽  
Tuning Parameter ◽  
Clustering Methods ◽  
Sampling Weights ◽  
Data Set ◽  
Information Theoretic ◽  
Nonparametric Clustering ◽  
Conditional Information

Clustering is a representative of unsupervised learning and one of the important approaches in exploratory data analysis. By its very nature, clustering without strong assumption on data distribution is desirable. Information-theoretic clustering is a class of clustering methods that optimize information-theoretic quantities such as entropy and mutual information. These quantities can be estimated in a nonparametric manner, and information-theoretic clustering algorithms are capable of capturing various intrinsic data structures. It is also possible to estimate information-theoretic quantities using a data set with sampling weight for each datum. Assuming the data set is sampled from a certain cluster and assigning different sampling weights depending on the clusters, the cluster-conditional information-theoretic quantities are estimated. In this letter, a simple iterative clustering algorithm is proposed based on a nonparametric estimator of the log likelihood for weighted data sets. The clustering algorithm is also derived from the principle of conditional entropy minimization with maximum entropy regularization. The proposed algorithm does not contain a tuning parameter. The algorithm is experimentally shown to be comparable to or outperform conventional nonparametric clustering methods.

Ant Custering Algorithms

2010 ◽  
Vol 1 (1) ◽  
pp. 1-15 ◽  
Author(s):  
Yu-Chiun Chiou ◽  
Shih-Ta Chou
Keyword(s):  
Clustering Algorithm ◽  
Clustering Algorithms ◽  
Small Scale ◽  
Solution Stability ◽  
Clustering Methods ◽  
Scale Problem ◽  
Clustering Problem ◽  
Genetic Clustering ◽  
Fully Connected ◽  
Pheromone Trail

This paper proposes three ant clustering algorithms (ACAs): ACA-1, ACA-2 and ACA-3. The core logic of the proposed ACAs is to modify the ant colony metaheuristic by reformulating the clustering problem into a network problem. For a clustering problem of N objects and K clusters, a fully connected network of N nodes is formed with link costs, representing the dissimilarity of any two nodes it connects. K ants are then to collect their own nodes according to the link costs and following the pheromone trail laid by previous ants. The proposed three ACAs have been validated on a small-scale problem solved by a total enumeration method. The solution effectiveness at different problem scales consistently shows that ACA-2 outperforms among these three ACAs. A further comparison of ACA-2 with other commonly used clustering methods, including agglomerative hierarchy clustering algorithm (AHCA), K-means algorithm (KMA) and genetic clustering algorithm (GCA), shows that ACA-2 significantly outperforms them in solution effectiveness for the most of cases and also performs considerably better in solution stability as the problem scales or the number of clusters gets larger.

scholarly journals Statistical methods of fracture characterization using acoustic borehole televiewer log interpretation

2021 ◽  
Author(s):  
C Massiot ◽  
John Townend ◽  
A Nicol ◽  
DD McNamara
Keyword(s):  
Probability Distributions ◽  
Likelihood Estimation ◽  
Information Criterion ◽  
Geothermal Field ◽  
Information Criteria ◽  
Clustering Methods ◽  
Agglomerative Clustering ◽  
Data Set ◽  
Data Points ◽  
Borehole Televiewer

©2017. American Geophysical Union. All Rights Reserved. Acoustic borehole televiewer (BHTV) logs provide measurements of fracture attributes (orientations, thickness, and spacing) at depth. Orientation, censoring, and truncation sampling biases similar to those described for one-dimensional outcrop scanlines, and other logging or drilling artifacts specific to BHTV logs, can affect the interpretation of fracture attributes from BHTV logs. K-means, fuzzy K-means, and agglomerative clustering methods provide transparent means of separating fracture groups on the basis of their orientation. Fracture spacing is calculated for each of these fracture sets. Maximum likelihood estimation using truncated distributions permits the fitting of several probability distributions to the fracture attribute data sets within truncation limits, which can then be extrapolated over the entire range where they naturally occur. Akaike Information Criterion (AIC) and Schwartz Bayesian Criterion (SBC) statistical information criteria rank the distributions by how well they fit the data. We demonstrate these attribute analysis methods with a data set derived from three BHTV logs acquired from the high-temperature Rotokawa geothermal field, New Zealand. Varying BHTV log quality reduces the number of input data points, but careful selection of the quality levels where fractures are deemed fully sampled increases the reliability of the analysis. Spacing data analysis comprising up to 300 data points and spanning three orders of magnitude can be approximated similarly well (similar AIC rankings) with several distributions. Several clustering configurations and probability distributions can often characterize the data at similar levels of statistical criteria. Thus, several scenarios should be considered when using BHTV log data to constrain numerical fracture models.

scholarly journals CLUSTERING TIME SERIES OF REPEATED SCAN DATA OF SANDY BEACHES

2019 ◽  
Vol XLII-2/W13 ◽  
pp. 1039-1046
Author(s):  
R. Lindenbergh ◽  
S. van der Kleij ◽  
M. Kuschnerus ◽  
S. Vos ◽  
S. de Vries
Keyword(s):  
Time Series ◽  
Laser Scanning ◽  
Sandy Beaches ◽  
Data Sets ◽  
Data Set ◽  
Spatially Correlated ◽  
Long Time ◽  
Clustering Approach ◽  
Scan Data ◽  
Year 2000

<p><strong>Abstract.</strong> Sandy beaches are highly dynamic areas affected by different natural and anthropogenic effects. Large changes, caused by a storm for example, are in general well-understood and easy to measure. Most times, only small changes, at the centimeter scale, are occurring, but these changes accumulate significantly over periods from weeks to months. Laser scanning is a suitable technique to measure such small signals, as it is able to obtain dense 3D terrain data at centimeter level in a time span of minutes. In this work we consider two repeated laser scan data sets of two different beaches in The Netherlands. The first data set is from around the year 2000 and consists of six consecutive yearly airborne laser scan data sets of a beach on Texel. The second data set is from 2017 and consists of 30 consecutive daily terrestrial scans of a beach near The Hague. So far, little work has been done on time series analysis of repeated scan data. To obtain a first grouping of morphologic processes, we propose to use a simple un-supervised clustering approach, k-means clustering, on de-leveled, cumulative point-wise time series. The results for both regions of interest, obtained using k=5 and k=10 clusters, indicate that such clustering gives a meaningful decomposition of the morphological laser scan data into clusters that exhibit similar change patterns. At the same time, we realize that the chosen approach is just a first step in a wide open topic of clustering spatially correlated long time series of morphological laser scan data as are now obtained by permanent laser scanning.</p>

scholarly journals METHODS OF BUILDING A MODEL OF USER BEHAVIOR

2020 ◽  
Vol 2 (1) ◽  
pp. 43-51
Author(s):  
N. B. Shakhovskaya Shakhovskaya ◽  
◽  
N. I. Melnykova ◽  
Keyword(s):  
Data Analysis ◽  
Data Processing ◽  
Large Scale ◽  
Clustering Algorithms ◽  
Free Form ◽  
Small Scale ◽  
Clustering Methods ◽  
User Behaviour ◽  
Input Parameters ◽  
Scale Data

The number of clustering methods and algorithms were analysed and the peculiarities of their application were singled out. The main advantages of density based clustering methods are the ability to detect free-form clusters of different sizes and resistance to noise and emissions, and the disadvantages include high sensitivity to input parameters, poor class description and unsuitability for large data. The analysis showed that the main problem of all clustering algorithms is their scalability with increasing amount of processed data. The main problems of most of them are the difficulty of setting the optimal input parameters (for density, grid or model algorithms), identification of clusters of different shapes and densities (distribution algorithms, grid-based algorithms), fuzzy completion criteria (hierarchical, partition and model-based). Since the clustering procedure is only one of the stages of data processing of the system as a whole, the chosen algorithm should be easy to use and easy to configure the input parameters. Results of researches show that hierarchical clustering methods include a number of algorithms suitable for both small-scale data processing and large-scale data analysis, which is relevant in the field of social networks. Based on the data analysis, information was collected within fill a smart user profile. Much attention is paid to the study of associative rules, based on which an algorithm for extracting associative rules is proposed, which allows to find statistically significant rules and to look only for dependencies defined by a common set of input data, and has high computational complexity if there are many classification rules. An approach has been developed that focuses on creating and understanding models of user behaviour, predicting future behaviour using the created template. Methods of modelling pre-processing of data (clustering) are investigated and regularities of planning of meetings of friends on the basis of the analysis of daily movement of people and their friends are revealed. Methods of creating and understanding models of user behaviour were presented. The k-means algorithm was used to group users to determine how well each object lay in its own cluster. The concept of association rules was introduced; the method of search of dependences is developed. The accuracy of the model was evaluated.

scholarly journals Similarity Preserving Representation Learning for Time Series Clustering

2019 ◽  
Author(s):  
Qi Lei ◽  
Jinfeng Yi ◽  
Roman Vaculin ◽  
Lingfei Wu ◽  
Inderjit S. Dhillon
Keyword(s):  
Time Series ◽  
Time Series Data ◽  
Clustering Algorithms ◽  
Representation Learning ◽  
Feature Representation ◽  
Series Data ◽  
Similarity Matrix ◽  
Clustering Methods ◽  
Time Series Clustering ◽  
Partially Observed

A considerable amount of clustering algorithms take instance-feature matrices as their inputs. As such, they cannot directly analyze time series data due to its temporal nature, usually unequal lengths, and complex properties. This is a great pity since many of these algorithms are effective, robust, efficient, and easy to use. In this paper, we bridge this gap by proposing an efficient representation learning framework that is able to convert a set of time series with various lengths to an instance-feature matrix. In particular, we guarantee that the pairwise similarities between time series are well preserved after the transformation , thus the learned feature representation is particularly suitable for the time series clustering task. Given a set of $n$ time series, we first construct an $n\times n$ partially-observed similarity matrix by randomly sampling $\mathcal{O}(n \log n)$ pairs of time series and computing their pairwise similarities. We then propose an efficient algorithm that solves a non-convex and NP-hard problem to learn new features based on the partially-observed similarity matrix. By conducting extensive empirical studies, we demonstrate that the proposed framework is much more effective, efficient, and flexible compared to other state-of-the-art clustering methods.

scholarly journals A Review article on Semi- Supervised Clustering Framework for High Dimensional Data

2019 ◽  
pp. 102-108
Author(s):  
M. Pavithra ◽  
R. M. S. Parvathi
Keyword(s):  
Expert Knowledge ◽  
Cluster Formation ◽  
Clustering Algorithms ◽  
Ensemble Member ◽  
Unsupervised Clustering ◽  
Outcome Variable ◽  
High Dimensional ◽  
Clustering Methods ◽  
Data Set ◽  
Supervised Clustering

Cluster analysis methods seek to partition a data set into homogeneous subgroups. It is useful in a wide variety of applications, including document processing and modern genetics. Conventional clustering methods are unsupervised, meaning that there is no outcome variable nor is anything known about the relationship between the observations in the data set. In many situations, however, information about the clusters is available in addition to the values of the features [2]. For example, the cluster labels of some observations may be known, or certain observations may be known to belong to the same cluster. In other cases, one may wish to identify clusters that are associated with a particular outcome variable. This review describes several clustering algorithms (known as �semi-supervised clustering� methods) that can be applied in these situations [3]. The majority of these methods are modifications of the popular k-means clustering method, and several of them will be described in detail. A brief description of some other semi-supervised clustering algorithms is also provided. Cluster formation has three types as supervised clustering, unsupervised clustering and semi supervised. This paper reviews traditional and state-of-the-art methods of clustering [1]. Clustering algorithms are based on active learning, with ensemble clustering-means algorithm, data streams with flock, fuzzy clustering for shape annotations, Incremental semi supervised clustering, Weakly supervised clustering, with minimum labeled data, self-organizing based on neural networks. Incremental semi-supervised clustering ensemble framework (ISSCE) which makes utilization of the advantage of the arbitrary subspace method, the limitation spread approach, the proposed incremental ensemble member choice process, and the normalized cut algorithm to perform high dimensional information clustering [4]. Semi-supervised clustering employs limited supervision in the form of labeled instances or pairwise instance constraints to aid unsupervised clustering and often significantly improves the clustering performance. Despite the vast amount of expert knowledge spent on this problem, most existing work is not designed for handling high-dimensional sparse data.

Sign in / Sign up

Export Citation Format

Share Document