Geometrical inspired pre-weighting enhances Markov clustering community detection in complex networks

AbstractMarkov clustering is an effective unsupervised pattern recognition algorithm for data clustering in high-dimensional feature space. However, its community detection performance in complex networks has been demonstrating results far from the state of the art methods such as Infomap and Louvain. The crucial issue is to convert the unweighted network topology in a ‘smart-enough’ pre-weighted connectivity that adequately steers the stochastic flow procedure behind Markov clustering. Here we introduce a conceptual innovation and we discuss how to leverage network latent geometry notions in order to design similarity measures for pre-weighting the adjacency matrix used in Markov clustering community detection. Our results demonstrate that the proposed strategy improves Markov clustering significantly, to the extent that it is often close to the performance of current state of the art methods for community detection. These findings emerge considering both synthetic ‘realistic’ networks (with known ground-truth communities) and real networks (with community metadata), and even when the real network connectivity is corrupted by noise artificially induced by missing or spurious links. Our study enhances the generalized understanding of how network geometry plays a fundamental role in the design of algorithms based on network navigability.

Download Full-text

Overlapping community detection based on the union of all maximum spanning trees

Library Hi Tech ◽

10.1108/lht-01-2019-0003 ◽

2020 ◽

Vol 38 (2) ◽

pp. 276-292

Author(s):

Khawla Asmi ◽

Dounia Lotfi ◽

Mohamed El Marraki

Keyword(s):

Community Detection ◽

Local Community ◽

Spanning Trees ◽

State Of The Art ◽

Ground Truth ◽

Actual State ◽

Overlapping Community Detection ◽

Content Type ◽

Art Methods ◽

Overlapping Community

Purpose The state-of-the-art methods designed for overlapping community detection are limited by their high execution time as in CPM or the need to provide some parameters like the number of communities in Bigclam and Nise_sph, which is a nontrivial information. Hence, there is a need to develop the accuracy that represents the primordial goal, where the actual state-of-the-art methods do not succeed to achieve high correspondence with the ground truth for many instances of networks. The paper aims to discuss this issue. Design/methodology/approach The authors offer a new method that explore the union of all maximum spanning trees (UMST) and models the strength of links between nodes. Also, each node in the UMST is linked with its most similar neighbor. From this model, the authors extract local community for each node, and then they combine the produced communities according to their number of shared nodes. Findings The experiments on eight real-world data sets and four sets of artificial networks show that the proposed method achieves obvious improvements over four state-of-the-art (BigClam, OSLOM, Demon, SE, DMST and ST) methods in terms of the F-score and ONMI for the networks with ground truth (Amazon, Youtube, LiveJournal and Orkut). Also, for the other networks, it provides communities with a good overlapping modularity. Originality/value In this paper, the authors investigate the UMST for the overlapping community detection.

Download Full-text

Community Detection Based on a Preferential Decision Model

Information ◽

10.3390/info11010053 ◽

2020 ◽

Vol 11 (1) ◽

pp. 53

Author(s):

Jinfang Sheng ◽

Ben Lu ◽

Bin Wang ◽

Jie Hu ◽

Kai Wang ◽

...

Keyword(s):

Community Structure ◽

Complex Networks ◽

Community Detection ◽

Decision Model ◽

Structural Characteristics ◽

Real Life ◽

Stable State ◽

Ground Truth ◽

Label Propagation ◽

Comparison Algorithms

The research on complex networks is a hot topic in many fields, among which community detection is a complex and meaningful process, which plays an important role in researching the characteristics of complex networks. Community structure is a common feature in the network. Given a graph, the process of uncovering its community structure is called community detection. Many community detection algorithms from different perspectives have been proposed. Achieving stable and accurate community division is still a non-trivial task due to the difficulty of setting specific parameters, high randomness and lack of ground-truth information. In this paper, we explore a new decision-making method through real-life communication and propose a preferential decision model based on dynamic relationships applied to dynamic systems. We apply this model to the label propagation algorithm and present a Community Detection based on Preferential Decision Model, called CDPD. This model intuitively aims to reveal the topological structure and the hierarchical structure between networks. By analyzing the structural characteristics of complex networks and mining the tightness between nodes, the priority of neighbor nodes is chosen to perform the required preferential decision, and finally the information in the system reaches a stable state. In the experiments, through the comparison of eight comparison algorithms, we verified the performance of CDPD in real-world networks and synthetic networks. The results show that CDPD not only has better performance than most recent algorithms on most datasets, but it is also more suitable for many community networks with ambiguous structure, especially sparse networks.

Download Full-text

Towards effective discovery of natural communities in complex networks and implications in e-commerce

Electronic Commerce Research ◽

10.1007/s10660-019-09395-y ◽

2020 ◽

Cited By ~ 1

Author(s):

Swarup Chattopadhyay ◽

Tanmay Basu ◽

Asit K. Das ◽

Kuntal Ghosh ◽

Late C. A. Murthy

Keyword(s):

Complex Networks ◽

Community Detection ◽

Similarity Measure ◽

Recommender System ◽

Data Clustering ◽

Ground Truth ◽

Normalized Mutual Information ◽

Detection Algorithms ◽

Node Similarity ◽

Art Community

AbstractAutomated community detection is an important problem in the study of complex networks. The idea of community detection is closely related to the concept of data clustering in pattern recognition. Data clustering refers to the task of grouping similar objects and segregating dissimilar objects. The community detection problem can be thought of as finding groups of densely interconnected nodes with few connections to nodes outside the group. A node similarity measure is proposed here that finds the similarity between two nodes by considering both neighbors and non-neighbors of these two nodes. Subsequently, a method is introduced for identifying communities in complex networks using this node similarity measure and the notion of data clustering. The significant characteristic of the proposed method is that it does not need any prior knowledge about the actual communities of a network. Extensive experiments on several real world and artificial networks with known ground-truth communities are reported. The proposed method is compared with various state of the art community detection algorithms by using several criteria, viz. normalized mutual information, f-measure etc. Moreover, it has been successfully applied in improving the effectiveness of a recommender system which is rapidly becoming a crucial tool in e-commerce applications. The empirical results suggest that the proposed technique has the potential to improve the performance of a recommender system and hence it may be useful for other e-commerce applications.

Download Full-text

3D Convolutional Neural Networks for Remote Pulse Rate Measurement and Mapping from Facial Video

Applied Sciences ◽

10.3390/app9204364 ◽

2019 ◽

Vol 9 (20) ◽

pp. 4364 ◽

Cited By ~ 10

Author(s):

Frédéric Bousefsaf ◽

Alain Pruski ◽

Choubeila Maaoui

Keyword(s):

Pulse Rate ◽

State Of The Art ◽

Synthetic Data ◽

Ground Truth ◽

Large Set ◽

Training Procedure ◽

Rate Measurement ◽

3D Network ◽

Public Data ◽

Art Methods

Remote pulse rate measurement from facial video has gained particular attention over the last few years. Research exhibits significant advancements and demonstrates that common video cameras correspond to reliable devices that can be employed to measure a large set of biomedical parameters without any contact with the subject. A new framework for measuring and mapping pulse rate from video is presented in this pilot study. The method, which relies on convolutional 3D networks, is fully automatic and does not require any special image preprocessing. In addition, the network ensures concurrent mapping by producing a prediction for each local group of pixels. A particular training procedure that employs only synthetic data is proposed. Preliminary results demonstrate that this convolutional 3D network can effectively extract pulse rate from video without the need for any processing of frames. The trained model was compared with other state-of-the-art methods on public data. Results exhibit significant agreement between estimated and ground-truth measurements: the root mean square error computed from pulse rate values assessed with the convolutional 3D network is equal to 8.64 bpm, which is superior to 10 bpm for the other state-of-the-art methods. The robustness of the method to natural motion and increases in performance correspond to the two main avenues that will be considered in future works.

Download Full-text

Minimax Bridgeness-Based Clustering for Hyperspectral Data

Remote Sensing ◽

10.3390/rs12071162 ◽

2020 ◽

Vol 12 (7) ◽

pp. 1162 ◽

Cited By ~ 1

Author(s):

Steven Le Moan ◽

Claude Cariou

Keyword(s):

Similarity Measures ◽

Feature Space ◽

Ground Truth ◽

Parameter Tuning ◽

Hyperspectral Data ◽

Ground Truth Data ◽

Sensing Applications ◽

Nearest Neighbours ◽

Silhouette Index ◽

Clustering Approach

Hyperspectral (HS) imaging has been used extensively in remote sensing applications like agriculture, forestry, geology and marine science. HS pixel classification is an important task to help identify different classes of materials within a scene, such as different types of crops on a farm. However, this task is significantly hindered by the fact that HS pixels typically form high-dimensional clusters of arbitrary sizes and shapes in the feature space spanned by all spectral channels. This is even more of a challenge when ground truth data is difficult to obtain and when there is no reliable prior information about these clusters (e.g., number, typical shape, intrinsic dimensionality). In this letter, we present a new graph-based clustering approach for hyperspectral data mining that does not require ground truth data nor parameter tuning. It is based on the minimax distance, a measure of similarity between vertices on a graph. Using the silhouette index, we demonstrate that the minimax distance is more suitable to identify clusters in raw hyperspectral data than two other graph-based similarity measures: mutual proximity and shared nearest neighbours. We then introduce the minimax bridgeness-based clustering approach, and we demonstrate that it can discover clusters of interest in hyperspectral data better than comparable approaches.

Download Full-text

Regularized Chained Deep Neural Network Classifier for Multiple Annotators

Applied Sciences ◽

10.3390/app11125409 ◽

2021 ◽

Vol 11 (12) ◽

pp. 5409

Author(s):

Julián Gil-González ◽

Andrés Valencia-Duque ◽

Andrés Álvarez-Meza ◽

Álvaro Orozco-Gutiérrez ◽

Andrea García-Moreno

Keyword(s):

Neural Network ◽

Deep Neural Network ◽

State Of The Art ◽

Feature Space ◽

Ground Truth ◽

Amazon Mechanical Turk ◽

Input Feature ◽

Multiple Annotators ◽

Classification Tasks ◽

Multiple Experts

The increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, changes how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), databases holding multiple annotators are provided. However, most state-of-the-art methods devoted to learning from multiple experts assume that the labeler’s behavior is homogeneous across the input feature space. Besides, independence constraints are imposed on annotators’ outputs. This paper presents a regularized chained deep neural network to deal with classification tasks from multiple annotators. The introduced method, termed RCDNN, jointly predicts the ground truth label and the annotators’ performance from input space samples. In turn, RCDNN codes interdependencies among the experts by analyzing the layers’ weights and includes l1, l2, and Monte-Carlo Dropout-based regularizers to deal with the over-fitting issue in deep learning models. Obtained results (using both simulated and real-world annotators) demonstrate that RCDNN can deal with multi-labelers scenarios for classification tasks, defeating state-of-the-art techniques.

Download Full-text

Fast and Effective Region-based Depth Map Upsampling with Application to Location Map-Free Reversible Data Hiding

Journal of Advances in Mathematics and Computer Science ◽

10.9734/jamcs/2020/v35i430268 ◽

2020 ◽

pp. 24-45

Author(s):

Kuo-Liang Chung ◽

Yu-Ling Tseng ◽

Tzu-Hsien Chan ◽

Ching-Sheng Wang

Keyword(s):

Data Hiding ◽

Reversible Data Hiding ◽

State Of The Art ◽

Depth Map ◽

Ground Truth ◽

The State ◽

Embedding Capacity ◽

Art Methods ◽

Location Map ◽

Depth Map Upsampling

In this paper, we rst propose a fast and eective region-based depth map upsampling method, and then propose a joint upsampling and location map-free reversible data hiding method, simpled called the JUR method. In the proposed upsampling method, all the missing depth pixels are partitioned into three disjoint regions: the homogeneous, semi-homogeneous, and non- homogeneous regions. Then, we propose the depth copying, mean value, and bicubic interpolation approaches to reconstruct the three kinds of missing depth pixels quickly, respectively. In the proposed JUR method, without any location map overhead, using the neighboring ground truth depth pixels of each missing depth pixel, achieving substantial quality, and embedding capacity merits. The comprehensive experiments have been carried out to not only justify the execution-time and quality merits of the upsampled depth maps by our upsampling method relative to the state-of-the-art methods, but also justify the embedding capacity and quality merits of our JUR method when compared with the state-of-the-art methods.

Download Full-text

Detecting communities by suspecting the maximum degree nodes

International Journal of Modern Physics B ◽

10.1142/s0217979219501339 ◽

2019 ◽

Vol 33 (13) ◽

pp. 1950133 ◽

Cited By ~ 1

Author(s):

Mei Chen ◽

Mei Zhang ◽

Ming Li ◽

Mingwei Leng ◽

Zhichong Yang ◽

...

Keyword(s):

Maximum Degree ◽

State Of The Art ◽

Ground Truth ◽

Underlying Structure ◽

Community Structures ◽

Large Size ◽

Low Degree ◽

Wide Range ◽

Art Methods ◽

High Degree

Detecting the natural communities in a real-world network can uncover its underlying structure and potential function. In this paper, a novel community algorithm SUM is introduced. The fundamental idea of SUM is that a node with relatively low degree stays faithful to its community, because it only has links with nodes in one community, while a node with relatively high degree not only has links with nodes within but also outside its community, and this may cause confusion when detecting communities. Based on this idea, SUM detects communities by suspecting the links of the maximum degree nodes to their neighbors within a community, and relying mainly on the nodes with relatively low degree simultaneously. SUM elegantly defines a similarity which takes into account both the commonality and the rejective degree of two adjacent nodes. After putting similar nodes into one community, SUM generates initial communities by reassigning the maximum degree nodes. Next, SUM assigns nodes without labels to the initial communities, and adjusts the border node to its most linked community. To evaluate the effectiveness of SUM, SUM is compared with seven baselines, including four classical and three state-of-the-art methods on a wide range of complex networks. On the small size networks with ground-truth community structures, results are visually demonstrated, as well as quantitatively measured with ARI, NMI and Modularity. On the relatively large size networks without ground-truth community structures, the performances of these algorithms are evaluated according to Modularity. Experimental results indicate that SUM can effectively determine community structures on small or relatively large size networks with high quality, and also outperforms the compared state-of-the-art methods.

Download Full-text

Improving future optical Earth Observation products using transfer learning

10.5194/egusphere-egu2020-9983 ◽

2020 ◽

Author(s):

Peter Kettig ◽

Eduardo Sanchez-Diaz ◽

Simon Baillarin ◽

Olivier Hagolle ◽

Jean-Marc Delvit ◽

...

Keyword(s):

Machine Learning ◽

Transfer Learning ◽

Imaging System ◽

State Of The Art ◽

Feature Space ◽

Ground Truth ◽

Earth Observation ◽

Physical Models ◽

Training Dataset ◽

Cloud Detection

Pixels covered by clouds in optical Earth Observation images are not usable for most applications. For this reason, only images delivered with reliable cloud masks are eligible for an automated or massive analysis. Current state of the art cloud detection algorithms, both physical models and machine learning models, are specific to a mission or a mission type, with limited transferability. A new model has to be developed every time a new mission is launched. Machine Learning may overcome this problem and, in turn obtain state of the art, or even better performances by training a same algorithm on datasets from different missions. However, simulating products for upcoming missions is not always possible and available actual products are not enough to create a training dataset until well after the launch. Furthermore, labelling data is time consuming. Therefore, even by the time when enough data is available, manually labelled data might not be available at all.&#160;To solve this bottleneck, we propose a transfer learning based method using the available products of the current generation of satellites. These existing products are gathered in a database that is used to train a deep convolutional neural network (CNN) solely on those products. The trained model is applied to images from other - unseen - sensors and the outputs are evaluated. We avoid labelling manually by automatically producing the ground data with existing algorithms. Only a few semi-manually labelled images are used for qualifying the model. Even those semi-manually labelled samples need very few user inputs. This drastic reduction of user input limits subjectivity and reduce the costs.&#160;We provide an example of such a process by training a model to detect clouds in Sentinel-2 images, using as ground-truth the masks of existing state-of-the-art processors. Then, we apply the trained network to detect clouds in previously unseen imagery of other sensors such as the SPOT family or the High-Resolution (HR) Pleiades imaging system, which provide a different feature space.The results demonstrate that the trained model is robust to variations within the individual bands resulting from different acquisition methods and spectral responses. Furthermore, the addition of geo-located auxiliary data that is independent from the platform, such as digital elevation models (DEMs), as well as simple synthetic bands such as the NDVI or NDSI, further improves the results.In the future, this approach opens up the possibility to be used on new CNES&#8217; missions, such as Microcarb or CO3D.

Download Full-text

FluidC+: A novel community detection algorithm based on fluid propagation

International Journal of Modern Physics C ◽

10.1142/s0129183119500219 ◽

2019 ◽

Vol 30 (04) ◽

pp. 1950021

Author(s):

Jinfang Sheng ◽

Kai Wang ◽

Zejun Sun ◽

Jie Hu ◽

Bin Wang ◽

...

Keyword(s):

Community Detection ◽

Real World ◽

Seed Set ◽

State Of The Art ◽

Ground Truth ◽

Detection Algorithm ◽

Original Algorithm ◽

Community Detection Algorithm ◽

Better Than

In recent years, community detection has gradually become a hot topic in the complex network data mining field. The research of community detection is helpful not only to understand network topology structure but also to explore network hiding function. In this paper, we improve FluidC which is a novel community detection algorithm based on fluid propagation, by ameliorating the quality of seed set based on positive feedback and determining the node update order. We first summarize the shortcomings of FluidC and analyze the reasons result in these drawbacks. Then, we took some effective measures to overcome them and proposed an efficient community detection algorithm, called FluidC+. Finally, experiments on the generated network and real-world network show that our method not only greatly improves the performance of the original algorithm FluidC but also is better than many state-of-the-art algorithms, especially in the performance on real-world network with ground truth.

Download Full-text