Using Grids for Distributed Knowledge Discovery

2011 ◽  
pp. 284-298 ◽  
Author(s):  
Antonio Congiusta ◽  
Domenico Talia ◽  
Paolo Trunfio
Keyword(s):  
Data Mining ◽  
Knowledge Discovery ◽  
High Performance ◽  
Data Transfer ◽  
Grid Services ◽  
Distributed Knowledge ◽  
Data Intensive ◽  
Knowledge Grid ◽  
Complex Knowledge ◽  
High Level

Knowledge discovery is a compute and data intensive process that allows for finding patterns, trends, and models in large datasets. The Grid can be effectively exploited for deploying knowledge discovery applications because of the high-performance it can offer and its distributed infrastructure. For effective use of Grids in knowledge discovery, the development of middleware is critical to support data management, data transfer, data mining and knowledge representation. To such purpose, we designed the Knowledge Grid, a high-level environment providing for Grid-based knowledge discovery tools and services. Such services allow users to create and manage complex knowledge discovery applications, composed as workflows that integrate data sources and data mining tools provided as distributed Grid services. This chapter describes the Knowledge Grid architecture and describes how its components can be used to design and implement distributed knowledge discovery applications. Then, the chapter describes how the Knowledge Grid services can be made accessible using the Open Grid Services Architecture (OGSA) model.

Data Mining and Knowledge Discovery in Databases

2019 ◽  
pp. 502-514 ◽  
Author(s):  
Ana Azevedo
Keyword(s):  
Data Mining ◽  
Knowledge Discovery ◽  
Knowledge Discovery In Databases ◽  
Great Relevance ◽  
The Subject ◽  
Inductive Databases ◽  
High Level

The term knowledge discovery in databases or KDD, for short, was coined in 1989 to refer to the broad process of finding knowledge in data, and to emphasize the “high-level” application of particular data mining (DM) methods. The DM phase concerns, mainly, the means by which the patterns are extracted and enumerated from data. Nowadays, the two terms are, usually, indistinctly used. Efforts are being developed in order to create standards and rules in the field of DM with great relevance being given to the subject of inductive databases. Within the context of inductive databases, a great relevance is given to the so-called DM languages. This chapter explores DM in KDD.

Arithmetic Research on Data Mining Technology and Associative Rules Mining

2014 ◽  
Vol 556-562 ◽  
pp. 3949-3951
Author(s):  
Jian Xin Zhu
Keyword(s):  
Data Mining ◽  
High Performance ◽  
Knowledge Engineering ◽  
Business Decisions ◽  
Depth Data ◽  
Source Data ◽  
Wide Availability ◽  
High Level ◽  
Performance Computing ◽  
Level Analysis

Data mining is a technique that aims to analyze and understand large source data reveal knowledge hidden in the data. It has been viewed as an important evolution in information processing. Why there have been more attentions to it from researchers or businessmen is due to the wide availability of huge amounts of data and imminent needs for turning such data into valuable information. During the past decade or over, the concepts and techniques on data mining have been presented, and some of them have been discussed in higher levels for the last few years. Data mining involves an integration of techniques from database, artificial intelligence, machine learning, statistics, knowledge engineering, object-oriented method, information retrieval, high-performance computing and visualization. Essentially, data mining is high-level analysis technology and it has a strong purpose for business profiting. Unlike OLTP applications, data mining should provide in-depth data analysis and the supports for business decisions.

Parallel and Distributed Data Mining through Parallel Skeletons and Distributed Objects

Data Mining ◽  
2011 ◽  
pp. 106-141
Author(s):  
Massimo Coppola ◽  
Marco Vanneschi
Keyword(s):  
Data Mining ◽  
High Performance ◽  
Distributed Databases ◽  
Sequential Algorithm ◽  
Distributed Data ◽  
Programming Environment ◽  
Programming Environments ◽  
Geographically Distributed ◽  
High Level ◽  
Performance Results

We consider the application of parallel programming environments to develop portable and efficient high performance data mining (DM) tools. We first assess the need of parallel and distributed DM applications, by pointing out the problems of scalability of some mining techniques and the need to mine large, eventually geographically distributed databases. We discuss the main issues of exploiting parallel and distributed computation for DM algorithms. A high-level programming language enhances the software engineering aspects of parallel DM, and it simplifies the problems of integration with existing sequential and parallel data management systems, thus leading to programming-efficient and high-performance implementations of applications. We describe a programming environment we have implemented that is based on the parallel skeleton model, and we examine the addition of object-like interfaces toward external libraries and system software layers. This kind of abstractions will be included in the forthcoming programming environment ASSIST. In the main part of the chapter, as a proof-of-concept we describe three well-known DM algorithms, Apriori, C4.5, and DBSCAN. For each problem, we explain the sequential algorithm and a structured parallel version, which is discussed and compared to parallel solutions found in the literature. We also discuss the potential gain in performance and expressiveness from the addition of external objects on the basis of the experiments we performed so far. We evaluate the approach with respect to performance results, design, and implementation considerations.

scholarly journals Building High-Resolution Sky Images Using the Cell/B.E.

2009 ◽  
Vol 17 (1-2) ◽  
pp. 113-134 ◽  
Author(s):  
Ana Lucia Varbanescu ◽  
Alexander S. van Amesfoort ◽  
Tim Cornwell ◽  
Ger van Diepen ◽  
Rob van Nieuwpoort ◽  
...  
Keyword(s):  
High Performance ◽  
Complete Solution ◽  
Performance Potential ◽  
Data Intensive ◽  
Irregular Data ◽  
Original Application ◽  
The One ◽  
High Level ◽  
Data Intensive Applications ◽  
Application Specific

The performance potential of the Cell/B.E., as well as its availability, have attracted a lot of attention from various high-performance computing (HPC) fields. While computation intensive kernels proved to be exceptionally well suited for running on the Cell, irregular data-intensive applications are usually considered as poor matches. In this paper, we present our complete solution for enabling such a data-intensive application to run efficiently on the Cell/B.E. processor. Specifically, we target radioastronomy data gridding and degridding, two resembling imaging filters based on convolutional resampling. Our solution is based on building a high-level application model, used to evaluate parallelization alternatives. Next, we choose the one with the best performance potential, and we gradually exploit this potential by applying platform-specific and application-specific optimizations. After several iterations, our target application shows a speed-up factor between 10 and 20 on a dual-Cell blade when compared with the original application running on a commodity machine. Given these results, and based on our empirical observations, we are able to pinpoint a set of ten guidelines for parallelizing similar applications on the Cell/B.E. Finally, we conclude the Cell/B.E. can provide high performance for data-intensive applications at the price of increased programming efforts and with a significant aid from aggressive application-specific optimizations.

scholarly journals Scientific workflows applied to the coupling of a continuum (Elmer v8.3) and a discrete element (HiDEM v1.0) ice dynamic model

2019 ◽  
Vol 12 (7) ◽  
pp. 3001-3015 ◽  
Author(s):  
Shahbaz Memon ◽  
Dorothée Vallot ◽  
Thomas Zwinger ◽  
Jan Åström ◽  
Helmut Neukirchen ◽  
...  
Keyword(s):  
Management System ◽  
High Performance ◽  
Heterogeneous Computing ◽  
Workflow Management ◽  
Scientific Workflow ◽  
Workflow Management System ◽  
Data Intensive ◽  
Cpu Utilization ◽  
Computing Environments ◽  
High Level

Abstract. Scientific computing applications involving complex simulations and data-intensive processing are often composed of multiple tasks forming a workflow of computing jobs. Scientific communities running such applications on computing resources often find it cumbersome to manage and monitor the execution of these tasks and their associated data. These workflow implementations usually add overhead by introducing unnecessary input/output (I/O) for coupling the models and can lead to sub-optimal CPU utilization. Furthermore, running these workflow implementations in different environments requires significant adaptation efforts, which can hinder the reproducibility of the underlying science. High-level scientific workflow management systems (WMS) can be used to automate and simplify complex task structures by providing tooling for the composition and execution of workflows – even across distributed and heterogeneous computing environments. The WMS approach allows users to focus on the underlying high-level workflow and avoid low-level pitfalls that would lead to non-optimal resource usage while still allowing the workflow to remain portable between different computing environments. As a case study, we apply the UNICORE workflow management system to enable the coupling of a glacier flow model and calving model which contain many tasks and dependencies, ranging from pre-processing and data management to repetitive executions in heterogeneous high-performance computing (HPC) resource environments. Using the UNICORE workflow management system, the composition, management, and execution of the glacier modelling workflow becomes easier with respect to usage, monitoring, maintenance, reusability, portability, and reproducibility in different environments and by different user groups. Last but not least, the workflow helps to speed the runs up by reducing model coupling I/O overhead and it optimizes CPU utilization by avoiding idle CPU cores and running the models in a distributed way on the HPC cluster that best fits the characteristics of each model.

scholarly journals Knowledge Discovery Web Service for Spatial Data Infrastructures

2020 ◽  
Vol 10 (1) ◽  
pp. 12
Author(s):  
Morteza Omidipoor ◽  
Ara Toomanian ◽  
Najmeh Neysani Samany ◽  
Ali Mansourian
Keyword(s):  
Data Mining ◽  
Knowledge Discovery ◽  
Web Service ◽  
Spatial Data ◽  
Spatial Clustering ◽  
Spatial Data Mining ◽  
Data Mining Techniques ◽  
Data Infrastructures ◽  
Spatial Data Infrastructures ◽  
High Level

The size, volume, variety, and velocity of geospatial data collected by geo-sensors, people, and organizations are increasing rapidly. Spatial Data Infrastructures (SDIs) are ongoing to facilitate the sharing of stored data in a distributed and homogeneous environment. Extracting high-level information and knowledge from such datasets to support decision making undoubtedly requires a relatively sophisticated methodology to achieve the desired results. A variety of spatial data mining techniques have been developed to extract knowledge from spatial data, which work well on centralized systems. However, applying them to distributed data in SDI to extract knowledge has remained a challenge. This paper proposes a creative solution, based on distributed computing and geospatial web service technologies for knowledge extraction in an SDI environment. The proposed approach is called Knowledge Discovery Web Service (KDWS), which can be used as a layer on top of SDIs to provide spatial data users and decision makers with the possibility of extracting knowledge from massive heterogeneous spatial data in SDIs. By proposing and testing a system architecture for KDWS, this study contributes to perform spatial data mining techniques as a service-oriented framework on top of SDIs for knowledge discovery. We implemented and tested spatial clustering, classification, and association rule mining in an interoperable environment. In addition to interface implementation, a prototype web-based system was designed for extracting knowledge from real geodemographic data in the city of Tehran. The proposed solution allows a dynamic, easier, and much faster procedure to extract knowledge from spatial data.

scholarly journals A DESCRIPTIVE FRAMEWORK FOR THE FIELD OF DATA MINING AND KNOWLEDGE DISCOVERY

2008 ◽  
Vol 07 (04) ◽  
pp. 639-682 ◽  
Author(s):  
YI PENG ◽  
GANG KOU ◽  
YONG SHI ◽  
ZHENGXIN CHEN
Keyword(s):  
Data Mining ◽  
Knowledge Discovery ◽  
Rapid Development ◽  
Theory Approach ◽  
Research Activities ◽  
Comprehensive Picture ◽  
Teaching Learning ◽  
High Level ◽  
Descriptive Framework ◽  
Grounded Theory Approach

Despite the rapid development, the field of data mining and knowledge discovery (DMKD) is still vaguely defined and lack of integrated descriptions. This situation causes difficulties in teaching, learning, research, and application. This paper surveys a large collection of DMKD literature to provide a comprehensive picture of current DMKD research and classify these research activities into high-level categories using grounded theory approach; it also evaluates the longitudinal changes of DMKD research activities during the last decade.

Sign in / Sign up

Export Citation Format

Share Document