A Framework for Using Redundant Data to Optimize Read-Intensive Database Applications

1997 ◽  
pp. 63-76 ◽  
Author(s):  
Timothy Griffin ◽  
Richard Hull ◽  
Bharat Kumar ◽  
Daniel Lieuwen ◽  
Gang Zhou
Keyword(s):  
Database Applications ◽  
Redundant Data

Using Machine Learning for Dependable Outlier Detection in Environmental Monitoring Systems

2021 ◽  
Vol 5 (3) ◽  
pp. 1-30
Author(s):  
Gonçalo Jesus ◽  
António Casimiro ◽  
Anabela Oliveira
Keyword(s):  
Machine Learning ◽  
Environmental Monitoring ◽  
Data Quality ◽  
Outlier Detection ◽  
Prediction Models ◽  
Sensor Data ◽  
Natural Phenomenon ◽  
Monitoring Systems ◽  
Data Errors ◽  
Redundant Data

Sensor platforms used in environmental monitoring applications are often subject to harsh environmental conditions while monitoring complex phenomena. Therefore, designing dependable monitoring systems is challenging given the external disturbances affecting sensor measurements. Even the apparently simple task of outlier detection in sensor data becomes a hard problem, amplified by the difficulty in distinguishing true data errors due to sensor faults from deviations due to natural phenomenon, which look like data errors. Existing solutions for runtime outlier detection typically assume that the physical processes can be accurately modeled, or that outliers consist in large deviations that are easily detected and filtered by appropriate thresholds. Other solutions assume that it is possible to deploy multiple sensors providing redundant data to support voting-based techniques. In this article, we propose a new methodology for dependable runtime detection of outliers in environmental monitoring systems, aiming to increase data quality by treating them. We propose the use of machine learning techniques to model each sensor behavior, exploiting the existence of correlated data provided by other related sensors. Using these models, along with knowledge of processed past measurements, it is possible to obtain accurate estimations of the observed environment parameters and build failure detectors that use these estimations. When a failure is detected, these estimations also allow one to correct the erroneous measurements and hence improve the overall data quality. Our methodology not only allows one to distinguish truly abnormal measurements from deviations due to complex natural phenomena, but also allows the quantification of each measurement quality, which is relevant from a dependability perspective. We apply the methodology to real datasets from a complex aquatic monitoring system, measuring temperature and salinity parameters, through which we illustrate the process for building the machine learning prediction models using a technique based on Artificial Neural Networks, denoted ANNODE ( ANN Outlier Detection ). From this application, we also observe the effectiveness of our ANNODE approach for accurate outlier detection in harsh environments. Then we validate these positive results by comparing ANNODE with state-of-the-art solutions for outlier detection. The results show that ANNODE improves existing solutions regarding accuracy of outlier detection.

scholarly journals Selecting Optimal Combination of Data Channels for Semantic Segmentation in City Information Modelling (CIM)

2021 ◽  
Vol 13 (7) ◽  
pp. 1367
Author(s):  
Yuanzhi Cai ◽  
Hong Huang ◽  
Kaiyang Wang ◽  
Cheng Zhang ◽  
Lei Fan ◽  
...  
Keyword(s):  
Large Scale ◽  
Semantic Segmentation ◽  
Optimal Combination ◽  
Reconstruction Technique ◽  
Process Data ◽  
Redundant Data ◽  
Multiple Data ◽  
Information Models ◽  
Efficient Data

Over the last decade, a 3D reconstruction technique has been developed to present the latest as-is information for various objects and build the city information models. Meanwhile, deep learning based approaches are employed to add semantic information to the models. Studies have proved that the accuracy of the model could be improved by combining multiple data channels (e.g., XYZ, Intensity, D, and RGB). Nevertheless, the redundant data channels in large-scale datasets may cause high computation cost and time during data processing. Few researchers have addressed the question of which combination of channels is optimal in terms of overall accuracy (OA) and mean intersection over union (mIoU). Therefore, a framework is proposed to explore an efficient data fusion approach for semantic segmentation by selecting an optimal combination of data channels. In the framework, a total of 13 channel combinations are investigated to pre-process data and the encoder-to-decoder structure is utilized for network permutations. A case study is carried out to investigate the efficiency of the proposed approach by adopting a city-level benchmark dataset and applying nine networks. It is found that the combination of IRGB channels provide the best OA performance, while IRGBD channels provide the best mIoU performance.

Reducing Layered Database Applications to their Essence through Vertical Integration

2015 ◽  
Vol 40 (3) ◽  
pp. 1-39 ◽  
Author(s):  
Kristian F. D. Rietveld ◽  
Harry A. G. Wijshoff
Keyword(s):  
Vertical Integration ◽  
Database Applications

scholarly journals A New Adaptive Method for the Extraction of Steel Design Structures from an Integrated Point Cloud

Sensors ◽  
2021 ◽  
Vol 21 (10) ◽  
pp. 3416
Author(s):  
Pawel Burdziakowski ◽  
Angelika Zakrzewska
Keyword(s):  
Data Integration ◽  
Point Cloud ◽  
Laser Scanning ◽  
Detection Threshold ◽  
Measurement Data ◽  
Steel Structures ◽  
Adaptive Method ◽  
Edge Extraction ◽  
Geometric Structures ◽  
Redundant Data

The continuous and intensive development of measurement technologies for reality modelling with appropriate data processing algorithms is currently being observed. The most popular methods include remote sensing techniques based on reflected-light digital cameras, and on active methods in which the device emits a beam. This research paper presents the process of data integration from terrestrial laser scanning (TLS) and image data from an unmanned aerial vehicle (UAV) that was aimed at the spatial mapping of a complicated steel structure, and a new automatic structure extraction method. We proposed an innovative method to minimize the data size and automatically extract a set of points (in the form of structural elements) that is vital from the perspective of engineering and comparative analyses. The outcome of the research was a complete technology for the acquisition of precise information with regard to complex and high steel structures. The developed technology includes such elements as a data integration method, a redundant data elimination method, integrated photogrammetric data filtration and a new adaptive method of structure edge extraction. In order to extract significant geometric structures, a new automatic and adaptive algorithm for edge extraction from a random point cloud was developed and presented herein. The proposed algorithm was tested using real measurement data. The developed algorithm is able to realistically reduce the amount of redundant data and correctly extract stable edges representing the geometric structures of a studied object without losing important data and information. The new algorithm automatically self-adapts to the received data. It does not require any pre-setting or initial parameters. The detection threshold is also adaptively selected based on the acquired data.

Design of New Intelligent Redundant Serial Bus with Isolated Driver in the Distributed Control System

2014 ◽  
Vol 527 ◽  
pp. 242-247
Author(s):  
Ling Wang ◽  
Bo Mo ◽  
Qin Hua Li ◽  
Hong Mei
Keyword(s):  
Control System ◽  
Distributed Control ◽  
Distributed Control System ◽  
Ip Core ◽  
Redundant Data ◽  
Working Together

In the distributed control system, to make sure the running of the system bus is long-term, stable and reliable, we design a new intelligent redundant serial bus as the system bus, and it will be introduced in this article. The new intelligent redundant serial bus (IRSBUS) is able to be configured into a dual redundant synchronous bus or a quadruple redundant asynchronous bus, and its redundancy is hot redundant that means two groups of the four signals are working together. We mainly design a redundant IP core, a redundant protocol and the byte format. The IP core of IRSBUS consists of the following major modules: Intelligent Signal Router, Signal Transceiver, Lines-connecting Status Detector, Synchronous Controller (Master), Synchronous Transponder (Slave), Redundant Results Decider, Interrupt Generator, Dual-port RAMs, Control and Status Registers, Master / Slave Redundant Logical Controller. The redundant protocol and byte format provide an extremely strict timing to synchronize the master-slaves and transmit the information. We show that our design allow to compare the redundant data to arrive at the correct results. It also provides a way to regroup the remaining signal lines into a system bus when one or two of the four signal lines are broken. And then it detects the lines-connection status every 100 milliseconds.

scholarly journals WWW interfaces for runtime relational database applications

2000 ◽  
Vol 48 (1-2) ◽  
pp. 63-68 ◽  
Author(s):  
Joshua Stillerman ◽  
Thomas W Fredian ◽  
Martin Greenwald
Keyword(s):  
Relational Database ◽  
Database Applications
Sign in / Sign up

Export Citation Format

Share Document