Data Extraction Formulation for Efficient Data Synchronization Between Mobile Databases and Server-Side Database

Web scraping is also known as data scraping and it is used for extracting data from sites. The software used for this may directly access the World Wide Web by using the Hypertext Transfer Protocol or by using a web browser. Over the years, due to advancements in web development and its technology, various frameworks have come in use and almost all of websites are dynamic with their content being served from CMS. This makes it tough to extract data since there is no common template for extracting data. Hence, we use RSS. Rich Site Summary is a kind of timeline allowing users and also applications to gain access to the updates on websites in a standardized, computer-readable format. This project combines the use of RSS to extract data from websites and serve users in a robust and easy way. The differentiation is that this project uses server side caching to serve users almost instantaneously without the need to perform data extraction from the requested site all over again. This is done using Redis and Django.

Download Full-text

Innovative Access and Query Schemes for Mobile Databases and Data Warehouses

Handbook of Research on Innovations in Database Technologies and Applications ◽

10.4018/978-1-60566-242-8.ch091 ◽

2009 ◽

pp. 860-871

Author(s):

Alfredo Cuzzocrea

Keyword(s):

Data Structures ◽

Data Availability ◽

Mobile Environments ◽

Enterprise Information ◽

Mobile Databases ◽

Enterprise Information Systems ◽

Data Engineering ◽

Efficient Data ◽

Software Platforms ◽

New Research

Thanks to the explosion of the wireless technology, mobile environments are becoming the leading software platforms for extracting knowledge and interacting with enterprise information systems. Data and services availability at all times is the major benefit coming from such deployment scenario, but new research challenges pose serious limitations concerning data engineering issues. In fact, although if one can suppose that re-writing and re-adapting data structures, algorithms, and data reliability/dependability schemes is the natural way to support efficient data management on mobile environments, new issues and old limitations arise, particularly for what concerns with data availability and consistency in wireless network environments.

Download Full-text

Technical Considerations for Building a Landslide Tracker Mobile App

10.5194/egusphere-egu21-6019 ◽

2021 ◽

Author(s):

Ramesh Guntha ◽

Maneesha Vinodini Ramesh

Keyword(s):

High Performance ◽

Monsoon Season ◽

Performance Testing ◽

Mobile App ◽

Mobile Databases ◽

Synchronization Mechanism ◽

Server Side ◽

Repeated Calls ◽

The World ◽

Client Side

Substantially complete landslide inventories aid the accurate landslide modelling of a region&#8217;s susceptibility and landslide forecasting. Recording of landslides soon after they have occurred is important as their presence can be quickly erased (e.g., the landslide removed by people or through erosion/vegetation). In this paper, we present the technical software considerations that went into building a Landslide Tracker app to aid in the collection of landslide information by non-technical local citizens, trained volunteers, and experts to create more complete inventories on a real-time basis through the model of crowdsourcing. The tracked landslide information is available for anyone across the world to view. This app is available on Google Play Store for free, and at http://landslides.amrita.edu, with software conceived and developed by Amrita University in the context of the UK NERC/FCDO funded LANDSLIP research project (http://www.landslip.org/).The three technical themes we discuss in this paper are the following: (i) security, (ii) performance, and (iii) network resilience. (i) Security considerations include authentication, authorization, and client/server-side enforcement. Authentication allows only the registered users to record and view the landslides, whereas authorization protects the data from illegal access. For example, landslides created by one user are not editable by others, and no user should be able to delete landslides. This validation is enforced at the client-side (mobile and web apps) and also at the server-side software to prevent unintentional and intentional illegal access. (ii) Performance considerations include designing high-performance data structures, mobile databases, client-side caching, server-side caching, cache synchronization, and push-notifications. The database is designed to ensure the best performance without sacrificing data integrity. Then the read-heavy data is cached in memory to get this data with very low latency. Similarly, the data, once fetched, is cached in memory on the app so that it can be re-used without making repeated calls to the server every time when the user visits a screen.&#160; The data persists in the mobile database so the app can load faster while reopening. A cache-synchronization mechanism is implemented to prevent the caches' data from becoming stale as new data comes into the database. The synchronization mechanism consists of push-notifications and incremental data pulls. (iii) Network resiliency considerations are achieved with the help of local storage on the app. This allows recording the landslides even when there is no internet connection. The app automatically pushes the updates to the server as soon as the connectivity resumes. We have observed over 300% reduction in time taken to load 2000 landslides, between the no-cache mode to cache mode during the performance testing.&#160;The Landslide tracker app was released during the 2020 monsoon season and more than 250 landslides were recorded through the app across India and the world.

Download Full-text

Locality-Based Visual Outlier Detection Algorithm for Time Series

Security and Communication Networks ◽

10.1155/2017/1869787 ◽

2017 ◽

Vol 2017 ◽

pp. 1-10

Author(s):

Zhihua Li ◽

Ziyuan Li ◽

Ning Yu ◽

Steven Wen

Keyword(s):

Time Series ◽

Outlier Detection ◽

Extraction Method ◽

Time Series Data ◽

Data Extraction ◽

Detection Algorithm ◽

Series Data ◽

Practical Application ◽

Detection Model ◽

Efficient Data

Physiological theories indicate that the deepest impression for time series data with respect to the human visual system is its extreme value. Based on this principle, by researching the strategies of extreme-point-based hierarchy segmentation, the hierarchy-segmentation-based data extraction method for time series, and the ideas of locality outlier, a novel outlier detection model and method for time series are proposed. The presented algorithm intuitively labels an outlier factor to each subsequence in time series such that the visual outlier detection gets relatively direct. The experimental results demonstrate the average advantage of the developed method over the compared methods and the efficient data reduction capability for time series, which indicates the promising performance of the proposed method and its practical application value.

Download Full-text