Efficient IoT Data Management for Geological Disasters Based on Big Data-Turbocharged Data Lake Architecture

Multi-source Internet of Things (IoT) data, archived in institutions’ repositories, are becoming more and more widely open-sourced to make them publicly accessed by scientists, developers, and decision makers via web services to promote researches on geohazards prevention. In this paper, we design and implement a big data-turbocharged system for effective IoT data management following the data lake architecture. We first propose a multi-threading parallel data ingestion method to ingest IoT data from institutions’ data repositories in parallel. Next, we design storage strategies for both ingested IoT data and processed IoT data to store them in a scalable, reliable storage environment. We also build a distributed cache layer to enable fast access to IoT data. Then, we provide users with a unified, SQL-based interactive environment to enable IoT data exploration by leveraging the processing ability of Apache Spark. In addition, we design a standard-based metadata model to describe ingested IoT data and thus support IoT dataset discovery. Finally, we implement a prototype system and conduct experiments on real IoT data repositories to evaluate the efficiency of the proposed system.

Designing RESTful API for the e-procurement system in private sector

The software for the e-procurement system was developed based on .NET Core RESTful API with Open API specifications. The server side uses RESTful API which ensures compatibility with the ma-jority of clients and enables them to exchange information in JSON format. The authentication and authorization flow was implemented using OAuth open standard paired with Microsoft Identity Service. User roles and functionality were handled with a standalone service for authentication and registration that made our system efficient and scalable. Business logic was designed to be split into micro-services accessible through rout-ing controllers. This approach allowed us to separate the responsibilities between the server and the client side. Special authorization headers passed during modi-fication queries allowed us to control and restrict access to particular resources for unauthorized users. The distributed cache mechanism inside the data repository level was used in order to increase the responsiveness of the system. The state handling subsystem was designed utilizing Finite State Machine concepts. The developed system was verified using unit and integration tests.