Telco Data Analytics using Open-Source Data Pipeline: Detailed Architecture and Technology Stack

Abstract: Open-source technology has influenced data analytics at each step from data storage to data analysis, and visualization. Open source for telco big data analytics enables sharp insights by enhancing problem discoverability and solution feasibility. This research paper talks about different technology stacks using open source for telco big data analytics that are used to deploy various tools including data collection, data storage, data processing, data analysis, and data visualization. This open source pipeline micro-services architecture built with modular technology stack and orchestrated by Kubernetes, can ingest data from multiple sources, process real-time data and provide business and network intelligence. Major idea of using open source technology in our architecture is to reduce cost and manage easily. Kubernetes is an industry adopted open source container orchestrator that offers fault-tolerance, application scaling, and load-balancing. The results can be displayed on the intuitive open source dashboard like Grafana for telecom operators. Our architecture is flexible and can be easily customized based on the telecommunication industry needs. Using the proposed architecture, the telecommunication sectors can get quick decision making with nearly 30% lower CapEX which is made possible using COTS hardware. Index Terms: Big data analytics, Data pipeline architecture, Open Source technologies, Real-time data processing, Faulttolerance, Load-balancing, Kubernetes, BDA, Open source dashboard

Download Full-text

Urban Planning and Smart City Decision Management Empowered by Real-Time Data Processing Using Big Data Analytics

Sensors ◽

10.3390/s18092994 ◽

2018 ◽

Vol 18 (9) ◽

pp. 2994 ◽

Cited By ~ 28

Author(s):

Bhagya Silva ◽

Murad Khan ◽

Changsu Jung ◽

Jihun Seo ◽

Diyan Muhammad ◽

...

Keyword(s):

Big Data ◽

Data Processing ◽

Real Time ◽

Smart City ◽

Data Analytics ◽

Smart Cities ◽

Big Data Analytics ◽

Time Data ◽

Real Time Data

The Internet of Things (IoT), inspired by the tremendous growth of connected heterogeneous devices, has pioneered the notion of smart city. Various components, i.e., smart transportation, smart community, smart healthcare, smart grid, etc. which are integrated within smart city architecture aims to enrich the quality of life (QoL) of urban citizens. However, real-time processing requirements and exponential data growth withhold smart city realization. Therefore, herein we propose a Big Data analytics (BDA)-embedded experimental architecture for smart cities. Two major aspects are served by the BDA-embedded smart city. Firstly, it facilitates exploitation of urban Big Data (UBD) in planning, designing, and maintaining smart cities. Secondly, it occupies BDA to manage and process voluminous UBD to enhance the quality of urban services. Three tiers of the proposed architecture are liable for data aggregation, real-time data management, and service provisioning. Moreover, offline and online data processing tasks are further expedited by integrating data normalizing and data filtering techniques to the proposed work. By analyzing authenticated datasets, we obtained the threshold values required for urban planning and city operation management. Performance metrics in terms of online and offline data processing for the proposed dual-node Hadoop cluster is obtained using aforementioned authentic datasets. Throughput and processing time analysis performed with regard to existing works guarantee the performance superiority of the proposed work. Hence, we can claim the applicability and reliability of implementing proposed BDA-embedded smart city architecture in the real world.

Download Full-text

Big data: The effect of analytics on marketing and business

Journal of Information Technology Teaching Cases ◽

10.1177/20438869211057284 ◽

2021 ◽

pp. 204388692110572

Author(s):

Barbara A. Manko

Keyword(s):

Social Media ◽

Big Data ◽

Data Analytics ◽

Big Data Analytics ◽

Time Data ◽

Bottom Line ◽

Real Time Data ◽

The Subject ◽

Successful Business

Big data analytics takes raw, real-time data and uses it to predict trends. Successful use of this data can have a powerful impact on a business’s effectiveness and ultimately their bottom line. As the amount of data increases, the need for analytics is growing. This teaching study discusses the role of social media in data analytics, how to approach the subject, and the desired outcomes. Students will explore the expansion of this field of study, familiarize themselves with the concept and where they may have encountered it in their lives so far, and discuss what analytics can contribute to running a successful business.

Download Full-text

Toward real-time data processing: an advanced approach in big data analytics

10.1049/pbpc037f_ch5 ◽

2021 ◽

pp. 157-174

Author(s):

Shafqat Ul Ahsaan ◽

Harleen Kaur ◽

Sameena Naaz

Keyword(s):

Big Data ◽

Data Processing ◽

Real Time ◽

Data Analytics ◽

Big Data Analytics ◽

Time Data ◽

Real Time Data ◽

Real Time Data Processing

Download Full-text

Big Data Analytics Demystified

Big Data ◽

10.4018/978-1-4666-9840-6.ch086 ◽

2016 ◽

pp. 1859-1894

Author(s):

Pethuru Raj

Keyword(s):

Big Data ◽

Data Analytics ◽

Big Data Analytics ◽

Use Cases ◽

Sensor Data ◽

New Paradigm ◽

Time Data ◽

Real Time Data ◽

Major Application

This chapter is mainly crafted in order to give a business-centric view of big data analytics. The readers can find the major application domains / use cases of big data analytics and the compelling needs and reasons for wholeheartedly embracing this new paradigm. The emerging use cases include the use of real-time data such as the sensor data to detect any abnormalities in plant and machinery and batch processing of sensor data collected over a period to conduct failure analysis of plant and machinery. The author describes the short-term as well as the long-term benefits and find and nullify all kinds of doubts and misgivings on this new idea, which has been pervading and penetrating into every tangible domain. The ultimate goal is to demystify this cutting-edge technology so that its acceptance and adoption levels go up significantly in the days to unfold.

Download Full-text

Big Data Analytics Demystified

Advances in Data Mining and Database Management - Handbook of Research on Cloud Infrastructures for Big Data Analytics ◽

10.4018/978-1-4666-5864-6.ch003 ◽

2014 ◽

pp. 38-73 ◽

Cited By ~ 2

Author(s):

Pethuru Raj

Keyword(s):

Big Data ◽

Data Analytics ◽

Big Data Analytics ◽

Use Cases ◽

Sensor Data ◽

New Paradigm ◽

Time Data ◽

Real Time Data ◽

Major Application

Download Full-text

An Intelligent Big Data Analytics System using Enhanced Map Reduce Techniques

International Journal of Innovative Technology and Exploring Engineering - Special Issue ◽

10.35940/ijitee.b1105.1292s219 ◽

2019 ◽

Vol 9 (2S2) ◽

pp. 1006-1010

Keyword(s):

Big Data ◽

Data Analysis ◽

Data Storage ◽

Data Analytics ◽

Data Privacy ◽

Big Data Analytics ◽

Map Reduce ◽

K Nearest Neighbor ◽

Real World Data ◽

Effective Manner

An Intelligent Big Data Analytics System using Enhanced Map Reduce Techniques include a set of Methods, applications and strategy which helps the organization and industry to bring together the data and information from outside sources and internal systems, as well as it is used to collect , classify, analysis and run the queries against the data and prepare the report for effective decision making. The Enhanced Map Reduced Techniques based on K-Nearest Neighbor (KNN) clustering Strategy works efficient as well as in an effective manner. We found that the existing MR – mafia sub space clustering Strategy have not performed effectively .Many clustering techniques are adopted in real world data analysis for example customer behavior analysis, medical data analysis, digital forensics, etc. The existing MR- mafia sub space clustering Strategy is inefficient because of continuously increase in the data size, and overlaying of the data blocks .The proposed KNN clustering Strategy mainly focused on the enhanced the Map Reduce techniques, and then to avoid the unnecessary input and output data, optimize the data storage in order to achieve the best out sourcing of data privacy. The proposed KNN clustering Strategy works effectively and that can be outsourced to cloud server.

Download Full-text

The Art of Data Science and Big Data Analytics: Inspecting and Transforming Data

Asian Journal of Computer Science and Technology ◽

10.51983/ajcst-2020.9.1.2151 ◽

2020 ◽

Vol 9 (1) ◽

pp. 45-56

Author(s):

Akella Subhadra

Keyword(s):

Big Data ◽

Data Analysis ◽

Data Storage ◽

Data Analytics ◽

Data Science ◽

New Technologies ◽

Big Data Analytics ◽

Business Strategies ◽

Science Data ◽

Using Data

Data Science is associated with new discoveries, the discovery of value from the data. It is a practice of deriving insights and developing business strategies through transformation of data in to useful information. It has been evaluated as a scientific field and research evolution in disciplines like statistics, computing science, intelligence science, and practical transformation in the domains like science, engineering, public sector, business and lifestyle. The field encompasses the larger areas of artificial intelligence, data analytics, machine learning, pattern recognition, natural language understanding, and big data manipulation. It also tackles related new scientific challenges, ranging from data capture, creation, storage, retrieval, sharing, analysis, optimization, and visualization, to integrative analysis across heterogeneous and interdependent complex resources for better decision-making, collaboration, and, ultimately, value creation. In this paper we entitled epicycles of analysis, formal modeling, from data analysis to data science, data analytics -A keystone of data science, The Big data is not a single technology but an amalgamation of old and new technologies that assistance companies gain actionable awareness. The big data is vital because it manages, store and manipulates large amount of data at the desirable speed and time. Big data addresses detached requirements, in other words the amalgamate of multiple un-associated datasets, processing of large amounts of amorphous data and harvesting of unseen information in a time-sensitive generation. As businesses struggle to stay up with changing market requirements, some companies are finding creative ways to use Big Data to their growing business needs and increasingly complex problems. As organizations evolve their processes and see the opportunities that Big Data can provide, they struggle to beyond traditional Business Intelligence activities, like using data to populate reports and dashboards, and move toward Data Science- driven projects that plan to answer more open-ended and sophisticated questions. Although some organizations are fortunate to have data scientists, most are not, because there is a growing talent gap that makes finding and hiring data scientists in a timely manner is difficult. This paper, aimed to demonstrate a close view about Data science, big data, including big data concepts like data storage, data processing, and data analysis of these technological developments, we also provide brief description about big data analytics and its characteristics , data structures, data analytics life cycle, emphasizes critical points on these issues.

Download Full-text