scholarly journals Towards a Concept for Building a Big Data Architecture with Microservices

2021 ◽  
pp. 83-94
Author(s):  
Aamir Shakir ◽  
Daniel Staegemann ◽  
Matthias Volk ◽  
Naoum Jamous ◽  
Klaus Turowski

Microservices and Big Data are renowned hot topics in computer science that have gained a lot of hype. While the use of microservices is an approach that is used in modern software development to increase flexibility, Big Data allows organizations to turn today’s information deluge into valuable insights. Many of those Big Data architectures have rather monolithic elements. However, a new trend arises in which monolithic architectures are replaced with more modularized ones, such as microservices. This transformation provides the benefits from microservices such as modularity, evolutionary design and extensibility while maintaining the old monolithic product’s functionality. This is also valid for Big Data architectures. To facilitate the success of this transformation, there are certain beneficial factors. In this paper, those aspects will be presented and the transformation of an exemplary Big Data architecture with somewhat monolithic elements into a microservice favoured one is outlined.

Author(s):  
Michael Goul ◽  
T. S. Raghu ◽  
Ziru Li

As procurement organizations increasingly move from a cost-and-efficiency emphasis to a profit-and-growth emphasis, flexible data architecture will become an integral part of a procurement analytics strategy. It is therefore imperative for procurement leaders to understand and address digitization trends in supply chains and to develop strategies to create robust data architecture and analytics strategies for the future. This chapter assesses and examines the ways companies can organize their procurement data architectures in the big data space to mitigate current limitations and to lay foundations for the discovery of new insights. It sets out to understand and define the levels of maturity in procurement organizations as they pertain to the capture, curation, exploitation, and management of procurement data. The chapter then develops a framework for articulating the value proposition of moving between maturity levels and examines what the future entails for companies with mature data architectures. In addition to surveying the practitioner and academic research literature on procurement data analytics, the chapter presents detailed and structured interviews with over fifteen procurement experts from companies around the globe. The chapter finds several important and useful strategies that have helped procurement organizations design strategic roadmaps for the development of robust data architectures. It then further identifies four archetype procurement area data architecture contexts. In addition, this chapter details exemplary high-level mature data architecture for each archetype and examines the critical assumptions underlying each one. Data architectures built for the future need a design approach that supports both descriptive and real-time, prescriptive analytics.


Author(s):  
M.Dolores Ruiz ◽  
Juan Gomez-Romero ◽  
Carlos Fernandez-Basso ◽  
Maria J. Martin-Bautista

Complexity ◽  
2019 ◽  
Vol 2019 ◽  
pp. 1-13
Author(s):  
Cristina Sánchez-Rebollo ◽  
Cristina Puente ◽  
Rafael Palacios ◽  
Claudia Piriz ◽  
Juan P. Fuentes ◽  
...  

Social networks are being used by terrorist organizations to distribute messages with the intention of influencing people and recruiting new members. The research presented in this paper focuses on the analysis of Twitter messages to detect the leaders orchestrating terrorist networks and their followers. A big data architecture is proposed to analyze messages in real time in order to classify users according to different parameters like level of activity, the ability to influence other users, and the contents of their messages. Graphs have been used to analyze how the messages propagate through the network, and this involves a study of the followers based on retweets and general impact on other users. Then, fuzzy clustering techniques were used to classify users in profiles, with the advantage over other classifications techniques of providing a probability for each profile instead of a binary categorization. Algorithms were tested using public database from Kaggle and other Twitter extraction techniques. The resulting profiles detected automatically by the system were manually analyzed, and the parameters that describe each profile correspond to the type of information that any expert may expect. Future applications are not limited to detecting terrorist activism. Human resources departments can apply the power of profile identification to automatically classify candidates, security teams can detect undesirable clients in the financial or insurance sectors, and immigration officers can extract additional insights with these techniques.


10.28945/4553 ◽  
2020 ◽  
Vol 19 ◽  
pp. 339-365
Author(s):  
Yasar Guneri Sahin ◽  
Ufuk Celikkan

Aim/Purpose: This paper investigates the gaps between industry and academia perceptions of information technology fields, such as computer science, software engineering, and computer engineering, and it identifies areas of asymmetry between curricula and industry expectations. The study mainly focuses on the skills required of IT professionals (graduated students) and on how higher education institutes equip students for industry. Background: Higher education institutes have several IT-related departments. However, it is not clear whether these departments have sufficient content to equip students with industry-related skills. Rapid advances mean that some curriculum topics are redundant before the end of a standard two- or four-year degree programs. Balancing the technical/non-technical skills and adjusting the curricula to better prepare the students for industry is a constant demand for higher education institutions. Several studies have demonstrated that a generic curriculum is inadequate to address current IT industry needs. Methodology: The study involved a comprehensive survey of IT professionals and companies using a Web-based questionnaire sent directly to individual companies, academics, and employers. 64 universities and 38 companies in 24 countries were represented by the 209 participants, of whom 99 were IT professionals, 72 academics, and 38 employers. Contribution: This paper is intended to guide academics in preparing dynamic curricula that can be easily adapted to current industry trends and technological developments, with content directly relevant to student’s careers. In addition, the results may identify the skills that students need to secure employment and the courses that will provide skills in line with current industry trends. Findings: The results indicate a lack of emphasis on personal and non-technical skills in undergraduate education compared to general computer science, software development, and coding courses. Employers’ and software experts’ responses emphasize that soft skills should not be ignored, and that, of these, analytical thinking and teamwork are the two most requested. Rather than a theoretical emphasis, courses should include hands-on projects. Rapid developments and innovations in information technologies demand that spiral and waterfall models are replaced with emerging software development models, such as Agile and Scrum development. Recommendations for Practitioners: A multidisciplinary approach should be taken to the teaching of soft skills, such as communication, ethics, leadership, and customer relations. Establishing multiple learning tracks in IT education would equip students with specialized knowledge and skills in IT. An effective communication channel should be established between students and industry. It is also important to reduce the distance between academics and students and to provide an interactive environment for technical discussions. Enterprise level computing and Framework use provide job market advantages. Recommendation for Researchers: Researchers and department heads, particularly those involved in curriculum design and accreditation, could use the results of this exemplary study to identify key topics for attention. Impact on Society: Changes of various degrees are required in the current curricula in many higher education institutions to better meet student needs. Societies and technology are dynamic in nature, and information technology-related curricula in higher education institutions should be equally dynamic. Future Research: Since technology (especially information technology) transforms and advances itself so rapidly, this study should be replicated t to investigate how these changes affect the gap between revised curricula and current industry expectations.


Author(s):  
Haixuan Zhu ◽  
◽  
Xiaoyu Jia ◽  
Pengluo Que ◽  
Xiaoyu Hou ◽  
...  

In the era of big data, with the development of computer technology, especially the comprehensive popularization of mobile terminal device and the gradual construction of the Internet of Things, the urban physical environment and social environment have been comprehensively digitized and quantified. Computational thinking mode has gradually become a new thinking mode for human beings to recognize and govern urban complex system. Meanwhile computational urban science has become the main discipline development aspect of modern urban planning. Computational thinking is the thinking of computer science using algorithms based on time complexity and space complexity, which provides a new paradigm for the construction of index system, data collection, data storage, data analysis, pattern recognition, dynamic governance in the process of scientific planning and urban management. Based on this, this paper takes the computational thinking mode of urban planning discipline in big data era as the research object, takes the scientific construction of computational urban planning as the research purpose, and adopts literature research methods and interdisciplinary research methods, comprehensively studies the connotation of the computing thinking mode of computer science. Meanwhile, this paper systematically discusses the system construction of urban computing, model generation, the theory and method of digital twinning, as well as the popularization of the computational thinking mode of urban and rural planning discipline and the scientific research of computational urban planning, which responds to the needs of the era of the development of urban and rural planning disciplines in the era of big data.


Bank marketers still have difficulties to find the best implementation for credit card promotion using above the line, particularly based on customers preferences in point of interest (POI) locations such as mall and shopping center. On the other hand, customers on those POIs are keen to have recommendation on what is being offered by the bank. On this paper we propose a design architecture and implementation of big data platform to support bank’s credit card’s program campaign that generating data and extracting topics from Twitter. We built a data pipeline that consist of a Twitter streamer, a text preprocessor, a topic extractor using Latent Dirichlet Allocation, and a dashboard that visualize the recommendation. As a result, we successfully generate topics that related to specific location in Jakarta during some time windows, that can be used as a recommendation for bank marketers to create promotion program for their customers. We also present the analysis of computing power usages that indicates the strategy is well implemented on the big data platform.


2017 ◽  
Vol 13 (02) ◽  
pp. 159-180 ◽  
Author(s):  
Mihai Horia Zaharia

Presented in this paper is a possible solution for speeding up the integration of various data in the big data mainstream. The data enrichment and convergence of all possible sources is still at the beginning. As a result, existing techniques must be retooled in order to increase the integration of already existing databases or of the ones specific to Internet of Things in order to use the advantages of the big data to fulfill the final goal of web of data creation. In this paper, semantic web-specific solutions are used to design a system based on intelligent agents. It tries to solve some problems specific to automation of the database migration system with the final goal of creating a common ontology over various data repositories or producers in order to integrate them into systems based on big data architecture.


Sign in / Sign up

Export Citation Format

Share Document