scholarly journals An Overview of Apache Pig and Apache Hive

2019 ◽  
pp. 432-436 ◽  
Author(s):  
Saiyam Arora ◽  
Abinesh Verma ◽  
Richa Vasuja ◽  
Richa Vasuja
Keyword(s):  
Big Data ◽  
Distributed Storage ◽  
Data Sets ◽  
Great Work ◽  
Apache Hadoop ◽  
The Social ◽  
Tremendous Amount ◽  
Hadoop Ecosystem ◽  
Apache Pig ◽  
Apache Hive

Ever since the enhancement of technology has taken place, the data is growing at an alarming rate. The most prominent factor of data growth is the “Social Media”, leads to the origination of a tremendous amount of data called Big Data. Big Data is a term used for data sets that are extremely large in size as well as complicated to store and process using traditional database processing applications. A saviour to deal with Big Data is “Hadoop” and two major components of Hadoop which are HDFS (Distributed Storage) and Map Reduce(Parallel Processing). Apache Pig and Hive is an essential part of the Hadoop Ecosystem. This paper covers an overview of both Apache Pig and Hive with their architecture. As Hadoop, no doubt is doing tremendously great work by storing and processing the huge volume of data but there are more frameworks now a days to increase the efficiency of Hadoop framework which are basically seen as the layers of Hadoop or a part of Apache Hadoop project. And that is why this paper includes the two most important layers namely Apache Pig and Apache Hive.

scholarly journals Bought and Sold: Exploring the Effects of Big Data on User Agency and Commodification

2021 ◽  
Author(s):  
Kristia M. Pavlakos
Keyword(s):  
Social Sciences ◽  
Big Data ◽  
Data Privacy ◽  
Large Data ◽  
Data Sets ◽  
Privacy Regulation ◽  
Scholarly Literature ◽  
User Interests ◽  
The Social ◽  
The Relationship

Big Data1is a phenomenon that has been increasingly studied in the academy in recent years, especially in technological and scientific contexts. However, it is still a relatively new field of academic study; because it has been previously considered in mainly technological contexts, more attention needs to be drawn to the contributions made in Big Data scholarship in the social sciences by scholars like Omar Tene and Jules Polonetsky, Bart Custers, Kate Crawford, Nick Couldry, and Jose van Dijk. The purpose of this Major Research Paper is to gain insight into the issues surrounding privacy and user rights, roles, and commodification in relation to Big Data in a social sciences context. The term “Big Data” describes the collection, aggregation, and analysis of large data sets. While corporations are usually responsible for the analysis and dissemination of the data, most of this data is user generated, and there must be considerations regarding the user’s rights and roles. In this paper, I raise three main issues that shape the discussion: how users can be more active agents in data ownership, how consent measures can be made to actively reflect user interests instead of focusing on benefitting corporations, and how user agency can be preserved. Through an analysis of social sciences scholarly literature on Big Data, privacy, and user commodification, I wish to determine how these concepts are being discussed, where there have been advancements in privacy regulation and the prevention of user commodification, and where there is a need to improve these measures. In doing this, I hope to discover a way to better facilitate the relationship between data collectors and analysts, and user-generators. 1 While there is no definitive resolution as to whether or not to capitalize the term “Big Data”, in capitalizing it I chose to conform with such authors as boyd and Crawford (2012), Couldry and Turow (2014), and Dalton and Thatcher (2015), who do so in the scholarly literature.

scholarly journals Bought and Sold: Exploring the Effects of Big Data on User Agency and Commodification

2021 ◽  
Author(s):  
Kristia M. Pavlakos
Keyword(s):  
Social Sciences ◽  
Big Data ◽  
Data Privacy ◽  
Large Data ◽  
Data Sets ◽  
Privacy Regulation ◽  
Scholarly Literature ◽  
User Interests ◽  
The Social ◽  
The Relationship

Big Data1is a phenomenon that has been increasingly studied in the academy in recent years, especially in technological and scientific contexts. However, it is still a relatively new field of academic study; because it has been previously considered in mainly technological contexts, more attention needs to be drawn to the contributions made in Big Data scholarship in the social sciences by scholars like Omar Tene and Jules Polonetsky, Bart Custers, Kate Crawford, Nick Couldry, and Jose van Dijk. The purpose of this Major Research Paper is to gain insight into the issues surrounding privacy and user rights, roles, and commodification in relation to Big Data in a social sciences context. The term “Big Data” describes the collection, aggregation, and analysis of large data sets. While corporations are usually responsible for the analysis and dissemination of the data, most of this data is user generated, and there must be considerations regarding the user’s rights and roles. In this paper, I raise three main issues that shape the discussion: how users can be more active agents in data ownership, how consent measures can be made to actively reflect user interests instead of focusing on benefitting corporations, and how user agency can be preserved. Through an analysis of social sciences scholarly literature on Big Data, privacy, and user commodification, I wish to determine how these concepts are being discussed, where there have been advancements in privacy regulation and the prevention of user commodification, and where there is a need to improve these measures. In doing this, I hope to discover a way to better facilitate the relationship between data collectors and analysts, and user-generators. 1 While there is no definitive resolution as to whether or not to capitalize the term “Big Data”, in capitalizing it I chose to conform with such authors as boyd and Crawford (2012), Couldry and Turow (2014), and Dalton and Thatcher (2015), who do so in the scholarly literature.

scholarly journals Integración de herramientas para la toma de decisiones en la congestión vehicular

DYNA ◽  
2018 ◽  
Vol 85 (205) ◽  
pp. 363-370
Author(s):  
Nelson Ivan Herrera-Herrera ◽  
Sergio Luján-Mora ◽  
Estevan Ricardo Gómez-Torres
Keyword(s):  
Big Data ◽  
Application Programming Interface ◽  
Apache Hadoop ◽  
Toma De Decisiones ◽  
Application Programming ◽  
La Red ◽  
Apache Pig ◽  
Programming Interface

Este estudio tiene como finalidad presentar un análisis de la utilización e integración de herramientas tecnológicas que ayudan a tomar decisiones en situaciones de congestión vehicular. La ciudad de Quito-Ecuador es considerada como un caso de estudio para el trabajo realizado. La investigación se presenta en función del desarrollo de una aplicación, haciendo uso de herramientas Big Data (Apache Flume, Apache Hadoop, Apache Pig), que permiten el procesamiento de gran cantidad de información que se requiere recolectar, almacenar y procesar. Uno de los aspectos innovadores de la aplicación es el uso de la red social Twitter como fuente de origen de datos. Para esto se utilizó su interfaz de programación de aplicaciones (Application Programming Interface, API), la cual permite tomar datos de esta red social en tiempo real e identificar puntos probables de congestión. Este estudio presenta resultados de pruebas realizadas con la aplicación, durante un período de 9 meses.

scholarly journals Implementation of Digital Signature Algorithm using Big Data Sensing Environment

2019 ◽  
Vol 9 (2) ◽  
pp. 4128-4131
Keyword(s):  
Big Data ◽  
Digital Signature ◽  
Distributed Storage ◽  
Data Retrieval ◽  
Map Reduce ◽  
Data Sets ◽  
Storage Process ◽  
Multiple Data ◽  
Digital Signature Algorithm ◽  
Multiple Data Sets

WBAN is a self-governing and perceptive used to informant the activities of a person and to improve the individuality of people, which satisfies the requirements of the user's needs. In this paper, we propose a Big data retrieval unit in WBAN using Elliptical Curve Cryptography. Big data transmit the data through Map reduce and retrieve the data safely using ECCDS algorithm. Map-reduce is a programming method for accessing multiple data sets on multi-node hardware efficiently using a distributed storage process and it incorporate the entire in-between requirements connected via the identical in-among key in . Cloud Sim extensible toolkit is used to enable the modeling and to enhance the application provision.

scholarly journals Efficient processing of complex XSD using Hive and Spark

2021 ◽  
Vol 7 ◽  
pp. e652
Author(s):  
Diana Martinez-Mosquera ◽  
Rosa Navarrete ◽  
Sergio Luján-Mora
Keyword(s):  
Big Data ◽  
Performance Management ◽  
Mobile Networks ◽  
Real Life ◽  
Real Data ◽  
Xml Schema ◽  
Apache Spark ◽  
Data Sets ◽  
Apache Hive

The eXtensible Markup Language (XML) files are widely used by the industry due to their flexibility in representing numerous kinds of data. Multiple applications such as financial records, social networks, and mobile networks use complex XML schemas with nested types, contents, and/or extension bases on existing complex elements or large real-world files. A great number of these files are generated each day and this has influenced the development of Big Data tools for their parsing and reporting, such as Apache Hive and Apache Spark. For these reasons, multiple studies have proposed new techniques and evaluated the processing of XML files with Big Data systems. However, a more usual approach in such works involves the simplest XML schemas, even though, real data sets are composed of complex schemas. Therefore, to shed light on complex XML schema processing for real-life applications with Big Data tools, we present an approach that combines three techniques. This comprises three main methods for parsing XML files: cataloging, deserialization, and positional explode. For cataloging, the elements of the XML schema are mapped into root, arrays, structures, values, and attributes. Based on these elements, the deserialization and positional explode are straightforwardly implemented. To demonstrate the validity of our proposal, we develop a case study by implementing a test environment to illustrate the methods using real data sets provided from performance management of two mobile network vendors. Our main results state the validity of the proposed method for different versions of Apache Hive and Apache Spark, obtain the query execution times for Apache Hive internal and external tables and Apache Spark data frames, and compare the query performance in Apache Hive with that of Apache Spark. Another contribution made is a case study in which a novel solution is proposed for data analysis in the performance management systems of mobile networks.

scholarly journals A Survey on Some Big Data Applications Tools and Technologies

2021 ◽  
Vol 9 (6) ◽  
pp. 239-242
Author(s):  
Nazia Tazeen ◽  
Sandhya Rani K.
Keyword(s):  
Decision Making ◽  
Big Data ◽  
Data Transmission ◽  
Data Sets ◽  
Complex Data ◽  
Broad Area ◽  
Apache Hadoop ◽  
Engineering Data ◽  
Big Data Applications ◽  
Science Media

Big Data is a broad area that deals with enormous chunks of data sets. It is a word for enormous data sets having huge volume, more diverse structures of data originating from diverse sources are growing rapidly. Many data being generated because of fast data transmission between devices concerning different sectors like healthcare, science, media, business, entertainment and engineering. Data collection capacity and its storage is big concern. Apache Hadoop software is a store of accessible source programs to store big data and perform analytics and various other operations related to big data. Many organizations base their decisions by extracting knowledge from huge and complex data, because of this prime cause of decision making, Big Data has to be accurately classified and analyzed. In order to overcome the complex challenges encountered by Big Data, various Big Data tools and technologies have developed. Big Data Applications, tools and technologies used to handle it are briefly discussed in this paper.

scholarly journals The research of social processes at the university using big data

2021 ◽  
Vol 348 ◽  
pp. 01003
Author(s):  
Abdullayev Vugar Hacimahmud ◽  
Ragimova Nazila Ali ◽  
Khalilov Matlab Etibar
Keyword(s):  
Big Data ◽  
Social Processes ◽  
Apache Hadoop ◽  
Big Data Applications ◽  
Big Data Technologies ◽  
Rapid Pace ◽  
Mapreduce Model ◽  
The University ◽  
Apache Pig ◽  
Apache Software Foundation

The volume of information in the 21st century is growing at a rapid pace. Big data technologies are used to process modern information. This article discusses the use of big data technologies to implement monitoring of social processes. Big data has its characteristics and principles, which reflect here. In addition, we also discussed big data applications in some areas. Particular attention in this article pays to the interactions of big data and sociology. For this, there consider digital sociology and computational social sciences. One of the main objects of study in sociology is social processes. The article shows the types of social processes and their monitoring. As an example, there is implemented monitoring of social processes at the university. There are used following technologies for the realization of social processes monitoring: products 1010data (1010edge, 1010connect, 1010reveal, 1010equities), products of Apache Software Foundation (Apache Hive, Apache Chukwa, Apache Hadoop, Apache Pig), MapReduce framework, language R, library Pandas, NoSQL, etc. Despite this, this article examines the use of the MapReduce model for social processes monitoring at the university.

Big Data

2021 ◽  
pp. 26-29
Author(s):  
Yannick Dufresne ◽  
Brittany I. Davidson
Keyword(s):  
Social Sciences ◽  
Social Media ◽  
Big Data ◽  
Online Communities ◽  
Data Analytics ◽  
Big Data Analytics ◽  
Data Sets ◽  
The Social ◽  
Data Volume ◽  
The Web

This chapter assesses big data. Within the social sciences, big data could refer to an emerging field of research that brings together academics from a variety of disciplines using and developing tools to widen perspective, to utilize latent data sets, as well as for the generation of new data. Another way to define big data in the social sciences refers to data corresponding to at least one of the three s of big data: volume, variety, or velocity.. These characteristics are widely used by researchers attempting to define and distinguish new types of data from conventional ones. However, there are a number of ethical and consent issues with big data analytics. For example, many studies across the social sciences utilize big data from the web, from social media, online communities, and the darknet, where there is a question as to whether users provided consent to the reuse of their posts, profiles, or other data shared when they signed up, knowing their profiles and information would be public. This has led to a number of issues regarding algorithms making decisions that cannot be explained. The chapter then considers the opportunities and pitfalls that come along with big data.

Big Data Analytics Using Apache Hive to Analyze Health Data

2022 ◽  
pp. 979-992
Author(s):  
Pavani Konagala
Keyword(s):  
Big Data ◽  
Stock Exchange ◽  
Big Data Analytics ◽  
Large Data ◽  
Massive Data ◽  
Data Sets ◽  
Related Data ◽  
Health Related ◽  
Relational Database Management ◽  
Apache Hive

A large volume of data is stored electronically. It is very difficult to measure the total volume of that data. This large amount of data is coming from various sources such as stock exchange, which may generate terabytes of data every day, Facebook, which may take about one petabyte of storage, and internet archives, which may store up to two petabytes of data, etc. So, it is very difficult to manage that data using relational database management systems. With the massive data, reading and writing from and into the drive takes more time. So, the storage and analysis of this massive data has become a big problem. Big data gives the solution for these problems. It specifies the methods to store and analyze the large data sets. This chapter specifies a brief study of big data techniques to analyze these types of data. It includes a wide study of Hadoop characteristics, Hadoop architecture, advantages of big data and big data eco system. Further, this chapter includes a comprehensive study of Apache Hive for executing health-related data and deaths data of U.S. government.

Sign in / Sign up

Export Citation Format

Share Document