scholarly journals How to Draw a Neighborhood? The Potential of Big Data, Regionalization, and Community Detection for Understanding the Heterogeneous Nature of Urban Neighborhoods.

2019 ◽  
Author(s):  
Ate Poorthuis
Keyword(s):  
Big Data ◽  
Social Life ◽  
Technical Problem ◽  
Data Availability ◽  
Data Sets ◽  
Temporal Scales ◽  
Data Set ◽  
Heterogeneous Nature ◽  
Spatio Temporal ◽  
Neighborhood Boundaries

How to draw neighborhood boundaries, or spatial regions in general, has been a long‐standing focus in Geography. This article examines this question from a methodological perspective, often referred to as regionalization, with an empirical study of neighborhoods in New York City. I argue that methodological advances, combined with the affordances of big data, enable a different, more nuanced approach to regionalization than has been possible in the past. Conventional data sets often dictate constraints in terms of data availability and spatio‐temporal granularity. However, big data is now available at much finer spatio‐temporal scales and covers a wider array of aspects of social life. The emergence of these data sets supports the notion that neighborhoods can be fuzzy and highly dependent on spatio‐temporal scales and socio‐economic variables. As such, these new data sets can help to bring quantitative analysis in line with social theory that has long emphasized the heterogeneous nature of neighborhoods. This article uses a data set of geotagged tweets to demonstrate how different “sets” of neighborhoods may exist at different spatio‐temporal scales and for different algorithms. Such varying neighborhood boundaries are not a technical problem in need of a solution but rather a reflection of the complexity of the underlying urban fabric.

scholarly journals INTERESTING SPATIO-TEMPORAL REGION DISCOVERY COMPUTATIONS OVER GPU AND MAPREDUCE PLATFORMS

2015 ◽  
Vol II-4/W2 ◽  
pp. 35-41 ◽  
Author(s):  
M. McDermott ◽  
S. K. Prasad ◽  
S. Shekhar ◽  
X. Zhou
Keyword(s):  
Parallel Architecture ◽  
Computation Time ◽  
Data Sets ◽  
Temporal Region ◽  
Data Set ◽  
Atmospheric Sciences ◽  
3 Dimensional ◽  
Linear Speedup ◽  
Spatio Temporal ◽  
Mapreduce Model

Discovery of interesting paths and regions in spatio-temporal data sets is important to many fields such as the earth and atmospheric sciences, GIS, public safety and public health both as a goal and as a preliminary step in a larger series of computations. This discovery is usually an exhaustive procedure that quickly becomes extremely time consuming to perform using traditional paradigms and hardware and given the rapidly growing sizes of today’s data sets is quickly outpacing the speed at which computational capacity is growing. In our previous work (Prasad et al., 2013a) we achieved a 50 times speedup over sequential using a single GPU. We were able to achieve near linear speedup over this result on interesting path discovery by using Apache Hadoop to distribute the workload across multiple GPU nodes. Leveraging the parallel architecture of GPUs we were able to drastically reduce the computation time of a 3-dimensional spatio-temporal interest region search on a single tile of normalized difference vegetative index for Saudi Arabia. We were further able to see an almost linear speedup in compute performance by distributing this workload across several GPUs with a simple MapReduce model. This increases the speed of processing 10 fold over the comparable sequential while simultaneously increasing the amount of data being processed by 384 fold. This allowed us to process the entirety of the selected data set instead of a constrained window.

scholarly journals Dynamic spatio-temporal generation of large-scale synthetic gridded precipitation: with improved spatial coherence of extremes

2019 ◽  
Vol 34 (9) ◽  
pp. 1369-1383 ◽  
Author(s):  
Dirk Diederen ◽  
Ye Liu
Keyword(s):  
Large Scale ◽  
Spatial Coherence ◽  
Original Data ◽  
Return Level ◽  
Data Sets ◽  
Large Set ◽  
Precipitation Data ◽  
Data Set ◽  
Spatio Temporal ◽  
Synthetic Precipitation

Abstract With the ongoing development of distributed hydrological models, flood risk analysis calls for synthetic, gridded precipitation data sets. The availability of large, coherent, gridded re-analysis data sets in combination with the increase in computational power, accommodates the development of new methodology to generate such synthetic data. We tracked moving precipitation fields and classified them using self-organising maps. For each class, we fitted a multivariate mixture model and generated a large set of synthetic, coherent descriptors, which we used to reconstruct moving synthetic precipitation fields. We introduced randomness in the original data set by replacing the observed precipitation fields in the original data set with the synthetic precipitation fields. The output is a continuous, gridded, hourly precipitation data set of a much longer duration, containing physically plausible and spatio-temporally coherent precipitation events. The proposed methodology implicitly provides an important improvement in the spatial coherence of precipitation extremes. We investigate the issue of unrealistic, sudden changes on the grid and demonstrate how a dynamic spatio-temporal generator can provide spatial smoothness in the probability distribution parameters and hence in the return level estimates.

Development of a Powerful Data-Analysis Tool Using Nonparametric Smoothing Models To Identify Drillsites in Tight Shale Reservoirs With High Economic Potential

SPE Journal ◽  
2017 ◽  
Vol 23 (03) ◽  
pp. 719-736 ◽  
Author(s):  
Quan Cai ◽  
Wei Yu ◽  
Hwa Chi Liang ◽  
Jenn-Tai Liang ◽  
Suojin Wang ◽  
...  
Keyword(s):  
Big Data ◽  
Data Analysis ◽  
Predictive Power ◽  
Oil And Gas ◽  
Predictor Variables ◽  
Data Sets ◽  
Analysis Tool ◽  
Data Set ◽  
Data Analysis Tool ◽  
Shale Reservoirs

Summary The oil-and-gas industry is entering an era of “big data” because of the huge number of wells drilled with the rapid development of unconventional oil-and-gas reservoirs during the past decade. The massive amount of data generated presents a great opportunity for the industry to use data-analysis tools to help make informed decisions. The main challenge is the lack of the application of effective and efficient data-analysis tools to analyze and extract useful information for the decision-making process from the enormous amount of data available. In developing tight shale reservoirs, it is critical to have an optimal drilling strategy, thereby minimizing the risk of drilling in areas that would result in low-yield wells. The objective of this study is to develop an effective data-analysis tool capable of dealing with big and complicated data sets to identify hot zones in tight shale reservoirs with the potential to yield highly productive wells. The proposed tool is developed on the basis of nonparametric smoothing models, which are superior to the traditional multiple-linear-regression (MLR) models in both the predictive power and the ability to deal with nonlinear, higher-order variable interactions. This data-analysis tool is capable of handling one response variable and multiple predictor variables. To validate our tool, we used two real data sets—one with 249 tight oil horizontal wells from the Middle Bakken and the other with 2,064 shale gas horizontal wells from the Marcellus Shale. Results from the two case studies revealed that our tool not only can achieve much better predictive power than the traditional MLR models on identifying hot zones in the tight shale reservoirs but also can provide guidance on developing the optimal drilling and completion strategies (e.g., well length and depth, amount of proppant and water injected). By comparing results from the two data sets, we found that our tool can achieve model performance with the big data set (2,064 Marcellus wells) with only four predictor variables that is similar to that with the small data set (249 Bakken wells) with six predictor variables. This implies that, for big data sets, even with a limited number of available predictor variables, our tool can still be very effective in identifying hot zones that would yield highly productive wells. The data sets that we have access to in this study contain very limited completion, geological, and petrophysical information. Results from this study clearly demonstrated that the data-analysis tool is certainly powerful and flexible enough to take advantage of any additional engineering and geology data to allow the operators to gain insights on the impact of these factors on well performance.

scholarly journals A multi-decade record of high-quality fCO<sub>2</sub> data in version 3 of the Surface Ocean CO<sub>2</sub> Atlas (SOCAT)

2016 ◽  
Author(s):  
Dorothee C. E. Bakker ◽  
Benjamin Pfeil ◽  
Camilla S. Landa ◽  
Nicolas Metzl ◽  
Kevin M. O'Brien ◽  
...  
Keyword(s):  
Carbon Dioxide ◽  
Surface Water ◽  
Data Collection ◽  
Data Availability ◽  
Data Sets ◽  
Science Data ◽  
High Quality ◽  
Data Set ◽  
Surface Ocean ◽  
Biogeochemical Models

Abstract. The Surface Ocean CO2 Atlas (SOCAT) is a synthesis of quality-controlled fCO2 (fugacity of carbon dioxide) values for the global surface oceans and coastal seas with regular updates. Version 3 of SOCAT has 14.5 million fCO2 values from 3646 data sets covering the years 1957 to 2014. This latest version has an additional 4.4 million fCO2 values relative to version 2 and extends the record from 2011 to 2014. Version 3 also significantly increases the data availability for 2005 to 2013. SOCAT has an average of approximately 1.2 million surface water fCO2 values per year for the years 2006 to 2012. Quality and documentation of the data has improved. A new feature is the data set quality control (QC) flag of E for data from alternative sensors and platforms. The accuracy of surface water fCO2 has been defined for all data set QC flags. Automated range checking has been carried out for all data sets during their upload into SOCAT. The upgrade of the interactive Data Set Viewer (previously known as the Cruise Data Viewer) allows better interrogation of the SOCAT data collection and rapid creation of high-quality figures for scientific presentations. Automated data upload has been launched for version 4 and will enable more frequent SOCAT releases in the future. High-profile scientific applications of SOCAT include quantification of the ocean sink for atmospheric carbon dioxide and its long-term variation, detection of ocean acidification, as well as evaluation of coupled-climate and ocean-only biogeochemical models. Users of SOCAT data products are urged to acknowledge the contribution of data providers, as stated in the SOCAT Fair Data Use Statement. This ESSD (Earth System Science Data) "Living Data" publication documents the methods and data sets used for the assembly of this new version of the SOCAT data collection and compares these with those used for earlier versions of the data collection (Pfeil et al., 2013; Sabine et al., 2013; Bakker et al., 2014).

scholarly journals An end-to-end Novel Forecasting Model for Crime Prediction based on Big Data

2020 ◽  
Vol 8 (6) ◽  
pp. 3704-3708
Keyword(s):  
Time Series ◽  
Big Data ◽  
Data Analytics ◽  
Linear Time ◽  
Big Data Analytics ◽  
Series Data ◽  
Data Sets ◽  
Data Set ◽  
Main Category ◽  
Crime Prediction

Big data analytics is a field in which we analyse and process information from large or convoluted data sets to be managed by methods of data-processing. Big data analytics is used in analysing the data and helps in predicting the best outcome from the data sets. Big data analytics can be very useful in predicting crime and also gives the best possible solution to solve that crime. In this system we will be using the past crime data set to find out the pattern and through that pattern we will be predicting the range of the incident. The range of the incident will be determined by the decision model and according to the range the prediction will be made. The data sets will be nonlinear and in the form of time series so in this system we will be using the prophet model algorithm which is used to analyse the non-linear time series data. The prophet model categories in three main category and i.e. trends, seasonality, and holidays. This system will help crime cell to predict the possible incident according to the pattern which will be developed by the algorithm and it also helps to deploy right number of resources to the highly marked area where there is a high chance of incidents to occur. The system will enhance the crime prediction system and will help the crime department to use their resources more efficiently.

scholarly journals Research on improved privacy publishing algorithm based on set cover

2019 ◽  
Vol 16 (3) ◽  
pp. 705-731
Author(s):  
Haoze Lv ◽  
Zhaobin Liu ◽  
Zhonglian Hu ◽  
Lihai Nie ◽  
Weijiang Liu ◽  
...  
Keyword(s):  
Privacy Protection ◽  
Data Privacy ◽  
Differential Privacy ◽  
Main Idea ◽  
Data Availability ◽  
Set Cover ◽  
Data Sets ◽  
Data Set ◽  
Query Cover ◽  
Privacy Model

With the invention of big data era, data releasing is becoming a hot topic in database community. Meanwhile, data privacy also raises the attention of users. As far as the privacy protection models that have been proposed, the differential privacy model is widely utilized because of its many advantages over other models. However, for the private releasing of multi-dimensional data sets, the existing algorithms are publishing data usually with low availability. The reason is that the noise in the released data is rapidly grown as the increasing of the dimensions. In view of this issue, we propose algorithms based on regular and irregular marginal tables of frequent item sets to protect privacy and promote availability. The main idea is to reduce the dimension of the data set, and to achieve differential privacy protection with Laplace noise. First, we propose a marginal table cover algorithm based on frequent items by considering the effectiveness of query cover combination, and then obtain a regular marginal table cover set with smaller size but higher data availability. Then, a differential privacy model with irregular marginal table is proposed in the application scenario with low data availability and high cover rate. Next, we obtain the approximate optimal marginal table cover algorithm by our analysis to get the query cover set which satisfies the multi-level query policy constraint. Thus, the balance between privacy protection and data availability is achieved. Finally, extensive experiments have been done on synthetic and real databases, demonstrating that the proposed method preforms better than state-of-the-art methods in most cases.

scholarly journals Locality-Sensitive Hashing for Information Retrieval System on Multiple GPGPU Devices

2020 ◽  
Vol 10 (7) ◽  
pp. 2539 ◽  
Author(s):  
Toan Nguyen Mau ◽  
Yasushi Inoguchi
Keyword(s):  
Big Data ◽  
Information Retrieval ◽  
Retrieval System ◽  
Hash Table ◽  
Information Retrieval System ◽  
Main Memory ◽  
Locality Sensitive Hashing ◽  
Data Sets ◽  
Similar Data ◽  
Data Set

It is challenging to build a real-time information retrieval system, especially for systems with high-dimensional big data. To structure big data, many hashing algorithms that map similar data items to the same bucket to advance the search have been proposed. Locality-Sensitive Hashing (LSH) is a common approach for reducing the number of dimensions of a data set, by using a family of hash functions and a hash table. The LSH hash table is an additional component that supports the indexing of hash values (keys) for the corresponding data/items. We previously proposed the Dynamic Locality-Sensitive Hashing (DLSH) algorithm with a dynamically structured hash table, optimized for storage in the main memory and General-Purpose computation on Graphics Processing Units (GPGPU) memory. This supports the handling of constantly updated data sets, such as songs, images, or text databases. The DLSH algorithm works effectively with data sets that are updated with high frequency and is compatible with parallel processing. However, the use of a single GPGPU device for processing big data is inadequate, due to the small memory capacity of GPGPU devices. When using multiple GPGPU devices for searching, we need an effective search algorithm to balance the jobs. In this paper, we propose an extension of DLSH for big data sets using multiple GPGPUs, in order to increase the capacity and performance of the information retrieval system. Different search strategies on multiple DLSH clusters are also proposed to adapt our parallelized system. With significant results in terms of performance and accuracy, we show that DLSH can be applied to real-life dynamic database systems.

Big Data has ascended as a Catchword in an extravagant Length of Time

2017 ◽  
Vol 8 (3) ◽  
Author(s):  
Subodh Kesharwani
Keyword(s):  
Big Data ◽  
Data Analytics ◽  
Credit Cards ◽  
Analysis Data ◽  
Data Sets ◽  
Data Set ◽  
Processing Application ◽  
Digital Footprint ◽  
Data As A Service ◽  
By Products

The whole ball of wax we create leaves a digital footprint. Big data had ascended as a catchword in recent years. Principally, it means a prodigious aggregate of information that is stimulated as trails or by-products of online and offline doings — what we get using credit cards, where we travel via GPS, what we ‘like’ on Facebook or retweet on Twitter, or what we bargain either through “apnidukaan” via amazon, and so on. In this era and stage, the Data as a Service (DaaS) battle is gaining force, spurring one of the fastest growing industries in the world.“Big data” is a jargon for data sets that are so gigantic or multi-layered that old-style data processing application software’s are deprived to concord with them. Challenges contain apprehension, storage, analysis, data curation, search, sharing, and transmission, visualization, querying, and updating information privacy. The term “big data” usually refers self-effacingly to the use of extrapolative analytics, user behaviour analytics, or sure other advanced data analytics methods that extract value from data, and infrequently to a separable size of data set

Hierarchy, Individuality and Paleoecosystems

1990 ◽  
Vol 5 ◽  
pp. 31-47 ◽  
Author(s):  
William Miller
Keyword(s):  
Spatial Variation ◽  
Temporal Dynamics ◽  
Data Sets ◽  
Field Observations ◽  
Important Research ◽  
Temporal Scales ◽  
Shell Beds ◽  
Spatio Temporal ◽  
Patterns And Processes ◽  
Fossil Data

Techniques and field observations that detect “spatial variation” and “temporal dynamics” in fossil deposits have become important research programs in paleosynecology. These studies attempt to delineate aggregates and sequences of fossils at varied scales that appear to result from processes encompassing larger areas and greater time spans than the processes familiar to neoecologists. Description and modeling of patterns and processes at these scales would be significant contributions to historical biology, but little attention has been given to the ontology of “natural” multispecies units discernable in fossil data sets at varied spatio-temporal scales of resolution. Do patterns at any of these nested levels of variation – patches within shell beds, shell beds within biofacies, and so on – represent the elusive original community of organisms?

scholarly journals Gap-Filling of NDVI Satellite Data Using Tucker Decomposition: Exploiting Spatio-Temporal Patterns

2021 ◽  
Vol 13 (19) ◽  
pp. 4007
Author(s):  
Andri Freyr Þórðarson ◽  
Andreas Baum ◽  
Mónica García ◽  
Sergio M. Vicente-Serrano ◽  
Anders Stockmarr
Keyword(s):  
Missing Data ◽  
Iberian Peninsula ◽  
Satellite Data ◽  
Simulated Data ◽  
Principal Component ◽  
Decomposition Methods ◽  
Data Sets ◽  
Gap Filling ◽  
Data Set ◽  
Spatio Temporal

Remote sensing satellite images in the optical domain often contain missing or misleading data due to overcast conditions or sensor malfunctioning, concealing potentially important information. In this paper, we apply expectation maximization (EM) Tucker to NDVI satellite data from the Iberian Peninsula in order to gap-fill missing information. EM Tucker belongs to a family of tensor decomposition methods that are known to offer a number of interesting properties, including the ability to directly analyze data stored in multidimensional arrays and to explicitly exploit their multiway structure, which is lost when traditional spatial-, temporal- and spectral-based methods are used. In order to evaluate the gap-filling accuracy of EM Tucker for NDVI images, we used three data sets based on advanced very-high resolution radiometer (AVHRR) imagery over the Iberian Peninsula with artificially added missing data as well as a data set originating from the Iberian Peninsula with natural missing data. The performance of EM Tucker was compared to a simple mean imputation, a spatio-temporal hybrid method, and an iterative method based on principal component analysis (PCA). In comparison, imputation of the missing data using EM Tucker consistently yielded the most accurate results across the three simulated data sets, with levels of missing data ranging from 10 to 90%.

Sign in / Sign up

Export Citation Format

Share Document