Application and Research on Distributed Collaborative Filtering Recommendation Algorithm Based on Hadoop

2015 ◽  
Vol 713-715 ◽  
pp. 1615-1621
Author(s):  
Xiu Juan Li ◽  
He Biao Yang
Keyword(s):  
Parallel Computing ◽  
Comparative Analysis ◽  
Collaborative Filtering ◽  
Computational Efficiency ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Recommendation Algorithm ◽  
Exponential Expansion ◽  
Hadoop Platform

Coupled with exponential expansion of the data, efficient computing of existing recommendation algorithm has become an important issue, and the traditional collaborative filtering recommendation algorithm also exist the problem of sparsity. Based on the detailed analysis, the article introduce Hadoop platform into improved collaborative filtering recommendation algorithm, the improved collaborative filtering recommendation algorithm solve the problem of data sparsity, MapReduce parallel computing of recommendation also solve the promble of computational efficiency. In the experiments, the comparative analysis between Hadoop platform implementation and the previous implementation draws the conclusion that the Hadoop platform improves collaborative filtering recommendation algorithm computation efficiently under conditions of large data sets.

scholarly journals Apache Hadoop, platform for the collection, processing and analysis of large data sets

2017 ◽  
Vol 4 ◽  
pp. 70-75
Author(s):  
Mateusz Gil
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
The Internet ◽  
Data Sets ◽  
Application Performance ◽  
Apache Hadoop ◽  
Hadoop Platform

The article presents the possibilities of using Hadoop platform to manage large data sets. The development of application performance has been shown based on available sources. Additionally, the article describes the organizations that have been successful in the Internet thanks to the implemented software.

Location-Based Collaborative Filtering for Web Service Recommendation

2019 ◽  
Vol 12 (1) ◽  
pp. 34-40
Author(s):  
Mareeswari Venkatachalaappaswamy ◽  
Vijayan Ramaraj ◽  
Saranya Ravichandran
Keyword(s):  
Collaborative Filtering ◽  
Web Service ◽  
Information Filtering ◽  
Service Selection ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Service Recommendation ◽  
Location Aware

Background: In many modern applications, information filtering is now used that exposes users to a collection of data. In such systems, the users are provided with recommended items’ list they might prefer or predict the rate that they might prefer for the items. So that, the users might be select the items that are preferred in that list. Objective: In web service recommendation based on Quality of Service (QoS), predicting QoS value will greatly help people to select the appropriate web service and discover new services. Methods: The effective method or technique for this would be Collaborative Filtering (CF). CF will greatly help in service selection and web service recommendation. It is the more general way of information filtering among the large data sets. In the narrower sense, it is the method of making predictions about a user’s interest by collecting taste information from many users. Results: It is easy to build and also much more effective for recommendations by predicting missing QoS values for the users. It also addresses the scalability problem since the recommendations are based on like-minded users using PCC or in clusters using KNN rather than in large data sources. Conclusion: In this paper, location-aware collaborative filtering is used to recommend the services. The proposed system compares the prediction outcomes and execution time with existing algorithms.

scholarly journals Accuracy and Computational Efficiency of a Graphical Modeling Approach to Linkage Disequilibrium Estimation

2011 ◽  
Vol 10 (1) ◽  
Author(s):  
Haley J Abel ◽  
Alun Thomas
Keyword(s):  
Linkage Disequilibrium ◽  
Missing Data ◽  
Computational Efficiency ◽  
Graphical Model ◽  
Optimal Parameter ◽  
Model Fitting ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Data Set

We develop recent work on using graphical models for linkage disequilibrium to provide efficient programs for model fitting, phasing, and imputation of missing data in large data sets. Two important features contribute to the computational efficiency: the separation of the model fitting and phasing-imputation processes into different programs, and holding in memory only the data within a moving window of loci during model fitting. Optimal parameter values were chosen by cross-validation to maximize the probability of correctly imputing masked genotypes. The best accuracy obtained is slightly below than that from the Beagle program of Browning and Browning, and our fitting program is slower. However, for large data sets, it uses less storage. For a reference set of n individuals genotyped at m markers, the time and storage required for fitting a graphical model are approximately O(nm) and O(n+m), respectively. To impute the phases and missing data on n individuals using an already fitted graphical model requires O(nm) time and O(m) storage. While the times for fitting and imputation are both O(nm), the imputation process is considerably faster; thus, once a model is estimated from a reference data set, the marginal cost of phasing and imputing further samples is very low.

An example of spectrum imaging used for comparison of EELS quantitative analysis techniques on Al-Li

1991 ◽  
Vol 49 ◽  
pp. 726-727
Author(s):  
John A. Hunt
Keyword(s):  
Quantitative Analysis ◽  
Large Data ◽  
Difference Spectrum ◽  
Large Data Sets ◽  
Foil Thickness ◽  
Data Sets ◽  
Analysis Techniques ◽  
Spectrum Imaging ◽  
Normal Spectrum ◽  
Electron Energy Loss

Spectrum-imaging is a useful technique for comparing different processing methods on very large data sets which are identical for each method. This paper is concerned with comparing methods of electron energy-loss spectroscopy (EELS) quantitative analysis on the Al-Li system. The spectrum-image analyzed here was obtained from an Al-10at%Li foil aged to produce δ' precipitates that can span the foil thickness. Two 1024 channel EELS spectra offset in energy by 1 eV were recorded and stored at each pixel in the 80x80 spectrum-image (25 Mbytes). An energy range of 39-89eV (20 channels/eV) are represented. During processing the spectra are either subtracted to create an artifact corrected difference spectrum, or the energy offset is numerically removed and the spectra are added to create a normal spectrum. The spectrum-images are processed into 2D floating-point images using methods and software described in [1].

Cluster analysis for large data sets: applications to individual aerosol particles from the mid-pacific

1992 ◽  
Vol 50 (2) ◽  
pp. 1488-1489
Author(s):  
Thomas W. Shattuck ◽  
James R. Anderson ◽  
Neil W. Tindale ◽  
Peter R. Buseck
Keyword(s):  
Cluster Analysis ◽  
Chemical Reactivity ◽  
Large Data ◽  
Large Data Sets ◽  
Particle Analysis ◽  
Data Sets ◽  
Halogen Chemistry ◽  
Complete Study ◽  
Components Analysis ◽  
Automated Scanning

Individual particle analysis involves the study of tens of thousands of particles using automated scanning electron microscopy and elemental analysis by energy-dispersive, x-ray emission spectroscopy (EDS). EDS produces large data sets that must be analyzed using multi-variate statistical techniques. A complete study uses cluster analysis, discriminant analysis, and factor or principal components analysis (PCA). The three techniques are used in the study of particles sampled during the FeLine cruise to the mid-Pacific ocean in the summer of 1990. The mid-Pacific aerosol provides information on long range particle transport, iron deposition, sea salt ageing, and halogen chemistry.Aerosol particle data sets suffer from a number of difficulties for pattern recognition using cluster analysis. There is a great disparity in the number of observations per cluster and the range of the variables in each cluster. The variables are not normally distributed, they are subject to considerable experimental error, and many values are zero, because of finite detection limits. Many of the clusters show considerable overlap, because of natural variability, agglomeration, and chemical reactivity.

NVESTIGATION OF THE EFFICIENCY OF DISTRIBUTED INFORMATION SYSTEMS BASED ON THE PROCESSING OF LARGE AMOUNTS OF DATA

2019 ◽  
Author(s):  
Mykhajlo Klymash ◽  
Olena Hordiichuk — Bublivska ◽  
Ihor Tchaikovskyi ◽  
Oksana Urikova
Keyword(s):  
Distributed Systems ◽  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Data Decomposition ◽  
Distributed Information ◽  
Software Model ◽  
Computing Performance ◽  
Mapreduce Model ◽  
Singular Data

In this article investigated the features of processing large arrays of information for distributed systems. A method of singular data decomposition is used to reduce the amount of data processed, eliminating redundancy. Dependencies of com­putational efficiency on distributed systems were obtained using the MPI messa­ging protocol and MapReduce node interaction software model. Were analyzed the effici­ency of the application of each technology for the processing of different sizes of data: Non — distributed systems are inefficient for large volumes of information due to low computing performance. It is proposed to use distributed systems that use the method of singular data decomposition, which will reduce the amount of information processed. The study of systems using the MPI protocol and MapReduce model obtained the dependence of the duration calculations time on the number of processes, which testify to the expediency of using distributed computing when processing large data sets. It is also found that distributed systems using MapReduce model work much more efficiently than MPI, especially with large amounts of data. MPI makes it possible to perform calculations more efficiently for small amounts of information. When increased the data sets, advisable to use the Map Reduce model.

Using Large Data Sets and Statistical Tools to Define “Failure” and Demonstrate the Safety of DPR

2018 ◽  
Vol 2018 (6) ◽  
pp. 38-39
Author(s):  
Austa Parker ◽  
Yan Qu ◽  
David Hokanson ◽  
Jeff Soller ◽  
Eric Dickenson ◽  
...  
Keyword(s):  
Large Data ◽  
Large Data Sets ◽  
Data Sets ◽  
Statistical Tools
Sign in / Sign up

Export Citation Format

Share Document