Application of Improved Recommendation System Based on Spark Platform in Big Data Analysis

Abstract In the era of big data, people have to face information filtration problem. For those cases when users do not or cannot express their demands clearly, recommender system can analyse user’s information more proactive and intelligent to filter out something users want. This property makes recommender system play a very important role in the field of e-commerce, social network and so on. The collaborative filtering recommendation algorithm based on Alternating Least Squares (ALS) is one of common algorithms using matrix factorization technique of recommendation system. In this paper, we design the parallel implementation process of the recommendation algorithm based on Spark platform and the related technology research of recommendation systems. Because of the shortcomings of the recommendation algorithm based on ALS model, a new loss function is designed. Before the model is trained, the similarity information of users and items is fused. The experimental results show that the performance of the proposed algorithm is better than that of algorithm based on ALS.

Download Full-text

Identifying Attack Models for Securing Cluster-Based Recommendation System

Recent Patents on Engineering ◽

10.2174/1872212114666200403091053 ◽

2020 ◽

Vol 14 ◽

Author(s):

Amreen Ahmad ◽

Tanvir Ahmad ◽

Ishita Tripathi

Keyword(s):

Collaborative Filtering ◽

Recommender System ◽

Recommendation System ◽

Relevant Information ◽

System Function ◽

Recommendation Algorithm ◽

Underlying Network ◽

Entire Dataset ◽

Attack Models ◽

Wide Usage

: The immense growth of information has led to the wide usage of recommender systems for retrieving relevant information. One of the widely used methods for recommendation is collaborative filtering. However, such methods suffer from two problems, scalability and sparsity. In the proposed research, the two issues of collaborative filtering are addressed and a cluster-based recommender system is proposed. For the identification of potential clusters from the underlying network, Shapley value concept is used, which divides users into different clusters. After that, the recommendation algorithm is performed in every respective cluster. The proposed system recommends an item to a specific user based on the ratings of the item’s different attributes. Thus, it reduces the running time of the overall algorithm, since it avoids the overhead of computation involved when the algorithm is executed over the entire dataset. Besides, the security of the recommender system is one of the major concerns nowadays. Attackers can come in the form of ordinary users and introduce bias in the system to force the system function that is advantageous for them. In this paper, we identify different attack models that could hamper the security of the proposed cluster-based recommender system. The efficiency of the proposed research is validated by conducting experiments on student dataset.

Download Full-text

CAN DISSIMILAR USERS CONTRIBUTE TO ACCURACY AND DIVERSITY OF PERSONALIZED RECOMMENDATION?

International Journal of Modern Physics C ◽

10.1142/s0129183110015786 ◽

2010 ◽

Vol 21 (10) ◽

pp. 1217-1227 ◽

Cited By ~ 24

Author(s):

WEI ZENG ◽

MING-SHENG SHANG ◽

QIAN-MING ZHANG ◽

LINYUAN LÜ ◽

TAO ZHOU

Keyword(s):

Collaborative Filtering ◽

Recommender System ◽

High Accuracy ◽

Personalized Recommendation ◽

Recommendation Algorithm ◽

Filtering Algorithm ◽

Recommendation Algorithms ◽

Collaborative Filtering Algorithm ◽

High Diversity ◽

Better Than

Recommender systems are becoming a popular and important set of personalization techniques that assist individual users with navigating through the rapidly growing amount of information. A good recommender system should be able to not only find out the objects preferred by users, but also help users in discovering their personalized tastes. The former corresponds to high accuracy of the recommendation, while the latter to high diversity. A big challenge is to design an algorithm that provides both highly accurate and diverse recommendation. Traditional recommendation algorithms only take into account the contributions of similar users, thus, they tend to recommend popular items for users ignoring the diversity of recommendations. In this paper, we propose a recommendation algorithm by considering both the effects of similar and dissimilar users under the framework of collaborative filtering. Extensive analyses on three datasets, namely MovieLens, Netflix and Amazon, show that our method performs much better than the standard collaborative filtering algorithm for both accuracy and diversity.

Download Full-text

Research on the Design of Mass Recommendation System Based on Lambda Architecture

Journal of Web Engineering ◽

10.13052/jwe1540-9589.20614 ◽

2021 ◽

Author(s):

Bowen Chen ◽

Li Zhu ◽

Da Wang ◽

JunHua Cheng

Keyword(s):

Big Data ◽

Data Collection ◽

Recommender System ◽

System Architecture ◽

Hybrid Algorithm ◽

Recommendation System ◽

Lambda Architecture ◽

System Data ◽

System Architecture Design ◽

Research Analysis

In the era of big data, in order to increasing the information data for conforms to the personalized needs of content, research scholars put forward based on the Lambda mass recommendation system architecture design, it can not only to the recessive and dominant behavior of users of the system data collection storage and research analysis, can also be based on the analysis of cascading hybrid algorithm to explore how to carry out real-time recommendation. Therefore, on the basis of understanding the research and development achievements of recommender systems at home and abroad in recent years, and based on the understanding and analysis of Lambda architecture and cascading hybrid algorithm, this paper aims at how to design a massive recommender system in line with users’ behavior, and makes clear the recommendation effect by combining with system testing.

Download Full-text

Design of the Piano Score Recommendation Image Analysis System Based on the Big Data and Convolutional Neural Network

Computational Intelligence and Neuroscience ◽

10.1155/2021/4953288 ◽

2021 ◽

Vol 2021 ◽

pp. 1-9

Author(s):

Yuanyuan Zhang

Keyword(s):

Neural Network ◽

Image Analysis ◽

Big Data ◽

Convolutional Neural Network ◽

Recommendation System ◽

Piano Music ◽

Music Recommendation ◽

Recommendation Algorithm ◽

Data Environment ◽

Music Recommendation System

In the era of big data, the problem of information overload is becoming more and more obvious. A piano music image analysis and recommendation system based on the CNN classifier and user preference is designed by using the convolutional neural network (CNN), which can realize accurate piano music recommendation for users in the big data environment. The piano music recommendation system based on the CNN is mainly composed of user modeling, music feature extraction, recommendation algorithm, and so on. In the recommendation algorithm module, the potential characteristics of music are predicted by the regression model, and the matching degree between users and music is calculated according to user preferences. Then, music that users may be interested in is generated and sorted in order to recommend new piano music to relevant users. The image analysis model contains four “convolution + pooling” layers. The classification accuracy and gradient change law of the CNN under RMSProp and Adam optimal controllers are compared. The image analysis results show that the Adam optimal controller can quickly find the direction, and the gradient decreases greatly. In addition, the accuracy of the recommendation system is 55.84%. Compared with the traditional CNN algorithm, this paper uses the convolutional neural network (CNN) to analyze and recommend piano music images according to users’ preferences, which can realize more accurate piano music recommendation for users in the big data environment. Therefore, the piano music recommendation system based on the CNN has strong feature learning ability and good prediction and recommendation ability.

Download Full-text

Research on Recommendation Algorithm in Social Networks

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.496-500.1865 ◽

2014 ◽

Vol 496-500 ◽

pp. 1865-1868

Author(s):

Hu Xin Tang ◽

Xu Qian

Keyword(s):

Social Networks ◽

Social Network ◽

Recommender System ◽

Recommendation System ◽

Simulation Experiment ◽

Recommendation Algorithm ◽

Simulation Experiments ◽

Correlation Algorithm ◽

Rapid Calculation ◽

Relative Attractiveness

Research status and development of the recommendation system are studied, the focus of evaluation of recommender system and recommender system based on social network in two aspects, and puts forward some improved algorithm, and achieved certain results. KDD Cup 2012 Track data for the simulation experiments on the correlation algorithm based on search engine, has been shown in different positions on the relative attractiveness of advertising, numerical user. At the same time, rapid calculation of the degree of correlation between a user and other users of an algorithm is given, and then quickly given the recommendation results. KDD Cup 2012 Track data for the simulation experiment of the algorithm, and the analysis result is given.

Download Full-text

Usability of a Machine-Learning Clinical Order Recommender System Interface for Clinical Decision Support and Physician Workflow

10.1101/2020.02.24.20025890 ◽

2020 ◽

Author(s):

Andre Kumar ◽

Jonathan Chiang ◽

Jason Hom ◽

Lisa Shieh ◽

Rachael Aikens ◽

...

Keyword(s):

Recommender System ◽

Recommendation System ◽

Clinical Decision ◽

Order Entry ◽

Manual Search ◽

System A ◽

The Difference ◽

System Order ◽

Better Than ◽

Electronic Order

AbstractObjectiveTo determine whether clinicians will use machine learned clinical order recommender systems for electronic order entry for simulated inpatient cases, and whether such recommendations impact the clinical appropriateness of the orders being placed.Materials and Methods43 physicians used a clinical order entry interface for five simulated medical cases, with each physician-case randomized whether to have access to a previously-developed clinical order recommendation system. A panel of clinicians determined whether orders placed were clinically appropriate. The primary outcome was the difference in clinical appropriateness scores of orders for cases randomized to the recommender system. Secondary outcomes included usage metrics and physician opinions.ResultsClinical appropriateness scores for orders were comparable for cases randomized to the recommender system (mean difference -0.1 order per score, 95% CI:[-0.4, 0.2]). Physicians using the recommender placed more orders (mean 17.3 vs. 15.7 orders; incidence ratio 1.09, 95% CI:[1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Approximately 95% of participants agreed the system would be useful for their workflows.DiscussionMachine-learned clinical order options can meet physician needs better than standard manual search systems. This may increase the number of clinical orders placed per case, while still resulting in similar overall clinically appropriate choices.ConclusionsClinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.

Download Full-text

Construction of E-commerce Personalized Information Recommendation System in the Era of Big Data

Journal of Physics Conference Series ◽

10.1088/1742-6596/2074/1/012085 ◽

2021 ◽

Vol 2074 (1) ◽

pp. 012085

Author(s):

Yaosheng Wang

Keyword(s):

Big Data ◽

Data Processing ◽

Recommendation System ◽

Personalized Recommendation ◽

Service Recommendation ◽

Actual System ◽

Recommendation Algorithm ◽

Dissimilarity Matrix ◽

Logistics Service ◽

Depth Study

Abstract With the continuous expansion of the scale of e-commerce, personalized recommendation technology has been widely used. However, the traditional recommendation system has been unable to meet the current needs of data processing, and good big data processing ability has become the basic requirement of the new personalized recommendation system. In addition, traditional recommendation systems are often limited to tangible goods recommendation, and pay less attention to e-commerce logistics service recommendation. In this paper, through the in-depth study of information personalized recommendation service in e-commerce environment, combined with the application background of big data: Taking the user dissimilarity matrix as the recommendation model, we propose IU usercf and UDB slope one recommendation algorithm. The two algorithms based on incremental update recommendation model have good scalability, can effectively deal with big data, and have high prediction accuracy. The proposed algorithm is applied to the actual system, taking e-commerce logistics service as the recommendation object and iu-usercf as the recommendation algorithm, the personalized recommendation system for e-commerce logistics service is constructed. The e-commerce logistics service recommendation system explores the application practice of recommendation algorithm under big data, and enriches the application scenarios of personalized recommendation technology.

Download Full-text

The application of e-commerce recommendation system in smart cities based on big data and cloud computing

Computer Science and Information Systems ◽

10.2298/csis200917026z ◽

2021 ◽

pp. 26-26

Author(s):

Yiman Zhang

Keyword(s):

Cloud Computing ◽

Big Data ◽

Collaborative Filtering ◽

Real Time ◽

Evaluation System ◽

Recommendation System ◽

Smart Cities ◽

Personalized Recommendation ◽

Current Recommendation ◽

Recommendation Algorithm

In the era of big data, the amount of Internet data is growing explosively. How to quickly obtain valuable information from massive data has become a challenging task. To effectively solve the problems faced by recommendation technology, such as data sparsity, scalability, and real-time recommendation, a personalized recommendation algorithm for e-commerce based on Hadoop is designed aiming at the problems in collaborative filtering recommendation algorithm. Hadoop cloud computing platform has powerful computing and storage capabilities, which are used to improve the collaborative filtering recommendation algorithm based on project, and establish a comprehensive evaluation system. The effectiveness of the proposed personalized recommendation algorithm is further verified through the analysis and comparison with some traditional collaborative filtering algorithms. The experimental results show that the e-commerce system based on cloud computing technology effectively improves the support of various recommendation algorithms in the system environment; the algorithm has good scalability and recommendation efficiency in the distributed cluster, and the recommendation accuracy is also improved, which can improve the sparsity, scalability and real-time problems in e-commerce personalized recommendation. This study greatly improves the recommendation performance of e-commerce, effectively solves the shortcomings of the current recommendation algorithm, and further promotes the personalized development of e-commerce.

Download Full-text

Customer Loyalty Improves the Effectiveness of Recommender Systems Based on Complex Network

10.20944/preprints202002.0367.v1 ◽

2020 ◽

Author(s):

Yun Bai ◽

Suling Jia ◽

Shuangzhe Wang ◽

Binkai Tan

Keyword(s):

Bipartite Graph ◽

Customer Loyalty ◽

Recommender System ◽

Dimensional Space ◽

Recommendation Algorithm ◽

Single Vertex ◽

Recommendation Algorithms ◽

Vertex Set ◽

Time And Energy ◽

Better Than

A good recommender system can infer customers’ preferences based on their historical purchase records, and recommend products that the customers may be interested in, saving them a lot of time and energy. For enterprises, it is difficult to recommend accurately to each customer, and the bad recommendation may be counterproductive. Customer loyalty is an indicator that measures the preference relationship between customers and products in the field of marketing. A hypothesis is proposed in this study: if companies can divide customers into different groups based on customer loyalty, the recommendation effect on certain groups is better than that on overall customers. In this study, customer loyalty is measured by four features of the RFML model. All customers are viewed as points on a four-dimensional space, which are clustered by the k-means model. Two recommendation algorithms based on complex networks are tested: recommendation algorithm based on bipartite graph and PersonalRank (BGPR), and recommendation algorithm based on a single vertex set network and DeepWalk (SVDW). The experimental results show that customer loyalty has improved the effectiveness of the two algorithms over 14%, and the recommendation effect is the best on customer groups with a loyalty level of 4 (the highest level is 5). The recommendation algorithms with customer loyalty are better than using them alone.

Download Full-text

A Big Data Analysis Method Based on Modified Collaborative Filtering Recommendation Algorithms

Open Physics ◽

10.1515/phys-2019-0102 ◽

2019 ◽

Vol 17 (1) ◽

pp. 966-974

Author(s):

Nan Yin

Keyword(s):

Big Data ◽

Data Analysis ◽

Collaborative Filtering ◽

Execution Time ◽

Recommendation System ◽

Big Data Analysis ◽

Analysis Method ◽

Recommendation Algorithm ◽

Filtering Algorithm ◽

Collaborative Filtering Algorithm

Abstract With the rapid development of e-commerce, collaborative filtering recommendation system has been widely used in various network platforms. Using recommendation system to accurately predict customers’ preferences for goods can solve the problem of information overload faced by users and improve users’ dependence on the network platform. Because the recommendation system based on collaborative filtering technology has the ability to recommend more abstract or difficult to describe goods in words, the research related to collaborative filtering technology has attracted more and more attention. According to the past research, in collaborative filtering algorithm, if Pearson correlation coefficient is used, errors will occur under special circumstances. In this study, the normal recovery similarity measure is used to modify the similarity value to correct the error value of a collaborative filtering recommendation algorithm. Based on this, a big data analysis method based on a modified collaborative filtering recommendation algorithm is proposed. This research implemented it in the cloud Hadoop environment, and measure the execution time with 2, 5 and 8 nodes. Then the research compared it with the execution time of a single machine, and analyze its speedup ratio and efficiency. The experimental results show that the execution time increases with the number of neighbors. When the number of nodes is 5 and 8, the execution time is greatly improved, which improves the efficiency of collaborative filtering algorithm and can cope with massive data in the future.

Download Full-text