Evaluating Recommender Systems

2021 ◽  
Vol 17 (2) ◽  
pp. 25-45
Author(s):  
Alaa Alslaity ◽  
Thomas Tran
Keyword(s):  
Recommender Systems ◽  
Quantitative Study ◽  
Random Search ◽  
Evaluation Methods ◽  
Search Process ◽  
Manual Search ◽  
Theoretical Side

Replicating the results of the recommender system's evaluation is one of the main concerns in the area. This paper discusses this issue from different angles: 1) It investigates the uniformity of recommenders' evaluation designs presented in practice and their consistency with the theoretical side. 2) It highlights some of the issues and challenges that face recommenders' evaluators. 3) It provides stepwise guidelines for offline evaluation settings. A quantitative study of articles published in the last decade is studied. The search process is a manual search for a conference and a random search of journals. The results show a lack of uniformity and consistency in presenting the evaluation methods. Most of the articles miss at least one evaluation aspect (i.e., some aspects are not presented in the article). These discrepancies and the wide variety of evaluation settings lead to non-replicable experiments. To mitigate this issue, the paper proposes the recommender evaluation guidelines (REval), which presents a roadmap for recommender systems' evaluators.

scholarly journals Urban sensing as a random search process

2021 ◽  
Vol 562 ◽  
pp. 125307
Author(s):  
Kevin O’Keeffe ◽  
Paolo Santi ◽  
Brandon Wang ◽  
Carlo Ratti
Keyword(s):  
Random Search ◽  
Search Process ◽  
Urban Sensing

scholarly journals Effective scanning of parameter space in a Desktop Grid for identification of a hydride decomposition model

2018 ◽  
Vol 9 (4) ◽  
pp. 53-68
Author(s):  
Ilya Aleksandrovich Chernov ◽  
Natalia Nikolaevna Nikitina
Keyword(s):  
Parameter Identification ◽  
Grid Computing ◽  
Parameter Space ◽  
Random Search ◽  
Additional Cost ◽  
Search Process ◽  
Desktop Grid ◽  
Decomposition Model

We consider parameter identification of a hydride decomposition model by scanning the parameter space in parallel. Such problem is resource demanding, but suits best for Desktop Grid computing. Considering task retrieval as a game, we show that the search process can be improved to produce solutions faster in comparison with random search, with no or minor additional cost.

Recommender System: Towards Classification of Human Intentions in E-Shopping Using Machine Learning

2019 ◽  
Vol 16 (10) ◽  
pp. 4280-4285
Author(s):  
Babaljeet Kaur ◽  
Richa Sharma ◽  
Shalli Rani ◽  
Deepali Gupta
Keyword(s):  
Machine Learning ◽  
Recommender Systems ◽  
Recommender System ◽  
Machine Learning Algorithms ◽  
Application Domain ◽  
Random Forest Algorithm ◽  
Manual Search ◽  
Online Shoppers ◽  
New Item

Recommender systems were introduced in mid-1990 for assisting the users to choose a correct product from innumerable choices available. The basic concept of a recommender system is to advise a new item or product to the users instead of the manual search, because when user wants to buy a new item, he is confused about which item will suit him better and meet the intended requirements. From google news to netflix and from Instagram to LinkedIn, recommender systems have spread their roots in almost every application domain possible. Now a days, lots of recommender system are available for every field. In this paper, overview of recommender system, recommender approaches, application areas and the challenges of recommender system, is given. Further, we study conduct an experiment on online shoppers’ intention to predict the behavior of shoppers using Machine learning algorithms. Based on the results, it is observed that Random forest algorithm performs the best with 93% ROC value.

Hyperparameter tuning for hidden unit conditional random fields

2017 ◽  
Vol 34 (6) ◽  
pp. 2054-2062 ◽  
Author(s):  
Eun-Suk Yang ◽  
Jong Dae Kim ◽  
Chan-Young Park ◽  
Hye-Jeong Song ◽  
Yu-Seop Kim
Keyword(s):  
Nonlinear Model ◽  
Random Fields ◽  
Conditional Random Fields ◽  
Random Search ◽  
Search Method ◽  
Grid Search ◽  
Content Type ◽  
Manual Search ◽  
Optimization Search ◽  
Better Than

Purpose In this paper, the problem of a nonlinear model – specifically the hidden unit conditional random fields (HUCRFs) model, which has binary stochastic hidden units between the data and the labels – exhibiting unstable performance depending on the hyperparameter under consideration. Design/methodology/approach There are three main optimization search methods for hyperparameter tuning: manual search, grid search and random search. This study shows that HUCRFs’ unstable performance depends on the hyperparameter values used and its performance is based on tuning that draws on grid and random searches. All experiments conducted used the n-gram features – specifically, unigram, bigram, and trigram. Findings Naturally, selecting a list of hyperparameter values based on a researchers’ experience to find a set in which the best performance is exhibited is better than finding it from a probability distribution. Realistically, however, it is impossible to calculate using the parameters in all combinations. The present research indicates that the random search method has a better performance compared with the grid search method while requiring shorter computation time and a reduced cost. Originality/value In this paper, the issues affecting the performance of HUCRF, a nonlinear model with performance that varies depending on the hyperparameters, but performs better than CRF, has been examined.

Genetic Algorithms as Global Random Search Methods: An Alternative Perspective

1995 ◽  
Vol 3 (1) ◽  
pp. 39-80 ◽  
Author(s):  
Charles C. Peck ◽  
Atam P. Dhawan
Keyword(s):  
Genetic Algorithm ◽  
Genetic Algorithms ◽  
Random Search ◽  
Schema Theory ◽  
Global Optimum ◽  
Global Search ◽  
Search Process ◽  
Sampling Distributions ◽  
Alternative Perspective ◽  
Global Random Search

Genetic algorithm behavior is described in terms of the construction and evolution of the sampling distributions over the space of candidate solutions. This novel perspective is motivated by analysis indicating that the schema theory is inadequate for completely and properly explaining genetic algorithm behavior. Based on the proposed theory, it is argued that the similarities of candidate solutions should be exploited directly, rather than encoding candidate solutions and then exploiting their similarities. Proportional selection is characterized as a global search operator, and recombination is characterized as the search process that exploits similarities. Sequential algorithms and many deletion methods are also analyzed. It is shown that by properly constraining the search breadth of recombination operators, convergence of genetic algorithms to a global optimum can be ensured.

A Characterisation and Framework for User-Centric Factors in Evaluation Methods for Recommender Systems

2017 ◽  
Vol 6 (1) ◽  
pp. 1-16 ◽  
Author(s):  
Tatenda D. Kavu ◽  
Kuda Dube ◽  
Peter G. Raeth ◽  
Gilford T. Hapanyengwi
Keyword(s):  
Conceptual Framework ◽  
Recommender Systems ◽  
User Experience ◽  
User Satisfaction ◽  
Optimal Solution ◽  
User Interaction ◽  
Evaluation Methods ◽  
User Preferences ◽  
Evaluation Framework ◽  
User Centric

Researchers have worked on-finding e-commerce recommender systems evaluation methods that contribute to an optimal solution. However, existing evaluations methods lack the assessment of user-centric factors such as buying decisions, user experience and user interactions resulting in less than optimum recommender systems. This paper investigates the problem of adequacy of recommender systems evaluation methods in relation to user-centric factors. Published work has revealed limitations of existing evaluation methods in terms of evaluating user satisfaction. This paper characterizes user-centric evaluation factors and then propose a user-centric evaluation conceptual framework to identify and expose a gap within literature. The researchers used an integrative review approach to formulate both the characterization and the conceptual framework for investigation. The results reveal a need to come up with a holistic evaluation framework that combines system-centric and user-centric evaluation methods as well as formulating computational user-centric evaluation methods. The conclusion reached is that, evaluation methods for e-commerce recommender systems lack full assessment of vital factors such as: user interaction, user experience and purchase decisions. A full consideration of these factors during evaluation will give birth to new types of recommender systems that predict user preferences using user decision-making process profiles, and that will enhance user experience and increase revenue in the long run.

Semantic Web Technologies in the Service of Personalization Tools

2010 ◽  
pp. 68-87 ◽  
Author(s):  
Yolanda Blanco-Fernández ◽  
José J. Pazos-Arias ◽  
Alberto Gil-Solla
Keyword(s):  
Semantic Web ◽  
Recommender Systems ◽  
Information Overload ◽  
Semantic Web Technologies ◽  
Web Technologies ◽  
Manual Search ◽  
Novel Approach ◽  
Search Processes ◽  
Matching Techniques

The so-called recommender systems have become assistance tools indispensable to the users in domains where the information overload hampers manual search processes. In literature, diverse personalization paradigms have been proposed to match automatically the preferences of each user (which are previously modelled in personal profiles) against the available items. All these paradigms are laid down on a common substratum that uses syntactic matching techniques, which greatly limit the quality of the offered recommendations due to their inflexible nature. To fight these limitations, this chapter explores a novel approach based on reasoning about the semantics of both the users’ preferences and considered items, by resorting to less rigid inference mechanisms borrowed from the Semantic Web.

Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems

2019 ◽  
Vol 61 (2) ◽  
pp. 1147-1178 ◽  
Author(s):  
Zohreh Dehghani Champiri ◽  
Adeleh Asemi ◽  
Salim Siti Salwah Binti
Keyword(s):  
Recommender Systems ◽  
Meta Analysis ◽  
Evaluation Methods ◽  
Context Aware

scholarly journals Evaluation of Neural Architecture Search Approaches for Offshore Platform Offset Prediction

2021 ◽  
Author(s):  
Tomaz M. Suller ◽  
Eric O. Gomes ◽  
Henrique B. Oliveira ◽  
Lucas P. Cotrim ◽  
Amir M. Sa’ad ◽  
...  
Keyword(s):  
Simulated Annealing ◽  
Comparative Study ◽  
Random Search ◽  
Center Of Gravity ◽  
Offshore Platform ◽  
Bayesian Optimization ◽  
Optimization Process ◽  
Search Process ◽  
Multi Layer Perceptron ◽  
Neural Architecture

This paper proposes a solution based on Multi-Layer Perceptron (MLP) to predict the offset of the center of gravity of an offshore platform. It also performs a comparative study with three optimization algorithms – Random Search, Simulated Annealing, and Bayesian Optimization (BO) – to find the best MLP architecture. Although BO obtained the best architecture in the shortest time, ablation studies developed in this paper with hyperparameters of the optimization process showed that the result is sensitive to them and deserves attention in the Neural Architecture Search process.

Sign in / Sign up

Export Citation Format

Share Document