A CNN-Based Advertisement Recommendation through Real-Time User Face Recognition

The advertising market’s use of smartphones and kiosks for non-face-to-face ordering is growing. An advertising video recommender system is needed that continuously shows advertising videos that match a user’s taste and displays other advertising videos quickly for unwanted advertisements. However, it is difficult to make a recommender system to identify users’ dynamic preferences in real time. In this study, we propose an advertising video recommendation procedure based on computer vision and deep learning, which uses changes in users’ facial expressions captured at every moment. Facial expressions represent a user’s emotions toward advertisements. We can utilize facial expressions to find a user’s dynamic preferences. For such a purpose, a CNN-based prediction model was developed to predict ratings, and a SIFT algorithm-based similarity model was developed to search for users with similar preferences in real time. To evaluate the proposed recommendation procedure, we experimented with food advertising videos. The experimental results show that the proposed procedure is superior to benchmark systems such as a random recommendation, an average rating approach, and a typical collaborative filtering approach in recommending advertising videos to both existing users and new users. From these results, we conclude that facial expressions are a critical factor for advertising video recommendations and are helpful in properly addressing the new user problem in existing recommender systems.