Vietnamese Complaint Detection on E-Commerce Websites

Customer product reviews play a role in improving the quality of products and services for business organizations or their brands. Complaining is an attitude that expresses dissatisfaction with an event or a product not meeting customer expectations. In this paper, we build a Vietnamese Open-domain Complaint Detection dataset (UIT-ViOCD), including 5,485 human-annotated reviews on four categories about product reviews on e-commerce sites. After the data collection phase, we proceed to the annotation task and achieve the inter-annotator agreement (Am) of 87%. Then, we present an extensive methodology for the research purposes and achieve 92.16% by F1-score for identifying complaints. With the results, in future, we aim to build a system for open-domain complaint detection on E-commerce websites.

Assessing the difficulty of annotating medical data in crowdworking with help of experiments

Background As healthcare-related data proliferate, there is need to annotate them expertly for the purposes of personalized medicine. Crowdworking is an alternative to expensive expert labour. Annotation corresponds to diagnosis, so comparing unlabeled records to labeled ones seems more appropriate for crowdworkers without medical expertise. We modeled the comparison of a record to two other records as a triplet annotation task, and we conducted an experiment to investigate to what extend sensor-measured stress, task duration, uncertainty of the annotators and agreement among the annotators could predict annotation correctness. Materials and methods We conducted an annotation experiment on health data from a population-based study. The triplet annotation task was to decide whether an individual was more similar to a healthy one or to one with a given disorder. We used hepatic steatosis as example disorder, and described the individuals with 10 pre-selected characteristics related to this disorder. We recorded task duration, electro-dermal activity as stress indicator, and uncertainty as stated by the experiment participants (n = 29 non-experts and three experts) for 30 triplets. We built an Artificial Similarity-Based Annotator (ASBA) and compared its correctness and uncertainty to that of the experiment participants. Results We found no correlation between correctness and either of stated uncertainty, stress and task duration. Annotator agreement has not been predictive either. Notably, for some tasks, annotators agreed unanimously on an incorrect annotation. When controlling for Triplet ID, we identified significant correlations, indicating that correctness, stress levels and annotation duration depend on the task itself. Average correctness among the experiment participants was slightly lower than achieved by ASBA. Triplet annotation turned to be similarly difficult for experts as for non-experts. Conclusion Our lab experiment indicates that the task of triplet annotation must be prepared cautiously if delegated to crowdworkers. Neither certainty nor agreement among annotators should be assumed to imply correct annotation, because annotators may misjudge difficult tasks as easy and agree on incorrect annotations. Further research is needed to improve visualizations for complex tasks, to judiciously decide how much information to provide, Out-of-the-lab experiments in crowdworker setting are needed to identify appropriate designs of a human-annotation task, and to assess under what circumstances non-human annotation should be preferred.

Leveraging the Generalization Ability of Deep Convolutional Neural Networks for Improving Classifiers for Color Fundus Photographs

Deep learning demands a large amount of annotated data, and the annotation task is often crowdsourced for economic efficiency. When the annotation task is delegated to non-experts, the dataset may contain data with inaccurate labels. Noisy labels not only yield classification models with sub-optimal performance, but may also impede their optimization dynamics. In this work, we propose exploiting the pattern recognition capacity of deep convolutional neural networks to filter out supposedly mislabeled cases while training. We suggest a training method that references softmax outputs to judge the correctness of the given labels. This approach achieved outstanding performance compared to the existing methods in various noise settings on a large-scale dataset (Kaggle 2015 Diabetic Retinopathy). Furthermore, we demonstrate a method mining positive cases from a pool of unlabeled images by exploiting the generalization ability. With this method, we won first place on the offsite validation dataset in pathological myopia classification challenge (PALM), achieving the AUROC of 0.9993 in the final submission. Source codes are publicly available.

Internalizing External Goals

The current study examined environmental contingencies (e.g., badgification) and individual factors (e.g., causality orientation) as a potential avenue for gamification to encourage adoption of issue-relevant behaviors via internalization processes. Drawing on the mini-theories of self-determination theory, we examined whether people’s causality orientation (i.e., autonomy vs. control) determined when different badges (i.e., reward vs. informational) encouraged adoption of issue-relevant behaviors. To examine this, an experiment was conducted with 215 participants using an image annotation task that implicitly addressed the issue of water conservation. The findings indicate that control-oriented people may have internalized the external goal of conserving water when presented with reward badges compared with informational badges. The results show that badgification warrants further examination to account for individual differences in causality orientation.

A Literature Research on Machine Learning Techniques used for Training Annotated Corpus

The development of research in the annotation area is growing. Researchers perform annotation task using various forms of datasets such as text, sound, images, and videos. Various algorithms are used to perform tasks. The purpose of this survey is to find out algorithms that are often used by researchers to perform annotation tasks, especially on text data. The literature surveys thirteen research papers on text annotation from the last 5 years. The results of this review indicate that SVM is the algorithm used for all three annotation methods: manual, automatic and semi-automatic annotation, with a significant accuracy above 80%. The result of this survey will be referred by the authors as the basis for subsequent research that will be conducted, especially in the semi-automatic annotation method.