Measuring Orthogonal Mechanics in Linguistic Annotation Games

Gamification has been recently growing in popularity among researchers investigating Information and Communication Technologies. Scholars have been trying to take advantage of this approach in the field of natural language processing (NLP), developing Games With A Purpose (GWAPs) for corpus annotation that have obtained encouraging results both in annotation quality and overall cost. However, GWAPs implement gamification in different ways and to different degrees. We propose a new framework to investigate the mechanics employed in the gamification process and their magnitude in terms of complexity. This framework is based on an analysis of some of the most important contributions in the field of NLP-related gamified applications and GWAP theory. Its primary purpose is to provide a first step towards classifying mechanics that mimic mainstream video games and may require skills that are not relevant to the annotation task, defined as orthogonal mechanics. In order to test our framework, we develop and evaluate Spacewords, a linguistic space game for synonymy annotation.

Vers un réseau lexico-sémantique de grande taille obtenu par crowdsourcing

Le projet JeuxDeMots vise à construire une grande base de connaissances de sens commun (et de spécialité), en français, à l’aide de jeux (GWAPs – Games With A Purpose), d’approches contributives, mais également de mécanismes d’inférences. Une dizaine de jeux ont été conçus dans le cadre du projet, chacun permettant de collecter des informations spécifiques, ou de vérifier la qualité de données acquises via un autre jeu. Cet article s’attachera à décrire la nature des données que nous avons collectées et construites, depuis le lancement du projet durant l’été 2007. Nous décrirons en particulier les aspects suivants : la structure du réseau lexical et sémantique JeuxDeMots, certains types de relations (sémantiques, ontologiques, subjectives, rôles sémantiques, associations d’idées, etc.), les questions d’activation et d’inhibition, l’annotation de relations (méta-informations), les raffinements sémantiques (gestion de la polysémie), la création de termes agglomérés permettant la représentation de connaissances plus riches (relations à n-arguments).

Refining Linked Data with Games with a Purpose

With the rise of linked data and knowledge graphs, the need becomes compelling to find suitable solutions to increase the coverage and correctness of data sets, to add missing knowledge and to identify and remove errors. Several approaches – mostly relying on machine learning and natural language processing techniques – have been proposed to address this refinement goal; they usually need a partial gold standard, i.e., some “ground truth” to train automatic models. Gold standards are manually constructed, either by involving domain experts or by adopting crowdsourcing and human computation solutions. In this paper, we present an open source software framework to build Games with a Purpose for linked data refinement, i.e., Web applications to crowdsource partial ground truth, by motivating user participation through fun incentive. We detail the impact of this new resource by explaining the specific data linking “purposes” supported by the framework (creation, ranking and validation of links) and by defining the respective crowdsourcing tasks to achieve those goals. We also introduce our approach for incremental truth inference over the contributions provided by players of Games with a Purpose (also abbreviated as GWAP): we motivate the need for such a method with the specificity of GWAP vs. traditional crowdsourcing; we explain and formalize the proposed process, explain its positive consequences and illustrate the results of an experimental comparison with state-of-the-art approaches. To show this resource's versatility, we describe a set of diverse applications that we built on top of it; to demonstrate its reusability and extensibility potential, we provide references to detailed documentation, including an entire tutorial which in a few hours guides new adopters to customize and adapt the framework to a new use case.

Chefs Know More than Just Recipes: Professional Vision in a Citizen Science Game

Some citizen science projects use “games with a purpose” (GWAPs) to integrate what humans and computers, respectively, can do well. One of these projects is Foldit, which invites talented players to predict three-dimensional (3D) models of proteins from their amino acid composition. This study investigated players’ professional vision and interpret their use of recipes, small scripts of computer code that automate some protein folding processes, to carry out their strategies more easily when solving game puzzles. Specifically, this study examined when, how and why the players ran recipes when solving the puzzles, and what actions those recipes performed in the gameplay.Autoethnographic accounts of players at different levels of experience (beginner, intermediate, and expert) with playing the game were analyzed using a grounded theory approach. The analysis of what these players observed and did visualized the professional vision necessary to use recipes sensibly and effectively. The findings highlight three key abilities: (a) seeing beauty; (b) repairing errors made by recipes, and (c) monitoring a large quantity of information to perform actions effectively. This study indicates that players indeed have to develop a professional vision independent of what the game itself can highlight. This is related to the nature of the game where it seems impossible for the game developers to show the affordances, because they are unknown. Players must learn to see the affordances and develop a professional vision, which means that they have to learn these skills through gaming.

Chefs Know More than Just Recipes: Professional Vision in a Citizen Science Game

Some citizen science projects use “games with a purpose” (GWAPs) to integrate what humans and computers, respectively, can do well. One of these projects is Foldit, which invites talented players to predict three-dimensional (3D) models of proteins from their amino acid composition. This study investigated players’ professional vision and interpret their use of recipes, small scripts of computer code that automate some protein folding processes, to carry out their strategies more easily when solving game puzzles. Specifically, this study examined when, how and why the players ran recipes when solving the puzzles, and what actions those recipes performed in the gameplay.Autoethnographic accounts of players at different levels of experience (beginner, intermediate, and expert) with playing the game were analyzed using a grounded theory approach. The analysis of what these players observed and did visualized the professional vision necessary to use recipes sensibly and effectively. The findings highlight three key abilities: (a) seeing beauty; (b) repairing errors made by recipes, and (c) monitoring a large quantity of information to perform actions effectively. This study indicates that players indeed have to develop a professional vision independent of what the game itself can highlight. This is related to the nature of the game where it seems impossible for the game developers to show the affordances, because they are unknown. Players must learn to see the affordances and develop a professional vision, which means that they have to learn these skills through gaming.

Performance of Players and Data Quality in a Citizen Science Casual Game

Background: Citizen science games are a type of Games with a Purpose (GWAPs), whose aim is to harness the skills of volunteers for solving scientific problems or contributing to action projects, where citizens intervene in social concerns. Employing games to collect data, classify images or even solve major scientific problems is a relatively new but growing phenomenon in citizen science. A main concern in citizen science is to ensure data quality. As games can be seen as having adverse effects on data quality, it is important to understand how citizen scientists produce data using games, how accurate this data can be, and whether and how games influence data quality. Objective: The objective of this study was to evaluate the performance of individual players’ data quality in MalariaSpot, a citizen science casual game in which volunteers are tasked with detecting parasites in digitized blood sample images.Methods: We used descriptive statistics to analyze a subset of the gameplays recorded and stored in the MalariaSpot database, comparing its clicks to the Gold Standard position of the parasites. This subset includes 15,546 gameplays played over 38 known images that correspond to 97,200 clicks from 1,278 different players. Gameplays have been played via the Android and iOS applications and via the web version of the game. Images were acquired in three different locations and therefore sample preparation have been done by different lab technicians. Two distinct technologies were used for sample digitalization.Results: The overall values for sensibility, specificity, and accuracy of the individual gameplays for the 38 images are 0.82, 0.60, and 0.29 respectively. High presence of parasites in an image makes it easier for players to detect them as their structures tend to look alike and can be compared. Being a simple casual game, the learning curve is very fast and after few minutes, players attend their typical performance level. Data quality is considerably lower in images acquired with mobile phones coupled to the microscope ocular compared to those digitized with standardized digitalization technologies. Conclusions: The results indicate that data quality is influenced by the game, the technologies for image digitalization and the sampling preparation.

Science Game Lab: tool for the unification of biomedical games with a purpose

Games with a purpose and other kinds of citizen science initiatives demonstrate great potential for advancing biomedical science and improving STEM education. Articles documenting the success of projects such as Fold.it and Eyewire in high impact journals have raised wide interest in new applications of the distributed human intelligence that these systems have tapped into. However, the path from a good idea to a successful citizen science game remains highly challenging. Apart from the scientific difficulties of identifying suitable problems and appropriate human-powered solutions, the games still need to be created, need to be fun, and need to reach a large audience that remain engaged for the long-term. Here, we describe Science Game Lab (SGL) (https://sciencegamelab.org), a platform for bootstrapping the production, facilitating the publication, and boosting both the fun and the value of the user experience for scientific games with a purpose.