A Game Theory Approach for Estimating Reliability of Crowdsourced Relevance Assessments

2022 ◽  
Vol 40 (3) ◽  
pp. 1-29
Author(s):  
Yashar Moshfeghi ◽  
Alvaro Francisco Huertas-Rosero
Keyword(s):  
Upper Bound ◽  
Cost Effective ◽  
Majority Voting ◽  
Task Completion ◽  
Theory Approach ◽  
Weighted Majority ◽  
Use Of Time ◽  
Optimal Function ◽  
Closing The Gap ◽  
Source Of Information

In this article, we propose an approach to improve quality in crowdsourcing (CS) tasks using Task Completion Time (TCT) as a source of information about the reliability of workers in a game-theoretical competitive scenario. Our approach is based on the hypothesis that some workers are more risk-inclined and tend to gamble with their use of time when put to compete with other workers. This hypothesis is supported by our previous simulation study. We test our approach with 35 topics from experiments on the TREC-8 collection being assessed as relevant or non-relevant by crowdsourced workers both in a competitive (referred to as “Game”) and non-competitive (referred to as “Base”) scenario. We find that competition changes the distributions of TCT, making them sensitive to the quality (i.e., wrong or right) and outcome (i.e., relevant or non-relevant) of the assessments. We also test an optimal function of TCT as weights in a weighted majority voting scheme. From probabilistic considerations, we derive a theoretical upper bound for the weighted majority performance of cohorts of 2, 3, 4, and 5 workers, which we use as a criterion to evaluate the performance of our weighting scheme. We find our approach achieves a remarkable performance, significantly closing the gap between the accuracy of the obtained relevance judgements and the upper bound. Since our approach takes advantage of TCT, which is an available quantity in any CS tasks, we believe it is cost-effective and, therefore, can be applied for quality assurance in crowdsourcing for micro-tasks.

scholarly journals Group decisions based on confidence weighted majority voting

2021 ◽  
Vol 6 (1) ◽  
Author(s):  
Sascha Meyen ◽  
Dorothee M. B. Sigg ◽  
Ulrike von Luxburg ◽  
Volker H. Franz
Keyword(s):  
Cognitive Modeling ◽  
Group Discussion ◽  
Group Processes ◽  
Majority Voting ◽  
Group Decisions ◽  
Optimal Method ◽  
Weighted Majority ◽  
Real Group ◽  
Lower Accuracy ◽  
Weighted Majority Voting

Abstract Background It has repeatedly been reported that, when making decisions under uncertainty, groups outperform individuals. Real groups are often replaced by simulated groups: Instead of performing an actual group discussion, individual responses are aggregated by a numerical computation. While studies have typically used unweighted majority voting (MV) for this aggregation, the theoretically optimal method is confidence weighted majority voting (CWMV)—if independent and accurate confidence ratings from the individual group members are available. To determine which simulations (MV vs. CWMV) reflect real group processes better, we applied formal cognitive modeling and compared simulated group responses to real group responses. Results Simulated group decisions based on CWMV matched the accuracy of real group decisions, while simulated group decisions based on MV showed lower accuracy. CWMV predicted the confidence that groups put into their group decisions well. However, real groups treated individual votes to some extent more equally weighted than suggested by CWMV. Additionally, real groups tend to put lower confidence into their decisions compared to CWMV simulations. Conclusion Our results highlight the importance of taking individual confidences into account when simulating group decisions: We found that real groups can aggregate individual confidences in a way that matches statistical aggregations given by CWMV to some extent. This implies that research using simulated group decisions should use CWMV instead of MV as a benchmark to compare real groups to.

scholarly journals COVID-19 Patient Detection Based on Fusion of Transfer Learning and Fuzzy Ensemble Models Using CXR Images

2021 ◽  
Vol 11 (23) ◽  
pp. 11423
Author(s):  
Chandrakanta Mahanty ◽  
Raghvendra Kumar ◽  
Panagiotis G. Asteris ◽  
Amir H. Gandomi
Keyword(s):  
Transfer Learning ◽  
Ensemble Methods ◽  
Cost Effective ◽  
Majority Voting ◽  
Learning Models ◽  
Medical Practitioners ◽  
Healthcare Facilities ◽  
Ensemble Techniques ◽  
Effective Manner ◽  
Detection Strategies

The COVID-19 pandemic has claimed the lives of millions of people and put a significant strain on healthcare facilities. To combat this disease, it is necessary to monitor affected patients in a timely and cost-effective manner. In this work, CXR images were used to identify COVID-19 patients. We compiled a CXR dataset with equal number of 2313 COVID positive, pneumonia and normal CXR images and utilized various transfer learning models as base classifiers, including VGG16, GoogleNet, and Xception. The proposed methodology combines fuzzy ensemble techniques, such as Majority Voting, Sugeno Integral, and Choquet Fuzzy, and adaptively combines the decision scores of the transfer learning models to identify coronavirus infection from CXR images. The proposed fuzzy ensemble methods outperformed each individual transfer learning technique and several state-of-the-art ensemble techniques in terms of accuracy and prediction. Specifically, VGG16 + Choquet Fuzzy, GoogleNet + Choquet Fuzzy, and Xception + Choquet Fuzzy achieved accuracies of 97.04%, 98.48%, and 99.57%, respectively. The results of this work are intended to help medical practitioners achieve an earlier detection of coronavirus compared to other detection strategies, which can further save millions of lives and advantageously influence society.

Role of the Pediatrician in Patient Education

PEDIATRICS ◽  
1984 ◽  
Vol 74 (5) ◽  
pp. 914-916
Author(s):  
Vincent A. Fulginiti
Keyword(s):  
Patient Education ◽  
Cost Effective ◽  
Educational Materials ◽  
Residency Programs ◽  
Personal Involvement ◽  
Use Of Time ◽  
Parents And Children ◽  
Printed Materials ◽  
Effective Use

Although most pediatricians agree on the importance of teaching parents and children about health care, they may not succeed in patient education because of a lack of the requisite communication skills, inadequate printed materials to augment personal involvement, a tendency to substitute such materials for personal involvement, and inadequate compensation. Physician recognition of an obligation to teach is a requisite for effective education. Adjustment of current practices is essential: use of educational materials must be systematically incorporated, educational efforts must make effective use of time and be cost-effective. Information should be readily understood, parents given an opportunity to ask questions, and effectiveness of the education evaluated. Pediatricians must have a constant source of supplementary material to use in patient education. Videotapes and interactive computer programs should be considered for more effective communication. Residency programs must teach future pediatricians how to educate parents and children.

scholarly journals Image georectification and feature tracking toolbox: ImGRAFT

2015 ◽  
Vol 4 (1) ◽  
pp. 23-34 ◽  
Author(s):  
A. Messerli ◽  
A. Grinsted
Keyword(s):  
Feature Tracking ◽  
Cost Effective ◽  
Time Lapse ◽  
Post Processing ◽  
Outlet Glacier ◽  
Use Of Time ◽  
Camera Systems ◽  
Wide Range ◽  
Oblique Images ◽  
Minimal Effort

Abstract. The use of time-lapse camera systems is becoming an increasingly popular method for data acquisition. The camera setup is often cost-effective and simple, allowing for a large amount of data to be accumulated over a variety of environments for relatively minimal effort. The acquired data can, with the correct post-processing, result in a wide range of useful quantitative and qualitative information in remote and dangerous areas. The post-processing requires a significant amount of steps to transform images into meaningful data for quantitative analysis, such as velocity fields. To the best of our knowledge at present a complete, openly available package that encompasses georeferencing, georectification and feature tracking of terrestrial, oblique images is still absent. This study presents a complete, yet adaptable, open-source package developed in MATLAB, that addresses and combines each of these post-processing steps into one complete suite in the form of an "Image GeoRectification and Feature Tracking" (ImGRAFT: http://imgraft.glaciology.net) toolbox. The toolbox can also independently produce other useful outputs, such as viewsheds, georectified and orthorectified images. ImGRAFT is primarily focused on terrestrial oblique images, for which there are currently limited post-processing options available. In this study, we illustrate ImGRAFT for glaciological applications on a small outlet glacier Engabreen, Norway.

UPPER BOUND ON SNR GAIN IN WAVELET TRANSFORM PREDICTIVE-ENTROPY IMAGE CODING

1998 ◽  
Vol 08 (02) ◽  
pp. 267-272
Author(s):  
DRAGORAD MILOVANOVIĆ ◽  
ZORAN BOJKOVIĆ ◽  
ANDREJA SAMČOVIĆ
Keyword(s):  
Wavelet Transform ◽  
Image Coding ◽  
Upper Bound ◽  
Rate Distortion ◽  
Upper Bounds ◽  
Discrete Wavelet ◽  
Theory Approach ◽  
Still Image ◽  
Rate Distortion Theory ◽  
Snr Gain

Using the rate-distortion theory approach and assuming the Laplacian probability density function of the quantizer input signal, we propose a function which gives the signal-to-quantizing-noise ratio (SNR) gain in discrete wavelet transform predictive-entropy coding over fullband predictive image coding. The upper bound on SNR gain is determined as a function of the subband number only. The practical SNR performances of realizable wavelet still image coders are compared with their theoretical bounds. The computer simulation results of wavelet based predictive coded test images show that the SNR gain grows faster with the subband number increment than its theoretical upper bounds do.

scholarly journals An investigation of ensemble combination schemes for genetic programming based hyper-heuristic approaches to dynamic job shop scheduling

2021 ◽  
Author(s):  
John Park ◽  
Yi Mei ◽  
Su Nguyen ◽  
Gang Chen ◽  
Mengjie Zhang
Keyword(s):  
Genetic Programming ◽  
Linear Combination ◽  
Job Shop ◽  
Job Shop Scheduling ◽  
Majority Voting ◽  
Dispatching Rules ◽  
Shop Scheduling ◽  
Weighted Majority ◽  
Dynamic Job Shop ◽  
Weighted Majority Voting

Genetic programming based hyper-heuristic (GP-HH) approaches that evolve ensembles of dispatching rules have been effectively applied to dynamic job shop scheduling (JSS) problems. Ensemble GP-HH approaches have been shown to be more robust than existing GP-HH approaches that evolve single dispatching rules for dynamic JSS problems. For ensemble learning in classification, the design of how the members of the ensembles interact with each other, e.g., through various combination schemes, is important for developing effective ensembles for specific problems. In this paper, we investigate and carry out systematic analysis for four popular combination schemes. They are majority voting, which has been applied to dynamic JSS, followed by linear combination, weighted majority voting and weighted linear combination, which have not been applied to dynamic JSS. In addition, we propose several mea-sures for analysing the decision making process in the ensembles evolved by GP. The results show that linear combination is generally better for the dynamic JSS problem than the other combination schemes investigated. In addition, the different combination schemes result in significantly different interactions between the members of the ensembles. Finally, the analysis based on the measures shows that the behaviours of the evolved ensembles are significantly affected by the combination schemes. Weighted majority voting has bias towards single members of the ensembles. © This manuscript version is made available under the CC-BY-NC-ND 4.0 license https://creativecommons.org/licenses/by-nc-nd/4.0/

scholarly journals Farmers’ Perceptions of Information and Communication Technology (ICT) Use in Extension Service Delivery in Northern Region, Ghana

2019 ◽  
Vol 6 (1) ◽  
pp. 21-29
Author(s):  
Francis K. OBENG ◽  
Salifu GUMAH ◽  
Stephen Mintah
Keyword(s):  
Service Delivery ◽  
Mobile Phone ◽  
Cost Effective ◽  
Northern Region ◽  
Extension Service ◽  
Positive Effects ◽  
Use Of Time ◽  
Region Data ◽  
Information And Communication ◽  
Time And Energy

Extension methodologies for communicating technologies to farmers have evolved over the past 200 years from so-called traditional methodologies to more advanced and technology-based methodologies that enable extension staff to reach many people within the shortest possible time in a more effective and efficient way. Though traditional methods are still relevant and effective, current trends require the use of more innovative and cost-effective methodologies.  This paper examined the perceptions of farmers on the use of ICTs in Extension Service delivery in the Northern Region of Ghana. Ninety farmers were randomly sampled from 6 communities in 6 districts in the region. Data was collected using semi-structured questionnaire. A 5-point Likert scale was used to determine farmers’ perceived effects of ICT on extension delivery. Data were analysed using means, standard deviations, t-test, frequencies and percentages. The most widely used ICTs by farmers are radio, mobile phone and television. Farmers perceive the use of mobile phone to have resulted in timely delivery of information, increased interaction among farmers and between farmers and AEAs and effective use of time and energy by AEAs. The use of radio has improved adoption of technologies and enhanced farmers’ awareness of innovations. It is concluded mobile phone, radio and television are used widely in the region and have very positive effects on extension service delivery.  

Sign in / Sign up

Export Citation Format

Share Document