Crowdsourcing expertise: Using Amazon's Mechanical Turk to develop scoring keys for situational judgment tests

It is common practice to rely on a convenience sample of subject matter experts (SMEs) when developing a scoring key for situational judgment tests (SJTs). However, the defining characteristics of what constitutes a SME are often ambiguous and inconsistent across studies. Other research fields have adopted crowdsourcing methods to replace or reproduce judgments thought to require subject matter expertise. Therefore, we conducted the current study to compare crowdsourced scoring keys to SME-based scoring keys for three SJTs in different domains, each varying in job-relatedness. Our results indicate that scoring keys derived from crowdsourced samples are likely to converge with keys based on SME judgment, regardless of test content (correlations ranging from r = .88 to .94). We observed the weakest agreement among individual MTurker and SME ratings for the more job-specific Medical SJT (classification consistency = 61%) but the aggregate scoring keys remained highly correlated. We observed stronger agreement in response option rankings for the Military and Communication SJTs (80% and 85%), which were both designed to require less procedural knowledge. Although general mental ability and conscientiousness were each related to greater expert similarity among MTurkers, the average crowd rating outperformed nearly all individual MTurk raters. Based on an analysis of randomly-drawn bootstrapped samples of MTurker ratings in each of the three samples, we found that as few as 30-40 raters may provide adequate estimates of SME judgments of most SJT items. We hope that these findings help inspire others to consider using crowdsourcing methods as an alternative to SMEs.

Download Full-text

Using situational judgment tests (SJTs) in training

Personnel Review ◽

10.1108/pr-05-2015-0137 ◽

2017 ◽

Vol 46 (1) ◽

pp. 36-45 ◽

Cited By ~ 4

Author(s):

Cody Brent Cox ◽

Laura G. Barron ◽

William Davis ◽

Bernardo de la Garza

Keyword(s):

Personnel Selection ◽

Procedural Knowledge ◽

Design Method ◽

Training Methods ◽

Training Design ◽

Situational Judgment ◽

Situational Judgment Tests ◽

Content Type ◽

Traditional Training ◽

Training Content

Purpose Situational judgment tests (SJTs) are widely used in personnel selection but have not been empirically explored as methods of training design. The purpose of this paper is to evaluate SJT-based training as a workplace training design method which utilizes active learning and structured feedback to enhance learning of both procedural and declarative knowledge. Design/methodology/approach Volunteers (n=416) were randomly assigned to full-length lecture-based training or abbreviated lecture-based training followed by 15 minutes of SJT-based training. Knowledge of training content was assessed at pre-test and three weeks after training. Findings SJT-based trainees showed greater improvement on declarative and procedural knowledge than those in traditional training. Research limitations/implications The results indicate that integrating the SJT methodology into training delivery may lead to greater mastery of declarative and procedural knowledge relative to exclusive use of lecture-based training methods. Practical implications Findings suggest that the relatively inexpensive, low-fidelity scenario-based training methodology the authors detail may increase retention of training material compared to more traditional training methods. Originality/value This is the first study to incorporate SJT methodology into the design of training content and to demonstrate that such content can produce greater retention of both declarative and procedural content.

Download Full-text

An empirical approach of identifying subject matter experts (SMEs) for the development of situational judgment tests

10.31234/osf.io/qkj8n ◽

2021 ◽

Author(s):

Don C. Zhang ◽

Yi Wang

Keyword(s):

Subject Matter ◽

Empirical Approach ◽

Subject Matter Experts ◽

Hypothetical Situation ◽

Situational Judgment Test ◽

Situational Judgment ◽

Situational Judgment Tests ◽

No Gold Standard ◽

Context Free ◽

Best Responses

The development of a scoring key for the situational judgment test (SJT) often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran-Weiss-Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS-index. Next, we demonstrate how the CWS-index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found the CWS index was associated with reflective thinking and intuition avoidance.

Download Full-text

Why Situational Judgment Is a Missing Component in the Theory of SJTs

Industrial and Organizational Psychology ◽

10.1017/iop.2015.111 ◽

2016 ◽

Vol 9 (1) ◽

pp. 29-34 ◽

Cited By ~ 5

Author(s):

Klaus G. Melchers ◽

Martin Kleinmann

Keyword(s):

Procedural Knowledge ◽

Situational Judgment ◽

Situational Judgment Tests

In their focal article, Lievens and Motowidlo (2016) consider procedural knowledge about effective actions in work situations as the key component of their theory of situational judgment tests (SJTs). In our commentary we want to suggest that situational judgment should nevertheless not be neglected in such a theory.

Download Full-text

An Empirical Approach to Identifying Subject Matter Experts for the Development of Situational Judgment Tests

Journal of Personnel Psychology ◽

10.1027/1866-5888/a000279 ◽

2021 ◽

pp. 1-13

Author(s):

Don C. Zhang ◽

Yi Wang

Keyword(s):

Subject Matter ◽

Empirical Approach ◽

Subject Matter Experts ◽

Hypothetical Situation ◽

Situational Judgment Test ◽

Situational Judgment ◽

Situational Judgment Tests ◽

No Gold Standard ◽

Context Free ◽

Best Responses

Abstract. The development of a scoring key for the situational judgment test often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran–Weiss–Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS index. Next, we demonstrate how the CWS index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found that the CWS index was associated with reflective thinking and intuition avoidance.

Download Full-text