The Response Vector for Mastery Method of Standard Setting

2021 ◽  
pp. 001316442110323
Author(s):  
Dimiter M. Dimitrov
Keyword(s):  
Item Response ◽  
Real Data ◽  
Standard Setting ◽  
Scoring Method ◽  
Single Item ◽  
Probability Judgments ◽  
Setting Process ◽  
Cut Score ◽  
Methodological Aspects ◽  
Response Vector

Proposed is a new method of standard setting referred to as response vector for mastery (RVM) method. Under the RVM method, the task of panelists that participate in the standard setting process does not involve conceptualization of a borderline examinee and probability judgments as it is the case with the Angoff and bookmark methods. Also, the RVM-based computation of a cut-score is not based on a single item (e.g., marked in an ordered item booklet) but, instead, on a response vector (1/0 scores) on items and their parameters calibrated in item response theory or under the recently developed D-scoring method. Illustrations with hypothetical and real-data scenarios of standard setting are provided and methodological aspects of the RVM method are discussed.

scholarly journals Modeling of Item Response Functions Under the D-Scoring Method

2019 ◽  
Vol 80 (1) ◽  
pp. 126-144 ◽  
Author(s):  
Dimiter M. Dimitrov
Keyword(s):  
Item Response ◽  
Logistic Regression Model ◽  
Real Data ◽  
Standard Errors ◽  
Response Functions ◽  
Inverse Regression ◽  
Scoring Method ◽  
Two Parameters ◽  
Function Models ◽  
True Scores

This study presents new models for item response functions (IRFs) in the framework of the D-scoring method (DSM) that is gaining attention in the field of educational and psychological measurement and largescale assessments. In a previous work on DSM, the IRFs of binary items were estimated using a logistic regression model (LRM). However, the LRM underestimates the item true scores at the top end of the D-scale (ranging from 0 to 1), especially for relatively difficult items. This entails underestimation of true D-scores, inaccuracy in the estimates of their standard errors, and other psychometric issues. The inverse-regression adjustments used to fix this problem are too complicated for regular applications of the DSM and not in line with its simplicity. This issue is resolved with the IRF models proposed in this study, referred to as rational function models (RFMs) with one parameter (RFM1), two parameters (RFM2), and three parameters (RFM3). The proposed RFMs are discussed and illustrated with simulated and real data.

scholarly journals Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee

2018 ◽  
Vol 15 ◽  
pp. 32 ◽  
Author(s):  
Janghee Park ◽  
Duck-Sun Ahn ◽  
Mi Kyoung Yim ◽  
Jaehyoung Lee
Keyword(s):  
Panel Discussion ◽  
Standard Setting ◽  
Angoff Method ◽  
Percentile Score ◽  
Setting Process ◽  
Cut Score ◽  
Licensing Examination ◽  
Practical Guidance ◽  
Radiological Technologist

Purpose: This study aimed to compare the possible standard-setting methods for the Korean Radiological Technologist Licensing Examination, which has a fixed cut score, and to suggest the most appropriate method. Methods: Six radiological technology professors set standards for 250 items on the Korean Radiological Technologist Licensing Examination administered in December 2016 using the Angoff, Ebel, bookmark, and Hofstee methods. Results: With a maximum percentile score of 100, the cut score for the examination was 71.27 using the Angoff method, 62.2 using the Ebel method, 64.49 using the bookmark method, and 62 using the Hofstee method. Based on the Hofstee method, an acceptable cut score for the examination would be between 52.83 and 70, but the cut score was 71.27 using the Angoff method. Conclusion: The above results suggest that the best standard-setting method to determine the cut score would be a panel discussion with the modified Angoff or Ebel method, with verification of the rated results by the Hofstee method. Since no standard-setting method has yet been adopted for the Korean Radiological Technologist Licensing Examination, this study will be able to provide practical guidance for introducing a standard-setting process.

Acid-Causing Emission Standards in Alberta: The Standard-Setting Process, Enforcement and Forums

1985 ◽  
Vol 23 (1) ◽  
pp. 1
Author(s):  
Shelley N. Phillips ◽  
Gale L. Pretash
Keyword(s):  
Standard Setting ◽  
The Public ◽  
Public Input ◽  
Emission Standards ◽  
Setting Process ◽  
Scientific Analysis

The implementation and enforcement of standards regulating acid-causing emissions in Alberta are examined in this paper. Also addressed are the available forums for public input, the need for further scientific analysis and better communication between government, industry and the public.

The Development of Auditing Standards at the PCAOB and Related Audit Practitioner Input

2020 ◽  
Author(s):  
L. Tyler Williams ◽  
W. Mark Wilder
Keyword(s):  
Standard Setting ◽  
The Public ◽  
Setting Process ◽  
Public Company ◽  
Auditing Standards ◽  
Audit Standard ◽  
Comment Letters ◽  
Feedback Responses

The Public Company Accounting Oversight Board (PCAOB; Board) maintains that constituent feedback plays an essential and dynamic role in its audit standard-setting process. We examine a major source of constituent feedback, responses to standard-setting questions, using a sample drawn from the original proposals of fourteen PCAOB auditing standards. We find that after receiving comment letter feedback to the standard-setting questions, the Board revises approximately half of its guidance tied to those questions before it finalizes auditing standards-a finding consistent with the Board's assertion that it carefully considers constituent perspectives as it develops new regulation. We also explore the related comment letters of eight professional auditing firms subject to the PCAOB's annual inspection program and discover varying levels of opposition to and support for the PCAOB's proposed authoritative guidance. We observe PCAOB revision to authoritative guidance highly contested by the firms in more than three-fourths of cases of standard-setting questions and PCAOB non-revision to guidance highly supported by the firms in more than ninety percent of cases.

The domain expert perspective: A qualitative study into the views expressed in a standard-setting exercise on a language for specific purposes (LSP) test for health professionals

2021 ◽  
pp. 026553222110107
Author(s):  
Simon Davidson
Keyword(s):  
Qualitative Data ◽  
Proficiency Test ◽  
Standard Setting ◽  
Domain Expert ◽  
Verbal Reports ◽  
Domain Experts ◽  
Language For Specific Purposes ◽  
Setting Process ◽  
Test Criteria ◽  
Textual Features

This paper investigates what matters to medical domain experts when setting standards on a language for specific purposes (LSP) English proficiency test: the Occupational English Test’s (OET) writing sub-test. The study explores what standard-setting participants value when making performance judgements about test candidates’ writing responses, and the extent to which their decisions are language-based and align with the OET writing sub-test criteria. Qualitative data is a relatively under-utilized component of standard setting and this type of commentary was garnered to gain a better understanding of the basis for performance decisions. Eighteen doctors were recruited for standard-setting workshops. To gain further insight, verbal reports in the form of a think-aloud protocol (TAP) were employed with five of the 18 participants. The doctors’ comments were thematically coded and the analysis showed that participants’ standard-setting judgements often aligned with the OET writing sub-test criteria. An overarching theme, ‘Audience Recognition’, was also identified as valuable to participants. A minority of decisions were swayed by features outside the OET’s communicative construct (e.g., clinical competency). Yet, overall, findings indicated that domain experts were undeniably focused on textual features associated with what the test is designed to assess and their views were vitally important in the standard-setting process.

Effective customer selection for marketing campaigns based on net scores

2017 ◽  
Vol 11 (1) ◽  
pp. 2-15 ◽  
Author(s):  
René Michel ◽  
Igor Schnakenburg ◽  
Tobias von Martens
Keyword(s):  
Decision Trees ◽  
Direct Marketing ◽  
Real Data ◽  
Business Case ◽  
Added Value ◽  
Scoring Method ◽  
Content Type ◽  
Effective Selection ◽  
Selection For ◽  
Response Modeling

Purpose This paper aims to address the effective selection of customers for direct marketing campaigns. It introduces a new method to forecast campaign-related uplifts (also known as incremental response modeling or net scoring). By means of these uplifts, only the most responsive customers are targeted by a campaign. This paper also aims at calculating the financial impact of the new approach compared to the classical (gross) scoring methods. Design/methodology/approach First, gross and net scoring approaches to customer selection for direct marketing campaigns are compared. After that, it is shown how net scoring can be applied in practice with regard to different strategical objectives. Then, a new statistic for net scoring based on decision trees is developed. Finally, a business case based on real data from the financial sector is calculated to compare gross and net scoring approaches. Findings Whereas gross scoring focuses on customers with a high probability of purchase, regardless of being targeted by a campaign, net scoring identifies those customers who are most responsive to campaigns. A common scoring procedure – decision trees – can be enhanced by the new statistic to forecast those campaign-related uplifts. The business case shows that the selected scoring method has a relevant impact on economical indicators. Practical implications The contribution of net scoring to campaign effectiveness and efficiency is shown by the business case. Furthermore, this paper suggests a framework for customer selection, given strategical objectives, e.g. minimizing costs or maximizing (gross or lift)-added value, and presents a new statistic that can be applied to common scoring procedures. Originality/value Despite its lever on the effectiveness of marketing campaigns, only few contributions address net scores up to now. The new χ2-statistic is a straightforward approach to the enhancement of decision trees for net scoring. Furthermore, this paper is the first to the application of net scoring with regard to different strategical objectives.

Sign in / Sign up

Export Citation Format

Share Document