Item Validation--Feeble-Minded Subjects.

In the social sciences, validity refers to the adequacy of a survey (or other mode of assessment) for its intended purpose. Validation refers to the activities undertaken during and after the construction of the survey to evaluate and improve validity. Item validation refers here to procedures for evaluating and improving respondents’ understanding of the questions and response options included in a survey. Ver- bal probing techniques such as cognitive interviews can be used to understand respondents’ response process, that is, what they are thinking as they answer the survey items. Although cognitive interviews can provide evidence for the validity of survey items, they are time-consuming and thus rarely used in practice. The Response Process Evaluation (RPE) method is a newly-developed technique that utilizes open-ended meta-surveys to rapidly collect evidence of validity across a population of interest, make quick revisions to items, and immediately test these revisions across new samples of respondents. Like cognitive interviews, the RPE method focuses on how participants interpret the item and select a re- sponse. The chapter demonstrates the process of validating one survey item taken from the Inventory of Non-Ordinary Experiences (INOE).

Download Full-text

The Development of an Integrated Assessment Instrument for Measuring Analytical Thinking and Science Process Skills

10.31227/osf.io/8cdpb ◽

2018 ◽

Author(s):

Irwanto Irwanto

Keyword(s):

Data Collection ◽

Integrated Assessment ◽

Assessment Instrument ◽

Process Skills ◽

Science Process Skills ◽

Analytical Thinking ◽

D Model ◽

Essay Test ◽

Item Validation ◽

Item Reliability

This research aims to develop instrument and determine the characteristics of an integrated assessment instrument. This research uses 4-D model, which includes define, design, develop, and disseminate. The primary product is validated by expert judgment, tested it’s readability by students, and assessed it’s feasibility by chemistry teachers. This research involved 246 students of grade XI of four senior high schools in Yogyakarta, Indonesia. Data collection techniques include interview, questionnaire, and test. Data collection instruments include interview guideline, item validation sheet, users’ response questionnaire, instrument readability questionnaire, and essay test. The results show that the integrated assessment instrument has Aiken validity value of 0.95. Item reliability was 0.99 and person reliability was 0.69. Teachers’ response to the integrated assessment instrument is very good. Therefore, the integrated assessment instrument is feasible to be applied to measure the students’ analytical thinking and science process skills.

Download Full-text

The Case for Cognitive Interviewing in Survey Item Validation: A Useful Approach for Improving the Measurement and Assessment of Substance Use Disorders

10.31219/osf.io/u35m4 ◽

2020 ◽

Author(s):

Cassandra L. Boness ◽

Kenneth J. Sher

Keyword(s):

Substance Use ◽

Substance Use Disorders ◽

Substance Use Disorder ◽

Measure Development ◽

Cognitive Interviewing ◽

Alcohol Tolerance ◽

Item Generation ◽

Impaired Control ◽

Reduce Measurement Error ◽

Item Validation

Background: In order to accurately identify substance use disorders, we must be confident of our ability to define and measure the construct itself. To date, research has demonstrated that the ways in which substance use disorder criteria are operationalized or assessed can significantly impact the information we obtain from these diagnoses. For example, differing operationalizations of the same construct (e.g., impaired control over substance use) can result in markedly different estimates of prevalence. This points to the need for approaches that aim to improve the validity of diagnostic assessments during the measure development phase. Methods: We performed a scoping review of the cognitive interviewing literature, a technique that aims to provide a systematic way to identify and reduce measurement error associated with the structure and content of interview items. Along with this, we present an applied example with alcohol tolerance. Findings: We argue that cognitive interviewing is well suited for improving the validity of substance use disorder assessment items. Conclusions: We suggest that cognitive interviewing be incorporated in the item generation stage of measure development for substance use disorder assessments.

Download Full-text