An Analytic Creativity Assessment Scale for Digital Game Story Design: Construct Validity, Internal Consistency and Interrater Reliability

2015 ◽  
Vol 27 (4) ◽  
pp. 322-327 ◽  
Author(s):  
Tsung-Yen Chuang ◽  
Yun-Hsuan Huang
2021 ◽  
Vol 33 (S1) ◽  
pp. 87-88
Author(s):  
J. Antonio Garcia-Casal ◽  
Natacha Coelho de Cunha Guimarães ◽  
Sofía Díaz Mosquera ◽  
María Alvarez Ariza ◽  
Raimundo Mateos Álvarez

Background:Rowland Universal Dementia Assessment Scale (RUDAS) is a brief cognitive test, appropriate for people with minimum completed level of education and sensitive to multicultural contexts. It could be a good instrument for cognitive impairment (CI) screening in Primary Health Care (PHC). It comprises the following areas: recent memory, body orientation, praxis, executive functions and language.Research Objective:The objective of this study is to assess the construct validity of RUDAS analysing its internal consistency and factorial structure.Method:Internal consistency will be calculated using ordinal Cronbach’s α, which reflects the average inter-item correlation score and, as such, will increase when correlations between the items increase. Exploratory Factor Analysis will be used to arrange the variables in domains using principal components extraction. The factorial analysis will include the extraction of five factors reflecting the neuropsychological areas assessed by the test. The result will be rotated under Varimax procedure to ease interpretation.Exploratory factor analysis will be used to arrange the variables in domains using principal components extraction. The analysis will include Kaiser–Meyer–Olkin measure of sampling adequacy and Bartlett’s test of sphericity. Estimations will be based based on Pearson’s correlations between indicators using a principal component analysis and later replicated with a tetrachoric correlation matrix. The variance in the tetrachoric model will be analysed to indentify convergent iterations and their explicative power.Preliminary results of the ongoing study:RUDAS is being administered to 321 participants older than 65 years, from seven PHC physicians’ consultations in O Grove Health Center. The data collection will be finished by August 2021 and in this poster we will present the final results of the exploratory factor analysis.Conclusions:We expect that the results of the exploratory factor analysis will replicate the results of previous studies of construct validity of the test in which explanatory factor weights were between 0.57 and 0.82, and all were above 40%. Confirming that RUDAS has a strong factor construct with high factor weights and variance ratio, and 6-item model is appropriate for measurement will support its recommendation as a valid screening instrument for PHC.


2019 ◽  
Author(s):  
Emma Joy McMahon ◽  
Rachael Jaenke ◽  
Julie Brimblecombe

BACKGROUND Consumer food environments are increasingly being recognized as influential determinants of food purchasing and subsequent intake and health. We developed a tool to enable efficient, but relatively comprehensive, appraisal of the in-store food environment. The Store Scout mobile app facilitates the evaluation of product (availability and range), placement (visibility, accessibility, proximity to high-traffic areas, and location relative to other products), price (price promotion), and promotion (displays and advertising) across 7 categories of food products, with appraisal given immediately as scores (0-100, where a higher score is more in line with best practice). Primary end users are public health nutritionists and nutritionists employed by store organizations; however, store managers and staff are also potential end users. OBJECTIVE This study aims to evaluate the reliability (interrater reliability and internal consistency), utility (distribution of scores), and construct validity (score by store type) of measurements using the Store Scout mobile app. METHODS The Store Scout mobile app was used independently by 2 surveyors to evaluate the store environment in 54 stores: 34 metropolitan stores (9 small and 11 large supermarkets, 10 convenience stores, and 4 petrol stations) in Brisbane, Australia, and 20 remote stores (19 small supermarkets and 1 petrol station) in Indigenous Australian communities in Northern Australia. The agreement between surveyors in the overall and category scores was evaluated using intraclass correlation coefficients (ICCs). Interrater reliability of measurement items was assessed using percentage agreement and the Gwet agreement coefficient (AC). Internal consistency was assessed by comparing the responses of items measuring similar aspects of the store environment. We examined the distribution of score values using boxplots and differences by store type using the Kruskal-Wallis test. RESULTS The median difference in the overall score between surveyors was 4.4 (range 0.0-11.1), with an ICC of 0.954 (95% CI 0.914-0.975). Most measurement items had very good (n=74/196, 37.8%) or good (n=81/196, 41.3%) interrater reliability using the Gwet AC. A minimal inconsistency of measurement was found. Overall scores ranged from 19.2 to 81.6. There was a significant difference in score by store type (<i>P</i>&lt;.001). Large Brisbane supermarkets scored highest (median 77.4, range 53.2-81.6), whereas small Brisbane supermarkets (median 63.9, range 41.0-71.3) and small remote supermarkets (median 63.8, range 56.5-74.9) scored significantly higher than Brisbane petrol stations (median 33.1, range 19.2-37.8) and convenience stores (median 39.0, range 22.4-63.8). CONCLUSIONS These findings suggest good reliability and internal consistency of food environment measurements using the Store Scout mobile app. We identified specific aspects that can be improved to further increase the reliability of this tool. We found a good distribution of score values and evidence that scoring could capture differences by store type in line with previous evidence, which gives an indication of construct validity. The Store Scout mobile app shows promise in its capability to measure and track the health-enabling characteristics of store environments.


2019 ◽  
Vol 12 (2) ◽  
pp. 152-171
Author(s):  
Sara Dias-Trindade ◽  
José António Moreira ◽  
Catarina S. Nunes

RESUMO: Este trabalho apresenta um estudo que descreve os procedimentos de construção de uma escala de autoavaliação de competências digitais de professores e as suas qualidades psicométricas. Depois de algumas considerações sobre o construto e a sua operacionalização, efetuou-se a análise dos procedimentos da sua construção, realizou-se a análise de consistência interna através do cálculo do coeficiente alfa de Cronbach e a análise da validade do construto (análise fatorial exploratória e confirmatória). Para além de bons indicadores de validade, globalmente, as medidas aplicadas caracterizam-se por uma fidelidade boa e estruturas fatoriais interpretáveis. PALAVRAS-CHAVE: escala de autoavaliação; competências digitais docentes; fluência digital.   ABSTRACT: This paper presents a study that describes the procedures for the construction of a self-assessment scale of teachers' digital competences and their psychometric qualities. After some considerations about the construct and its operationalization, the procedures of its construction were analysed, the internal consistency analysis was carried out through the calculation of the Cronbach alpha coefficient and the analysis of the construct validity (exploratory and confirmatory factorial analysis). In addition to good indicators of validity, overall, the applied measures are characterized by a good fidelity and interpretable factorial structures. KEYWORDS: self-evaluation scale; teachers' digital competences; digital fluency.


10.2196/16971 ◽  
2020 ◽  
Vol 8 (7) ◽  
pp. e16971 ◽  
Author(s):  
Emma Joy McMahon ◽  
Rachael Jaenke ◽  
Julie Brimblecombe

Background Consumer food environments are increasingly being recognized as influential determinants of food purchasing and subsequent intake and health. We developed a tool to enable efficient, but relatively comprehensive, appraisal of the in-store food environment. The Store Scout mobile app facilitates the evaluation of product (availability and range), placement (visibility, accessibility, proximity to high-traffic areas, and location relative to other products), price (price promotion), and promotion (displays and advertising) across 7 categories of food products, with appraisal given immediately as scores (0-100, where a higher score is more in line with best practice). Primary end users are public health nutritionists and nutritionists employed by store organizations; however, store managers and staff are also potential end users. Objective This study aims to evaluate the reliability (interrater reliability and internal consistency), utility (distribution of scores), and construct validity (score by store type) of measurements using the Store Scout mobile app. Methods The Store Scout mobile app was used independently by 2 surveyors to evaluate the store environment in 54 stores: 34 metropolitan stores (9 small and 11 large supermarkets, 10 convenience stores, and 4 petrol stations) in Brisbane, Australia, and 20 remote stores (19 small supermarkets and 1 petrol station) in Indigenous Australian communities in Northern Australia. The agreement between surveyors in the overall and category scores was evaluated using intraclass correlation coefficients (ICCs). Interrater reliability of measurement items was assessed using percentage agreement and the Gwet agreement coefficient (AC). Internal consistency was assessed by comparing the responses of items measuring similar aspects of the store environment. We examined the distribution of score values using boxplots and differences by store type using the Kruskal-Wallis test. Results The median difference in the overall score between surveyors was 4.4 (range 0.0-11.1), with an ICC of 0.954 (95% CI 0.914-0.975). Most measurement items had very good (n=74/196, 37.8%) or good (n=81/196, 41.3%) interrater reliability using the Gwet AC. A minimal inconsistency of measurement was found. Overall scores ranged from 19.2 to 81.6. There was a significant difference in score by store type (P<.001). Large Brisbane supermarkets scored highest (median 77.4, range 53.2-81.6), whereas small Brisbane supermarkets (median 63.9, range 41.0-71.3) and small remote supermarkets (median 63.8, range 56.5-74.9) scored significantly higher than Brisbane petrol stations (median 33.1, range 19.2-37.8) and convenience stores (median 39.0, range 22.4-63.8). Conclusions These findings suggest good reliability and internal consistency of food environment measurements using the Store Scout mobile app. We identified specific aspects that can be improved to further increase the reliability of this tool. We found a good distribution of score values and evidence that scoring could capture differences by store type in line with previous evidence, which gives an indication of construct validity. The Store Scout mobile app shows promise in its capability to measure and track the health-enabling characteristics of store environments.


Author(s):  
Juliette Raulic ◽  
Vivian SY Leung ◽  
Grayson A Doss ◽  
Jennifer E Graham ◽  
Krista A Keller ◽  
...  

In biomedical research, rabbits are commonly sedated to facilitate a variety of procedures. Developing a sedation assessment scale enables standardization of levels of sedation and comparisons of sedation protocols, and may help in predicting sedation level requirements for different procedures. The goal of this study was to develop a rabbit sedation assessment scale using a psychometric approach. We hypothesized that the sedation scale would have construct validity, good internal consistency, and reliability. In a prospective, randomized, blinded study design, 15 (8 females, 7 males) healthy 1-y-old New Zealand white rabbits received 3 intramuscular treatments: midazolam (0.5 mg/kg; n = 6); midazolam (1.5 mg/kg)–ketamine (5 mg/kg; n = 7); and alfaxalone (4 mg/kg)–dexmedetomidine (0.1 mg/kg)–midazolam (0.2 mg/kg; n = 3). One rabbit received 2 treatments. A sedation scale was developed by using psychometric methods, with assessment performed by 6 independent raters who were blind to treatment. Final sedation scale items included posture, palpebral reflex, orbital tightening, lateral recumbency, loss of righting reflex, supraglottic airway device placement, toe pinch, and general appearance. The scale showed construct validity, good to very good interrater reliability for individual items (6 raters; intraclass correlation coefficient, 0.671 to 0.940), very good intrarater reliability (5 raters; intraclass correlation coefficient, 0.951 to 0.987), and excellent internal consistency (Cronbach α, 0.947). The sedation scale performed well under the conditions tested, suggesting that it can be applied in a wider range of settings (different populations, raters, sedation protocols).


1984 ◽  
Vol 14 (1) ◽  
pp. 159-166 ◽  
Author(s):  
B. F. Fuller

SynopsisThe reliability and validity of Psychological Stress Evaluator (PSE) output and frequency modulation suppression as an inaudible vocal measure of stress-arousal was further explored, using a new interval scoring method. ‘Yes’ answers to 6 neutral questions were obtained from 106 volunteers 2 weeks before, the day before and one week after a stress-provoking event (oral comprehensive examinations for an MS degree). The data show excellent construct validity and good but not high internal consistency (0·77–0·81) and test-retest (0·72–0·74) reliability. Interrater reliability ranged from 0·94 to 0·98. Correlations of PSE output with other psychological and physiological measures (criterion-related validity) was very poor.


Crisis ◽  
2013 ◽  
Vol 34 (1) ◽  
pp. 13-21 ◽  
Author(s):  
Philip J. Batterham ◽  
Alison L. Calear ◽  
Helen Christensen

Background: There are presently no validated scales to adequately measure the stigma of suicide in the community. The Stigma of Suicide Scale (SOSS) is a new scale containing 58 descriptors of a “typical” person who completes suicide. Aims: To validate the SOSS as a tool for assessing stigma toward suicide, to examine the scale’s factor structure, and to assess correlates of stigmatizing attitudes. Method: In March 2010, 676 staff and students at the Australian National University completed the scale in an online survey. The construct validity of the SOSS was assessed by comparing its factors with factors extracted from the Suicide Opinion Questionnaire (SOQ). Results: Three factors were identified: stigma, isolation/depression, and glorification/normalization. Each factor had high internal consistency and strong concurrent validity with the Suicide Opinion Questionnaire. More than 25% of respondents agreed that people who suicided were “weak,” “reckless,” or “selfish.” Respondents who were female, who had a psychology degree, or who spoke only English at home were less stigmatizing. A 16-item version of the scale also demonstrated robust psychometric properties. Conclusions: The SOSS is the first attitudes scale designed to directly measure the stigma of suicide in the community. Results suggest that psychoeducation may successfully reduce stigma.


Sign in / Sign up

Export Citation Format

Share Document