Evaluating subscore uses across multiple levels: A case of reading and listening subscores for young EFL learners

2019 ◽  
Vol 37 (2) ◽  
pp. 254-279 ◽  
Author(s):  
Ikkyu Choi ◽  
Spiros Papageorgiou
Keyword(s):  
Classical Test Theory ◽  
Test Theory ◽  
Added Value ◽  
School Level ◽  
Individual Level ◽  
Foreign Language Learners ◽  
Language Tests ◽  
Multiple Levels ◽  
The Individual ◽  
Level Analysis

Stakeholders of language tests are often interested in subscores. However, reporting a subscore is not always justified; a subscore should provide reliable and distinct information to be worth reporting. When a subscore is used for decisions across multiple levels (e.g., individual test takers and schools), it needs to be justified for its reliability and distinctiveness at every relevant level. In this study, we examined whether reporting seven Reading and Listening subscores of the TOEFL Primary® test, a standardized English proficiency test for young English as a foreign language learners, could be justified for reporting at individual and school levels. We analyzed data collected in pilot administrations, in which 4776 students from 51 schools participated. We employed the classical test theory (CTT) based approaches of Haberman (2008) and Haberman, Sinharay, and Puhan (2009) for the individual and school-level investigations, respectively. We also supplemented the CTT-based approaches with a factor analytic approach for the individual level analysis and a multilevel modeling approach for the school-level analysis. The results differed across the two levels: we found little support for reporting the subscores at the individual level, but strong evidence supporting the added-value of the school-level subscores when the sample size for each school exceeds 50.

scholarly journals The comparison of the scores obtained by Bayesian nonparametric model and classical test theory methods

2021 ◽  
Vol 104 (3) ◽  
pp. 003685042110283
Author(s):  
Meltem Yurtcu ◽  
Hülya Kelecioglu ◽  
Edward L Boone
Keyword(s):  
Classical Test Theory ◽  
Small Sample ◽  
Test Theory ◽  
Nonparametric Model ◽  
Bayesian Nonparametric ◽  
Test Equating ◽  
Classical Test ◽  
A Value ◽  
Item Functioning ◽  
The Individual

Bayesian Nonparametric (BNP) modelling can be used to obtain more detailed information in test equating studies and to increase the accuracy of equating by accounting for covariates. In this study, two covariates are included in the equating under the Bayes nonparametric model, one is continuous, and the other is discrete. Scores equated with this model were obtained for a single group design for a small group in the study. The equated scores obtained with the model were compared with the mean and linear equating methods in the Classical Test Theory. Considering the equated scores obtained from three different methods, it was found that the equated scores obtained with the BNP model produced a distribution closer to the target test. Even the classical methods will give a good result with the smallest error when using a small sample, making equating studies valuable. The inclusion of the covariates in the model in the classical test equating process is based on some assumptions and cannot be achieved especially using small groups. The BNP model will be more beneficial than using frequentist methods, regardless of this limitation. Information about booklets and variables can be obtained from the distributors and equated scores that obtained with the BNP model. In this case, it makes it possible to compare sub-categories. This can be expressed as indicating the presence of differential item functioning (DIF). Therefore, the BNP model can be used actively in test equating studies, and it provides an opportunity to examine the characteristics of the individual participants at the same time. Thus, it allows test equating even in a small sample and offers the opportunity to reach a value closer to the scores in the target test.

Levels of Aggregation in Conjoint Analysis: An Empirical Comparison

1980 ◽  
Vol 17 (4) ◽  
pp. 516-523 ◽  
Author(s):  
William L. Moore
Keyword(s):  
Conjoint Analysis ◽  
Predictive Power ◽  
Aggregate Level ◽  
Segmentation Method ◽  
Empirical Comparison ◽  
Individual Level ◽  
Aggregate Analysis ◽  
The Individual ◽  
Level Analysis

Two segmented methods of performing conjoint anal/sis, clustered and componential segmentation, are compared with each other as well as with individual level and totally aggregate level analyses. The two segmented methods provide insights to the data that (1) are not obtainable at the aggregate level and (2) are in a form that is more easily communicated than the information from the individual level analysis. The predictive power of the clustered segmentation method is higher than that of componential segmentation, and both are superior to the aggregate analysis but inferior to individual level analysis.

Does School Matter?

2008 ◽  
Vol 13 (1) ◽  
pp. 12-23 ◽  
Author(s):  
Katariina Salmela-Aro ◽  
Noona Kiuru ◽  
Minna Pietikäinen ◽  
Jukka Jokela
Keyword(s):  
School Climate ◽  
Secondary School ◽  
Secondary School Students ◽  
School Level ◽  
School Students ◽  
Individual Level ◽  
Upper Secondary ◽  
School Burnout ◽  
The Individual ◽  
Positive Motivation

School burnout can be defined as consisting of exhaustion due to school demands, cynical, and detached attitude toward one’s school, and feelings of inadequacy as a student ( Kiuru, Aunola, Nurmi, Leskinen, & Salmela-Aro, 2008 ; Salmela-Aro & Näätänen, 2005 ; Schaufeli, Martínez, Pinto, Salanova, & Bakker, 2002 ). The first aim of the present study was to examine the extent to which schools differ in school-related burnout. Moreover, the aim was to examine the extent to which school-related and background factors are associated with school burnout at the school level and at the individual level. The participants were 58,657 students from 431 comprehensive schools and 29,515 students from 228 upper secondary schools who filled in a questionnaire measuring their school burnout, school-related variables (i.e., negative school climate, positive motivation received from teachers, support from the school), and background variables (i.e., gender, grade-point average, socio-economic status, and family structure). The results revealed only small differences between schools in school burnout. Among the comprehensive school students the results at the school-level showed that negative school climate typical of the school was positively related, while support from school shared among school members was negatively related to school-related burnout. Among upper secondary school students, in turn, positive motivation received from teachers typical of the school was negatively related to school-related burnout. At the individual level, negative school climate was positively related, and support from school and positive motivation received from teachers were negatively related to burnout among both the comprehensive and upper secondary school students. In addition, girls and those with lower GPA experienced higher levels of school burnout compared to boys and those with higher GPA.

scholarly journals Using Differential Item Functioning to Analyze the Domain Generality of a Common Scientific Reasoning Test

2021 ◽  
Author(s):  
Ansgar Opitz ◽  
Moritz Heene ◽  
Frank Fischer
Keyword(s):  
Differential Item Functioning ◽  
Scientific Reasoning ◽  
Classical Test Theory ◽  
Test Theory ◽  
Added Value ◽  
Biology Students ◽  
Classical Test ◽  
Reasoning Test ◽  
Domain Generality ◽  
Item Functioning

Abstract. A significant problem that assessments of scientific reasoning face at the level of higher education is the question of domain generality, that is, whether a test will produce biased results for students from different domains. This study applied three recently developed methods of analyzing differential item functioning (DIF) to evaluate the domain generality assumption of a common scientific reasoning test. Additionally, we evaluated the usefulness of these new, tree- and lasso-based, methods to analyze DIF and compared them with methods based on classical test theory. We gave the scientific reasoning test to 507 university students majoring in physics, biology, or medicine. All three DIF analysis methods indicated a domain bias present in about one-third of the items, mostly benefiting biology students. We did not find this bias by using methods based on classical test theory. Those methods indicated instead that all items were easier for physics students compared to biology students. Thus, the tree- and lasso-based methods provide a clear added value to test evaluation. Taken together, our analyses indicate that the scientific reasoning test is neither entirely domain-general, nor entirely domain-specific. We advise against using it in high-stakes situations involving domain comparisons.

scholarly journals Nonergodicity in Load and Recovery: Group Results Do Not Generalize to Individuals

2021 ◽  
pp. 1-9
Author(s):  
Niklas D. Neumann ◽  
Nico W. Van Yperen ◽  
Jur J. Brauers ◽  
Wouter Frencken ◽  
Michel S. Brink ◽  
...  
Keyword(s):  
Perceived Exertion ◽  
Rating Of Perceived Exertion ◽  
Group Level ◽  
Total Quality ◽  
Individual Level ◽  
Quality Of Recovery ◽  
Level Statistics ◽  
The Individual ◽  
Level Analysis

Purpose: The study of load and recovery gained significant interest in the last decades, given its important value in decreasing the likelihood of injuries and improving performance. So far, findings are typically reported on the group level, whereas practitioners are most often interested in applications at the individual level. Hence, the aim of the present research is to examine to what extent group-level statistics can be generalized to individual athletes, which is referred to as the “ergodicity issue.” Nonergodicity may have serious consequences for the way we should analyze, and work with, load and recovery measures in the sports field. Methods: The authors collected load, that is, rating of perceived exertion × training duration, and total quality of recovery data among youth male players of a professional football club. This data were collected daily across 2 seasons and analyzed on both the group and the individual level. Results: Group- and individual-level analysis resulted in different statistical outcomes, particularly with regard to load. Specifically, SDs within individuals were up to 7.63 times larger than SDs between individuals. In addition, at either level, the authors observed different correlations between load and recovery. Conclusions: The results suggest that the process of load and recovery in athletes is nonergodic, which has important implications for the sports field. Recommendations for training programs of individual athletes may be suboptimal, or even erroneous, when guided by group-level outcomes. The utilization of individual-level analysis is key to ensure the optimal balance of individual load and recovery.

scholarly journals Experiences of managing a gluten-free diet on multiple levels of society: a qualitative study

BMC Nutrition ◽  
2020 ◽  
Vol 6 (1) ◽  
Author(s):  
Lisa Garnweidner-Holme ◽  
Karla Sende ◽  
Monica Hellmann ◽  
Christine Henriksen ◽  
Knut E. A. Lundin ◽  
...  
Keyword(s):  
Interpretative Phenomenological Analysis ◽  
Political Action ◽  
Gluten Free ◽  
Free Products ◽  
Individual Level ◽  
Individual Interviews ◽  
Immune Mediated ◽  
Multiple Levels ◽  
The Individual ◽  
Selection Of

Abstract Background Coeliac disease (CD) is an immune-mediated enteropathy against dietary gluten. The treatment for CD is a strict life-long gluten-free (GF) diet, which has a profound effect on a person’s life. In recent years, there has been an increase in the availability of gluten-free products. This study investigates how people with CD experience and manage a GF diet. Methods Semi-structured, individual interviews were conducted in different areas of Norway. The analysis was guided by Interpretative Phenomenological Analysis. Participants with CD (n = 12) varied in terms of gender, age, family composition and time since diagnosed. Results The analysis revealed challenges for a GF diet at the individual, interpersonal, community and policy levels. At the individual level, the participants explained that it took time to gain knowledge about a GF diet, and they expressed uncertainty about the healthiness of a GF diet. At the interpersonal level, the feeling of being different and the fear of gluten contamination were barriers to the enjoyment of social meals. At the community level, the participants asked for a wider selection of tastier GF products to purchase and increased knowledge about CD among those who prepare and sell GF foods. At the policy level, the participants asked for political action to make GF products more affordable. Conclusions This study indicates that people with CD should be given information about how to manage a GF diet right after being diagnosed with CD. The food industry should be encouraged to produce healthy and tasty GF products.

scholarly journals Racialised professionals’ experiences of selective incivility in organisations: A multi-level analysis of subtle racism

2020 ◽  
pp. 001872672095772
Author(s):  
Mustafa Bilgehan Ozturk ◽  
Aykut Berber
Keyword(s):  
Structured Interviews ◽  
Dynamic Changes ◽  
Individual Level ◽  
Subtle Racism ◽  
Societal Level ◽  
Conditions Of Possibility ◽  
Multi Level ◽  
The Individual ◽  
Level Analysis ◽  
Organisational Level

This article explores how racialised professionals experience selective incivility in UK organisations. Analysing 22 in-depth, semi-structured interviews, we provide multi-level findings that relate to individual, organisational and societal phenomena to illuminate the workings of subtle racism. On the individual level, selective incivility appears as articulated through ascriptions of excess and deficit that marginalise racialised professionals; biased actions by white employees who operate as honest liars or strategic coverers; and white defensiveness against selective incivility claims. On the organisational level, organisational whitewashing, management denial and upstream exclusion constitute the key enablers of selective incivility. On the societal level, dynamic changes relating to increasing intolerance outside organisations indirectly yet sharply fuel selective incivility within organisations. Finally, racialised professionals experience intersectional (dis-)advantages at the imbrications of individual, organisation and society levels, shaping within-group variations in experiences of workplace selective incivility. Throughout all three levels of analysis and their interplay, differences in power and privilege inform the conditions of possibility for and the continual reproduction of selective incivility.

Introduction

2018 ◽  
pp. 1-13
Author(s):  
Ron Avi Astor ◽  
Rami Benbenishty
Keyword(s):  
School Violence ◽  
School Level ◽  
Cultural Groups ◽  
Traditional Bullying ◽  
Base Rates ◽  
Similarities And Differences ◽  
Definition Of ◽  
The Individual ◽  
Level Analysis ◽  
Over Time

This introduction chapter presents the historical, personal, and academic background that led to this book that examines a model of school violence in context. The chapter presents the book chapters: a revised model of school violence in evolving contexts; the definition of bullying and school violence; exploring similarities and differences between groups in base rates of victimization and in the structures of victimization types; the multiple ways of exposure to a range of weapons on school grounds, focusing on school-level analysis; sexual victimization and its association with cultural groups; suicide and its relations with bullying both on the individual and school levels; the victimization of teachers by students and of students by their teachers and their interrelationships on a student and school-level; multiple forms of cyberbullying and their relationships with “traditional” bullying; school climate, its definitions, and relationships with victimization and academic achievement over time; policy and intervention implications.

Individual- and School-Level Factors Contributing to Disproportionate Suspension Rates: A Multilevel Analysis of One State

2018 ◽  
Vol 27 (4) ◽  
pp. 209-220 ◽  
Author(s):  
Kristine A. Camacho ◽  
Michael P. Krezmien
Keyword(s):  
High Schools ◽  
Students With Disabilities ◽  
Multilevel Model ◽  
English Language ◽  
School Enrollment ◽  
School Level ◽  
Highly Qualified ◽  
Individual Level ◽  
Increased Risk ◽  
The Individual

Data from middle schools ( n = 219), high schools ( n = 200), and combined middle and high schools ( n = 20) were used to examine individual- and school-level factors within a multilevel model associated with an increased risk of suspension for minority students and students with disabilities. Results indicate that the individual-level variables of race and disability status were associated with an increased risk of suspension. Multiple school-level factors were also found to be associated with an increased risk of suspension including school enrollment, attendance, mobility, the percent of highly qualified teachers, the percent of students receiving free and reduced priced meals, the percent of students receiving special education services, the school’s Title I status, the student-to-teacher ratio, English Language Arts state exam scores, and the percent of White students in the school. In both analyses, the majority of variance was associated with the multilevel model which indicates the importance of examining individual factors within the context of school-level factors when trying to understand and respond to disproportionate suspension practices.

EXPRESS: Trust Propensity across Cultures: The Role of Collectivism

2021 ◽  
pp. 1069031X2110366
Author(s):  
Stanford A. Westjohn ◽  
Peter Magnusson ◽  
George R. Franke ◽  
Yi Peng
Keyword(s):  
Past Research ◽  
Trust Game ◽  
Outcome Variable ◽  
Individual Level ◽  
The Us ◽  
The Individual ◽  
Trust Propensity ◽  
Explanatory Mechanism ◽  
Level Analysis

Does collectivism influence an individual's willingness to trust others? Conflicting empirical results from past research and the role of trust in international marketing make this question important to resolve. We investigate this question across cultures and at the individual level with four studies using multiple methods. Study 1 establishes correlational evidence between societal-level collectivism and individual-level trust propensity with results from a multi-level analysis of data from over 6,000 respondents in 36 different countries. Study 2 offers an individual-level analysis using the trust game, introducing a more rigorous behavioral outcome variable. Study 3 contributes causal evidence at the individual level based on experiments in both the US and China and offers evidence of social projection as the explanatory mechanism. Finally, Study 4 demonstrates managerial relevance by using advertising to prime collectivism and assessing its effect on trust in the firm.

Sign in / Sign up

Export Citation Format

Share Document