scholarly journals Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

2017 ◽  
Vol 48 (3) ◽  
pp. 319-334 ◽  
Author(s):  
Jia He ◽  
Janine Buchholz ◽  
Eckhard Klieme
Keyword(s):  
Classroom Management ◽  
Predictive Validity ◽  
Low Socioeconomic Status ◽  
Teacher Support ◽  
International Student ◽  
Math Achievement ◽  
Student Assessment ◽  
Self Report ◽  
Anchoring Vignettes ◽  
International Student Assessment

Anchoring vignettes are item batteries especially designed for correcting responses that might be affected by incomparability. This article investigates the effects of anchoring vignettes on the validity of student self-report data in 64 cultures. Using secondary data analysis from the 2012 Programme for International Student Assessment (PISA), we checked the validity of ratings on vignette questions, and investigated how rescaled item responses of two student scales, Teacher Support and Classroom Management, enhanced comparability and predictive validity. The main findings include that (a) responses to vignette questions represent valid individual and cultural differences; in particular, violations in these responses (i.e., misorderings) are related to low socioeconomic status and low cognitive sophistication; (b) the rescaled responses tend to show higher levels of comparability; and (c) the associations of rescaled Teacher Support and Classroom Management with math achievement, Student-Oriented Instruction, and Teacher-Directed Instruction are slightly different from raw scores of the two target constructs, and the associations with rescaled scores seem to be more in line with the literature. Namely, the associations among all self-report Likert-type scales are weaker with rescaled scores, presumably reducing common method variance, and both rescaled scale scores are more positively related to math achievement. The country ranking also changes substantially; in particular, Asian cultures top the ranking on Teacher Support after rescaling. However, anchoring vignettes are not a cure-all in solving measurement bias in cross-cultural surveys; we discuss the technicality and directions for further research on this technique.

scholarly journals Students’ Math Self-Concept, Math Anxiety, and Math Achievement: The Moderating Role of Teacher Support

2021 ◽  
Vol 5 (1) ◽  
pp. 45
Author(s):  
Satoshi Oda ◽  
Chiaki Konishi ◽  
Takashi Oba ◽  
Tracy K. Y. Wong ◽  
Xiaoxue Kong ◽  
...  
Keyword(s):  
Emotional Support ◽  
Teacher Support ◽  
International Student ◽  
Math Achievement ◽  
Math Anxiety ◽  
Student Assessment ◽  
Self Concept ◽  
Instrumental Support ◽  
International Student Assessment

This study explored the moderating roles of teacher instrumental and emotional support on the association between students’ math anxiety/math self-concept and math achievement. Participants included 21,544 Canadian students aged 15 years (10,943 girls) who participated in the 2012 Program for International Student Assessment. Results indicated that instrument support and emotional support were positively associated with math achievement. A significant moderation effect was evident between instrumental support and math anxiety; higher levels of instrumental support were associated with higher math achievement at low levels of math anxiety. Emotional support did not interact with math anxiety or math self-concept. The present findings highlight the importance to consider not only individual factors (i.e., math anxiety and math self-concept) but also the role of teacher support in supporting math achievement. 

National Gross Domestic Product, Science Interest, and Science Achievement: A Direct Replication and Extension of the Tucker-Drob, Cheung, and Briley (2014) Study

2019 ◽  
Vol 30 (5) ◽  
pp. 776-788 ◽  
Author(s):  
Anqing Zheng ◽  
Elliot M. Tucker-Drob ◽  
Daniel A. Briley
Keyword(s):  
Gross Domestic Product ◽  
Low Socioeconomic Status ◽  
Science Achievement ◽  
International Student ◽  
Student Assessment ◽  
Domestic Product ◽  
Student Interest ◽  
Similar Data ◽  
Science Interest ◽  
International Student Assessment

We replicated the study by Tucker-Drob, Cheung, and Briley (2014), who found that the association between science interest and science knowledge depended on economic resources at the family, school, and national levels, using data from the 2006 Programme for International Student Assessment (PISA). In more economically prosperous families, schools, and nations, student interest was more strongly correlated with actual knowledge. Here, we investigated whether these results still held despite substantial changes to educational and economic systems over roughly a decade. Using similar data from PISA 2015 ( N = 537,170), we found largely consistent results. Students from more economically advantaged homes, schools, and nations exhibited a stronger link between interests and knowledge. However, these moderation effects were substantially reduced, and the main effect of science interest increased by nearly 25%, driven almost entirely by families of low socioeconomic status and nations with low gross domestic product. The interdependence of interests and resources is robust but perhaps weakening with educational progress.

scholarly journals Examining school variables affecting PISA 2012 math achievement in Turkey and Shanghai-China

2017 ◽  
Vol 6 (4) ◽  
pp. 167-174
Author(s):  
Mustafa Kale
Keyword(s):  
International Student ◽  
Math Achievement ◽  
Student Assessment ◽  
International Student Assessment ◽  
Pisa 2012 ◽  
Math Scores ◽  
Model Research ◽  
School Variables ◽  
Research Population ◽  
The Difference

The main purpose of the research is to examine school variables that have effect on Programme for International Student Assessment (PISA) 2012 math achievement in Turkey and Shanghai-China. The research was designed in casual comparison model. Research population was constituted by student in age group of 15 in Turkey and Shanghai-China in 2012.The sample consists of 4848 students and 170 schools in Turkey and 5177 students and 155 schools in  Shanghai-China that participated in PISA 2012. Two-leveled Hierarchical Linear Modelling (HLM) was used to analyze data because the data collected in PISA 2012 had a hierarchical data structure. As a result of analysis, variability in math scores, 63% in Turkey and  47% in Shanghai-China, was found due to the difference between the mean math scores of schools. It was determined that  MACTIV, SCMATEDU and TCMORALE in Turkey and MACTIV, in Shanghai-China statistically affect on math achievement. Keywords: PISA, school administration, school variables, HLM

The Role of Personality in Predicting (Change in) Students’ Academic Success Across Four Years of Secondary School

2016 ◽  
Vol 32 (1) ◽  
pp. 95-103 ◽  
Author(s):  
Marion Spengler ◽  
Martin Brunner ◽  
Romain Martin ◽  
Oliver Lüdtke
Keyword(s):  
Academic Success ◽  
Predictive Validity ◽  
Big Five ◽  
Educational Outcomes ◽  
International Student ◽  
Longitudinal Research ◽  
Student Assessment ◽  
Big Five Personality ◽  
International Student Assessment ◽  
Incremental Predictive Validity

Abstract. Little longitudinal research has addressed the question of whether the Big Five personality traits (i.e., O, C, E, A, N) predict educational outcomes and whether the Big Five provide incremental predictive validity for educational outcomes when prior achievement, intelligence, and academic self-concept are controlled for. Also, little is known about whether noncognitive factors are related to change in academic success, especially grades, after controlling for its stability. To address these research questions, we used data from the Luxembourg longitudinal extension of the 2009 cycle of the Program for International Student Assessment (PISA). We included data from two student cohorts that were followed across 4 years: The first cohort was in ninth grade when they participated in PISA (N = 240); the second cohort in 10th grade (N = 276). Correlational results showed that Conscientiousness and Openness were substantially related to subject-specific grades in Mathematics, French, and German across several school years. There was evidence for incremental predictive validity beyond intelligence and academic self-concepts. When controlling for the stability of grades, there were only small effects of all predictors on later grades. In summary, students’ personality (but also intelligence and academic self-concepts) predicted the stable part of grades rather than change.

A Note on Construct Validity of the Anchoring Method in PISA 2012

2017 ◽  
Vol 36 (7) ◽  
pp. 709-724 ◽  
Author(s):  
Lazar Stankov ◽  
Jihyun Lee ◽  
Matthias von Davier
Keyword(s):  
Academic Achievement ◽  
Construct Validity ◽  
International Student ◽  
Student Assessment ◽  
Achievement Scores ◽  
Individual Level ◽  
Anchoring Vignettes ◽  
International Student Assessment ◽  
Pisa 2012 ◽  
Country Level

We examine construct validity of the anchoring method used with 12 noncognitive scales from the Programme for International Student Assessment (PISA) 2012 project. This method combines individuals’ responses to vignettes and self-rated scores based on Likert-type items. It has been reported that the use of anchoring vignettes can reverse country-level correlations between academic achievement scores and noncognitive measures from negative to positive, and therefore align them with the typically reported individual-level correlations. Using the PISA 2012 data, we show that construct validity of this approach may be open to question because the anchored scales produce a different set of latent dimensions in comparison with nonanchored scales, even though both scales were created from the same set of individual responses. We also demonstrate that only one of three vignettes may be responsible for the resolution of the “paradox” highlighting that the choice of vignettes may be more important than what was previously reported.

Three Paradoxical Effects on Academic Self-Concept Across Countries, Schools, and Students

2019 ◽  
Vol 24 (3) ◽  
pp. 231-242 ◽  
Author(s):  
Herbert W. Marsh ◽  
Philip D. Parker ◽  
Reinhard Pekrun
Keyword(s):  
International Student ◽  
Student Assessment ◽  
Cross Cultural ◽  
Frame Of Reference ◽  
School Level ◽  
Self Concept ◽  
Positive Effects ◽  
International Student Assessment ◽  
Country Level ◽  
Starting School

Abstract. We simultaneously resolve three paradoxes in academic self-concept research with a single unifying meta-theoretical model based on frame-of-reference effects across 68 countries, 18,292 schools, and 485,490 15-year-old students. Paradoxically, but consistent with predictions, effects on math self-concepts were negative for: • being from countries where country-average achievement was high; explaining the paradoxical cross-cultural self-concept effect; • attending schools where school-average achievement was high; demonstrating big-fish-little-pond-effects (BFLPE) that generalized over 68 countries, Organisation for Economic Co-operation and Development (OECD)/non-OECD countries, high/low achieving schools, and high/low achieving students; • year-in-school relative to age; unifying different research literatures for associated negative effects for starting school at a younger age and acceleration/skipping grades, and positive effects for starting school at an older age (“academic red shirting”) and, paradoxically, even for repeating a grade. Contextual effects matter, resulting in significant and meaningful effects on self-beliefs, not only at the student (year in school) and local school level (BFLPE), but remarkably even at the macro-contextual country-level. Finally, we juxtapose cross-cultural generalizability based on Programme for International Student Assessment (PISA) data used here with generalizability based on meta-analyses, arguing that although the two approaches are similar in many ways, the generalizability shown here is stronger in terms of support for the universality of the frame-of-reference effects.

Are There Test Administrator Effects in Large-Scale Educational Assessments?

Methodology ◽  
2007 ◽  
Vol 3 (4) ◽  
pp. 149-159 ◽  
Author(s):  
Oliver Lüdtke ◽  
Alexander Robitzsch ◽  
Ulrich Trautwein ◽  
Frauke Kreuter ◽  
Jan Marten Ihme
Keyword(s):  
Large Scale ◽  
International Student ◽  
Student Assessment ◽  
Student Groups ◽  
Mathematics Scores ◽  
International Student Assessment ◽  
Sample Attrition ◽  
Educational Assessments ◽  
Test Administrator ◽  
Practical Implications

Abstract. In large-scale educational assessments such as the Third International Mathematics and Sciences Study (TIMSS) or the Program for International Student Assessment (PISA), sizeable numbers of test administrators (TAs) are needed to conduct the assessment sessions in the participating schools. TA training sessions are run and administration manuals are compiled with the aim of ensuring standardized, comparable, assessment situations in all student groups. To date, however, there has been no empirical investigation of the effectiveness of these standardizing efforts. In the present article, we probe for systematic TA effects on mathematics achievement and sample attrition in a student achievement study. Multilevel analyses for cross-classified data using Markov Chain Monte Carlo (MCMC) procedures were performed to separate the variance that can be attributed to differences between schools from the variance associated with TAs. After controlling for school effects, only a very small, nonsignificant proportion of the variance in mathematics scores and response behavior was attributable to the TAs (< 1%). We discuss practical implications of these findings for the deployment of TAs in educational assessments.

Sign in / Sign up

Export Citation Format

Share Document