A Note on Construct Validity of the Anchoring Method in PISA 2012

2017 ◽  
Vol 36 (7) ◽  
pp. 709-724 ◽  
Author(s):  
Lazar Stankov ◽  
Jihyun Lee ◽  
Matthias von Davier
Keyword(s):  
Academic Achievement ◽  
Construct Validity ◽  
International Student ◽  
Student Assessment ◽  
Achievement Scores ◽  
Individual Level ◽  
Anchoring Vignettes ◽  
International Student Assessment ◽  
Pisa 2012 ◽  
Country Level

We examine construct validity of the anchoring method used with 12 noncognitive scales from the Programme for International Student Assessment (PISA) 2012 project. This method combines individuals’ responses to vignettes and self-rated scores based on Likert-type items. It has been reported that the use of anchoring vignettes can reverse country-level correlations between academic achievement scores and noncognitive measures from negative to positive, and therefore align them with the typically reported individual-level correlations. Using the PISA 2012 data, we show that construct validity of this approach may be open to question because the anchored scales produce a different set of latent dimensions in comparison with nonanchored scales, even though both scales were created from the same set of individual responses. We also demonstrate that only one of three vignettes may be responsible for the resolution of the “paradox” highlighting that the choice of vignettes may be more important than what was previously reported.

scholarly journals The Influence of Foreign-born Population on Immigrant and Native-born Students’ Academic Achievement

2019 ◽  
Vol 5 ◽  
pp. 237802311984525 ◽  
Author(s):  
Florencia Silveira ◽  
Mikaela J. Dufur ◽  
Jonathan A. Jarvis ◽  
Kristie J. Rowley
Keyword(s):  
Academic Achievement ◽  
Socioeconomic Status ◽  
International Student ◽  
Immigrant Students ◽  
Student Assessment ◽  
Immigrant Status ◽  
Foreign Born ◽  
International Student Assessment ◽  
Country Level ◽  
The Relationship

With recent increases in international migration, some political and academic narratives argue for limiting migration because of possible negative effects on the host country. Among other outcomes, these groups argue that immigrant students have an impact on education, negatively affecting native-born students’ academic performance. The authors contextualize the relationship between immigrant status and academic achievement by considering a macro social setting: country-level foreign-born population. The authors examine achievement from the 2015 Programme for International Student Assessment in 41 high-income countries. The authors use within- and cross-level interactions to examine (1) the relationship between immigrant status and academic achievement, (2) the moderating effect of student socioeconomic status on achievement, and (3) how country-level foreign-born population affects both immigrant and native-born students’ performance. The findings indicate that immigrant students perform similarly to native-born students when considering other contextual factors, with socioeconomic status moderating the effect of immigrant status. Furthermore, all students, immigrant and nonimmigrant students alike, benefit academically from more immigration.

scholarly journals Benchmarking London in the PISA rankings

2016 ◽  
Author(s):  
John Jerrim ◽  
Gill Wyness
Keyword(s):  
Academic Achievement ◽  
Confidence Interval ◽  
International Student ◽  
Student Assessment ◽  
International Study ◽  
Future Research ◽  
World Cities ◽  
International Student Assessment ◽  
Country Level ◽  
City State

The Programme for International Student Assessment (PISA) is an important international study of 15-year-olds' academic achievement. Although PISA has traditionally been used to draw comparisons across countries, there is growing interest in the production of regional (i.e. city, state, or provincial level) results. In this paper we present the first attempt to benchmark London in the PISA rankings. Pooling data across the 2009 and 2012 survey waves, we estimated a 95 per cent confidence interval for London's mathematics, reading, and science PISA scores. These are compared not only to country-level averages, but also to the scores of other major world cities and states. The paper concludes by discussing how these results should be interpreted, and possible directions for future research.

Three Paradoxical Effects on Academic Self-Concept Across Countries, Schools, and Students

2019 ◽  
Vol 24 (3) ◽  
pp. 231-242 ◽  
Author(s):  
Herbert W. Marsh ◽  
Philip D. Parker ◽  
Reinhard Pekrun
Keyword(s):  
International Student ◽  
Student Assessment ◽  
Cross Cultural ◽  
Frame Of Reference ◽  
School Level ◽  
Self Concept ◽  
Positive Effects ◽  
International Student Assessment ◽  
Country Level ◽  
Starting School

Abstract. We simultaneously resolve three paradoxes in academic self-concept research with a single unifying meta-theoretical model based on frame-of-reference effects across 68 countries, 18,292 schools, and 485,490 15-year-old students. Paradoxically, but consistent with predictions, effects on math self-concepts were negative for: • being from countries where country-average achievement was high; explaining the paradoxical cross-cultural self-concept effect; • attending schools where school-average achievement was high; demonstrating big-fish-little-pond-effects (BFLPE) that generalized over 68 countries, Organisation for Economic Co-operation and Development (OECD)/non-OECD countries, high/low achieving schools, and high/low achieving students; • year-in-school relative to age; unifying different research literatures for associated negative effects for starting school at a younger age and acceleration/skipping grades, and positive effects for starting school at an older age (“academic red shirting”) and, paradoxically, even for repeating a grade. Contextual effects matter, resulting in significant and meaningful effects on self-beliefs, not only at the student (year in school) and local school level (BFLPE), but remarkably even at the macro-contextual country-level. Finally, we juxtapose cross-cultural generalizability based on Programme for International Student Assessment (PISA) data used here with generalizability based on meta-analyses, arguing that although the two approaches are similar in many ways, the generalizability shown here is stronger in terms of support for the universality of the frame-of-reference effects.

Fabrication of interview data

2018 ◽  
Vol 26 (2) ◽  
pp. 213-226 ◽  
Author(s):  
Jörg Blasius
Keyword(s):  
Survey Research ◽  
International Student ◽  
Student Assessment ◽  
Content Type ◽  
International Student Assessment ◽  
Pisa 2012 ◽  
Research Organizations ◽  
Using Data ◽  
Copy And Paste ◽  
Practical Implications

Purpose Evidence from past surveys suggests that some interviewees simplify their responses even in very well-organized and highly respected surveys. This paper aims to demonstrate that some interviewers, too, simplify their task by at least partly fabricating their data, and that, in some survey research institutes, employees simplify their task by fabricating entire interviews via copy and paste. Design/methodology/approach Using data from the principal questionnaires in the Programme for International Student Assessment (PISA) 2012 and the Programme for the International Assessment of Adult Competencies (PIAAC) data, the author applies statistical methods to search for fraudulent methods used by interviewers and employees at survey research organizations. Findings The author provides empirical evidence for potential fraud performed by interviewers and employees of survey research organizations in several countries that participated in PISA 2012 and PIAAC. Practical implications The proposed methods can be used as early as the initial phase of fieldwork to flag potentially problematic interviewer behavior such as copying responses. Originality/value The proposed methodology may help to improve data quality in survey research by detecting fabricated data.

Principals and Student Achievement

2018 ◽  
pp. 84-117 ◽  
Author(s):  
S. Marshall Perry ◽  
Karen M. Sealy ◽  
Héctor X. Ramírez-Pérez ◽  
Thomas C. DeNicola ◽  
Yair Cohen
Keyword(s):  
Academic Achievement ◽  
Student Achievement ◽  
Instructional Leadership ◽  
Teaching And Learning ◽  
International Student ◽  
Student Assessment ◽  
School Context ◽  
School Level ◽  
International Student Assessment ◽  
Context Specific

Connections between principal leadership activities, school context, and student achievement are examined within this paper. Data for this quantitative study are from the 2013 Teaching and Learning International Survey (TALIS) and the 2012 Programme for International Student Assessment (PISA). The eight countries of examination participated in both the TALIS and PISA and the researchers merged datasets, yielding a study sample of 1,301 schools. This paper supports a context-specific view of instructional leadership. When looking across countries, the researchers found different practices were more strongly associated with the academic achievement of students, and suggest that school leaders have a meaningful overall relationship with academic achievement, both directly and indirectly. This study therefore supports prior research about the direct and indirect effects of instructional leadership. Further study, which accounts for differences in family academic resources and school-level opportunities to learn, will better illuminate the connection between instructional leadership practices and academic achievement.

Does student grade contribute to the declining trend in Programme for International Student Assessment reading and mathematics in Australia?

2020 ◽  
Vol 64 (3) ◽  
pp. 205-226
Author(s):  
John Ainley ◽  
Dan Cloney ◽  
Jessica Thompson
Keyword(s):  
Multiple Regression ◽  
International Student ◽  
Student Assessment ◽  
Socioeconomic Background ◽  
Achievement Scores ◽  
Regression Methods ◽  
International Student Assessment ◽  
And Gender ◽  
And Mathematics ◽  
Reading And Mathematics

Declines in the scores of Australian 15-year-old students from the Programme for International Student Assessment are a matter of policy interest. Some of the declines may have resulted from shifts in the age-grade distributions of students in the Programme for International Student Assessment samples. We use multiple regression methods to model the student-level effects of grade for each Programme for International Student Assessment cycle allowing for the effects of student characteristics (e.g. socioeconomic background and gender) and jurisdiction. We estimate an average net effect of grade over the Programme for International Student Assessment cycles since 2006 as 42 scale points with no difference between reading and mathematics. We explore the extent to which differences between grades in achievement and changes in the grade distributions of students contributed to changes in average Programme for International Student Assessment achievement scores. We conclude that the relatively greater decline in Grade 11, compared to Grade 10 achievement, contributed to the overall decline and that shifts in distributions may have also contributed a little to those declines.

scholarly journals Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

2017 ◽  
Vol 48 (3) ◽  
pp. 319-334 ◽  
Author(s):  
Jia He ◽  
Janine Buchholz ◽  
Eckhard Klieme
Keyword(s):  
Classroom Management ◽  
Predictive Validity ◽  
Low Socioeconomic Status ◽  
Teacher Support ◽  
International Student ◽  
Math Achievement ◽  
Student Assessment ◽  
Self Report ◽  
Anchoring Vignettes ◽  
International Student Assessment

Anchoring vignettes are item batteries especially designed for correcting responses that might be affected by incomparability. This article investigates the effects of anchoring vignettes on the validity of student self-report data in 64 cultures. Using secondary data analysis from the 2012 Programme for International Student Assessment (PISA), we checked the validity of ratings on vignette questions, and investigated how rescaled item responses of two student scales, Teacher Support and Classroom Management, enhanced comparability and predictive validity. The main findings include that (a) responses to vignette questions represent valid individual and cultural differences; in particular, violations in these responses (i.e., misorderings) are related to low socioeconomic status and low cognitive sophistication; (b) the rescaled responses tend to show higher levels of comparability; and (c) the associations of rescaled Teacher Support and Classroom Management with math achievement, Student-Oriented Instruction, and Teacher-Directed Instruction are slightly different from raw scores of the two target constructs, and the associations with rescaled scores seem to be more in line with the literature. Namely, the associations among all self-report Likert-type scales are weaker with rescaled scores, presumably reducing common method variance, and both rescaled scale scores are more positively related to math achievement. The country ranking also changes substantially; in particular, Asian cultures top the ranking on Teacher Support after rescaling. However, anchoring vignettes are not a cure-all in solving measurement bias in cross-cultural surveys; we discuss the technicality and directions for further research on this technique.

scholarly journals HAPPINESS OF PREGNANT WOMEN AND CHILDREN'S ACADEMIC ACHIEVEMENTS VALUE IN KINDERGARTEN KHADIJAH

2021 ◽  
Vol 5 (4) ◽  
pp. 340
Author(s):  
Dian Lestari Hidayah ◽  
Nining Febriyana ◽  
Atika Atika
Keyword(s):  
Academic Achievement ◽  
Pregnant Women ◽  
International Student ◽  
Student Assessment ◽  
Rank Correlation ◽  
Cross Sectional ◽  
Academic Field ◽  
Biological Mothers ◽  
International Student Assessment ◽  
Demographic Questionnaire

Background: Indonesia's education ranked sixth bottom of the 2018 world rankings compiled by the Program for International Student Assessment (PISA). The value of academic achievement is an indicator of the quality of students in the academic field. One of the factors that influence achievement is children's intelligence. Child's intelligence is influenced by the happiness of his mother during pregnancy. This study aims to analyze the relationship between the happiness of pregnant women and the value of children's academic achievement in TK Khadijah Gedangan. Method: This type of research is an observational analytic study, using a cross sectional approach. The study population was all TK-A and TK-B students at TK Khadijah Gedangan and their biological mothers. The total sampling method was carried out according to the inclusion criteria and exclusion criteria. Collecting data using a demographic questionnaire and OHQ which was translated into Indonesian. SPSS software was used to test the Spearman rank correlation statistical test. Result: Obtained N = 54, significance value 0.514, correlation coefficient value -.091. Conclusion: The happiness of pregnant women is not related to the achievement scores of kindergarten children.Keywords: happiness, pregnant women, academic achievement 

scholarly journals Examining school variables affecting PISA 2012 math achievement in Turkey and Shanghai-China

2017 ◽  
Vol 6 (4) ◽  
pp. 167-174
Author(s):  
Mustafa Kale
Keyword(s):  
International Student ◽  
Math Achievement ◽  
Student Assessment ◽  
International Student Assessment ◽  
Pisa 2012 ◽  
Math Scores ◽  
Model Research ◽  
School Variables ◽  
Research Population ◽  
The Difference

The main purpose of the research is to examine school variables that have effect on Programme for International Student Assessment (PISA) 2012 math achievement in Turkey and Shanghai-China. The research was designed in casual comparison model. Research population was constituted by student in age group of 15 in Turkey and Shanghai-China in 2012.The sample consists of 4848 students and 170 schools in Turkey and 5177 students and 155 schools in  Shanghai-China that participated in PISA 2012. Two-leveled Hierarchical Linear Modelling (HLM) was used to analyze data because the data collected in PISA 2012 had a hierarchical data structure. As a result of analysis, variability in math scores, 63% in Turkey and  47% in Shanghai-China, was found due to the difference between the mean math scores of schools. It was determined that  MACTIV, SCMATEDU and TCMORALE in Turkey and MACTIV, in Shanghai-China statistically affect on math achievement. Keywords: PISA, school administration, school variables, HLM

Sign in / Sign up

Export Citation Format

Share Document