Cross-cultural Measurement Invariance of the Items in the Science Literacy Test in the Programme for International Student Assessment (PISA-2015)

2020 ◽  
Vol 8 (2) ◽  
pp. 16
Author(s):  
Betül Alatlı
Keyword(s):  
Measurement Invariance ◽  
International Student ◽  
Science Literacy ◽  
Student Assessment ◽  
Cross Cultural ◽  
The Other ◽  
Item Bias ◽  
International Student Assessment ◽  
Significant Difference ◽  
Literacy Test

This study aimed to investigate cross-cultural measurement invariance of the PISA (Programme for International Student Assessment, 2015) science literacy test and items and to carry out a bias study on the items which violate measurement invariance. The study used a descriptive review model. The sample of the study consisted of 2224 students taking the S12 test booklet from Australia, France, Singapore, and Turkey. Measurement invariance analyses for the test were done using Multi-Group Confirmatory Factor Analysis (MGCFA). Differential Item Functioning (DIF), in other words, measurement invariance of the test items, was analyzed using the item response theory log-likelihood ratio (IRTLR), Hierarchical Generalized Linear Model (HGLM), and the Simultaneous Item Bias Test (SIBTEST) methods.According to the findings, the test was determined to exhibit structural invariance across cultures. The highest number of items showing DIF was observed in the comparisons of Australia-Singapore and Australia-France with 35%. The number of items showing DIF, with 24%, determined in bilateral comparisons which included Turkey, the only country taking the translated form among other countries, did not show a significant difference compared to the other comparisons. While the lowest number of items showing DIF was obtained from Singapore-France samples with 12%, the rate of items indicating DIF in the France-Turkey samples was 18%. On the other hand, 35% of the items showed cross cultural measurement invariance. An item bias study was carried out based on expert opinions on items identified and released as showing DIF in the comparisons of Turkey with Australia and Singapore.According to the findings, translation-bound differentiation of the items, familiarity of a culture group with the contents of the items, polysemy in the expressions or words used in the items, the format, or the stylistic characteristics of the items were determined to be the cause of the bias in the skills measured with the items.

Three Paradoxical Effects on Academic Self-Concept Across Countries, Schools, and Students

2019 ◽  
Vol 24 (3) ◽  
pp. 231-242 ◽  
Author(s):  
Herbert W. Marsh ◽  
Philip D. Parker ◽  
Reinhard Pekrun
Keyword(s):  
International Student ◽  
Student Assessment ◽  
Cross Cultural ◽  
Frame Of Reference ◽  
School Level ◽  
Self Concept ◽  
Positive Effects ◽  
International Student Assessment ◽  
Country Level ◽  
Starting School

Abstract. We simultaneously resolve three paradoxes in academic self-concept research with a single unifying meta-theoretical model based on frame-of-reference effects across 68 countries, 18,292 schools, and 485,490 15-year-old students. Paradoxically, but consistent with predictions, effects on math self-concepts were negative for: • being from countries where country-average achievement was high; explaining the paradoxical cross-cultural self-concept effect; • attending schools where school-average achievement was high; demonstrating big-fish-little-pond-effects (BFLPE) that generalized over 68 countries, Organisation for Economic Co-operation and Development (OECD)/non-OECD countries, high/low achieving schools, and high/low achieving students; • year-in-school relative to age; unifying different research literatures for associated negative effects for starting school at a younger age and acceleration/skipping grades, and positive effects for starting school at an older age (“academic red shirting”) and, paradoxically, even for repeating a grade. Contextual effects matter, resulting in significant and meaningful effects on self-beliefs, not only at the student (year in school) and local school level (BFLPE), but remarkably even at the macro-contextual country-level. Finally, we juxtapose cross-cultural generalizability based on Programme for International Student Assessment (PISA) data used here with generalizability based on meta-analyses, arguing that although the two approaches are similar in many ways, the generalizability shown here is stronger in terms of support for the universality of the frame-of-reference effects.

An Examination of Different Scale Usage Correction Procedures to Enhance Cross-Cultural Data Comparability

2020 ◽  
Vol 51 (10) ◽  
pp. 794-813
Author(s):  
Jia He ◽  
Joanne M. Chung ◽  
Fons J. R. van de Vijver
Keyword(s):  
International Student ◽  
A Priori ◽  
Student Assessment ◽  
Cross Cultural ◽  
International Student Assessment ◽  
Multilevel Confirmatory Factor Analysis ◽  
Extreme Response ◽  
Original Item ◽  
Item Scores ◽  
Scale Usage

This study aims to examine different scale usage correction procedures that are meant to enhance the cross-cultural comparability of Likert scale data. Specifically, we examined a priori study design (i.e., anchoring vignettes and overclaiming) and post hoc statistical procedures (i.e., ipsatization and extreme response style correction) in data from the 2012 Programme for International Student Assessment across 64 countries. We analyzed both original item responses and corrected item scores from two targeted scales in an integrative fashion by using multilevel confirmatory factor analysis and multilevel regressions. Results indicate that mean levels and structural relations varied across the correction procedures, although the psychological meaning of the constructs examined did not change. Furthermore, scores were least affected by these procedures for females who did not repeat a grade and students with higher math achievement. We discuss the implications of our findings and offer recommendations for researchers who are considering scale usage correction procedures.

Enhancing the Cross-Cultural Comparability of Self-Reports Using the Overclaiming Technique: An Analysis of Accuracy and Exaggeration in 64 Cultures

2018 ◽  
Vol 49 (8) ◽  
pp. 1247-1268 ◽  
Author(s):  
Hana Vonkova ◽  
Ondrej Papajoanu ◽  
Jiri Stipek
Keyword(s):  
International Student ◽  
Western Europe ◽  
Student Assessment ◽  
Cross Cultural ◽  
Response Patterns ◽  
International Student Assessment ◽  
World Regions ◽  
The Cross ◽  
Scale Usage ◽  
Survey Responses

The overclaiming technique (OCT) is a novel way of measuring how socially desirable responding influences survey responses. It has the potential to enhance the cross-cultural comparability of survey data. It allows the identification of respondents’ knowledge accuracy and exaggeration by comparing their assessments of familiarity with existing and nonexisting concepts in a particular field of knowledge. We aim to compare the response patterns of countries and world regions based on their OCT accuracy and exaggeration-index values and validate these OCT scores using external variables. We also introduce a general model for the categorization of respondents based on their OCT indices values. We use the Programme for International Student Assessment (PISA) 2012 data from 64 countries ( N = 275,904). We found considerable differences in response patterns across world regions: high accuracy values in East Asia, low accuracy values in Southern and Central America, high exaggeration values in Southern Europe, and low exaggeration values in Western Europe. Furthermore, we show that familiarity with math concepts changes substantially after adjustment using the OCT. The correlation between unadjusted math familiarity and math test scores is weak and nonsignificant (.13) whereas after adjustment the correlation becomes strong and significant (.66). Concerning other indicators such as gross domestic product (GDP), public expenditure in education, and Corruption Perceptions Index (CPI), before adjustment the correlation is negative and even significant for CPI (–.11, –.22, and –.45) whereas after adjustment the correlation becomes, though nonsignificant, positive (.05, .19, and .07). We also discuss the OCT in the context of other methods indicative of culturally preferred scale usage.

scholarly journals Multigroup CFA and alignment approaches for testing measurement invariance and factor score estimation: Illustration with the schoolwork-related anxiety survey across countries and gender

Methodology ◽  
2021 ◽  
Vol 17 (1) ◽  
pp. 22-38
Author(s):  
Jason C. Immekus
Keyword(s):  
Measurement Invariance ◽  
Large Scale ◽  
International Student ◽  
Student Assessment ◽  
International Studies ◽  
Invariance Testing ◽  
International Student Assessment ◽  
Item Parameters ◽  
And Gender ◽  
Gender Groups

Within large-scale international studies, the utility of survey scores to yield meaningful comparative data hinges on the degree to which their item parameters demonstrate measurement invariance (MI) across compared groups (e.g., culture). To-date, methodological challenges have restricted the ability to test the measurement invariance of item parameters of these instruments in the presence of many groups (e.g., countries). This study compares multigroup confirmatory factor analysis (MGCFA) and alignment method to investigate the MI of the schoolwork-related anxiety survey across gender groups within the 35 Organisation for Economic Co-operation and Development (OECD) countries (gender × country) of the Programme for International Student Assessment 2015 study. Subsequently, the predictive validity of MGCFA and alignment-based factor scores for subsequent mathematics achievement are examined. Considerations related to invariance testing of noncognitive instruments with many groups are discussed.

scholarly journals THE EFFECTS OF USING BAR MODEL TOWARDS ALGEBRAIC THINKING SKILL IN ALGEBRAIC PROBLEM SOLVING AMONG STUDENTS

2021 ◽  
Vol 6 (44) ◽  
pp. 38-51
Author(s):  
Janet Jahudin ◽  
Nyet Moi Siew
Keyword(s):  
International Student ◽  
Student Assessment ◽  
Thinking Skills ◽  
Algebraic Thinking ◽  
International Student Assessment ◽  
Significant Difference ◽  
Post Test ◽  
Inference Test ◽  
Quasi Experimental ◽  
The Mean

Algebraic thinking skills are one of the elements emphasized in the benchmarking of PISA (Program for International Student Assessment). However, the algebraic thinking skills among students were found to be still at a very weak level. The purpose of this study was to look at the effect of the bar model method on students’ algebraic thinking skills. Pre-test and post -test were used as research instruments. The study was an experimental study conducted on 90 Form One students from a school in Tuaran District, Sabah. This study used a quasi-experimental design of pre-and post-test. The sample was divided into 3 groups, namely the group of learning methods i) Bar Model (MB, n = 30), ii) Bar Model and Cooperative Learning (MBPK, n = 30), and conventional (TR, n = 30). Statistical inference test, One-way Analysis of Variances Test (ANOVA) was used to analyze the findings of this study. The results of one-way ANOVA analysis showed that there was a significant difference in the mean score of the post-test between the MB group, MBPK group, and TR group (F (2, 87) = 9.316, p <.05). Significant differences in mean scores for post-test could only be seen between TR group with MB group (P = 0.019) and between TR group with MBPK group (P = 0.000) while insignificant differences were shown between MB and MBPK group (P = 0.304). Therefore, it is suggested that mathematics teachers use the Bar Model as a teaching aid to improve students' algebraic thinking skills.

scholarly journals Finnish Education System and its Triumph in Pisa: Lessons to Learn for Pakistan

2020 ◽  
Vol V (I) ◽  
pp. 479-487
Author(s):  
Sobia Altaf ◽  
Abid Shehzad ◽  
Aneela Sana Akhtar
Keyword(s):  
Education System ◽  
International Student ◽  
Educational Achievement ◽  
Student Assessment ◽  
The Other ◽  
Educational Policies ◽  
Educational Structure ◽  
International Student Assessment ◽  
Long Run ◽  
System 2

Finland has emerged as world leader in terms of educational achievement and become a symbol for school development and reforms since the first exam of Programme for International Student Assessment (PISA), a test structured for the fifteen-year-old students, in science and language literacy, on which it ranked highest among Organization for Economic Cooperation and Development (OECD) nations. The major aim of the study was 1) to review the factors behind the phenomenal success of Finnish education system, 2) what lessons possibly Pakistan could learn while considering the differences. The findings were that there were no isolated factors that have a contribution in the success of Finnish Education system rather there were numerous interrelated factors that had added strength to Finland�s successful educational structure. Moreover, the educational policies in the long run, the culture of trust and the passion of Finnish citizens for reading are the other reasons for this triumph.

scholarly journals STUDENTS' PERCEPTION OF TEACHER GUIDANCE ON READING LEARNING BASED ON RESULTS OF PISA 2018

2020 ◽  
Vol 3 (1) ◽  
pp. 32
Author(s):  
NFN Safari
Keyword(s):  
Performance Improvement ◽  
International Student ◽  
Student Assessment ◽  
Real Life ◽  
Average Score ◽  
Teacher Guidance ◽  
International Student Assessment ◽  
Reading Material ◽  
Life Challenges ◽  
Significant Difference

The main purpose of this study is to answer the following question; is there any difference between international students' perceptions of teacher guide on learning to read to meet real-life challenges whose country position is above and below the 2018 average Program for International Student Assessment (PISA) score? The data in this study taken from PISA 2018 data that has been released to the public and has been permitted for research and development purposes. The population of this study was 15-year-old students (SMP) throughout the world who were studying in 2018, while the sample was 15-year-old students. PISA test participants was spread in 80 countries. The sample was students who participated in the 2018 PISA test. The data in this study were taken from the students’ test scores and questionnaires that were answered by 612.004 students from 80 countries. Based on the results of the analysis of variants of one path obtained the following results; there is a statistically significant difference (p<0,000) between students’ perceptions of countries below and above the PISA average on the following issues; (1) teacher feedback, (2) improvement of material, and (3) performance improvement in reading, learning to meet real life challenges where the country's position is above and below the average score average PISA 2018, (Sig. 0,000). The conclusion indicates that teachers in countries above the PISA average often provide feedback and provide improved reading material to their students compared to teachers in countries below the PISA average. Teachers in countries below the PISA average provide feedback for performance improvement more often than to teachers in countries above the PISA average. Because the learning dependence of students in countries above the PISA average is higher than students in countries below the PISA average, the frequency of feedback is considered unnecessary.  

scholarly journals Finland, A Package Deal: Disciplinary Climate in Science Classes, Science Dispositions and Science Literacy

2021 ◽  
Vol 13 (24) ◽  
pp. 13857
Author(s):  
Larry J. Grabau ◽  
Jari Lavonen ◽  
Kalle Juuti
Keyword(s):  
International Student ◽  
Science Literacy ◽  
Multilevel Models ◽  
Student Assessment ◽  
International Student Assessment ◽  
Teaching Support ◽  
Hierarchical Nature ◽  
Pisa Data ◽  
Science Classes ◽  
Finnish Study

Finland’s educational prowess, though tempered by recent international assessments, has remained intact. This report focused on lessons that could be learned regarding secondary-level science education from the Program for International Student Assessment (PISA) 2015, science-focused assessment. That PISA iteration included not only science literacy but also students’ science dispositions (epistemology, enjoyment, interest, and self-efficacy) and the schools’ science climate measures (disciplinary climate and teaching support). Due to the hierarchical nature of the PISA data, multilevel models were employed in this Finnish study, involving 5582 students from 167 schools. Science dispositions (as outcome measures) were differently associated with teaching support and disciplinary climate (epistemology with neither; enjoyment and interest, with both). Science literacy (as an outcome measure) was associated with all four science dispositions, whether modeled with each science disposition separately or all four simultaneously. Science literacy was also associated with the disciplinary climate in science classes for all tested models. We concluded that, in the Finnish context, science dispositions and the disciplinary climate were predictive of science literacy. Furthermore, we presented evidence from the literature indicating that these conclusions may well extend to other international contexts.

Indirect Effect of Economic, Social, and Cultural Status on Immigrant Students’ Science Performance Through Science Dispositions: A Multilevel Analysis

2020 ◽  
pp. 001312452092860
Author(s):  
Safiye Bilican Demir ◽  
Ozen Yildirim
Keyword(s):  
Hierarchical Linear Modeling ◽  
Indirect Effect ◽  
International Student ◽  
Science Literacy ◽  
Immigrant Students ◽  
Student Assessment ◽  
Linear Modeling ◽  
International Student Assessment ◽  
Cultural Status ◽  
Science Performance

Determining the ethnic differences in academic performance among immigrant students is important in terms of adapting them into the system they live in. Examining the variables related to students’ performance will be a guide in overcoming the issue. The purpose of this study is to examine the direct effect of economic, social, and cultural status (ESCS) on science performance and the indirect effect of ESCS on the relationship between science dispositions and science literacy in Canadian immigrant students using hierarchical linear modeling. Data were obtained from Program for International Student Assessment (PISA) 2015 science literacy test and student questionnaire and data sets were provided from the official website of PISA. The results of the research show that ESCS had a direct and indirect effect through science dispositions with medium to large on science performance of immigrant students. The impacts of significant variables were discussed and implications for further research were provided.

Sign in / Sign up

Export Citation Format

Share Document