scholarly journals A correct method to calculate the IDD value-added indicator in the Brazillian higher education quality assurance framework

Author(s):  
Ewout ter Haar

This paper investigates the value-added indicator used in the Brazilian higher education quality assurance framework, the so-called IDD indicator for undergraduate programmes (“Indicator of the difference between observed and expected outcomes''). The two main claims are that since 2014 this indicator is calculated incorrectly and that this mistake has relevance for public policy. INEP, the educational statistical agency responsible for educational quality indicators in Brazil, incorrectly uses multilevel modeling in their value-added analysis. The IDD indicator is calculated by estimating a varying intercept linear mixed model, but instead of identifying the intercepts with the value added of courses, INEP uses the mean of the student residuals. That this was indeed the error made is shown by reproducing exactly INEP’s published values using the incorrect method with the microdata for the 2019 assessment cycle. I then compare these values to the ones obtained with the same model and same data, but using the correct value-added measure. A comparison of reliability estimates for both methods shows that this measure of internal consistency is indeed higher for the correct method. As an example of policy relevance, I calculate the number of courses that would change from “satisfactory” to “unsatisfactory” and vice-versa, using the usual criteria established by INEP, if the correct method is applied.

2020 ◽  
Author(s):  
Ewout ter Haar

This paper investigates the “value added” indicator used in the Brazilian higher education quality assurance framework, the so-called IDD indicator for undergraduate programmes. The two main claims are that since 2014 this indicator is calculated incorrectly and that this error has relevance for public policy. I explain how INEP, the educational statistical agency responsible for educational quality indicators in Brazil, incorrectly uses multilevel modeling in their value added analysis. The IDD is calculated by estimating a varying intercept linear mixed model, but instead of identifying the intercepts with the value added of courses, INEP uses the mean of the student residuals. I check that this was indeed the error made by reproducing INEP’s published values using the microdata for the 2019 assessment cycle. I then compare these to the values obtained with the same model and same data, but using the correct value added measure. I calculate reliability estimates for both methods and this measure of internal consistency is indeed higher for the correct methodology. As an example of policy relevance, I calculate the number of courses that would go from “satisfactory” to “unsatisfactory” using the usual criteria established by INEP.


Author(s):  
Francis Ansah

The traditional tension between external and internal quality assurance implementation in higher education appears to be declining, based on a rethinking of the relationship between the two concepts. Although there are quality assurance agencies that still consider external and internal quality assurance as separate entities, most quality assurance agencies now regard the two concepts as complementary. In this paper, a case is put that the present rethinking of external and internal quality assurance in most higher education settings is guided by pragmatism, but not explicitly acknowledged in the literature. For a better appreciation of pragmatists’ influence on the current understanding of the relationship between external and internal quality assurance in higher education, this paper provides a further pragmatist conceptualisation of the two concepts to enhance stakeholders’ appreciation of employing a pragmatist approach to quality assurance practices in higher education. The conceptualisation is done through a pragmatist analysis of selected international accounts on higher education quality assurance. The paper concludes that pragmatism helps to understand external and internal quality assurance as nested concepts with reciprocities of accountability and improvement roles, and influences which call for alignment of perspectives through negotiations and settlements in order to focus on their practical relevance for implementation in higher education. La tension traditionnelle entre l’implémentation de systèmes d’assurance qualité interne et externe dans l’enseignement supérieur semble s’affaiblir grâce à la reconsidération de la relation entre ces deux concepts. Bien qu’il existe des agences d’assurance qualité qui continuent à considérer les assurances qualité interne et externe comme deux entités distinctes, la plupart des agences considèrent désormais qu’elles sont complémentaires. Cet article soutient que la nouvelle manière de penser les assurances qualité interne et externe dans l’enseignement supérieur est guidée par un souci de pragmatisme mais est encore peu reconnue dans la littérature. Pour mieux apprécier l’influence des pragmatistes sur la compréhension actuelle de la relation entre les assurances qualité interne et externe, cet article offre une conceptualisation pragmatique approfondie de ces deux concepts dans le but d’augmenter l’appréciation des parties prenantes pour l’utilisation d’une telle approche. La conceptualisation proposée provient d’une analyse pragmatique d’un choix d’expériences internationales en matière d’assurance qualité pour l’enseignement supérieur. En conclusion, cet article affirme que le pragmatisme aide à comprendre les assurances qualité interne et externe comme des concepts imbriqués qui ont des rôles réciproques en ce qui concerne la responsabilisation du système et son amélioration. Ces rôles ainsi que l’influence exercée par ces deux types d’assurance qualité requièrent des négociations et accords, pour s’accorder sur les perspectives et pouvoir ensuite se concentrer pleinement sur la pertinence pratique de leur implémentation dans les systèmes d’enseignement supérieur. 


2014 ◽  
Vol 22 (3) ◽  
pp. 255-272 ◽  
Author(s):  
Amélia Veiga ◽  
Maria João Rosa ◽  
Sónia Cardoso ◽  
Alberto Amaral

Purpose – The purpose of this article is to discuss Portuguese academics’ views on quality assessment and the elements that are important for a better understanding of what ascribes meaning to “quality cultures” in Portuguese higher education. Design/methodology/approach – The discussion was based on the results of a survey run in 2010 among Portuguese academics on quality assessment objectives and purposes. Descriptive statistics was used to investigate academics’ support to what quality assessment was supposed to guarantee (its purposes) and which should be its objectives. Furthermore, a factorial analysis using Promax rotation (oblique) was performed to investigate if the different purposes could be grouped according to the different areas they address in terms of quality assessment, helping to uncover a rationale that could explain the answers obtained. Theoretically, the results have been analysed in the light of the “quality culture” concept. Findings – Perceptions of Portuguese academics that support internal processes of quality assurance correspond either to the responsive quality culture or the regenerative quality culture. The viable form of ideal cultures is analytically limited, and the perceptions gathered encourage “quality cultures” biased by stronger group control. Originality/value – The paper offers new insights into academics’ perceptions on quality assessment, a theme that so far has been relatively absent from higher education quality assurance studies. Furthermore, the results obtained could be useful to policymakers and quality assurance agencies when setting up evaluation and accreditation systems capable of balancing improvement associated with the group dimension and accountability coupled with the grid dimension.


Sign in / Sign up

Export Citation Format

Share Document