The effect of insufficient effort responding on the validity of student evaluation of teaching

2019 ◽  
Vol 11 (3) ◽  
pp. 604-615
Author(s):  
Mahmoud AlQuraan
Keyword(s):  
Higher Education ◽  
Construct Validity ◽  
Teaching Effectiveness ◽  
Detection Method ◽  
Student Evaluation ◽  
Principal Component ◽  
Data Sets ◽  
Factorial Validity ◽  
Content Type ◽  
Before And After

Purpose The purpose of this paper is to investigate the effect of insufficient effort responding (IER) on construct validity of student evaluations of teaching (SET) in higher education. Design/methodology/approach A total of 13,340 SET surveys collected by a major Jordanian university to assess teaching effectiveness were analyzed in this study. The detection method was used to detect IER, and the construct (factorial) validity was assessed using confirmatory factor analysis (CFA) and principal component analysis (PCA) before and after removing detected IER. Findings The results of this study show that 2,160 SET surveys were flagged as insufficient effort responses out of 13,340 surveys. This figure represents 16.2 percent of the sample. Moreover, the results of CFA and PCA show that removing detected IER statistically enhanced the construct (factorial) validity of the SET survey. Research limitations/implications Since IER responses are often ignored by researchers and practitioners in industrial and organizational psychology (Liu et al., 2013), the results of this study strongly suggest that higher education administrations should give the necessary attention to IER responses, as SET results are used in making critical decisions Practical implications The results of the current study recommend universities to carefully design online SET surveys, and provide the students with clear instructions in order to minimize students’ engagement in IER. Moreover, since SET results are used in making critical decisions, higher education administrations should give the necessary attention to IER by examining the IERs rate in their data sets and its consequences on the data quality. Originality/value Reviewing the related literature shows that this is the first study that investigates the effect of IER on construct validity of SET in higher education using an IRT-based detection method.

scholarly journals Lessons Learned from Research on Student Evaluation of Teaching in Higher Education

2021 ◽  
pp. 237-256
Author(s):  
Bob Uttl
Keyword(s):  
Higher Education ◽  
Effective Teaching ◽  
Teaching Effectiveness ◽  
Student Evaluation ◽  
High Stakes ◽  
Student Evaluation Of Teaching ◽  
Knowledge Class ◽  
Teaching Awards ◽  
Student Preference ◽  
Evaluation Of Teaching

AbstractIn higher education, anonymous student evaluation of teaching (SET) ratings are used to measure faculty’s teaching effectiveness and to make high-stakes decisions about hiring, firing, promotion, merit pay, and teaching awards. SET have many desirable properties: SET are quick and cheap to collect, SET means and standard deviations give aura of precision and scientific validity, and SET provide tangible seemingly objective numbers for both high-stake decisions and public accountability purposes. Unfortunately, SET as a measure of teaching effectiveness are fatally flawed. First, experts cannot agree what effective teaching is. They only agree that effective teaching ought to result in learning. Second, SET do not measure faculty’s teaching effectiveness as students do not learn more from more highly rated professors. Third, SET depend on many teaching effectiveness irrelevant factors (TEIFs) not attributable to the professor (e.g., students’ intelligence, students’ prior knowledge, class size, subject). Fourth, SET are influenced by student preference factors (SPFs) whose consideration violates human rights legislation (e.g., ethnicity, accent). Fifth, SET are easily manipulated by chocolates, course easiness, and other incentives. However, student ratings of professors can be used for very limited purposes such as formative feedback and raising alarm about ineffective teaching practices.

scholarly journals Understanding and evaluating teaching effectiveness in the UK higher education sector using experimental design

2019 ◽  
Vol 36 (2) ◽  
pp. 202-216 ◽  
Author(s):  
Jiju Antony ◽  
Stavros Karamperidis ◽  
Frenie Antony ◽  
Elizabeth A. Cudney
Keyword(s):  
Higher Education ◽  
Experimental Design ◽  
Teaching Effectiveness ◽  
School Setting ◽  
Content Type ◽  
Postgraduate Students ◽  
Interesting Study ◽  
Designed Experiment ◽  
The Uk

PurposeThe purpose of this paper is to demonstrate the power of experimental design as a technique to understand and evaluate the most important factors which influence teaching effectiveness for a postgraduate course in a higher education (HE) context.Design/methodology/approachThe methodology involves the execution of a case study in the form of an experiment in a business school setting. The experiment was carried out with the assistance of over 100 postgraduate students from 26 countries. The data were collected over a two year period (2015 and 2016) from a postgraduate course offered by the same tutor for repeatability reasons.FindingsThe key findings of the experiment have clearly indicated that students’ perceptions of teaching effectiveness based on intuition and guesswork are not identical to the outcomes from a simple designed experiment. Moreover, the results of the experiment provided a greater stimulus for the wider applications of the technique to other processes across the case study HE sector.Research limitations/implicationsOne of the limitations of the study is that the experiment was conducted for a popular postgraduate course. It would be beneficial to understand the results of the experiment for less popular postgraduate courses in the university in order to drive improvements. Moreover, this research was conducted only for postgraduate courses and the results may vary for undergraduate courses. This would be an interesting study to understand the differences in the factors between undergraduate and postgraduate teaching effectiveness.Practical implicationsThe outcome of this experiment would help everyone who is involved in teaching to understand the factors and their influences to improve students’ satisfaction scores during the delivery of teaching.Originality/valueThis paper shows how experimental design as a pure manufacturing technique can be extended to a HE setting.

scholarly journals The student as customer and quality in higher education

2020 ◽  
Vol 34 (8) ◽  
pp. 1221-1235
Author(s):  
Angelito Calma ◽  
Camille Dickson-Deane
Keyword(s):  
Higher Education ◽  
Teaching And Learning ◽  
Student Evaluation ◽  
Poor Quality ◽  
Quality Initiatives ◽  
Content Type ◽  
Customer Expectations ◽  
Management Concepts ◽  
Quality In Higher Education

PurposeThis paper explores some management concepts and how applying these concepts from business to higher education can be problematic, let alone incompatible, particularly in relation to measuring quality in higher education.Design/methodology/approachIt provides a conceptual understanding of the literature on quality in the higher education context. It does so by examining the literature on students as customers, customer expectations, customer satisfaction and other management theories that have been applied to higher education.FindingsIt argues that the current bases for perceiving quality such as meeting customer expectations, satisfying the customer, ensuring quality control, meeting standards and assessing the cost associated with poor quality are in disagreement with the principal aims and measures of quality in higher education.Research limitations/implicationsThis paper can certainly benefit from many other concepts in business that have been applied in higher education, which it lacks. It only focussed on a number of key and popular ideas in management theory that have been used in higher education more broadly.Practical implicationsStudent-focussed quality initiatives can be devoid of the student as customer concept. How programs, subjects and experiences are curated can be solely for the purpose of continuous improvement. Second, universities that choose to treat the student as a customer may find it beneficial to apply a relationship marketing approach to higher education. Lastly, those against the student as customer concept may focus on the long-term impact of quality initiatives such as promoting lifelong learning, building long-term relationships with alumni and employers and those that further promote academic integrity.Originality/valueSome considerations have been offered. These considerations revisit the basic notions of teaching and learning in higher education. It puts an emphasis on sidestepping the student as customer metaphor, that learning is not expressed in dollar terms, and that the quality of the student experience cannot be measured by student evaluation alone because it is felt much later in life.

Teaching performance: determinants of the student assessment

2014 ◽  
Vol 27 (3) ◽  
pp. 402-418
Author(s):  
Adriana Morales Rodríguez ◽  
Joan‐Lluís Capelleras ◽  
Víctor M. Gimenez Garcia
Keyword(s):  
Higher Education ◽  
Student Evaluation ◽  
Student Assessment ◽  
Multivariate Techniques ◽  
Teaching Performance ◽  
Research Activity ◽  
Content Type ◽  
Teacher Student ◽  
Wide Range ◽  
Performance Determinants

Purpose The purpose of this paper is to identify the nature and factors that influence student evaluation of the teaching performance of university teachers by integrating two areas of research: services marketing and higher education. Design/methodology/approach A set of hypotheses were developed taking into consideration customer (student), employee (teacher) and service (course) characteristics. They were then tested using data from 952 courses for a three‐year period and employing different multivariate techniques. Findings Students basically evaluate the expertise, attitude and behavior of teachers. The results also indicate that this evaluation is a complex phenomenon that depends on factors related to teacher, student and course profiles. Research limitations/implications Given the nature of the data used here, future studies should extend the scope of research to other institutions, examine quality from an objective standpoint and include teachers’ perceptions and the outcomes of their research activity. Practical implications Based on the results of this paper, the authors recommend the following: to permit teachers to teach the same courses repeatedly, allowing them to consolidate their practice; to provide training in teaching techniques and ethics; to pay particular attention to those students who move to another degree program; and to maintain an appropriate class size. Originality/value This study integrates two areas of research and proposes a wide range of service quality determinants in the context of higher education, including several factors that had not been previously considered.

Teaching effectiveness attributes in business schools

2017 ◽  
Vol 31 (6) ◽  
pp. 780-800
Author(s):  
Earl Simendinger ◽  
Abdul-Nasser El-Kassar ◽  
Maria Alejandra Gonzalez-Perez ◽  
John Crawford ◽  
Stephanie Thomason ◽  
...  
Keyword(s):  
Teaching Effectiveness ◽  
Student Evaluation ◽  
Business Schools ◽  
Faculty Perceptions ◽  
Personal Characteristics ◽  
Extensive Literature ◽  
Personal Communications ◽  
Content Type ◽  
Business Teachers ◽  
The Usa

Purpose Determining the attributes of effective business teachers is critical to schools as they strive to attract and retain students, assure learning, obtain and maintain reputation and accreditations, and place their alumni in the competitive job market. The purpose of this paper is to examine students and faculty perceptions of teaching effectiveness in five culturally disparate countries: Colombia, France, Lebanon, Sweden, and the USA. Design/methodology/approach A survey was designed based on previous research complemented by an extensive literature review as well as personal communications with faculty in different international business schools. The survey considered 39 teaching attributes related to three specific dimensions: class delivery, class preparation and design, and instructional traits and personal characteristics. The survey targeted students and faculty from seven business schools located in five countries. Findings This study offers new conceptual and analytical analyses from a cross-country comparative perspective. Rankings of the importance of perceived teaching attributes for both major groups involved in the teaching of business, faculty and students, are reported. The attributes are also ranked by teaching taxonomy and examined across countries. Practical implications This study provides practical results that can be useful to instructors wishing to increase their teaching effectiveness and to universities considering revising their student evaluation forms. Originality/value This study includes data collected from faculty and students from several schools located in culturally disparate countries and, thus, increases the applicability of the results in a cross-cultural manner and provides implications for practice internationally.

The skill of discipline – measuring F.I.R.S.T discipline principles in higher education

2020 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Hana Krskova ◽  
Chris Baumann ◽  
Yvonne Breyer ◽  
Leigh Norma Wood
Keyword(s):  
Higher Education ◽  
Cognitive Skills ◽  
Personal Development ◽  
Online Survey ◽  
Critical Role ◽  
Self Regulation ◽  
Principal Component ◽  
The United States ◽  
Student Discipline ◽  
Content Type

PurposeHuman capital theory suggests that any increase in skills translates into greater productivity of the workforce. Non-cognitive skills, in particular, play a critical role in many domains in life. The aim of this study is to gain a greater understanding of one such skill, discipline. Viewing discipline as a tool for enhancing learning, personal development and increasing overall achievement, this study offers an alternative way to measure discipline in higher education.Design/methodology/approachThis paper presents the results of an online survey of 537 current students and recent graduates from the United States, South Korea and China. Principal component analysis was used to test the overarching assumption that student discipline is composed of five dimensions. Multiple analysis of variance (ANOVA) with post hoc analyses and t-tests were applied to test for country and gender-related differences between the three country groups. Cluster analysis was used to profile the respondent groups based on similarities across the samples.FindingsThe results confirm that student discipline is a construct comprising five discipline dimensions – focus, intention, responsibility, structure and time (F.I.R.S.T). In addition, the identification of low, medium and high discipline levels among the respondents provides support for the recently introduced concept of a layered “threshold of Discipline”.Originality/valueA F.I.R.S.T. discipline measurement questionnaire for capturing student discipline – underpinned by a conceptual model encompassing self-determination, goal setting, self-efficacy, self-regulation and time management principles – was developed and tested. Suggestions for enhancing graduate work readiness through increasing levels of the skill of discipline are outlined.

Evaluating the validity of Student Evaluation of Teaching Effectiveness (SET) in India

2015 ◽  
Vol 57 (6) ◽  
pp. 623-638 ◽  
Author(s):  
Sanjiv Mittal ◽  
Rajat Gera ◽  
Dharminder Kumar Batra
Keyword(s):  
Teaching Effectiveness ◽  
Student Evaluation ◽  
Emerging Market ◽  
Two Dimensions ◽  
Cross Cultural ◽  
Student Evaluation Of Teaching ◽  
Content Type ◽  
Evaluation Of Teaching ◽  
Relationship Of ◽  
Indian Students

Purpose – There is a debate in literature about the generalizability of the structure and the validity of the measures of Student Evaluation of Teaching Effectiveness (SET). This debate spans the dimensionality and validity of the construct, and the use of the measure for summative and formative purposes of teachers valuation and feedback. The purpose of this paper is to contribute to the debate on the aforementioned issues. Specifically the paper tests the relationship of teacher’s “charisma” trait with a measure of SET consisting of the two dimensions of “lecturer ability” and “module attributes.” The market characteristics of the paper are those of an emerging market and cross-cultural context with a specific reference to India. Design/methodology/approach – In this study, a two-dimensional scale of SET, which was originally developed by Shevlin et al. (2000) in their study in the UK, was empirically tested with Indian students and modified. Empirical data were collected from Indian students pursuing their MBA program in a north Indian university and statistical testing using exploratory and confirmatory factor analyses was undertaken. The proposed relationship of a teacher’s “charisma” trait was tested as a reflective construct comprising of the two dimensions of SET with the help of the software package Amos ver 4.0. Findings – The results indicate that the measure of SET is influenced by the teacher’s “Charisma” (trait), thus providing evidence of a halo effect. This raises the issue of validity of SET as an instrument for measuring teaching effectiveness (TE). The results provide support to the hypothesis that structure of SET is multidimensional along with the need for adapting the instrument in diverse cultural and market contexts. Originality/value – This study contributes to the debate on the validity, structure and use of SET as an instrument for measuring TE in a developing market with cross-cultural implications such as India.

Considering pollution in the higher education (HE) service ecosystem

2017 ◽  
Vol 29 (6) ◽  
pp. 767-782 ◽  
Author(s):  
Montserrat Díaz-Méndez ◽  
Michael Saren ◽  
Evert Gummesson
Keyword(s):  
Higher Education ◽  
Literature Review ◽  
Student Evaluation ◽  
Ecosystem Approach ◽  
Teaching Quality ◽  
Content Type ◽  
Teachers And Students ◽  
Service Ecosystem ◽  
Operant Resources

Purpose From a service ecosystem perspective, the purpose of this paper is to examine students’ evaluation surveys as a tool used by most higher education (HE) institutions worldwide to measure teaching quality with consequences for tenure and promotion. Design/methodology/approach This study is based on the service-dominant (S-D) logic and specifically on the service ecosystem approach. Through an in-depth literature review and analysis the authors explore the effect of student evaluation surveys on the value co-creation process, describe the role they play in the HE ecosystem and critically evaluate their efficacy. The research is based on empirical data from the literature and previous studies findings. Findings The literature review highlights the detrimental consequences of the use of students’ evaluation surveys for teachers and students and for the rest of actors of this service. The authors argue that institutions should embrace a service ecosystem perspective based on S-D logic. The authors highlight the role of institutions as moderators of the interactions between actors and, finally, the authors have introduced the concept of “service ecosystem pollution” which the authors define as the presence or introduction of disruptive elements in the service ecosystem adversely affecting the nature of value co-creation. Practical implications This study emphasizes the importance for HE institutions to embrace a service ecosystem approach so as to foster and preserve the value co-creation processes taking place within the interactions among the HE actors. Conclusions drawn from this paper suggest that HE institutions should focus on strategies such as investing in improving students’ and teachers’ operant resources rather than polluting the HE service ecosystem with the use of students’ evaluation surveys. Originality/value Hitherto there are no studies analyzing the tools HE institutions use to measure teaching quality from a service ecosystem perspective. The study is especially valuable due to the consequences and the use of these measures entails for teachers, students and society.

Graduates’ job satisfaction across domains of study

2014 ◽  
Vol 35 (4) ◽  
pp. 470-499 ◽  
Author(s):  
Tomasz Gajderowicz ◽  
Gabriela Grotkowska ◽  
Leszek Wincenciak
Keyword(s):  
Higher Education ◽  
Job Satisfaction ◽  
Labour Market ◽  
Principal Component ◽  
Educational Process ◽  
Model Parameters ◽  
Capital Theory ◽  
Content Type ◽  
Explanatory Variables ◽  
Work Related

Purpose – The purpose of this paper is to address the problem of job satisfaction determinants of higher education graduates across six selected study domains. Design/methodology/approach – Based on the theoretical considerations, derived from human capital theory and signalling theory, the authors formulate the model explaining job satisfaction as a broad measure of labour market success. Explanatory variables include various socio-demographic characteristics as well as market environment and process of learning, modes of teaching and study programme characteristics. Data used in the analysis comes from two special surveys of European research projects REFLEX and HEGESCO. Principal component analysis method and OLS regression were used to estimate model parameters. Findings – The results of our research show the important role of characteristics of educational process, as well as individual graduates’ early work-related experience in predicting job satisfaction. Differences in job satisfaction determinants across domains may be to some extent explained by the differences in the labour market characteristics for graduates in given discipline. Variety of education-related characteristics taken into account in the empirical analysis of determinants of job satisfaction is a key valuable contribution to the research in the field. Originality/value – Research findings indicate the areas of potential actions aimed at improving future job satisfaction which can be undertaken by higher education institutions’ management bodies.

Sign in / Sign up

Export Citation Format

Share Document