scholarly journals Bridging the edtech evidence gap

2016 ◽  
Vol 18 (1) ◽  
pp. 18-40 ◽  
Keyword(s):  
Rapid Change ◽  
Evidence Base ◽  
Realist Evaluation ◽  
Evaluation Framework ◽  
Content Type ◽  
Reference Models ◽  
Realist Approach ◽  
Community Contribution ◽  
Complex Technology ◽  
Evaluation Approaches

Purpose – There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives. Approach – An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives. Findings – A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping. Practical implications – Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity. Originality – This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.

The impact of a refurbishment programme on older people living in sheltered housing

2021 ◽  
Vol 24 (2) ◽  
pp. 54-64
Author(s):  
Zeibeda (Zeb) Sattar ◽  
Stephanie Wilkie ◽  
Jonathan Ling
Keyword(s):  
Well Being ◽  
Realist Evaluation ◽  
Evaluation Framework ◽  
Sheltered Housing ◽  
Social Activities ◽  
Content Type ◽  
Social And Emotional ◽  
Communal Areas ◽  
Physical Changes ◽  
The Impact

Purpose This paper aims to explore residents' perceptions of a refurbishment programme to sheltered housing schemes and its impact on their well-being. Design/methodology/approach The methodology draws upon a realist evaluation framework. Four participatory appraisals (PAs) and 19 interviews with residents were conducted in the sheltered housing schemes. Ages of participants ranged from 50 to 99 years. Findings Two categories of residents were identified: healthy active older adults and older frail adults (or over 85+). Residents said their social and emotional well-being improved from the provision of indoor and outdoor communal areas. Older frail residents only accessed the new communal spaces when staff took them in their wheelchairs. The physical changes increased opportunities for social connections for residents. Conservatories and sensory gardens were most popular. Residents felt that structured activities in the new spaces and digital training would improve their social activities. Research limitations/implications The participatory methods spanned over an hour, and some residents felt too tired to complete the full session. Practical implications A practical limitation was that some sensory rooms were not fully completed at the time of the evaluation. Originality/value This paper adds the following: Perceptions of residents of a refurbishment programme in sheltered housing and the impact on their well-being. Perceptions of residents about social activities after a refurbishment programme. Perceptions of residents about the impact of physical changes to their sheltered housing schemes and impact on their internal accessibility to the improvements.

scholarly journals The Stopit! programme to reduce bullying and undermining behaviour in hospitals

2018 ◽  
Vol 32 (3) ◽  
pp. 428-443 ◽  
Author(s):  
Graham Benmore ◽  
Steven Henderson ◽  
Joanna Mountfield ◽  
Brian Wink
Keyword(s):  
Evaluation Method ◽  
Realist Evaluation ◽  
Evaluation Framework ◽  
Content Type ◽  
Training Interventions ◽  
Research Findings ◽  
Past Experiences ◽  
The Impact ◽  
Q Sort ◽  
Practical Implications

Purpose The impact of bullying and undermining behaviours on the National Health Service on costs, patient safety and retention of staff was well understood even before the Illing report, published in 2013, that reviewed the efficacy of training interventions designed to reduce bullying and harassment in the outputs. The purpose of this paper is to provide an example of a good programme well evaluated. Design/methodology/approach The methodology follows a broad realist approach, by specifying the underlying programme assumptions and intention of the designers. Three months after the event, Q-sort methodology was employed to group participants into one of three contexts – mechanism – output groups. Interviews were then undertaken with members of two of these groups, to evaluate how the programme had influenced each. Findings Q-sort identified a typology of three beneficiaries from the Stopit! workshops, characterised as professionals, colleagues and victims. Each group had acted upon different parts of the programme, depending chiefly upon their current and past experiences of bullying in hospitals. Research limitations/implications The paper demonstrates the effectiveness of using Q-sort method to identify relevant CMOs in a realist evaluation framework. Practical implications The paper considers the effectiveness of the programme to reduce bullying, rather than teach victims to cope, and how it may be strengthened based upon the research findings and Illing recommendations. Social implications Workplace bullying is invariably implicated in scandals concerning poor hospital practice, poor patient outcomes and staff illness. All too frequently, the sector responds by offering training in resilience, which though helpful, places the onus on the victim to cope rather than the employer to reduce or eliminate the practice. This paper documents and evaluates an attempt to change workplace practices to directly address bullying and undermining. Originality/value The paper describes a new programme broadly consistent with Illing report endorsements. Second, it illustrates a novel evaluation method that highlights rigorously the contexts, mechanisms and outcomes at the pilot stage of an intervention identifies contexts and mechanisms via factor analysis using Q-sort methodology.

Quality outcomes in NHS library and knowledge services

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Clare Edwards ◽  
Dominic Gilroy
Keyword(s):  
Evidence Base ◽  
Evaluation Framework ◽  
Strategic Development ◽  
Quality Outcomes ◽  
Content Type ◽  
Development Framework ◽  
High Level ◽  
Impact Planning ◽  
The Impact ◽  
Knowledge Services

PurposeThis paper aims to demonstrate the approach taken in delivering the quality and impact elements of Knowledge for Healthcare, the strategic development framework for National Health Service (NHS) library and knowledge services in England. It examines the work undertaken to enhance quality and demonstrate the value and impact of health library and knowledge services. It describes the interventions developed and implemented over a five-year period 2015–2020 and the move towards an outcome rather than process approach to impact and quality.Design/methodology/approachThe case study illustrates a range of interventions that have been developed, including the outcomes of implementation to date. The methodology behind each intervention is informed by the evidence base and includes professional engagement.FindingsThe outcomes approach to the development and implementation of quality and impact interventions and assets provides evidence to demonstrate the value of library and knowledge staff to the NHS in England to both high-level decision-makers and service users.Originality/valueThe interventions are original concepts developed within the NHS to demonstrate system-wide impacts and change. The Evaluation Framework has been developed based on the impact planning and assessment (IPA) methodology. The interventions can be applied to other healthcare systems, and the generic learning is transferable to other library and knowledge sectors, such as higher education.

scholarly journals Realism and rhetoric in the evaluation of a new care model

2019 ◽  
Vol 27 (3) ◽  
pp. 204-214 ◽  
Author(s):  
Tom Grimwood
Keyword(s):  
Integrated Care ◽  
Large Scale ◽  
Realist Evaluation ◽  
Scale Transformation ◽  
Care Model ◽  
Content Type ◽  
Realist Approach ◽  
Dynamic Form ◽  
Multiple Audiences ◽  
Objective System

Purpose The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care in England, and make the case for a modified realist approach to this kind of evaluation. Design/methodology/approach The paper considers three challenges to evaluating the NCM in this particular vanguard: complexity, strategy and rhetoric. It reflects on how the realist approach negotiates these philosophical challenges to delivering integrated care, in order to provide contextualised accounts of who a programme works for, in what context, and why. Findings The paper argues that, in the case of this particular vanguard site, the tangible benefit of the realist approach was not in providing a firm epistemological basis for evaluation, but rather in drawing out and articulating the ontological rhetoric of such large-scale transformation programmes. By understanding the work of the NCM less as an objective “system”, and more as a dynamic form of persuasion, aimed at securing the “adherence of minds” (Perelman and Olbrechts-Tyteca, 2008, p. 8) in multiple audiences, the paper suggests that realist evaluation can be used to address both the systematic issues and localised successes the NCMs encountered. Originality/value The paper identifies a number of aspects of new models of integrated care for evaluators to consider. It offers ways of negotiating the challenges to conventional outcome-focused evaluation, by drawing attention to the need for contextualised, time-situated and audience-sensitive value of NCMs.

scholarly journals Utilizing a realist evaluative research approach to investigate complex technology implementations

2017 ◽  
Vol 19 (1/2) ◽  
pp. 22-41 ◽  
Author(s):  
Melanie Rose Nova King ◽  
Ray J. Dawson ◽  
Steve J. Rothberg ◽  
Firat Batmaz
Keyword(s):  
Educational Technology ◽  
Policy Development ◽  
Evaluation Framework ◽  
Future Research ◽  
Specific Reference ◽  
Research Approach ◽  
Technological Determinism ◽  
Content Type ◽  
Complex Technology ◽  
Evaluative Research

Purpose This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations. Design/methodology/approach An institution wide e-learning implementation of lecture capture (LC), within a UK University, was chosen, and a realist evaluation framework was used, tailored for educational technology. The research was conducted over four, increasingly focused, evaluation cycles combining engagement analytics, user interviews and theory to refine what works (or does not work), for whom, in which contexts and why. Findings Despite explicit demand and corresponding investment, overall student engagement is lower than expected. Increased student use appears linked to particular staff attitudes and behaviours and not to specific disciplines or course content. The main benefits of LC are providing reassurance to the majority, aiding revision and understanding for the many and enabling catch-up for the few. Recommendations for future research are based on some unexpected outcomes uncovered, including evolving detrimental student behaviours, policy development based on technological determinism and future learner-centred system development for next-generation LC technologies. Practical implications The realist approach taken, and evaluation framework used, can be adopted (and adapted) for future evaluative research. Domain specific reference models, categorizing people and technology, supported analysis across multiple contexts. Originality/value This study responds to a call for more theory-based research in the field of educational technology. The authors demonstrate that a theory-driven approach provides real and practical recommendations for institutions and allows for greater insight into the political, economic and social complexity of technology implementation.

scholarly journals Contextualising Youth Justice Interventions: Making the Case for Realist Synthesis

2022 ◽  
Vol 14 (2) ◽  
pp. 854
Author(s):  
Charlie E. Sutton ◽  
Mark Monaghan ◽  
Stephen Case ◽  
Joanne Greenhalgh ◽  
Judy Wright
Keyword(s):  
Critical Review ◽  
Evidence Base ◽  
Evaluation Framework ◽  
Youth Justice ◽  
Intervention Evaluation ◽  
Realist Approach ◽  
Context Sensitive ◽  
Intervention Selection ◽  
What Works ◽  
Justice Practice

This article examines the problematic reductionism and decontextualising nature of hegemonic youth justice intervention evaluation and offers a way ahead for a realistic, context-sensitive approach to intervention evaluation in the youth justice field. It opens by considering how the development of risk-based youth justice interventions in England and Wales flowed from and fed into the modernisation and resultant partiality of the ‘evidence-base’, which shaped youth justice practice. It then moves to a critical review of the emergence and continued influence of risk-based interventions and the ‘What Works’ intervention evaluation framework in youth justice. In the closing discussion, this article envisages the potential of taking a realist approach to the evaluation of youth justice interventions to mitigate the limitations of current approaches to intervention selection and the evaluation of their ‘effectiveness’.

scholarly journals Total risk evaluation framework

2020 ◽  
Vol 37 (4) ◽  
pp. 575-608 ◽  
Author(s):  
Zsolt Tibor Kosztyán ◽  
Tibor Csizmadia ◽  
Zoltán Kovács ◽  
István Mihálcz
Keyword(s):  
Risk Evaluation ◽  
Evaluation Framework ◽  
Total Risk ◽  
Effect Analysis ◽  
Content Type ◽  
Product Service ◽  
Organizational Demands ◽  
Evaluation Approaches ◽  
New Framework

PurposeThe purpose of this paper is to generalize the traditional risk evaluation methods and to specify a multi-level risk evaluation framework, in order to prepare customized risk evaluation and to enable effectively integrating the elements of risk evaluation.Design/methodology/approachA real case study of an electric motor manufacturing company is presented to illustrate the advantages of this new framework compared to the traditional and fuzzy failure mode and effect analysis (FMEA) approaches.FindingsThe essence of the proposed total risk evaluation framework (TREF) is its flexible approach that enables the effective integration of firms’ individual requirements by developing tailor-made organizational risk evaluation.Originality/valueIncreasing product/service complexity has led to increasingly complex yet unique organizational operations; as a result, their risk evaluation is a very challenging task. Distinct structures, characteristics and processes within and between organizations require a flexible yet robust approach of evaluating risks efficiently. Most recent risk evaluation approaches are considered to be inadequate due to the lack of flexibility and an inappropriate structure for addressing the unique organizational demands and contextual factors. To address this challenge effectively, taking a crucial step toward customization of risk evaluation.

Experience and expression of social isolation by inner-city high-rise residents

2014 ◽  
Vol 17 (3) ◽  
pp. 151-166 ◽  
Author(s):  
Love M. Chile ◽  
Xavier M. Black ◽  
Carol Neill
Keyword(s):  
Older Adults ◽  
Inner City ◽  
Social Isolation ◽  
Age Groups ◽  
Evidence Base ◽  
Structured Interviews ◽  
Content Type ◽  
High Rise ◽  
Starting Point ◽  
Health And Social Care

Purpose – The purpose of this paper is to examine the significance of social isolation and the factors that create social isolation for residents of inner-city high-rise apartment communities. We critically examine how the physical environment and perceptions of safety in apartment buildings and the inner-city implicate the quality of interactions between residents and with their neighbourhood community. Design/methodology/approach – The authors used mixed-methods consisting of survey questionnaires supplemented by semi-structured interviews and focus group discussions using stratified random sampling to access predetermined key strata of inner-city high-rise resident population. Using coefficient of correlation we examine the significance of the association between social isolation, age and ethnicity amongst Auckland's inner-city high-rise residents. Findings – The authors found the experience and expression of social isolation consistent across all age groups, with highest correlation between functional social isolation and “being student”, and older adults (60+ years), length of tenure in current apartment and length of time residents have lived in the inner-city. Research limitations/implications – As a case study, we did not seek in this research to compare the experience and expressions of social isolation in different inner-city contexts, nor of inner-city high-rise residents in New Zealand and other countries, although these will be useful areas to explore in future studies. Practical implications – This study is a useful starting point to build evidence base for professionals working in health and social care services to develop interventions that will help reduce functional social isolation amongst young adults and older adults in inner-city high-rise apartments. This is particularly important as the inner-city population of older adults grow due to international migration, and sub-national shifts from suburbs to the inner-cities in response to governmental policies of urban consolidation. Originality/value – By identifying two forms of social isolation, namely functional and structural social isolation, we have extended previous analysis of social isolation and found that “living alone” or structural social isolation did not necessarily lead to functional social isolation. It also touched on the links between functional social isolation and self-efficacy of older adults, particularly those from immigrant backgrounds.

Embedding employability within higher education for the profession of policing

2019 ◽  
Vol 9 (3) ◽  
pp. 319-328
Author(s):  
Ian Pepper ◽  
Ruth McGrath
Keyword(s):  
Higher Education ◽  
Design Methodology ◽  
Research Study ◽  
Longitudinal Research ◽  
Evidence Base ◽  
Content Type ◽  
Educational Provision ◽  
Professional Body ◽  
The Impact ◽  
Practical Implications

Purpose The purpose of this paper is to evaluate the impact of an employability module, the College of Policing Certificate in Knowledge of Policing (CKP), on students’ career aspirations, their confidence and wish to join the police along with the appropriateness of the module. This will inform the implementation of employability as part of the College of Policing-managed Police Education Qualifications Framework (PEQF). Design/methodology/approach A three-year longitudinal research study used mixed methods across four points in time to evaluate the impact on students studying the employability module. Findings The research suggests that the employability-focussed CKP was useful as an introduction to policing, it developed interest in the police and enhanced the confidence of learners applying to join. Lessons learnt from the CKP should be considered during the implementation of the PEQF. Research limitations/implications The ability to generalise findings across different groups is limited as other influences may impact on a learner’s confidence and employability. However, the implications for the PEQF curriculum are worthy of consideration. Practical implications As the police service moves towards standardised higher educational provision and evolution of policing as a profession, lessons can be learnt from the CKP with regards to the future employability of graduates. Originality/value Enhancing the employability evidence base, focussing on policing, the research identified aspects which may impact on graduates completing a degree mapped to the PEQF. The research is therefore of value to higher education and the professional body for policing.

Entrepreneurship in regions: differentiated impacts of the socio cultural and gender types

2015 ◽  
Vol 28 (1) ◽  
pp. 45-76 ◽  
Author(s):  
Claudio Mancilla ◽  
José Ernesto Amorós
Keyword(s):  
Negative Impact ◽  
Positive Impact ◽  
Fear Of Failure ◽  
Content Type ◽  
Reference Models ◽  
Cultural Aspects ◽  
And Gender ◽  
The Individual ◽  
Practical Implications ◽  
Secondary City

Purpose The purpose of this paper is to study the differentiated impact of factors that influence the propensity to entrepreneur in a sample of people in Chile. A distinction is made between individuals that live in primary cities and secondary cities. The differentiating factors are socio‐cultural aspects (reference models – positive examples of entrepreneurs – and perception of social fear of failure) and the gender of the individual. Design/methodology/approach For the research data from the survey used in Chile by the Global Entrepreneurship Monitor for the years 2010 and 2011 were used. A logit model was used to determine the differentiated impact of the analysed factors and interactions were done using the method proposed by Corneliâen and Sonderhof (2009). Findings These showed that the fact that an individual lives in a secondary city decreases his entrepreneurship probability. The positive impact that the reference models have is weaker in women. Contrary to what was expected, the negative impact of the fear of failure perception is weaker in women. Practical implications These results have the implications to suggest focused public policies and differentiations that consider the socio‐cultural, territorial (focused in cities) and gender aspects. Originality/value The research contributes by giving empirical evidence of the existence of the negative impact of living in a secondary city and of differentiated effects of socio‐cultural factors from the gender perspective.

Sign in / Sign up

Export Citation Format

Share Document