Discussion on Scoring Issues in Second Signed or Spoken Language Assessment

This chapter is a joint discussion of key items related to scoring issues related to signed and spoken language assessment that were discussed in Chapters 9.1 and 9.2. One aspect of signed language assessment that has the potential to stimulate new research in spoken second language (L2) assessment is the scoring of nonverbal speaker behaviors. This aspect is rarely represented in the scoring criteria of spoken assessments and in many cases not even available to raters during the scoring process. The authors argue, therefore, for a broadening of the construct of spoken language assessment to also include elements of nonverbal communication in the scoring descriptors. Additionally, the importance of rater training for signed language assessments, application of Rasch analysis to investigate possible reasons of disagreement between raters, and the need to conduct research on rasting scales are discussed.

Scoring Spoken Second Language Assessment

Achieving scores that adequately reflect the test-takers’ proficiency level, as evidenced in spoken assessment tasks, has been the subject of a large body of research in second language assessment. In this chapter, the author outlines the work that has been undertaken in relation to the scoring of spoken assessments by human raters and automated scoring. The chapter focuses on research on rater effects, rater training and feedback, rater characteristics, interlocutor/interviewer effects, rating scales, and score resolution techniques. The section on automated scoring discusses research on the underlying construct and what limits this puts on the types of tasks that can be used in the assessment. The chapter concludes by setting out some future directions for the scoring of spoken responses.

How can surgical skills in laparoscopic colon surgery be objectively assessed?—a scoping review

Background In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature. Methods A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools. Results Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings. Conclusion The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

The Effects of Different Rater Training Procedures on ESL Essay Raters’ Rating Accuracy

The study investigated the effects of three commonly employed rater training procedures on the rating accuracy of novice ESL essay raters. The first training procedure involved going through a set of benchmarked scripts with scores, the second involved assessing benchmarked scripts before viewing the scores. The third was a combination of the former and latter. A pre, post and delayed post-experimental research design was employed. Data were collected before, immediately after and one month after training. Actual IELTS scripts with benchmarked scores determined by subjecting expert IELTS raters’ scores through Multi-Faceted Rasch (MFR) analysis were used for the training and data collection purposes. Sixty-three TESL trainees were selected based on their pre-training rating accuracy to form three equally matched experimental groups. The trainees’ scores for the essays before, immediately after and one month after the assigned training procedure were compared with the official scores for the operational essays. Although the findings indicate that generally, rater training improves raters’ rating accuracy by narrowing the gap between their scores and the official scores, different training procedures seem to have different effects. The first training procedure significantly improved raters’ rating accuracy but showed a decreasing effect with time. The second training procedure showed immediate as well as delayed positive effects on raters’ rating accuracy. The third training did not lead to significant immediate improvement, but rating accuracy improved significantly after some time. This paper discusses the implications of the findings in planning efficient and effective rater training programmes.

Linking the International English Language Competency Assessment suite of examinations to the Common European Framework of Reference

The study investigated the alignment process of the International English Language Competency Assessment (IELCA) suite examinations’ four levels, B1, B2, C1 and C2, onto the Common European Framework of Reference (CEFR) by explaining and discussing the five linking stages (Council of Europe (CoE 2009). Unlike previous studies, this study used the five linking stages altogether to make fair judgements and informed decisions about the practical consequences and validity arguments of this mapping task. Findings indicated that the useful and in-depth discussions of the relevant CEFR descriptors resulted in a deeper awareness of establishing succinct re-familiarisations and re-definitions of the salient features of the different skills and items, thus making them more specific to reflect the CEFR descriptors. The ample alignment activities provided fertile ground for dependable results. For instance, teacher estimates confirmed the cut scores with high agreement percentages, ranging from 74.4 to 99.34. Also, the FACETS analyses showed a good global model fit with a high reliability value of the judgement process, only after undergoing rater training sessions. Specifically, the majority of item difficulty estimates were within the typical range, thus indicating that the IELCA examinations were measuring the underlying construct traits; however, the empirical validation called for additional data and further implementation practices regarding other judgements on the levels’ boundary for IELCA examinations. Further mapping challenges, implications, and future research were also discussed.

Development of an adapted Clinical Global Impression scale for use in Angelman syndrome

Background The Clinical Global Impression-Severity (CGI-S) and CGI-Improvement (CGI-I) scales are widely accepted tools that measure overall disease severity and change, synthesizing the clinician’s impression of the global state of an individual. Frequently employed in clinical trials for neuropsychiatric disorders, the CGI scales are typically used in conjunction with disease-specific rating scales. When no disease-specific rating scale is available, the CGI scales can be adapted to reflect the specific symptom domains that are relevant to the disorder. Angelman syndrome (AS) is a rare, clinically heterogeneous condition for which there is no disease-specific rating scale. This paper describes efforts to develop standardized, adapted CGI scales specific to AS for use in clinical trials. Methods In order to develop adapted CGI scales specific to AS, we (1) reviewed literature and interviewed caregivers and clinicians to determine the most impactful symptoms, (2) engaged expert panels to define and operationalize the symptom domains identified, (3) developed detailed rating anchors for each domain and for global severity and improvement ratings, (4) reviewed the anchors with expert clinicians and established minimally clinically meaningful change for each symptom domain, and (5) generated mock patient vignettes to test the reliability of the resulting scales and to standardize rater training. This systematic approach to developing, validating, and training raters on a standardized, adapted CGI scale specifically for AS is described herein. Results The resulting CGI-S/I-AS scales capture six critical domains (behavior, gross and fine motor function, expressive and receptive communication, and sleep) defined by caregivers and expert clinicians as the most challenging for patients with AS and their families. Conclusions Rigorous training and careful calibration for clinicians will allow the CGI-S/-I-AS scales to be reliable in the context of randomized controlled trials. The CGI-S/-I-AS scales are being utilized in a Phase 3 trial of gaboxadol for the treatment of AS.