The Effects of Different Rater Training Procedures on ESL Essay Raters’ Rating Accuracy

The study investigated the effects of three commonly employed rater training procedures on the rating accuracy of novice ESL essay raters. The first training procedure involved going through a set of benchmarked scripts with scores, the second involved assessing benchmarked scripts before viewing the scores. The third was a combination of the former and latter. A pre, post and delayed post-experimental research design was employed. Data were collected before, immediately after and one month after training. Actual IELTS scripts with benchmarked scores determined by subjecting expert IELTS raters’ scores through Multi-Faceted Rasch (MFR) analysis were used for the training and data collection purposes. Sixty-three TESL trainees were selected based on their pre-training rating accuracy to form three equally matched experimental groups. The trainees’ scores for the essays before, immediately after and one month after the assigned training procedure were compared with the official scores for the operational essays. Although the findings indicate that generally, rater training improves raters’ rating accuracy by narrowing the gap between their scores and the official scores, different training procedures seem to have different effects. The first training procedure significantly improved raters’ rating accuracy but showed a decreasing effect with time. The second training procedure showed immediate as well as delayed positive effects on raters’ rating accuracy. The third training did not lead to significant immediate improvement, but rating accuracy improved significantly after some time. This paper discusses the implications of the findings in planning efficient and effective rater training programmes.

Social Media Sharing Reduces Truth Discernment

There is widespread concern about fake news and other misinformation circulating on social media. In particular, many argue that the context of social media itself may make people particularly susceptible to the influence of false claims. Here, we test that claim by asking whether simply considering whether to share news on social media reduces people’s ability to identify truth versus falsehood. In a large online experiment (N=3,157 Americans quota-matched to the national distribution of age, gender, ethnicity, and geographic region) examining COVID-19 and political news, we find support for this possibility. Compared to a baseline where participants judged only the accuracy of each headline, we observed worse truth discernment when participants also indicated their sharing intentions. Conversely, sharing discernment was substantially higher when participants also rated accuracy, relative to a baseline where sharing intentions were elicited without rating accuracy. These results suggest people may be particularly vulnerable to believing false claims on social media due to fundamental features of these platforms – which is particularly concerning given that it is hard to imagine social media without sharing.

Assessing critical thinking in L2: An exploratory study

Critical thinking (CT) is taking on an increasingly important role in Japanese tertiary education. Teachers tasked with developing CT in a second-language (L2) context may need a way of assessing students’ abilities. However, a number of difficulties face L2 students taking a test designed for first-language (L1) speakers. They may be disadvantaged by linguistic and perhaps cultural issues. This study describes an exploratory attempt to make a CT test that can be administered to learners of English and which allows them to display selected elements of CT, specifically analyzing arguments and judging or evaluating. A comparison of L1 and L2 performance in the test showed the results to be comparable. Analysis of two different question topics showed differences in CT skills displayed. Issues with rating accuracy are linked to the format of the test. We argue that this test format is suitable for many students in Japan and elsewhere who have intermediate levels of English.

Assessing critical thinking in L2: An exploratory study

Critical thinking (CT) is taking on an increasingly important role in Japanese tertiary education. Teachers tasked with developing CT in a second-language (L2) context may need a way of assessing students’ abilities. However, a number of difficulties face L2 students taking a test designed for first-language (L1) speakers. They may be disadvantaged by linguistic and perhaps cultural issues. This study describes an exploratory attempt to make a CT test that can be administered to learners of English and which allows them to display selected elements of CT, specifically analyzing arguments and judging or evaluating. A comparison of L1 and L2 performance in the test showed the results to be comparable. Analysis of two different question topics showed differences in CT skills displayed. Issues with rating accuracy are linked to the format of the test. We argue that this test format is suitable for many students in Japan and elsewhere who have intermediate levels of English.

Effect of Training With Anchors on Auditory-Perceptual Evaluation of Dysphonia in Speech-Language Pathology Students

Purpose Auditory-perceptual evaluation is essential for the clinical assessment of voice disorders. Unstable perceptual voice evaluation has been shown for inexperienced listeners as compared to expert listeners. We examined the effects of perceptual training with external auditory anchors with and without immediate feedback on the evaluation of roughness and breathiness of natural, nonsynthesized speech stimuli (reading of a standard passage) in speech-language pathology students. Method Perceptual voice evaluation and training with anchors using a visual analog scale was implemented with a computer software. Forty-eight speech-language pathology students were randomly assigned into three groups, feedback group (Group F), no feedback group (Group NF), and control group (Group C), attending one training session and four assessment sessions (before training, immediately after training, and 1 and 7 weeks after training). Group F received training with anchors with immediate feedback, Group NF received training without immediate feedback, and Group C received sham training (exposure session). Results Training with anchors significantly increased the rating accuracy (agreement with expert ratings) on both roughness and breathiness for Group F, with the effects lasting for 7 weeks. No significant changes in rating accuracy with training were observed for Group NF and Group C. No improvements in intra- and interrater reliability as well as intrarater agreement were observed in all three groups, whereas interrater agreement on breathiness (but not roughness) significantly increased for all groups, with the effect lasting for 7 weeks only for Group F. Conclusions These findings suggested that perceptual training with external auditory anchors and the use of immediate feedback could be effective for facilitating the development of perceptual voice evaluation skills in speech-language pathology students. Further studies involving more extensive training with stimuli covering a full range of dysphonia severity categories and improvements in design of the training protocol are recommended to verify these results.

CNAs’ Ratings of Nursing Home Residents’ Pain: The Role of CNA and Resident Traits

Long-term care residents with and without cognitive impairment may experience undertreatment of persistent pain (Fain et. al, 2017). Certified nursing assistants (CNAs) are important sources of information about resident pain as they provide the majority of residents’ hands-on care. Therefore, assessing the accuracy of CNAs’ pain assessments and potential influencing factors may provide insight regarding the undertreatment of pain. Informed by prior research, this study examined resident pain catastrophizing and cognitive status as predictors of CNAs’ pain assessment accuracy. CNA empathy was examined as a moderating variable. Analyses confirmed a relationship between pain catastrophizing and CNA pain rating accuracy (R^2 = .205, p < .01), reflecting lower accuracy of ratings for residents higher in catastrophizing. Hypotheses predicting a relationship between resident cognitive status and CNA pain rating accuracy and moderating effects of empathy were disconfirmed. Challenges of conducting research in long-term care are discussed.

Teachers' Accuracy in Estimating Social Inclusion of Students With and Without Special Educational Needs

It is unclear to what extent teachers can accurately assess the social inclusion of their students with and without SEN. The study aims to shed light on these desiderata. Students (N = 1.644) with SEN (learning, behavior, and language problems) and without SEN and their teachers (N = 79) participated in the study. Sociometric peer nominations, students' self-perceived social inclusion, and teachers' assessments regarding students' social inclusion and self-perceived social inclusion were administered. The results suggest that teachers are moderately accurate in identifying social acceptance and social rejection, while accuracy is low when assessing students' self-perceived social inclusion. That said, rating accuracy varied strongly between teachers, ranging from no agreement to a perfect concordance. Teachers seem to be more accurate in estimating the social acceptance of students with learning problems. The results emphasize the importance of differentiating between various social inclusion criteria (i.e., students' self-report vs. peer nominations) and accounting for inter-individual differences in teachers' rating accuracy.

Management Tenure and the Quality of Corporate Bond Ratings

We examine the implication of management for credit rating quality by focusing on the relation between management tenure and rating quality. Using a large sample of corporate bond issues in the U.S., we find robust evidence that firms with longer-tenured CEOs have lower rating quality, as reflected in lower rating accuracy, informativeness, and timeliness. Further analyses uncover two channels that underlie this relation. One channel is through learned confidence: as CEO tenure increases, rating agencies learn about how the CEO influences firm value, which leads agencies to reduce their caution and effort in management assessment. The other channel is through developed relationships: as CEO tenure increases, rating agencies develop relationships with the CEO, which leads agencies to reduce scrutiny of or cater to the CEO and her firm. Overall, we show that management tenure has important implications for the external oversight of rating agencies.