The (real) need for a human touch: testing a human–machine hybrid topic classification workflow on a New York Times corpus

The classification of the items of ever-increasing textual databases has become an important goal for a number of research groups active in the field of computational social science. Due to the increased amount of text data there is a growing number of use-cases where the initial effort of human classifiers was successfully augmented using supervised machine learning (SML). In this paper, we investigate such a hybrid workflow solution classifying the lead paragraphs of New York Times front-page articles from 1996 to 2006 according to policy topic categories (such as education or defense) of the Comparative Agendas Project (CAP). The SML classification is conducted in multiple rounds and, within each round, we run the SML algorithm on n samples and n times if the given algorithm is non-deterministic (e.g., SVM). If all the SML predictions point towards a single label for a document, then it is classified as such (this approach is also called a “voting ensemble"). In the second step, we explore several scenarios, ranging from using the SML ensemble without human validation to incorporating active learning. Using these scenarios, we can quantify the gains from the various workflow versions. We find that using human coding and validation combined with an ensemble SML hybrid approach can reduce the need for human coding while maintaining very high precision rates and offering a modest to a good level of recall. The modularity of this hybrid workflow allows for various setups to address the idiosyncratic resource bottlenecks that a large-scale text classification project might face.

Use of commercially available natural language processing software to identify bleeding from the medical record

Background: Real-world evidence derived from the electronic medical record (EMR) is increasingly prevalent. How best to ascertain cardiovascular outcomes from EMRs is unknown. We sought to validate a commercially available natural language processing (NLP) software to extract bleeding events. Methods: We included patients with atrial fibrillation and cancer seen at our cancer center from 1/1/2016 to 12/31/2019. A query set based on SNOMED CT expressions was created to represent bleeding from 11 different organ systems. We ran the query against the clinical notes and randomly selected a sample of notes for physician validation. The primary outcome was the positive predictive value (PPV) of the software to identify bleeding events stratified by organ system. Results: We included 1370 patients with mean age 72 years old (SD 1.5) and 35% female. We processed 66,130 notes; the NLP software identified 6522 notes including 654 unique patients with possible bleeding events. Among 1269 randomly selected notes, the PPV of the software ranged from 0.921 for neurologic bleeds to 0.571 for OB/GYN bleeds. Patterns related to false positive bleeding events identified by the software included historic bleeds, hypothetical bleeds, missed negatives, and word errors. Conclusions: NLP may provide an alternative for population-level screening for bleeding outcomes in cardiovascular studies. Human validation is still needed, but an NLP-driven screening approach may improve efficiency.

Revealing Developers’ Arguments on Validating the Incidence of Code Smells: A Focus Group Experience

Identifying code smells is considered a subjective task. Unfortunately, current automated detection tools cannot deal with such subjectivity, requiring human validation. Developers tend to follow different, albeit complementary, strategies when validating the identified smells. Intending to find out developers' arguments when validating the incidence of code smells, we conducted a focus group session with developers familiar with identifying code smells. We distributed them among two groups, in which they had to argue about the incidence of a code smell: either accepting or rejecting its presence. Based on their arguments, we compiled a set of general heuristics that developers follow when validating smells. We then used these heuristics for composing validation items. We understand that the set of validation items proposed may support developers in reflecting on the incidence of code smells. However, further studies are needed for reaching a more comprehensive and optimized set. The experience of this study reveals that conducting focus group sessions is helpful to emerge the tacit knowledge of developers when validating code smells.

P–280 Changes in oolemma height during ICSI injection on day 0 is associated with day 5–6 blastocyst formation

Study question Does the oolemma response to ICSI injection on day 0 affect blastocyst formation on day 5–6 (d5/6)? Summary answer A large change in oolemma height during ICSI injection on day 0 is associated with lower blastocyst formation rates on d5/6. What is known already The oolemma changes in all dimensions (i.e. height, width and depth) and can exhibit different reactions in ICSI during needle injection. This is seen as instant rupture or with little needle pressure, normal rupture with the needle pushed approximately halfway through, or difficult rupture with repeated attempts or the needle passing 3/4 of the oocyte width. Previous studies have shown that these responses can affect degeneration and fertilisation rates on day 1, however, there is little research on its effect on blastocyst formation rates. Furthermore, most previous studies have used qualitative methods to assess oolemma response. Study design, size, duration This is a retrospective study using ICSI procedure videos conducted by four embryologists in a private clinic from 2013–2015. All videos of procedures which did not result in 2PN or in which the oocyte was not fully visible were excluded. Six operators categorised 455 videos (by majority vote) into four groups based on the oolemma response: oolemma breakage within 1/4, between 1/4 and 1/2, between 1/2 and 3/4 and beyond 3/4 of the oocyte’s width. Participants/materials, setting, methods A U-Net neural network model was trained to extract the frame of maximum oolemma indent from each video which were validated by a human operator; any in which maximum indent occured after breaking of the oolemma were excluded. The ratio of starting to maximal indent width/height were calculated automatically and human-validated. Chi-squared tests were performed for each ratio vs d5/6 blastocyst formation. These results were compared with those obtained from purely human annotations. Main results and the role of chance From the purely human annotations, the percentages of oocytes in groups 1–4 respectively were: 3.3%, 85.3%, 11.4% and 0%. This variation in oolemma response may be due to the arrangement of thick and thin microfilaments or cortical granules in the cytoskeleton. When analysed with d5/6 blastocyst formation, these showed no significant result (p = 0.12) which is consistent with findings using the model. The artificial intelligence (AI) model processed 26 frames per second. During human validation of the ratios calculated at maximal indentation, 36% of width ratios and 31% of height ratios were rejected. The proportion of blastocysts formed in the upper and lower quartile for each ratio was analysed. Both the upper (0.49) and lower (0.41) quartiles of the width ratios were not significant for d5/6 blastocyst formation. The lower (1.12) quartile of height ratios showed no significance, however there were significantly fewer blastocysts formed on d5/6 for the upper (1.18) quartile of height ratios (p < 0.025). This subtle change in the height ratio, which was significant for d5/6 blastocyst formation was not taken into account when grouping oocytes any previous literature (and our human labelling). Limitations, reasons for caution This study was conducted at a single clinic so variations between clinics were not captured in the study and would need further collaborations to confirm the proportion of oocytes responses. Due to the small sample size, this study also did not identify any group 4 oocytes cultured until d5/6. Wider implications of the findings: The grouping criteria in this study were more quantitative than previous work yet indicated no correlation between the oolemma group and d5/6 blastocyst formation. However, changes in the height which are hard to assess in real-time (and which have been neglected in previous literature) were seen to be significant. Trial registration number NA

Negative sentiment towards COVID-19 vaccines: A comparative study of USA and UK social media posts before vaccination rollout

Introduction: The global spread of the COVID-19 pandemic was rapid and devastating to humanity. The public health response to the pandemic was rapid too. Completion of COVID-19 vaccine development was achieved in under a year. The USA and the UK were the first countries to rollout COVID-19 vaccines to contain the pandemic. Successful rollout of the vaccines hinges on many factors, among which is public trust. Aim: To investigate the sentiments towards COVID-19 vaccines in the USA and UK prior to vaccination rollout. Methods: Neuro-linguistic programming with human validation was used to analyse a sample of 243,883 COVID-19 vaccine related social media posts from the USA and the UK in the period 28 July to 28 August 2020. The sentiment analysis measured polarity (positive, neutral, negative), and the themes present in negative comments. Results: In the sample of 243,883 social media posts, both the USA and the UK had a net sentiment profile of approximately 28% positive, 8% negative and 63% neutral sentiment. On further analysis, there were distinct differences between the two country’s social media sentiment towards COVID-19 vaccines. The differences were seen in the themes behind the negative sentiment. In the USA, the negative sentiments were mainly due to health and safety concerns, the fear of making a vaccine mandatory, and the role that pharmaceutical companies would play with the release of vaccines. In the UK the main driver of negative sentiment was the fear of making the vaccine mandatory (almost double the size of the sentiment in the USA). Conclusions: Negative sentiments towards COVID-19 vaccines were prevalent in the third quarter of 2020 in the USA and the UK. Reasons behind the negative sentiments can be used by authorities in the two countries to design evidence-based interventions to address the refusal of vaccination against COVID-19.

Towards Automated Psychotherapy via Language Modeling

In this experiment, a model was devised, trained, and evaluated to automate psychotherapist/client text conversations through state-of-the-art, Seq2Seq Transformer-based Natural Language Generation (NLG) systems. Through training upon a mix of the Cornell Movie Dialogue Corpus for language understanding and an open-source, anonymized, and public licensed psychotherapeutic dataset, the model achieved statistically significant performance in published, standardized qualitative benchmarks against human validation data — meeting or exceeding human-written response performance in 59.7% and 67.1% of the test set two independent test methods respectively. Although the model cannot replace the work of psychotherapy, its ability to synthesize human-appearing utterances for the majority of the test set serves as a promising step towards communizing and easing tensions at the psychotherapeutic point-of-care.

Powering electronic implants by high frequency volume conduction: in human validation

Aiming at miniaturization, wireless power transfer (WPT) is frequently used in biomedical electronic implants as an alternative to batteries. However, WPT methods in use still require integrating bulky parts within the receiver, thus hindering the development of devices implantable by minimally invasive procedures, particularly when powers above 1 mW are required in deep locations. In this regard, WPT based on volume conduction of high frequency currents is an advantageous alternative relatively unexplored, and never demonstrated in humans. We describe an experimental study in which ac and dc electric powers in the order of milliwatts are obtained from pairs of needle electrodes (diameter = 0.4 mm, separation = 30 mm) inserted into the arms or lower legs of five healthy participants while innocuous and imperceptible high frequency (6.78 MHz) currents are delivered through two textile electrodes strapped around the considered limb. In addition, we demonstrate a procedure to model WPT based on volume conduction which characterizes coupling between the transmitters and the receivers by means of two-port impedance models which are generated from participants' medical images.

Unaltered Liver Regeneration in Post-Cholestatic Rats Treated with the FXR Agonist Obeticholic Acid

In a previous study, obeticholic acid (OCA) increased liver growth before partial hepatectomy (PHx) in rats through the bile acid receptor farnesoid X-receptor (FXR). In that model, OCA was administered during obstructive cholestasis. However, patients normally undergo PHx several days after biliary drainage. The effects of OCA on liver regeneration were therefore studied in post-cholestatic Wistar rats. Rats underwent sham surgery or reversible bile duct ligation (rBDL), which was relieved after 7 days. PHx was performed one day after restoration of bile flow. Rats received 10 mg/kg OCA per day or were fed vehicle from restoration of bile flow until sacrifice 5 days after PHx. Liver regeneration was comparable between cholestatic and non-cholestatic livers in PHx-subjected rats, which paralleled liver regeneration a human validation cohort. OCA treatment induced ileal Fgf15 mRNA expression but did not enhance post-PHx hepatocyte proliferation through FXR/SHP signaling. OCA treatment neither increased mitosis rates nor recovery of liver weight after PHx but accelerated liver regrowth in rats that had not been subjected to rBDL. OCA did not increase biliary injury. Conclusively, OCA does not induce liver regeneration in post-cholestatic rats and does not exacerbate biliary damage that results from cholestasis. This study challenges the previously reported beneficial effects of OCA in liver regeneration in cholestatic rats.