Ablations over transformer models for biomedical relationship extraction

Background: Masked language modelling approaches have enjoyed success in improving benchmark performance across many general and biomedical domain natural language processing tasks, including biomedical relationship extraction (RE). However, the recent surge in both the number of novel architectures and the volume of training data they utilise may lead us to question whether domain specific pretrained models are necessary. Additionally, recent work has proposed novel classification heads for RE tasks, further improving performance. Here, we perform ablations over several pretrained models and classification heads to try to untangle the perceived benefits of each. Methods: We use a range of string preprocessing strategies, combined with Bidirectional Encoder Representations from Transformers (BERT), BioBERT and RoBERTa architectures to perform ablations over three RE datasets pertaining to drug-drug and chemical protein interactions, and general domain relationship extraction. We explore the use of the RBERT classification head, compared to a simple linear classification layer across all architectures and datasets. Results: We observe a moderate performance benefit in using the BioBERT pretrained model over the BERT base cased model, although there appears to be little difference when comparing BioBERT to RoBERTa large. In addition, we observe a substantial benefit of using the RBERT head on the general domain RE dataset, but this is not consistently reflected in the biomedical RE datasets. Finally, we discover that randomising the token order of training data does not result in catastrophic performance degradation in our selected tasks. Conclusions: We find a recent general domain pretrained model performs approximately the same as a biomedical specific one, suggesting that domain specific models may be of limited use given the tendency of recent model pretraining regimes to incorporate ever broader sets of data. In addition, we suggest that care must be taken in RE model training, to prevent fitting to non-syntactic features of datasets.

Memory Skill Acquisition for Young and Old Adults: Does Training Order Affect Learning?

The purpose of the present experiment was to identify the effects of altering the order of training for a memory search task in old and young adults. We provided subjects with extensive practice on consistently mapped (CM) and variably mapped (VM) versions of a memory search task. Half of the subjects in each age group received CM training followed by VM training and the other half received VM first followed by CM. Based on previous findings (Fisk, Rogers, and Giambra, 1990), in which older adults did not switch to a more efficient search strategy (i. e., from serial exhaustive to serial self-terminating) we predicted that older subjects who received VM training first would not adopt the most efficient strategy on subsequent CM training compared to old adults who received the CM training first. The results supported our prediction: namely, the comparison slopes were shallower (i. e., more efficient) for the older adults who received CM training first, relative to those who received VM training prior to the CM training. Order of practice did not significantly affect the performance of the young adults. These data have important implications for the development of training programs in which subjects will be required to learn several task components.

Statistical Process Control Charting Applied to the Analysis of Human Performance in Computer-Supported Environments

To develop usable Human-Machine Systems, we need Tools to evaluate and measure the length of learning periods, error rate, response time, and transfer of learning in the human operators of these systems (Whiteside, Bennett, and Holtzblatt, 1988). This research explores the use of Statistical Process Control (SPC) charts as a tool to visualize and analyze performance in a decision-making task. The data submitted to control charting was collected in an experiment that explored the effect of order of training or experience in working with alternate display formats. Results for an individual subject as well as a summary for one of the four experimental groups are discussed. Suggestions for further applications of these techniques are offered.

The Effects of Different Types of Advance Organizers on Classification Learning

The effects of three types of advance organizer lessons containing high-order social studies concept (AO1), high-order rules for hierarchical classification (AO2), or both (AO3), on the learning of social studies concepts and hierarchical classification (as defined by Piaget) were evaluated for a sample of 237 rural children in kindergarten, third, and fifth grades. The overall order of training effect was AO3 → AO2 → AO1 → C. Effects on delayed posttests and or transfer tasks are also presented.