Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing

Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. In this article, we challenge this assumption by showing that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. To facilitate this investigation, we compile a comprehensive biomedical NLP benchmark from publicly available datasets. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks, leading to new state-of-the-art results across the board. Further, in conducting a thorough evaluation of modeling choices, both for pretraining and task-specific fine-tuning, we discover that some common practices are unnecessary with BERT models, such as using complex tagging schemes in named entity recognition. To help accelerate research in biomedical NLP, we have released our state-of-the-art pretrained and task-specific models for the community, and created a leaderboard featuring our BLURB benchmark (short for Biomedical Language Understanding & Reasoning Benchmark) at https://aka.ms/BLURB .

A functional division of Drosophila sweet taste neurons that is value-based and task-specific

Sucrose is an attractive feeding substance and a positive reinforcer for Drosophila. But Drosophila females have been shown to robustly reject a sucrose-containing option for egg-laying when given a choice between a plain and a sucrose-containing option in specific contexts. How the sweet taste system of Drosophila promotes context-dependent devaluation of an egg-laying option that contains sucrose, an otherwise highly appetitive tastant, is unknown. Here, we report that devaluation of sweetness/sucrose for egg-laying is executed by a sensory pathway recruited specifically by the sweet neurons on the legs of Drosophila. First, silencing just the leg sweet neurons caused acceptance of the sucrose option in a sucrose versus plain decision, whereas expressing the channelrhodopsin CsChrimson in them caused rejection of a plain option that was “baited” with light over another that was not. Analogous bidirectional manipulations of other sweet neurons did not produce these effects. Second, circuit tracing revealed that the leg sweet neurons receive different presynaptic neuromodulations compared to some other sweet neurons and were the only ones with postsynaptic partners that projected prominently to the superior lateral protocerebrum (SLP) in the brain. Third, silencing one specific SLP-projecting postsynaptic partner of the leg sweet neurons reduced sucrose rejection, whereas expressing CsChrimson in it promoted rejection of a light-baited option during egg-laying. These results uncover that the Drosophila sweet taste system exhibits a functional division that is value-based and task-specific, challenging the conventional view that the system adheres to a simple labeled-line coding scheme.

Motor restrictions impair divergent thinking during walking and during sitting

Creativity, specifically divergent thinking, has been shown to benefit from unrestrained walking. Despite these findings, it is not clear if it is the lack of restriction that leads to the improvement. Our goal was to explore the effects of motor restrictions on divergent thinking for different movement states. In addition, we assessed whether spontaneous eye blinks, which are linked to motor execution, also predict performance. In experiment 1, we compared the performance in Guilford’s alternate uses task (AUT) during walking vs. sitting, and analysed eye blink rates during both conditions. We found that AUT scores were higher during walking than sitting. Albeit eye blinks differed significantly between movement conditions (walking vs. sitting) and task phase (baseline vs. thinking vs. responding), they did not correlate with task performance. In experiment 2 and 3, participants either walked freely or in a restricted path, or sat freely or fixated on a screen. When the factor restriction was explicitly modulated, the effect of walking was reduced, while restriction showed a significant influence on the fluency scores. Importantly, we found a significant correlation between the rate of eye blinks and creativity scores between subjects, depending on the restriction condition. Our study shows a movement state-independent effect of restriction on divergent thinking. In other words, similar to unrestrained walking, unrestrained sitting also improves divergent thinking. Importantly, we discuss a mechanistic explanation of the effect of restriction on divergent thinking based on the increased size of the focus of attention and the consequent bias towards flexibility.

Group Mobility Model for Complex Multimission Cooperation of UAV Swarm

In the unmanned aerial vehicle (UAV) swarm combat system, multiple UAVs’ collaborative operations can solve the bottleneck of the limited capability of a single UAV when they carry out complicated missions in complex combat scenarios. As one of the critical technologies of UAV collaborative operation, the mobility model is the basic infrastructure that plays an important role for UAV networking, routing, and task scheduling, especially in high dynamic and real-time scenarios. Focused on real-time guarantee and complex mission cooperative execution, a multilevel reference node mobility model based on the reference node strategy, namely, the ML-RNGM model, is proposed. In this model, the task decomposition and task correlation of UAV cluster execution are realized by using the multilayer task scheduling model. Based on the gravity model of spatial interaction and the correlation between tasks, the reference node selection algorithm is proposed to select the appropriate reference node in the process of node movement. This model can improve the real-time performance of individual tasks and the overall mission group carried out by UAVs. Meanwhile, this model can enhance the connectivity between UAVs when they are performing the same mission group. Finally, OMNeT++ is used to simulate the ML-RNGM model with three experiments, including the different number of nodes and clusters. Within the three experiments, the ML-RNGM model is compared with the random class mobility model, the reference class mobility model, and the associated class mobility model for the network connectivity rate, the average end-to-end delay, and the overhead caused by algorithms. The experimental results show that the ML-RNGM model achieves an obvious improvement in network connectivity and real-time performance for missions and tasks.

Exploring Functional Brain Network Modularity in Educational Contexts

The brain's modular functional organization facilitates adaptability. Modularity has been linked with a wide range of cognitive abilities such as intelligence, memory, and learning. However, much of this work has (1) considered modularity while a participant is at rest rather than during tasks conditions and/or (2) relied primarily on lab-based cognitive assessments. Thus, the extent to which modularity can provide information about real-word behavior remains largely unknown. Here, we investigated whether functional modularity during resting-state and task-based fMRI was associated with academic learning (measured by GPA) and ability (measured by PSAT) in a large sample of high school students. Additional questions concerned the extent to which modularity differs between rest and task conditions, and across spatial scales. Results indicated that whole-brain modularity during task conditions was significantly associated with academic learning. In contrast to prior work, no such associations were observed for resting-state modularity. We further showed that differences in modularity between task conditions and resting-state varied across spatial scales. Taken together, the present findings inform how functional brain network modularity - during task conditions and while at rest - relate to a range of cognitive abilities.

EFL PHONOLOGY: A CASE STUDY OF ENGLISH FRICATIVE PRODUCTION BY INDONESIAN LEARNERS/FONOLOGI BAHASA INGGRIS SEBAGAI BAHASA ASING: STUDI KASUS PELAFALAN BUNYI GESER OLEH MAHASISWA INDONESIA

This study reports the difficulties 40 first-year Indonesian college students, majoring in English, had in pronouncing the English fricatives. The aim of this paper is, first, to reveal how these Indonesian EFL learners produced English fricatives, the order of difficulties, and the pronunciation constraints they experienced. The second aim is to identify the possible causes of the pronunciation difficulties. In collecting the data, two types of tasks were administered: a word-list-reading task (Task 1) and a sentence-list-reading task (Task 2). By using Wilcoxon based T-Test, it was revealed that there was a significant difference in the number of errors in Task 1 and Task 2. There was also an increase in errors in Task 2. The results show that the order of difficulties Indonesian learners had in producing fricative sounds (from the most to the least problematic) were: /v/, /ʃ/, /ð/, /θ/, /z/, /ʒ/, /f/, and /s/. It is likely that the influence of the challenging English spelling system played the most important role in the students’ errors. Other factors such as transfer of L1 and the developmental process also contributed to the errors. This research implies that teachers need to apply certain strategies to meet students’ needs. Keywords: English, fricative, Indonesian students, errors AbstrakPenelitian ini melaporkan kesulitan yang dialami mahasiswa tahun pertama Jurusan Pendidikan Bahasa Inggris dalam melafalkan bunyi geser , urutan kesulitan dan kendala dalam pelafalan bunyi tersebut. Tujuan kedua penelitian ini adalah mengidentifikasi kemungkian sebab-sebab dari kesulitan ini. Dua jenis tugas diberikan kepada siswa untuk mengumpulkan data yaitu tugas membaca daftar kata (Tugas 1) dan tugas membaca daftar kalimat (Tugas 2). Berdasarkan hasil Uji T Wilcoxon terungkap bahwa terdapat perbedaan yang signifikan dari segi jumlah kesalahan dalam mengerjakan tugas I dan tugas 2. Juga terdapat peningkatan kesalahan dalam Tugas 2. Hasil-hasil ini memperlihatkan bahwa urutan kesulitan yang dialami mahasiswa Indonesia dalam melafalkan bunyi frikatif ( dari yang paling bermasalah hingga yang kurang bermasalah ) adalah : /v/, /ʃ/, /ð/, /θ/, /z/, /ʒ/, /f/, dan /s/. Ada kecenderungan bahwa sistem ejaan Bahasa Inggris yang rumit sangat besar pengaruhnya terhadap kesalahan-kesalahan yang dibuat oleh mahasiswa. Faktor-faktor lain seperti transfer dari BI dan proses perkembangan juga berkontribusi terhadap kesalahan-kesalahan tersebut. Implikasi penelitian ini adalah dosen atau guru dapat menerapkan strategi-srategi tertentu untuk memenuhi kebutuhan mahasiswa atau siswa. Kata kunci: bahasa Inggris, bunyi geser, mahasiswa Indonesia, kesalahan