Combining Self-supervised Learning and Active Learning for Disfluency Detection

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

Exploring a Bilingual Next Word Predictor for a Federated Learning Mobile Application

We explore a bilingual next-word predictor (NWP) under federated optimization for a mobile application. A character-based LSTM is server-trained on English and Dutch texts from a custom parallel corpora. This is used as the target performance. We simulate a federated learning environment to assess the feasibility of distributed training for the same model. The popular Federated Averaging (FedAvg) algorithm is used as the aggregation method. We show that the federated LSTM achieves decent performance, yet it is still sub-optimal. We suggest possible next steps to bridge this performance gap. Furthermore, we explore the effects of language imbalance varying the ratio of English and Dutch training texts (or clients). We show the model upholds performance (of the balanced case) up and until a 80/20 imbalance before decaying rapidly. Lastly, we describe the implementation of local client training, word prediction and client-server communication in a custom virtual keyboard for Android platforms. Additionally, homomorphic encryption is applied to provide with secure aggregation guarding the user from malicious servers.

Exploring a Bilingual Next Word Predictor for a Federated Learning Mobile Application

We explore a bilingual next-word predictor (NWP) under federated optimization for a mobile application. A character-based LSTM is server-trained on English and Dutch texts from a custom parallel corpora. This is used as the target performance. We simulate a federated learning environment to assess the feasibility of distributed training for the same model. The popular Federated Averaging (FedAvg) algorithm is used as the aggregation method. We show that the federated LSTM achieves decent performance, yet it is still sub-optimal. We suggest possible next steps to bridge this performance gap. Furthermore, we explore the effects of language imbalance varying the ratio of English and Dutch training texts (or clients). We show the model upholds performance (of the balanced case) up and until a 80/20 imbalance before decaying rapidly. Lastly, we describe the implementation of local client training, word prediction and client-server communication in a custom virtual keyboard for Android platforms. Additionally, homomorphic encryption is applied to provide with secure aggregation guarding the user from malicious servers.

MEMe: A Mutually Enhanced Modeling Method for Efficient and Effective Human Pose Estimation

In this paper, a mutually enhanced modeling method (MEMe) is presented for human pose estimation, which focuses on enhancing lightweight model performance, but with low complexity. To obtain higher accuracy, a traditional model scale is largely expanded with heavy deployment difficulties. However, for a more lightweight model, there is a large performance gap compared to the former; thus, an urgent need for a way to fill it. Therefore, we propose a MEMe to reconstruct a lightweight baseline model, EffBase transferred intuitively from EfficientDet, into the efficient and effective pose (EEffPose) net, which contains three mutually enhanced modules: the Enhanced EffNet (EEffNet) backbone, the total fusion neck (TFNeck), and the final attention head (FAHead). Extensive experiments on COCO and MPII benchmarks show that our MEMe-based models reach state-of-the-art performances, with limited parameters. Specifically, in the same conditions, our EEffPose-P0 with 256 × 192 can use only 8.98 M parameters to achieve 75.4 AP on the COCO val set, which outperforms HRNet-W48, but with only 14% of its parameters.

An Experiment-Based Comparison between Fully Digital and Hybrid Beamforming Radio Architectures for Many-Antenna Full-Duplex Wireless Communication

Full-duplex (FD) communication in many-antenna base stations (BSs) is hampered by self-interference (SI). This is because a FD node’s transmitting signal generates significant interference to its own receiver. Recent works have shown that it is possible to reduce/eliminate this SI in fully digital many-antenna systems, e.g., through transmit beamforming by using some spatial degrees of freedom to reduce SI instead of increasing the beamforming gain. On a parallel front, hybrid beamforming has recently emerged as a radio architecture that uses multiple antennas per FR chain. This can significantly reduce the cost of the end device (e.g., BS) but may also reduce the capacity or SI reduction gains of a fully digital radio system. This is because a fully digital radio architecture can change both the amplitude and phase of the wireless signal and send different data streams from each antenna element. Our goal in this paper is to quantify the performance gap between these two radio architectures in terms of SI cancellation and system capacity, particularly in multi-user MIMO setups. To do so, we experimentally compare the performance of a state-of-the-art fully digital many antenna FD solution to a hybrid beamforming architecture and compare the corresponding performance metrics leveraging a fully programmable many-antenna testbed and collecting over-the-air wireless channel data. We show that SI cancellation through beam design on a hybrid beamforming radio architecture can achieve capacity within 16% of that of a fully digital architecture. The performance gap further shrinks with a higher number of quantization bits in the hybrid beamforming system.

Does Teacher Training Narrow the Educational Gap between Urban and Rural Students? Empirical Evidence from CEPS Baseline Data

Based on the baseline data of the China Education Panel Survey, this paper explored the relationship between teacher training and academic performance in urban and rural samples respectively and the impact of teacher training on the urban-rural gap of students' academic performance. The results showed that: firstly, there was a significant urban-rural gap in academic performance, and the gap in high quantiles and language subjects were even larger. Secondly, the results of unconditional quantile regression showed that teacher training could improve the performance of urban students with different academic levels and rural students with intermediate or above academic levels, but it cannot improve the performance of rural students with lower academic levels. In addition, the overall effect of teacher training in urban areas is significantly higher than that in rural areas. Thirdly, different quantiles of Oaxaca-Blinder decomposition found that the endowment effect and the coefficient effect of teacher training were the important causes of the urban-rural performance gap, but the relative sizes of the two were different according to the different grades and different quantiles of performance distribution. Therefore, to increase the training opportunities and improve the training quality of rural teachers as well as enhance the resource conversion rate of rural students are of great practical significance for narrowing the urban-rural performance gap.