Combining Self-supervised Learning and Active Learning for Disfluency Detection

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

Photon-Driven Neural Reconstruction for Path Guiding

Although Monte Carlo path tracing is a simple and effective algorithm to synthesize photo-realistic images, it is often very slow to converge to noise-free results when involving complex global illumination. One of the most successful variance-reduction techniques is path guiding, which can learn better distributions for importance sampling to reduce pixel noise. However, previous methods require a large number of path samples to achieve reliable path guiding. We present a novel neural path guiding approach that can reconstruct high-quality sampling distributions for path guiding from a sparse set of samples, using an offline trained neural network. We leverage photons traced from light sources as the primary input for sampling density reconstruction, which is effective for challenging scenes with strong global illumination. To fully make use of our deep neural network, we partition the scene space into an adaptive hierarchical grid, in which we apply our network to reconstruct high-quality sampling distributions for any local region in the scene. This allows for effective path guiding for arbitrary path bounce at any location in path tracing. We demonstrate that our photon-driven neural path guiding approach can generalize to diverse testing scenes, often achieving better rendering results than previous path guiding approaches and opening up interesting future directions.

Alpha particle microdosimetry calculations using a shallow neural network

A shallow neural network was trained to accurately calculate the microdosimetric parameters, <z1> and <z1 2> (the first and second moments of the single-event specific energy spectra, respectively) for use in alpha-particle microdosimetry calculations. The regression network of four inputs and two outputs was created in MATLAB and trained on a data set consisting of both previously published microdosimetric data and recent Monte Carlo simulations. The input data consisted of the alpha-particle energies (3.97–8.78 MeV), cell nuclei radii (2–10 µm), cell radii (2.5–20 µm), and eight different source-target configurations. These configurations included both single cells in suspension and cells in geometric clusters. The mean square error (MSE) was used to measure the performance of the network. The sizes of the hidden layers were chosen to minimize MSE without overfitting. The final neural network consisted of two hidden layers with 13 and 20 nodes, respectively, each with tangential sigmoid transfer functions, and was trained on 1932 data points. The overall training/validation resulted in a MSE = 3.71×10-7. A separate testing data set included input values that were not seen by the trained network. The final test on 892 separate data points resulted in a MSE = 2.80×10-7. The 95th percentile testing data errors were within ±1.4% for <z1> outputs and ±2.8% for <z1 2> outputs, respectively. Cell survival was also predicted using actual vs. neural network generated microdosimetric moments and showed overall agreement within ±3.5%. In summary, this trained neural network can accurately produce microdosimetric parameters used for the study of alpha-particle emitters. The network can be exported and shared for tests on independent data sets and new calculations.

Architecture and Model of Neural Network Based Service for Choice of the Penetration Testing Tools

During penetration testing of web applications, different tools are actively used to relieve the tester from repeating monotonous operations. The difficulty of the choice is in the fact that there are tools with similar functionality, and it is hard to define which tool is best to choose for a particular case. In this paper, a solution of the problem with making a choice by creating a Web service that will use a neural network on the server side is proposed. The neural network is trained on data obtained from experts in the field of penetration testing. A trained neural network will be able to select tools in accordance with specified requirements. Examples of the operation of a neural network trained on a small sample of data are shown. The effect of the number of neural network learning epochs on the results of work is shown. An example of input data is given, in which the neural network could not select the tool due to insufficient data for training. The advantages of the method shown are the simplicity of implementation (the number of lines of code is used as a metric) and the possibility of using opinions about tools from various experts. The disadvantages include the search for data for training, the need for experimental selection of the parameters of the neural network and the possibility of situations where the neural network will not be able to select tool that meets the specified requirements.

Image-based taxonomic classification of bulk biodiversity samples using deep learning and domain adaptation

Complex bulk samples of invertebrates from biodiversity surveys present a great challenge for taxonomic identification, especially if obtained from unexplored ecosystems. High-throughput imaging combined with machine learning for rapid classification could overcome this bottleneck. Developing such procedures requires that taxonomic labels from an existing source data set are used for model training and prediction of an unknown target sample. Yet the feasibility of transfer learning for the classification of unknown samples remains to be tested. Here, we assess the efficiency of deep learning and domain transfer algorithms for family-level classification of below-ground bulk samples of Coleoptera from understudied forests of Cyprus. We trained neural network models with images from local surveys versus global databases of above-ground samples from tropical forests and evaluated how prediction accuracy was affected by: (a) the quality and resolution of images, (b) the size and complexity of the training set and (c) the transferability of identifications across very disparate source-target pairs that do not share any species or genera. Within-dataset classification accuracy reached 98% and depended on the number and quality of training images and on dataset complexity. The accuracy of between-datasets predictions was reduced to a maximum of 82% and depended greatly on the standardisation of the imaging procedure. When the source and target images were of similar quality and resolution, albeit from different faunas, the reduction of accuracy was minimal. Application of algorithms for domain adaptation significantly improved the prediction performance of models trained by non-standardised, low-quality images. Our findings demonstrate that existing databases can be used to train models and successfully classify images from unexplored biota, when the imaging conditions and classification algorithms are carefully considered. Also, our results provide guidelines for data acquisition and algorithmic development for high-throughput image-based biodiversity surveys.

Double Deep Recurrent Reinforcement Learning for Centralized Dynamic Multichannel Access

We consider the problem of dynamic multichannel access for transmission maximization in multiuser wireless communication networks. The objective is to find a multiuser strategy that maximizes global channel utilization with a low collision in a centralized manner without any prior knowledge. Obtaining an optimal solution for centralized dynamic multichannel access is an extremely difficult problem due to the large-state and large-action space. To tackle this problem, we develop a centralized dynamic multichannel access framework based on double deep recurrent Q-network. The centralized node first maps current state directly to channel assignment actions, which can overcome prohibitive computation compared with reinforcement learning. Then, the centralized node can be easy to select multiple channels by maximizing the sum of value functions based on a trained neural network. Finally, the proposed method avoids collisions between secondary users through centralized allocation policy.

Untangling the Galaxy. III. Photometric Search for Pre-main-sequence Stars with Deep Learning

A reliable census of pre-main-sequence stars with known ages is critical to our understanding of early stellar evolution, but historically there has been difficulty in separating such stars from the field. We present a trained neural network model, Sagitta, that relies on Gaia DR2 and 2 Micron All-Sky Survey photometry to identify pre-main-sequence stars and to derive their age estimates. Our model successfully recovers populations and stellar properties associated with known star-forming regions up to five kpc. Furthermore, it allows for a detailed look at the star-forming history of the solar neighborhood, particularly at age ranges to which we were not previously sensitive. In particular, we observe several bubbles in the distribution of stars, the most notable of which is a ring of stars associated with the Local Bubble, which may have common origins with Gould’s Belt.

MRlogP: Transfer Learning Enables Accurate logP Prediction Using Small Experimental Training Datasets

Small molecule lipophilicity is often included in generalized rules for medicinal chemistry. These rules aim to reduce time, effort, costs, and attrition rates in drug discovery, allowing the rejection or prioritization of compounds without the need for synthesis and testing. The availability of high quality, abundant training data for machine learning methods can be a major limiting factor in building effective property predictors. We utilize transfer learning techniques to get around this problem, first learning on a large amount of low accuracy predicted logP values before finally tuning our model using a small, accurate dataset of 244 druglike compounds to create MRlogP, a neural network-based predictor of logP capable of outperforming state of the art freely available logP prediction methods for druglike small molecules. MRlogP achieves an average root mean squared error of 0.988 and 0.715 against druglike molecules from Reaxys and PHYSPROP. We have made the trained neural network predictor and all associated code for descriptor generation freely available. In addition, MRlogP may be used online via a web interface.

Prediction of the fatigue strength of metals at low temperatures based on artificial intelligence

An intelligent system for predicting the fatigue strength of metals in a wide temperature range is developed using a specially trained neural network. The system makes it possible to predict the number of load cycles of a part to failure, as well as the start of formation and growth rate of fatigue cracks for different test conditions, including at low temperatures. Keywords: neural network, prediction of loading cycles, low temperatures, fatigue strength. [email protected]