A HYBRID MODEL USING THE PRETRAINED BERT AND DEEP NEURAL NETWORKS WITH RICH FEATURE FOR EXTRACTIVE TEXT SUMMARIZATION

Deep neural networks have been applied successfully to extractive text summarization tasks with the accompany of large training datasets. However, when the training dataset is not large enough, these models reveal certain limitations that affect the quality of the system’s summary. In this paper, we propose an extractive summarization system basing on a Convolutional Neural Network and a Fully Connected network for sentence selection. The pretrained BERT multilingual model is used to generate embeddings vectors from the input text. These vectors are combined with TF-IDF values to produce the input of the text summarization system. Redundant sentences from the output summary are eliminated by the Maximal Marginal Relevance method. Our system is evaluated with both English and Vietnamese languages using CNN and Baomoi datasets, respectively. Experimental results show that our system achieves better results comparing to existing works using the same dataset. It confirms that our approach can be effectively applied to summarize both English and Vietnamese languages.

Download Full-text

Evaluation of Power Insulator Detection Efficiency with the Use of Limited Training Dataset

Applied Sciences ◽

10.3390/app10062104 ◽

2020 ◽

Vol 10 (6) ◽

pp. 2104

Author(s):

Michał Tomaszewski ◽

Paweł Michalski ◽

Jakub Osuchowski

Keyword(s):

Neural Network ◽

Neural Networks ◽

Object Detection ◽

Convolutional Neural Network ◽

Deep Neural Networks ◽

Detection Efficiency ◽

Training Data ◽

Training Dataset ◽

Training Set ◽

Convolutional Network

This article presents an analysis of the effectiveness of object detection in digital images with the application of a limited quantity of input. The possibility of using a limited set of learning data was achieved by developing a detailed scenario of the task, which strictly defined the conditions of detector operation in the considered case of a convolutional neural network. The described solution utilizes known architectures of deep neural networks in the process of learning and object detection. The article presents comparisons of results from detecting the most popular deep neural networks while maintaining a limited training set composed of a specific number of selected images from diagnostic video. The analyzed input material was recorded during an inspection flight conducted along high-voltage lines. The object detector was built for a power insulator. The main contribution of the presented papier is the evidence that a limited training set (in our case, just 60 training frames) could be used for object detection, assuming an outdoor scenario with low variability of environmental conditions. The decision of which network will generate the best result for such a limited training set is not a trivial task. Conducted research suggests that the deep neural networks will achieve different levels of effectiveness depending on the amount of training data. The most beneficial results were obtained for two convolutional neural networks: the faster region-convolutional neural network (faster R-CNN) and the region-based fully convolutional network (R-FCN). Faster R-CNN reached the highest AP (average precision) at a level of 0.8 for 60 frames. The R-FCN model gained a worse AP result; however, it can be noted that the relationship between the number of input samples and the obtained results has a significantly lower influence than in the case of other CNN models, which, in the authors’ assessment, is a desired feature in the case of a limited training set.

Download Full-text

Automatic Text Summarization of COVID-19 Research Articles Using Recurrent Neural Networks and Coreference Resolution

Frontiers in Biomedical Technologies ◽

10.18502/fbt.v7i4.5321 ◽

2021 ◽

Author(s):

Mahsa Afsharizadeh ◽

Hossein Ebrahimpour-Komleh ◽

Ayoub Bagheri

Keyword(s):

Neural Network ◽

Recurrent Neural Network ◽

Text Summarization ◽

Medical Community ◽

Research Articles ◽

Coreference Resolution ◽

Extractive Summarization ◽

Automatic Text Summarization ◽

Summarization System ◽

Automatic Text

Purpose: Pandemic COVID-19 has created an emergency for the medical community. Researchers require extensive study of scientific literature in order to discover drugs and vaccines. In this situation where every minute is valuable to save the lives of hundreds of people, a quick understanding of scientific articles will help the medical community. Automatic text summarization makes this possible. Materials and Methods: In this study, a recurrent neural network-based extractive summarization is proposed. The extractive method identifies the informative parts of the text. Recurrent neural network is very powerful for analyzing sequences such as text. The proposed method has three phases: sentence encoding, sentence ranking, and summary generation. To improve the performance of the summarization system, a coreference resolution procedure is used. Coreference resolution identifies the mentions in the text that refer to the same entity in the real world. This procedure helps to summarization process by discovering the central subject of the text. Results: The proposed method is evaluated on the COVID-19 research articles extracted from the CORD-19 dataset. The results show that the combination of using recurrent neural network and coreference resolution embedding vectors improves the performance of the summarization system. The Proposed method by achieving the value of ROUGE1-recall 0.53 demonstrates the improvement of summarization performance by using coreference resolution embedding vectors in the RNN-based summarization system. Conclusion: In this study, coreference information is stored in the form of coreference embedding vectors. Jointly use of recurrent neural network and coreference resolution results in an efficient summarization system.

Download Full-text

Bias in Deep Neural Networks in Land Use Characterization for International Development

Remote Sensing ◽

10.3390/rs13152908 ◽

2021 ◽

Vol 13 (15) ◽

pp. 2908

Author(s):

Do-Hyung Kim ◽

Guzmán López ◽

Diego Kiedanski ◽

Iyke Maduako ◽

Braulio Ríos ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Land Use ◽

International Development ◽

Deep Neural Networks ◽

Satellite Image ◽

Fine Tuning ◽

Training Dataset ◽

Model Accuracy ◽

The Neural Network

Understanding the biases in Deep Neural Networks (DNN) based algorithms is gaining paramount importance due to its increased applications on many real-world problems. A known problem of DNN penalizing the underrepresented population could undermine the efficacy of development projects dependent on data produced using DNN-based models. In spite of this, the problems of biases in DNN for Land Use and Land Cover Classification (LULCC) have not been a subject of many studies. In this study, we explore ways to quantify biases in DNN for land use with an example of identifying school buildings in Colombia from satellite imagery. We implement a DNN-based model by fine-tuning an existing, pre-trained model for school building identification. The model achieved overall 84% accuracy. Then, we used socioeconomic covariates to analyze possible biases in the learned representation. The retrained deep neural network was used to extract visual features (embeddings) from satellite image tiles. The embeddings were clustered into four subtypes of schools, and the accuracy of the neural network model was assessed for each cluster. The distributions of various socioeconomic covariates by clusters were analyzed to identify the links between the model accuracy and the aforementioned covariates. Our results indicate that the model accuracy is lowest (57%) where the characteristics of the landscape are predominantly related to poverty and remoteness, which confirms our original assumption on the heterogeneous performances of Artificial Intelligence (AI) algorithms and their biases. Based on our findings, we identify possible sources of bias and present suggestions on how to prepare a balanced training dataset that would result in less biased AI algorithms. The framework used in our study to better understand biases in DNN models would be useful when Machine Learning (ML) techniques are adopted in lieu of ground-based data collection for international development programs. Because such programs aim to solve issues of social inequality, MLs are only applicable when they are transparent and accountable.

Download Full-text

PIPxResNet: Penalty Induced Prototype-Based eXplainable Residual Neural Network for Heartbeat Classification

10.21203/rs.3.rs-852812/v1 ◽

2021 ◽

Author(s):

Deepankar Nankani ◽

Rashmi Dutta Baruah

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Neural Networks ◽

Early Stage ◽

Black Box ◽

Training Dataset ◽

Medical Attention ◽

Generative Adversarial Network ◽

Heartbeat Classification ◽

General Physicians

Abstract Early stage heartbeat classification using the electrocardiogram signals can prevent cardiovascular diseases that causes millions of deaths annually around the world. In the past, researchers have used deep neural networks to achieve significant performance for heartbeat classification but their black-box nature and prediction rationale limits real-world deployment. We propose a Penalty Induced Prototype based eXplainable Residual Neural Network (PIPxResNet) that addresses the black-box nature of deep neural networks. PIPxResNet encodes the temporal variations of heartbeats by employing pretrained residual neural network following the concept of task transfer learning. The algorithm further extracts prototypes that are most representative of the training dataset that explain model predictions to general physicians, making them clinically relevant. The prototypes of a particular class having close resemblance to other class prototypes are penalised and their contribution towards corresponding class is reduced. In addition, the classification performance is improved by synthesising regular and irregular heartbeats using a deep convolution conditional generative adversarial network. The proposed method can easily be adopted to other domains that requires explanations for the classification tasks. The PIPxResNet performs at par with existing state-of-the-art algorithms without compromising individual class performance when tested on four publicly available annotated datasets. The proposed model is capable to perform automated screening and provide medical attention by simulating a clinical decision support system for general physicians.

Download Full-text

Memristor crossbar architectures for implementing deep neural networks

Complex & Intelligent Systems ◽

10.1007/s40747-021-00282-4 ◽

2021 ◽

Author(s):

Xiaoyang Liu ◽

Zhigang Zeng

Keyword(s):

Neural Network ◽

Neural Networks ◽

Pattern Recognition ◽

Deep Neural Networks ◽

Activation Functions ◽

Simulation Results ◽

Memristor Crossbars ◽

Memristor Crossbar ◽

Fully Connected

AbstractThe paper presents memristor crossbar architectures for implementing layers in deep neural networks, including the fully connected layer, the convolutional layer, and the pooling layer. The crossbars achieve positive and negative weight values and approximately realize various nonlinear activation functions. Then the layers constructed by the crossbars are adopted to build the memristor-based multi-layer neural network (MMNN) and the memristor-based convolutional neural network (MCNN). Two kinds of in-situ weight update schemes, which are the fixed-voltage update and the approximately linear update, respectively, are used to train the networks. Consider variations resulted from the inherent characteristics of memristors and the errors of programming voltages, the robustness of MMNN and MCNN to these variations is analyzed. The simulation results on standard datasets show that deep neural networks (DNNs) built by the memristor crossbars work satisfactorily in pattern recognition tasks and have certain robustness to memristor variations.

Download Full-text

Quantization of Deep Neural Networks for Accurate Edge Computing

ACM Journal on Emerging Technologies in Computing Systems ◽

10.1145/3451211 ◽

2021 ◽

Vol 17 (4) ◽

pp. 1-11

Author(s):

Wentao Chen ◽

Hailong Qiu ◽

Jian Zhuang ◽

Chutong Zhang ◽

Yu Hu ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Image Segmentation ◽

Automatic Speech Recognition ◽

Deep Neural Networks ◽

Biomedical Image ◽

Wide Range ◽

Improve Accuracy ◽

Memory Reduction ◽

Fully Connected

Deep neural networks have demonstrated their great potential in recent years, exceeding the performance of human experts in a wide range of applications. Due to their large sizes, however, compression techniques such as weight quantization and pruning are usually applied before they can be accommodated on the edge. It is generally believed that quantization leads to performance degradation, and plenty of existing works have explored quantization strategies aiming at minimum accuracy loss. In this paper, we argue that quantization, which essentially imposes regularization on weight representations, can sometimes help to improve accuracy. We conduct comprehensive experiments on three widely used applications: fully connected network for biomedical image segmentation, convolutional neural network for image classification on ImageNet, and recurrent neural network for automatic speech recognition, and experimental results show that quantization can improve the accuracy by 1%, 1.95%, 4.23% on the three applications respectively with 3.5x-6.4x memory reduction.

Download Full-text

Deep distributed convolutional neural networks: Universality

Analysis and Applications ◽

10.1142/s0219530518500124 ◽

2018 ◽

Vol 16 (06) ◽

pp. 895-919 ◽

Cited By ~ 19

Author(s):

Ding-Xuan Zhou

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Learning ◽

Convolutional Neural Networks ◽

Deep Neural Network ◽

Deep Neural Networks ◽

Mathematical Foundation ◽

Deep Convolutional Neural Networks ◽

The Family ◽

Fully Connected

Deep learning based on structured deep neural networks has provided powerful applications in various fields. The structures imposed on the deep neural networks are crucial, which makes deep learning essentially different from classical schemes based on fully connected neural networks. One of the commonly used deep neural network structures is generated by convolutions. The produced deep learning algorithms form the family of deep convolutional neural networks. Despite of their power in some practical domains, little is known about the mathematical foundation of deep convolutional neural networks such as universality of approximation. In this paper, we propose a family of new structured deep neural networks: deep distributed convolutional neural networks. We show that these deep neural networks have the same order of computational complexity as the deep convolutional neural networks, and we prove their universality of approximation. Some ideas of our analysis are from ridge approximation, wavelets, and learning theory.

Download Full-text

Using Summary Layers to Probe Neural Network Behaviour

South African Computer Journal ◽

10.18489/sacj.v32i2.861 ◽

2020 ◽

Vol 32 (2) ◽

Author(s):

Marelie Hattingh Davel

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Neural Networks ◽

Network Architectures ◽

Feedforward Networks ◽

Empirical Results ◽

Analysis Process ◽

Generalisation Ability ◽

Neural Network Architectures ◽

Fully Connected

No framework exists that can explain and predict the generalisation ability of deep neural networks in general circumstances. In fact, this question has not been answered for some of the least complicated of neural network architectures: fully-connected feedforward networks with rectified linear activations and a limited number of hidden layers. For such an architecture, we show how adding a summary layer to the network makes it more amenable to analysis, and allows us to define the conditions that are required to guarantee that a set of samples will all be classified correctly. This process does not describe the generalisation behaviour of these networks, but produces a number of metrics that are useful for probing their learning and generalisation behaviour. We support the analytical conclusions with empirical results, both to confirm that the mathematical guarantees hold in practice, and to demonstrate the use of the analysis process.

Download Full-text

Physics Inspired Deep Neural Networks for Top Quark Reconstruction

EPJ Web of Conferences ◽

10.1051/epjconf/202024506029 ◽

2020 ◽

Vol 245 ◽

pp. 06029

Author(s):

Kevin Greif ◽

Kevin Lannon

Keyword(s):

Neural Network ◽

Neural Networks ◽

Computer Vision ◽

Natural Language Processing ◽

Natural Language ◽

Language Processing ◽

Top Quark ◽

Deep Neural Networks ◽

Great Success ◽

Fully Connected

Deep neural networks (DNNs) have been applied to the fields of computer vision and natural language processing with great success in recent years. The success of these applications has hinged on the development of specialized DNN architectures that take advantage of specific characteristics of the problem to be solved, namely convolutional neural networks for computer vision and recurrent neural networks for natural language processing. This research explores whether a neural network architecture specific to the task of identifying t → Wb decays in particle collision data yields better performance than a generic, fully-connected DNN. Although applied here to resolved top quark decays, this approach is inspired by an DNN technique for tagging boosted top quarks, which consists of defining custom neural network layers known as the combination and Lorentz layers. These layers encode knowledge of relativistic kinematics applied to combinations of particles, and the output of these specialized layers can then be fed into a fully connected neural network to learn tasks such as classification. This research compares the performance of these physics inspired networks to that of a generic, fully-connected DNN, to see if there is any advantage in terms of classification performance, size of the network, or ease of training.

Download Full-text

Deep neural networks using a single neuron: folded-in-time architecture using feedback-modulated delay loops

Nature Communications ◽

10.1038/s41467-021-25427-4 ◽

2021 ◽

Vol 12 (1) ◽

Author(s):

Florian Stelzer ◽

André Röhm ◽

Raul Vicente ◽

Ingo Fischer ◽

Serhiy Yanchuk

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Neural Network ◽

Single Neuron ◽

Deep Neural Networks ◽

Back Propagation ◽

Local Network ◽

Multiple Time ◽

Learning Tools ◽

Back Propagation Algorithm

AbstractDeep neural networks are among the most widely applied machine learning tools showing outstanding performance in a broad range of tasks. We present a method for folding a deep neural network of arbitrary size into a single neuron with multiple time-delayed feedback loops. This single-neuron deep neural network comprises only a single nonlinearity and appropriately adjusted modulations of the feedback signals. The network states emerge in time as a temporal unfolding of the neuron’s dynamics. By adjusting the feedback-modulation within the loops, we adapt the network’s connection weights. These connection weights are determined via a back-propagation algorithm, where both the delay-induced and local network connections must be taken into account. Our approach can fully represent standard Deep Neural Networks (DNN), encompasses sparse DNNs, and extends the DNN concept toward dynamical systems implementations. The new method, which we call Folded-in-time DNN (Fit-DNN), exhibits promising performance in a set of benchmark tasks.

Download Full-text