(Invited) Analog Memory Fully Connected Networks for Deep Neural Network Accelerated Training

In this paper, the structure of a separable convolutional neural network that consists of an embedding layer, separable convolutional layers, convolutional layer and global average pooling is represented for binary and multiclass text classifications. The advantage of the proposed structure is the absence of multiple fully connected layers, which is used to increase the classification accuracy but raises the computational cost. The combination of low-cost separable convolutional layers and a convolutional layer is proposed to gain high accuracy and, simultaneously, to reduce the complexity of neural classifiers. Advantages are demonstrated at binary and multiclass classifications of written texts by means of the proposed networks under the sigmoid and Softmax activation functions in convolutional layer. At binary and multiclass classifications, the accuracy obtained by separable convolutional neural networks is higher in comparison with some investigated types of recurrent neural networks and fully connected networks.

Download Full-text

Challenging the Recognition of Facial Expression via Deep Learning

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.571-572.717 ◽

2014 ◽

Vol 571-572 ◽

pp. 717-720

Author(s):

De Kun Hu ◽

Yong Hong Liu ◽

Li Zhang ◽

Gui Duo Duan

Keyword(s):

Neural Network ◽

Facial Expression ◽

Gradient Descent ◽

Deep Neural Network ◽

Back Propagation ◽

Stochastic Gradient Descent ◽

Excellent Performance ◽

Proposed Model ◽

Input Layer ◽

Fully Connected

A deep Neural Network model was trained to classify the facial expression in unconstrained images, which comprises nine layers, including input layer, convolutional layer, pooling layer, fully connected layers and output layer. In order to optimize the model, rectified linear units for the nonlinear transformation, weights sharing for reducing the complexity, “mean” and “max” pooling for subsample, “dropout” for sparsity are applied in the forward processing. With large amounts of hard training faces, the model was trained via back propagation method with stochastic gradient descent. The results of shows the proposed model achieves excellent performance.

Download Full-text

Partial Label Learning Based on Fully Connected Deep Neural Network

International Journal of Circuits, Systems and Signal Processing ◽

10.46300/9106.2022.16.35 ◽

2022 ◽

Vol 16 ◽

pp. 287-297

Author(s):

Houjie Li ◽

Lei Wu ◽

Jianjun He ◽

Ruirui Zheng ◽

Yu Zhou ◽

...

Keyword(s):

Neural Network ◽

Process Model ◽

Deep Neural Network ◽

Learning Algorithm ◽

Learning Algorithms ◽

Ground Truth ◽

Training Sample ◽

Support Vector ◽

Partial Label Learning ◽

Fully Connected

The ambiguity of training samples in the partial label learning framework makes it difficult for us to develop learning algorithms and most of the existing algorithms are proposed based on the traditional shallow machine learn- ing models, such as decision tree, support vector machine, and Gaussian process model. Deep neu- ral networks have demonstrated excellent perfor- mance in many application fields, but currently it is rarely used for partial label learning frame- work. This study proposes a new partial label learning algorithm based on a fully connected deep neural network, in which the relationship between the candidate labels and the ground- truth label of each training sample is established by defining three new loss functions, and a regu- larization term is added to prevent overfitting. The experimental results on the controlled U- CI datasets and real-world partial label datasets reveal that the proposed algorithm can achieve higher classification accuracy than the state-of- the-art partial label learning algorithms.

Download Full-text

A Parametric Level Set Method for Topology Optimization based on Deep Neural Network (DNN)

Journal of Mechanical Design ◽

10.1115/1.4050105 ◽

2021 ◽

pp. 1-14

Author(s):

Hao Deng ◽

Albert C. To

Keyword(s):

Neural Network ◽

Topology Optimization ◽

Differential Equations ◽

Level Set Method ◽

Level Set ◽

Deep Neural Network ◽

Implicit Function ◽

Zero Level ◽

Fully Connected ◽

Parametric Level Set Method

Abstract This paper proposes a new parametric level set method for topology optimization based on Deep Neural Network (DNN). In this method, the fully connected deep neural network is incorporated into the conventional level set methods to construct an effective approach for structural topology optimization. The implicit function of level set is described by fully connected deep neural networks. A DNN-based level set optimization method is proposed, where the Hamilton-Jacobi partial differential equations (PDEs) are transformed into parametrized ordinary differential equations (ODEs). The zero-level set of implicit function is updated through updating the weights and biases of networks. The parametrized reinitialization is applied periodically to prevent the implicit function from being too steep or too flat in the vicinity of its zero-level set. The proposed method is implemented in the framework of minimum compliance, which is a well-known benchmark for topology optimization. In practice, designers desire to have multiple design options, where they can choose a better conceptual design base on their design experience. One of the major advantages of DNN-based level set method is capable to generate diverse and competitive designs with different network architectures. Several numerical examples are presented to verify the effectiveness of proposed DNN-based level set method.

Download Full-text

AN INTELLIGENT DENGUE HEMORRHAGIC FEVER SEVERITY LEVEL DETECTION BASED ON DEEP NEURAL NETWORK APPROACH

Jurnal Ilmu Komputer dan Informasi ◽

10.21609/jiki.v12i2.642 ◽

2019 ◽

Vol 12 (2) ◽

pp. 57

Author(s):

Dian Pratiwi ◽

Gatot Budi Santoso ◽

Leni Muslimah ◽

Raden Davin Rizki

Keyword(s):

Neural Network ◽

Dengue Hemorrhagic Fever ◽

Deep Neural Network ◽

Hemorrhagic Fever ◽

Training Data ◽

Network Approach ◽

Neural Network Approach ◽

Severity Level ◽

Fully Connected ◽

Learning Data

Dengue hemorrhagic fever is one of the most dangerous diseases which often leads to death for the sufferer due to delays or improper handling of the severity that has occurred. In determining that severity level, a specialist analyzes it from the symptoms and blood testing results. This research was developed to produce a system by applying Deep Neural Network approach that is able to give the same analytical ability as a doctor, so that it can give fast and precise decision of dengue handling. The research stages consisted of normalizing data to 0 – 1 intervals by Min-Max method, training data into multilayer networks with fully connected and partially connected schemes to produce the best weights, validating data and final testing. From the use of network parameters as much as 10 input units, 1 bias, 2 hidden layers, 2 output units, learning rate of 0.3, epoch 1000, tolerance rate 0.02, threshold 0.5, the system succeeded in generating a maximum accuracy of 95% in data learning (60 data), 87.5% on data learning and non-learning (40 data), 85% on non-learning data (20 data).

Download Full-text

A Fully Connected Deep Neural Network approach with multiple sub-frame consideration and phase recompense for noise suppression

10.36227/techrxiv.15156324.v1 ◽

2021 ◽

Author(s):

Rohun Nisa

Keyword(s):

Neural Network ◽

Mean Square Error ◽

Deep Neural Network ◽

Speech Quality ◽

Hybrid Network ◽

Speech Communication ◽

Mean Square ◽

Perceptual Evaluation ◽

Speech Spectrum ◽

Fully Connected

In the speech communication process, the desirable speech needs to be addressed under the influence of noise encountered in diverse environments that degrade the speech quality and intelligibility. In opposition to the unfavorable scenario particularly lowered signal-to-noiseratio, the progress of traditional noise suppressive algorithms is hindered, introducing further distortion in speech, making them non-applicable for real-time applications. In order to reduce the complicacies of current algorithms, a hybrid approach for upgrading the quality together with intelligibility of speech is proposed for dealing with real-world hearing scenario. For improving the intelligibility of speech of interest, multiple sub-frame analysis using over-spectral subtractive factor with phase recompense approach is implemented on the multi-channel noise corrupted speech, yielding approximated speech spectrum, that constitutes the pre-processing stage. The approximated speech spectrum and clean speech spectrum forming the training set are further fed to Fully Connected Layered Deep Neural Network to reduce the mean square error with the incorporation of regression network resulting in improved quality for speech. The proposed hybrid network results in upgraded intelligibility and quality in speech signal with improved SNR measured in terms of Short-Time-Objective-Intelligibility (STOI) score, Perceptual-Evaluation-of-Speech-Quality (PESQ) score, Segmental SNR level, and Mean Square Error (MSE) in contrast to prior noise suppressive algorithms together with less complexity of the hybrid network.<br>

Download Full-text

A robust deep neural network for denoising task-based fMRI data: An application to working memory and episodic memory

10.1101/746313 ◽

2019 ◽

Cited By ~ 1

Author(s):

Zhengshi Yang ◽

Xiaowei Zhuang ◽

Karthik Sreenivasan ◽

Virendra Mishra ◽

Tim Curran ◽

...

Keyword(s):

Neural Network ◽

Working Memory ◽

Episodic Memory ◽

Gray Matter ◽

Deep Neural Network ◽

Memory Task ◽

Fmri Data ◽

Elderly Subjects ◽

Fully Connected ◽

Time Point

ABSTRACTIn this study, a deep neural network (DNN) is proposed to reduce the noise in task-based fMRI data without explicitly modeling noise. The DNN artificial neural network consists of one temporal convolutional layer, one long short-term memory (LSTM) layer, one time-distributed fully-connected layer, and one unconventional selection layer in sequential order. The LSTM layer takes not only the current time point but also what was perceived in a previous time point as its input to characterize the temporal autocorrelation of fMRI data. The fully-connected layer weights the output of the LSTM layer, and the output denoised fMRI time series is selected by the selection layer. Assuming that task-related neural response is limited to gray matter, the model parameters in the DNN network are optimized by maximizing the correlation difference between gray matter voxels and white matter or ventricular cerebrospinal fluid voxels. Instead of targeting a particular noise source, the proposed neural network takes advantage of the task design matrix to better extract task-related signal in fMRI data. The DNN network, along with other traditional denoising techniques, has been applied on simulated data, working memory task fMRI data acquired from a cohort of healthy subjects and episodic memory task fMRI data acquired from a small set of healthy elderly subjects. Qualitative and quantitative measurements were used to evaluate the performance of different denoising techniques. In the simulation, DNN improves fMRI activation detection and also adapts to varying hemodynamic response functions across different brain regions. DNN efficiently reduces physiological noise and generates more homogeneous task-response correlation maps in real data.

Download Full-text

Coarse Return Prediction in a Cement Industry’s Closed Grinding Circuit System through a Fully Connected Deep Neural Network (FCDNN) Model

Applied Sciences ◽

10.3390/app11041361 ◽

2021 ◽

Vol 11 (4) ◽

pp. 1361

Author(s):

Morad Danishvar ◽

Sebelan Danishvar ◽

Francisco Souza ◽

Pedro Sousa ◽

Alireza Mousavi

Keyword(s):

Neural Network ◽

Control Strategy ◽

Deep Neural Network ◽

Milling Process ◽

Event Modeling ◽

Look Ahead ◽

Milling Operations ◽

Corrective Actions ◽

Fully Connected ◽

The Impact

Milling operations in various production processes are among the most important factors in determining the quality, stability, and consumption of energy. Optimizing and stabilizing the milling process is a non-linear multivariable control problem. In specific processes that deal with natural materials (e.g., cement, pulp and paper, beverage brewery and water/wastewater treatment industries). A novel data-driven approach utilizing real-time monitoring control technology is proposed for the purpose of optimizing the grinding of cement processing. A combined event modeling for feature extraction and the fully connected deep neural network model to predict the coarseness of cement particles is proposed. The resulting prediction allows a look ahead control strategy and corrective actions. The proposed solution has been deployed in a number of cement plants around the world. The resultant control strategy has enabled the operators to take corrective actions before the coarse return increases, both in autonomous and manual mode. The impact of the solution has improved efficiency resource use by 10% of resources, the plant stability, and the overall energy efficiency of the plant.

Download Full-text

A Novel Deep Neural Network for Robust Detection of Seizures Using EEG Signals

Computational and Mathematical Methods in Medicine ◽

10.1155/2020/9689821 ◽

2020 ◽

Vol 2020 ◽

pp. 1-9 ◽

Cited By ~ 1

Author(s):

Wei Zhao ◽

Wenbing Zhao ◽

Wenfeng Wang ◽

Xiaolu Jiang ◽

Xiaodong Zhang ◽

...

Keyword(s):

Neural Network ◽

Deep Neural Network ◽

Model Performance ◽

Classification Problem ◽

Automatic Identification ◽

Robust Detection ◽

Heavy Burden ◽

The University ◽

Fully Connected

The detection of recorded epileptic seizure activity in electroencephalogram (EEG) segments is crucial for the classification of seizures. Manual recognition is a time-consuming and laborious process that places a heavy burden on neurologists, and hence, the automatic identification of epilepsy has become an important issue. Traditional EEG recognition models largely depend on artificial experience and are of weak generalization ability. To break these limitations, we propose a novel one-dimensional deep neural network for robust detection of seizures, which composes of three convolutional blocks and three fully connected layers. Thereinto, each convolutional block consists of five types of layers: convolutional layer, batch normalization layer, nonlinear activation layer, dropout layer, and max-pooling layer. Model performance is evaluated on the University of Bonn dataset, which achieves the accuracy of 97.63%∼99.52% in the two-class classification problem, 96.73%∼98.06% in the three-class EEG classification problem, and 93.55% in classifying the complicated five-class problem.

Download Full-text

Automated detection of mouse scratching behaviour using convolutional recurrent neural network

Scientific Reports ◽

10.1038/s41598-020-79965-w ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Koji Kobayashi ◽

Seiji Matsushita ◽

Naoyuki Shimizu ◽

Sakura Masuko ◽

Masahito Yamamoto ◽

...

Keyword(s):

Neural Network ◽

Recurrent Neural Network ◽

Lysophosphatidic Acid ◽

Deep Neural Network ◽

Detection Method ◽

Automated Detection ◽

Experimental Animals ◽

Novel Method ◽

Predictive Rate ◽

Fully Connected

AbstractScratching is one of the most important behaviours in experimental animals because it can reflect itching and/or psychological stress. Here, we aimed to establish a novel method to detect scratching using deep neural network. Scratching was elicited by injecting a chemical pruritogen lysophosphatidic acid to the back of a mouse, and behaviour was recorded using a standard handy camera. Images showing differences between two consecutive frames in each video were generated, and each frame was manually labelled as showing scratching behaviour or not. Next, a convolutional recurrent neural network (CRNN), composed of sequential convolution, recurrent, and fully connected blocks, was constructed. The CRNN was trained using the manually labelled images and then evaluated for accuracy using a first-look dataset. Sensitivity and positive predictive rates reached 81.6% and 87.9%, respectively. The predicted number and durations of scratching events correlated with those of the human observation. The trained CRNN could also successfully detect scratching in the hapten-induced atopic dermatitis mouse model (sensitivity, 94.8%; positive predictive rate, 82.1%). In conclusion, we established a novel scratching detection method using CRNN and showed that it can be used to study disease models.

Download Full-text