scholarly journals DRHNet: A Deep Residual Network Based on Heterogeneous Kernel for Steganalysis

2020 ◽  
Vol 2020 ◽  
pp. 1-9
Author(s):  
Yang Xu ◽  
Zixi Fu ◽  
Guiyong Xu ◽  
Sicong Zhang ◽  
Xiaoyao Xie
Keyword(s):  
Neural Networks ◽  
Heterogeneous Network ◽  
State Of The Art ◽  
Training Phase ◽  
Image Size ◽  
Residual Network ◽  
Training Time ◽  
Learning Framework ◽  
Residual Learning ◽  
The Rich

Convolutional neural networks as steganalysis have problems such as poor versatility, long training time, and limited image size. For these problems, we present a heterogeneous kernel residual learning framework called DRHNet—Dual Residual Heterogeneous Network—to save time on the networks during the training phase. Instead of using the image as an input of the network, we extract and merge the images into a feature matrix using the rich model and use the generated feature matrix as the real input of the network. The architecture we proposed has good versatility and can reduce the computation and the number of parameters while still getting higher accuracy. On BOSSbase 1.01, we evaluate the performance of DRHNet in the setting of the spatial domain and frequency domain. The preliminary experimental results show that DRHNet shows excellent steganalysis performance against the state-of-the-art steganographic algorithms.

scholarly journals Soft Threshold Ternary Networks

2020 ◽  
Author(s):  
Weixiang Xu ◽  
Xiangyu He ◽  
Tianli Zhao ◽  
Qinghao Hu ◽  
Peisong Wang ◽  
...  
Keyword(s):  
Neural Networks ◽  
Mobile Devices ◽  
State Of The Art ◽  
Performance Gap ◽  
Training Time ◽  
Current State ◽  
The Arts ◽  
Soft Threshold ◽  
And Storage ◽  
Selection Of

Large neural networks are difficult to deploy on mobile devices because of intensive computation and storage. To alleviate it, we study ternarization, a balance between efficiency and accuracy that quantizes both weights and activations into ternary values. In previous ternarized neural networks, a hard threshold Δ is introduced to determine quantization intervals. Although the selection of Δ greatly affects the training results, previous works estimate Δ via an approximation or treat it as a hyper-parameter, which is suboptimal. In this paper, we present the Soft Threshold Ternary Networks (STTN), which enables the model to automatically determine quantization intervals instead of depending on a hard threshold. Concretely, we replace the original ternary kernel with the addition of two binary kernels at training time, where ternary values are determined by the combination of two corresponding binary values. At inference time, we add up the two binary kernels to obtain a single ternary kernel. Our method dramatically outperforms current state-of-the-arts, lowering the performance gap between full-precision networks and extreme low bit networks. Experiments on ImageNet with AlexNet (Top-1 55.6%), ResNet-18 (Top-1 66.2%) achieves new state-of-the-art.

scholarly journals Gaussian Transformer: A Lightweight Approach for Natural Language Inference

2019 ◽  
Vol 33 ◽  
pp. 6489-6496 ◽  
Author(s):  
Maosheng Guo ◽  
Yu Zhang ◽  
Ting Liu
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
State Of The Art ◽  
Research Area ◽  
High Order Interaction ◽  
Training Time ◽  
Attention Networks ◽  
Local Dependency ◽  
Active Research ◽  
Active Research Area

Natural Language Inference (NLI) is an active research area, where numerous approaches based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), and self-attention networks (SANs) has been proposed. Although obtaining impressive performance, previous recurrent approaches are hard to train in parallel; convolutional models tend to cost more parameters, while self-attention networks are not good at capturing local dependency of texts. To address this problem, we introduce a Gaussian prior to selfattention mechanism, for better modeling the local structure of sentences. Then we propose an efficient RNN/CNN-free architecture named Gaussian Transformer for NLI, which consists of encoding blocks modeling both local and global dependency, high-order interaction blocks collecting the evidence of multi-step inference, and a lightweight comparison block saving lots of parameters. Experiments show that our model achieves new state-of-the-art performance on both SNLI and MultiNLI benchmarks with significantly fewer parameters and considerably less training time. Besides, evaluation using the Hard NLI datasets demonstrates that our approach is less affected by the undesirable annotation artifacts.

scholarly journals FEW-GROUP CROSS SECTIONS MODELING BY ARTIFICIAL NEURAL NETWORKS

2021 ◽  
Vol 247 ◽  
pp. 06029
Author(s):  
E. Szames ◽  
K. Ammar ◽  
D. Tomatis ◽  
J.M. Martinez
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Cross Sections ◽  
Linear Interpolation ◽  
Sensitivity Study ◽  
Library Size ◽  
Training Time ◽  
Learning Framework ◽  
Artificial Neural ◽  
And Training

This work deals with the modeling of homogenized few-group cross sections by Artificial Neural Networks (ANN). A comprehensive sensitivity study on data normalization, network architectures and training hyper-parameters specifically for Deep and Shallow Feed Forward ANN is presented. The optimal models in terms of reduction in the library size and training time are compared to multi-linear interpolation on a Cartesian grid. The use case is provided by the OECD-NEA Burn-up Credit Criticality Benchmark [1]. The Pytorch [2] machine learning framework is used.

scholarly journals Learning from Web Data Using Adversarial Discriminative Neural Networks for Fine-Grained Classification

2019 ◽  
Vol 33 ◽  
pp. 273-280 ◽  
Author(s):  
Xiaoxiao Sun ◽  
Liyi Chen ◽  
Jufeng Yang
Keyword(s):  
Neural Networks ◽  
Large Scale ◽  
State Of The Art ◽  
Training Data ◽  
Web Data ◽  
Fine Grained ◽  
Learning Framework ◽  
Attractive Option ◽  
Public Datasets ◽  
Noisy Labels

Fine-grained classification is absorbed in recognizing the subordinate categories of one field, which need a large number of labeled images, while it is expensive to label these images. Utilizing web data has been an attractive option to meet the demands of training data for convolutional neural networks (CNNs), especially when the well-labeled data is not enough. However, directly training on such easily obtained images often leads to unsatisfactory performance due to factors such as noisy labels. This has been conventionally addressed by reducing the noise level of web data. In this paper, we take a fundamentally different view and propose an adversarial discriminative loss to advocate representation coherence between standard and web data. This is further encapsulated in a simple, scalable and end-to-end trainable multi-task learning framework. We experiment on three public datasets using large-scale web data to evaluate the effectiveness and generalizability of the proposed approach. Extensive experiments demonstrate that our approach performs favorably against the state-of-the-art methods.

scholarly journals AAR-CNNs: Auto Adaptive Regularized Convolutional Neural Networks

2018 ◽  
Author(s):  
Yao Lu ◽  
Guangming Lu ◽  
Yuanrong Xu ◽  
Bob Zhang
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
State Of The Art ◽  
Experimental Results ◽  
Training Phase ◽  
Low Resolution ◽  
Adaptive Regularization ◽  
End To End ◽  
Overfitting Problem

In order to address the overfitting problem caused by the small or simple training datasets and the large model’s size in Convolutional Neural Networks (CNNs), a novel Auto Adaptive Regularization (AAR) method is proposed in this paper. The relevant networks can be called AAR-CNNs. AAR is the first method using the “abstraction extent” (predicted by AE net) and a tiny learnable module (SE net) to auto adaptively predict more accurate and individualized regularization information. The AAR module can be directly inserted into every stage of any popular networks and trained end to end to improve the networks’ flexibility. This method can not only regularize the network at both the forward and the backward processes in the training phase, but also regularize the network on a more refined level (channel or pixel level) depending on the abstraction extent’s form. Comparative experiments are performed on low resolution ImageNet, CIFAR and SVHN datasets. Experimental results show that the AAR-CNNs can achieve state-of-the-art performances on these datasets.

scholarly journals Effective Plug-Ins for Reducing Inference-Latency of Spiking Convolutional Neural Networks During Inference Phase

2021 ◽  
Vol 15 ◽  
Author(s):  
Xuan Chen ◽  
Xiaopeng Yuan ◽  
Gaoming Fu ◽  
Yuanyong Luo ◽  
Tao Yue ◽  
...  
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
State Of The Art ◽  
Low Cost ◽  
Training Phase ◽  
Working Mechanism ◽  
Simulation Results ◽  
New Perspective ◽  
Fully Connected

Convolutional Neural Networks (CNNs) are effective and mature in the field of classification, while Spiking Neural Networks (SNNs) are energy-saving for their sparsity of data flow and event-driven working mechanism. Previous work demonstrated that CNNs can be converted into equivalent Spiking Convolutional Neural Networks (SCNNs) without obvious accuracy loss, including different functional layers such as Convolutional (Conv), Fully Connected (FC), Avg-pooling, Max-pooling, and Batch-Normalization (BN) layers. To reduce inference-latency, existing researches mainly concentrated on the normalization of weights to increase the firing rate of neurons. There are also some approaches during training phase or altering the network architecture. However, little attention has been paid on the end of inference phase. From this new perspective, this paper presents 4 stopping criterions as low-cost plug-ins to reduce the inference-latency of SCNNs. The proposed methods are validated using MATLAB and PyTorch platforms with Spiking-AlexNet for CIFAR-10 dataset and Spiking-LeNet-5 for MNIST dataset. Simulation results reveal that, compared to the state-of-the-art methods, the proposed method can shorten the average inference-latency of Spiking-AlexNet from 892 to 267 time steps (almost 3.34 times faster) with the accuracy decline from 87.95 to 87.72%. With our methods, 4 types of Spiking-LeNet-5 only need 24–70 time steps per image with the accuracy decline not more than 0.1%, while models without our methods require 52–138 time steps, almost 1.92 to 3.21 times slower than us.

scholarly journals HalluciNet-ing Spatiotemporal Representations Using a 2D-CNN

Signals ◽  
2021 ◽  
Vol 2 (3) ◽  
pp. 604-618
Author(s):  
Paritosh Parmar ◽  
Brendan Morris
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Scene Recognition ◽  
Multitask Learning ◽  
Training Phase ◽  
Dynamic Scene ◽  
Improve Performance ◽  
Computing Power ◽  
Practical Standpoint ◽  
3D Cnn

Spatiotemporal representations learned using 3D convolutional neural networks (CNN) are currently used in state-of-the-art approaches for action-related tasks. However, 3D-CNN are notorious for being memory and compute resource intensive as compared with more simple 2D-CNN architectures. We propose to hallucinate spatiotemporal representations from a 3D-CNN teacher with a 2D-CNN student. By requiring the 2D-CNN to predict the future and intuit upcoming activity, it is encouraged to gain a deeper understanding of actions and how they evolve. The hallucination task is treated as an auxiliary task, which can be used with any other action-related task in a multitask learning setting. Thorough experimental evaluation, it is shown that the hallucination task indeed helps improve performance on action recognition, action quality assessment, and dynamic scene recognition tasks. From a practical standpoint, being able to hallucinate spatiotemporal representations without an actual 3D-CNN can enable deployment in resource-constrained scenarios, such as with limited computing power and/or lower bandwidth. We also observed that our hallucination task has utility not only during the training phase, but also during the pre-training phase.

scholarly journals Selectively Connected Self-Attentions for Semantic Role Labeling

2019 ◽  
Vol 9 (8) ◽  
pp. 1716
Author(s):  
Jaehui Park
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Hierarchical Structures ◽  
Large Data ◽  
Neural Model ◽  
Semantic Role ◽  
Semantic Role Labeling ◽  
Data Set ◽  
Training Time

Semantic role labeling is an effective approach to understand underlying meanings associated with word relationships in natural language sentences. Recent studies using deep neural networks, specifically, recurrent neural networks, have significantly improved traditional shallow models. However, due to the limitation of recurrent updates, they require long training time over a large data set. Moreover, they could not capture the hierarchical structures of languages. We propose a novel deep neural model, providing selective connections among attentive representations, which remove the recurrent updates, for semantic role labeling. Experimental results show that our model performs better in accuracy compared to the state-of-the-art studies. Our model achieves 86.6 F1 scores and 83.6 F1 scores on the CoNLL 2005 and CoNLL 2012 shared tasks, respectively. The accuracy gains are improved by capturing the hierarchical information using the connection module. Moreover, we show that our model can be parallelized to avoid the repetitive updates of the model. As a result, our model reduces the training time by 62 percentages from the baseline.

scholarly journals Hybrid Attention Based Residual Network for Pansharpening

2021 ◽  
Vol 13 (10) ◽  
pp. 1962
Author(s):  
Qin Liu ◽  
Letong Han ◽  
Rui Tan ◽  
Hongfei Fan ◽  
Weiqi Li ◽  
...  
Keyword(s):  
Neural Network ◽  
State Of The Art ◽  
Object Identification ◽  
Spectral Information ◽  
Spectral Distortion ◽  
Residual Network ◽  
Spatial Features ◽  
Fused Image ◽  
Ground Object ◽  
The Rich

Pansharpening aims at fusing the rich spectral information of multispectral(MS) images and the spatial details of panchromatic(PAN) images to generate a fused image with both high resolutions. In general, the existing pansharpening methods suffer from the problems of spectral distortion and lack of spatial detail information, which might prevent the accuracy computation for ground object identification. To alleviate these problems, we propose a Hybrid Attention mechanism-based Residual Neural Network(HARNN) . In the proposed network, we develop an encoder attention module in the feature extraction part to better utilize the spectral and spatial features of MS and PAN images. Furthermore, the fusion attention module is designed to alleviate spectral distortion and improve contour details of the fused image. A series of ablation and contrast experiments are conducted on GF-1 and GF-2 datasets. The fusion results with less distorted pixels and more spatial details demonstrate that HARNN can implement the pansharpening task effectively, which outperforms the state-of-the-art algorithms.

Text classification by untrained sentence embeddings

2021 ◽  
Vol 14 (2) ◽  
pp. 245-259
Author(s):  
Daniele Di Sarli ◽  
Claudio Gallicchio ◽  
Alessio Micheli
Keyword(s):  
Neural Networks ◽  
Text Classification ◽  
Recurrent Neural Networks ◽  
Learning Algorithm ◽  
State Of The Art ◽  
Sequential Data ◽  
Training Time ◽  
Reference Models ◽  
Gating Mechanisms ◽  
Heavy Training

Recurrent Neural Networks (RNNs) represent a natural paradigm for modeling sequential data like text written in natural language. In fact, RNNs and their variations have long been the architecture of choice in many applications, however in practice they require the use of labored architectures (such as gating mechanisms) and computationally heavy training processes. In this paper we address the question of whether it is possible to generate sentence embeddings via completely untrained recurrent dynamics, on top of which to apply a simple learning algorithm for text classification. This would allow to obtain extremely efficient models in terms of training time. Our work investigates the extent to which this approach can be used, by analyzing the results on different tasks. Finally, we show that, within certain limits, it is possible to build extremely efficient models for text classification that remain competitive in accuracy with reference models in the state-of-the-art.

Sign in / Sign up

Export Citation Format

Share Document