Revisiting Batch Normalization for Training Low-Latency Deep Spiking Neural Networks From Scratch

Spiking Neural Networks (SNNs) have recently emerged as an alternative to deep learning owing to sparse, asynchronous and binary event (or spike) driven processing, that can yield huge energy efficiency benefits on neuromorphic hardware. However, SNNs convey temporally-varying spike activation through time that is likely to induce a large variation of forward activation and backward gradients, resulting in unstable training. To address this training issue in SNNs, we revisit Batch Normalization (BN) and propose a temporal Batch Normalization Through Time (BNTT) technique. Different from previous BN techniques with SNNs, we find that varying the BN parameters at every time-step allows the model to learn the time-varying input distribution better. Specifically, our proposed BNTT decouples the parameters in a BNTT layer along the time axis to capture the temporal dynamics of spikes. We demonstrate BNTT on CIFAR-10, CIFAR-100, Tiny-ImageNet, event-driven DVS-CIFAR10 datasets, and Sequential MNIST and show near state-of-the-art performance. We conduct comprehensive analysis on the temporal characteristic of BNTT and showcase interesting benefits toward robustness against random and adversarial noise. Further, by monitoring the learnt parameters of BNTT, we find that we can do temporal early exit. That is, we can reduce the inference latency by ~5 − 20 time-steps from the original training latency. The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/BNTT-Batch-Normalization-Through-Time.

A framework for multiplex imaging optimization and reproducible analysis

Multiplex imaging technologies are increasingly used for single-cell phenotyping and spatial characterization of tissues; however, transparent methods are needed for comparing the performance of platforms, protocols and analytical pipelines. We developed a python software, jinxif, for reproducible image processing and utilize Jupyter notebooks to share our optimization of signal removal, antibody specificity, background correction and batch normalization of the multiplex imaging with a focus on cyclic immunofluorescence (CyCIF). Our work both improves the CyCIF methodology and provides a framework for multiplexed image analytics that can be easily shared and reproduced.

Tessarine and Quaternion-Valued Deep Neural Networks for Image Classification

Many image processing and analysis tasks are performed with deep neural networks. Although the vast majority of advances have been made with real numbers, recent works have shown that complex and hypercomplex-valued networks may achieve better results. In this paper, we address quaternion-valued and introduce tessarine-valued deep neural networks, including tessarine-valued 2D convolutions. We also address initialization schemes and hypercomplex batch normalization. Finally, a tessarine-valued ResNet model with hypercomplex batch normalization outperformed the corresponding real and quaternion-valued networks on the CIFAR dataset.

Convolutional Neural Network With Batch Normalization for Classification of Emotional Expressions Based on Facial Images

Emotion recognition through facial images is one of the most challenging topics in human psychological interactions with machines. Along with advances in robotics, computer graphics, and computer vision, research on facial expression recognition is an important part of intelligent systems technology for interactive human interfaces where each person may have different emotional expressions, making it difficult to classify facial expressions and requires training data. large, so the deep learning approach is an alternative solution., The purpose of this study is to propose a different Convolutional Neural Network (CNN) model architecture with batch normalization consisting of three layers of multiple convolution layers with a simpler architectural model for the recognition of emotional expressions based on human facial images in the FER2013 dataset from Kaggle. The experimental results show that the training accuracy level reaches 98%, but there is still overfitting where the validation accuracy level is still 62%. The proposed model has better performance than the model without using batch normalization.

Addressing catastrophic forgetting for medical domain expansion

Model brittleness is a key concern when deploying deep learning models in real-world medical settings. A model that has high performance at one dataset may suffer a significant decline in performance when tested at on different datasets. While pooling datasets from multiple hospitals and re-training may provide a straightforward solution, it is often infeasible and may compromise patient privacy. An alternative approach is to fine-tune the model on subsequent datasets after training on the original dataset. Notably,this approach degrades model performance at the original datasets, a phenomenon known as catastrophic forgetting. In this paper, we develop an approach to address catastrophic forgetting based on elastic weight consolidation combined with modulation of batch normalization statistics under three scenarios: 1) for expanding the domain from one imaging system’s data to another imaging system’s 2) for expanding the domain from a large multi-hospital dataset to another single hospital dataset 3) for expanding the domain from dataset from one geographic region to a dataset from another geographic region. Focusing on the clinical uses cases of mammographic breast density detection and retinopathy of prematurity stage diagnosis, we show that our approach outperforms several other state-of-the-art approaches and provide theoretical justification for the efficacy of batch normalization modulation. The results of this study are generally applicable to the deployment of any clinical deep learning model which requires domain expansion.

The Shrank YoloV3-tiny for spinal fracture lesions detection

Purpose: at present, more and more deep learning algorithms are used to detect and segment lesions from spinal CT (Computed Tomography) images. But these algorithms usually require computers with high performance and occupy large resources, so they are not suitable for the clinical embedded and mobile devices, which only have limited computational resources and also expect a relative good performance in detecting and segmenting lesions. Methods: in this paper, we present a model based on Yolov3-tiny to detect three spinal fracture lesions, cfracture (cervical fracture), tfracture (thoracic fracture), and lfracture (lumbar fracture) with a small size model. We construct this novel model by replacing the traditional convolutional layers in YoloV3-tiny with fire modules from SqueezeNet, so as to reduce the parameters and model size, meanwhile get accurate lesions detection. Then we remove the batch normalization layers in the fire modules after the comparative experiments, though the overall performance of fire module without batch normalization layers is slightly improved, we can reduce computation complexity and low occupations of computer resources for fast lesions detection. Results: the experiments show that the shrank model only has a size of 13 MB (almost a third of Yolov3-tiny), while the mAP (mean Average Precsion) is 91.3%, and IOU (intersection over union) is 90.7. The detection time is 0.015 second per CT image, and BFLOP/s (Billion Floating Point Operations per Second) value is less than Yolov3-tiny. Conclusion: the model we presented can be deployed in clinical embedded and mobile devices, meanwhile has a relative accurate and rapid real-time lesions detection.

Mapbatch: Conservative Batch Normalization for Single Cell RNA-Sequencing Data Enables Discovery of Rare Cell Populations in a Multiple Myeloma Cohort

Introduction Many cancers involve the participation of rare cell populations that may only be found in a subset of patients. Single-cell RNA sequencing (scRNA-seq) can identify distinct cell populations across multiple samples with batch normalization used to reduce processing-based effects between samples. However, aggressive normalization obscures rare cell populations, which may be erroneously grouped with other cell types. There is a need for conservative batch normalization that maintains the biological signal necessary to detect rare cell populations. MapBatch We designed a batch normalization tool, MapBatch, based on two principles: an autoencoder trained with a single sample learns the underlying gene expression structure of cell types without batch effect; and an ensemble model combines multiple autoencoders, allowing the use of multiple samples for training. Each autoencoder is trained on one sample, learning a projection into the biological space S representing the real expression differences between cells in that sample (Figure 1a, middle). When other samples are projected into S, the projection reduces expression differences orthogonal to S, while preserving differences along S. The reverse projection transforms the data back into gene space at the autoencoder's output, sans expression differences orthogonal to S (Figure 1a, right). Since batch-based technical differences are not represented in S, this transformation selectively removes batch effect between samples, while preserving biological signal. The autoencoder output thus represents normalized expression data, conditioned on the training sample. To incorporate multiple samples into training, MapBatch uses an ensemble of autoencoders, each trained with a single sample (Figure 1b). We train with a minimal number of samples necessary to cover the different cell populations in the dataset. We implement regularization using dropout and noise layers, and an a priori feature extraction layer using KEGG gene modules. The autoencoders' outputs are concatenated for downstream analysis. For visualization and clustering, we use the top principal components of the concatenated outputs. For differential expression (DE), we perform DE on each of the gene matrices output by each model, then take the result with the lowest P-value. To test MapBatch, we generated a synthetic dataset based on 7 batches of publicly available PBMC data. For each batch we simulated rare cell populations by selecting one of three cell types to perturb by up and down-regulating 40 genes in 0.5%-2% of the cells (Figure 1c). We simulated additional batch effect by scaling each gene in each batch with a scaling factor. Upon visualization and clustering, cells grouped largely by batch (Figure 1d). After batch normalization, cells grouped by cell type rather than batch, and all three perturbed cell populations were successfully delineated (Figure 1e). DE between each perturbed population and its mother cells accurately retrieved the perturbed genes, showing that normalization maintained real expression differences (Figure 1e). In contrast, three methods tested Seurat (Stuart et al., 2019), Harmony (Korsunsky et al., 2019), and Liger (Welch et al., 2019) could only derive a subset of the perturbed populations (Figures 1f-h). MapBatch identifies rare populations in multiple myeloma (MM) We used MapBatch to process bone marrow scRNA-seq data from 14 MM samples and 2 healthy controls. After batch normalization, unsupervised clustering identified 20 clusters, which we annotated using MapCell (Koh & Hoon, 2019) (Figures 2a, 2b). We identified 3 small clusters of cells that could not be reliably annotated, comprising less than 1% of total cells and found in only a subset of patients (Figures 2c, 2d). As validation, we observed that these cells were present in distinct clusters in individual samples using their uncorrected expression data, providing evidence that these clusters were not driven by batch effect nor MapBatch (Figure 2e). Conclusion Batch normalization of scRNA-seq data involves a trade-off between minimizing batch effect and maximizing the remaining biological signal. While most methods lean towards the former, MapBatch maintains more biological signal for downstream analysis, enabling the discovery of previously difficult to find cell populations. Figure 1 Figure 1. Disclosures Xu: Proteona Pte Ltd: Current Employment. Scolnick: Proteona Pte Ltd: Current holder of individual stocks in a privately-held company. Huo: Proteona Pte Ltd: Ended employment in the past 24 months. Lovci: Proteona Pte Ltd: Current Employment. Chng: Amgen: Honoraria, Research Funding; Abbvie: Honoraria; Janssen: Honoraria, Research Funding; Novartis: Honoraria; Celgene: Honoraria, Research Funding.

BNDNN: Batch Normalization Based Deep Neural Network for Predicting Flood in Urban Areas

Disaster is a very serious dissipation that arises for a short time period, but the impact of that disaster on human society is very dangerous and very long-lasting. Disasters are categorized into two types like natural disasters and manmade disasters. Among all disasters, of all the natural disasters, flood is the commonplace natural disaster. Flood disaster that causes huge loss of human life, diversity as well as economic loss, which is very dangerous for the developing countries and developed countries also. Nowadays during the monsoon season flood is dangerous for all the geographical areas located nearby water bodies. Much research has been done for flood detection. Machine Learning and many other recent technologies are playing a vital role in predicting the occurrence of floods. For prediction purposes, a huge amount of data is requiring collected from sensors deployed in various locations. In this paper, we used the Batch normalization with Deep Neural Network (BNDNN) technique for the classification of data in three classes as Low, Moderate, and High. The result obtained from our proposed model is compared with some other models like Decision Tree (DT), Support Vector Machine (SVM), Artificial Neural Network (ANN), and Deep Neural Network (DNN). In this our proposed BNDNN provides 89% accuracy which is higher among all existing models. Models are compared based on some parameters like Accuracy, Precision, Recall, F –Score. The compression among all the models used in this paper shows that our proposed model provides better results.