Modifying the softening process for knowledge distillation

2021 ◽  
pp. 1-13
Author(s):  
Chao Tan ◽  
Jie Liu
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Error Rate ◽  
Experimental Results ◽  
Teacher Student ◽  
Resource Limited ◽  
Visual Tasks ◽  
Knowledge Distillation ◽  
Softening Process ◽  
Performance Gains

The prime focus of knowledge distillation (KD) seeks a light proxy termed student to mimic the outputs of its heavy neural networks termed teacher, and makes the student run real-time on the resource-limited devices. This paradigm requires aligning the soft logits of both teacher and student. However, few doubts whether the process of softening the logits truly give full play to the teacher-student paradigm. In this paper, we launch several analyses to delve into this issue from scratch. Subsequently, several simple yet effective functions are devised to replace the vanilla KD. The ultimate function can be an effective alternative to its original counterparts and work well with other skills like FitNets. To claim this point, we conduct several visual tasks on individual benchmarks, and experimental results verify the potential of our proposed function in terms of performance gains. For example, when the teacher and student networks are ShuffleNetV2-1.0 and ShuffleNetV2-0.5, our proposed method achieves 40.88%top-1 error rate on Tiny ImageNet.

scholarly journals Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction

2020 ◽  
Vol 34 (05) ◽  
pp. 9620-9627 ◽  
Author(s):  
Zhenyu Zhang ◽  
Xiaobo Shu ◽  
Bowen Yu ◽  
Tingwen Liu ◽  
Jiapeng Zhao ◽  
...  
Keyword(s):  
Neural Networks ◽  
Bipartite Graph ◽  
Prior Knowledge ◽  
Semantic Information ◽  
Relation Extraction ◽  
Important Task ◽  
Experimental Results ◽  
Plain Text ◽  
Knowledge Distillation ◽  
The Rich

Extracting relations from plain text is an important task with wide application. Most existing methods formulate it as a supervised problem and utilize one-hot hard labels as the sole target in training, neglecting the rich semantic information among relations. In this paper, we aim to explore the supervision with soft labels in relation extraction, which makes it possible to integrate prior knowledge. Specifically, a bipartite graph is first devised to discover type constraints between entities and relations based on the entire corpus. Then, we combine such type constraints with neural networks to achieve a knowledgeable model. Furthermore, this model is regarded as teacher to generate well-informed soft labels and guide the optimization of a student network via knowledge distillation. Besides, a multi-aspect attention mechanism is introduced to help student mine latent information from text. In this way, the enhanced student inherits the dark knowledge (e.g., type constraints and relevance among relations) from teacher, and directly serves the testing scenarios without any extra constraints. We conduct extensive experiments on the TACRED and SemEval datasets, the experimental results justify the effectiveness of our approach.

Real-time Detection of Aortic Valve in Echocardiography using Convolutional Neural Networks

2020 ◽  
Vol 16 (5) ◽  
pp. 584-591 ◽  
Author(s):  
Muhammad Hanif Ahmad Nizar ◽  
Chow Khuen Chan ◽  
Azira Khalil ◽  
Ahmad Khairuddin Mohamed Yusof ◽  
Khin Wee Lai
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Heart Disease ◽  
Aortic Valve ◽  
Real Time ◽  
Convolutional Neural Networks ◽  
Valvular Heart Disease ◽  
Detection System ◽  
Processing Unit ◽  
Real Time Detection

Background: Valvular heart disease is a serious disease leading to mortality and increasing medical care cost. The aortic valve is the most common valve affected by this disease. Doctors rely on echocardiogram for diagnosing and evaluating valvular heart disease. However, the images from echocardiogram are poor in comparison to Computerized Tomography and Magnetic Resonance Imaging scan. This study proposes the development of Convolutional Neural Networks (CNN) that can function optimally during a live echocardiographic examination for detection of the aortic valve. An automated detection system in an echocardiogram will improve the accuracy of medical diagnosis and can provide further medical analysis from the resulting detection. Methods: Two detection architectures, Single Shot Multibox Detector (SSD) and Faster Regional based Convolutional Neural Network (R-CNN) with various feature extractors were trained on echocardiography images from 33 patients. Thereafter, the models were tested on 10 echocardiography videos. Results: Faster R-CNN Inception v2 had shown the highest accuracy (98.6%) followed closely by SSD Mobilenet v2. In terms of speed, SSD Mobilenet v2 resulted in a loss of 46.81% in framesper- second (fps) during real-time detection but managed to perform better than the other neural network models. Additionally, SSD Mobilenet v2 used the least amount of Graphic Processing Unit (GPU) but the Central Processing Unit (CPU) usage was relatively similar throughout all models. Conclusion: Our findings provide a foundation for implementing a convolutional detection system to echocardiography for medical purposes.

scholarly journals Assurance monitoring of learning-enabled cyber-physical systems using inductive conformal prediction based on distance learning

2021 ◽  
Vol 35 (2) ◽  
pp. 251-264
Author(s):  
Dimitrios Boursinos ◽  
Xenofon Koutsoukos
Keyword(s):  
Neural Networks ◽  
Distance Learning ◽  
Real Time ◽  
Speaker Recognition ◽  
Deep Neural Networks ◽  
Cyber Physical Systems ◽  
Error Rates ◽  
Traffic Sign ◽  
Conformal Prediction ◽  
Physical Systems

AbstractMachine learning components such as deep neural networks are used extensively in cyber-physical systems (CPS). However, such components may introduce new types of hazards that can have disastrous consequences and need to be addressed for engineering trustworthy systems. Although deep neural networks offer advanced capabilities, they must be complemented by engineering methods and practices that allow effective integration in CPS. In this paper, we proposed an approach for assurance monitoring of learning-enabled CPS based on the conformal prediction framework. In order to allow real-time assurance monitoring, the approach employs distance learning to transform high-dimensional inputs into lower size embedding representations. By leveraging conformal prediction, the approach provides well-calibrated confidence and ensures a bounded small error rate while limiting the number of inputs for which an accurate prediction cannot be made. We demonstrate the approach using three datasets of mobile robot following a wall, speaker recognition, and traffic sign recognition. The experimental results demonstrate that the error rates are well-calibrated while the number of alarms is very small. Furthermore, the method is computationally efficient and allows real-time assurance monitoring of CPS.

Hybrid Approach and Architecture to Detect Fake News on Twitter in Real-Time using Neural Networks

2020 ◽  
Author(s):  
Madusha Prasanjith Thilakarathna ◽  
Vihanga Ashinsana Wijayasekara ◽  
Yasiru Gamage ◽  
Kavindi Hanshani Peiris ◽  
Chanuka Abeysinghe ◽  
...  
Keyword(s):  
Neural Networks ◽  
Real Time ◽  
Hybrid Approach ◽  
Fake News
Sign in / Sign up

Export Citation Format

Share Document