Hierarchical BERT with an adaptive fine-tuning strategy for document classification

2021 ◽  
pp. 107872
Author(s):  
Jun Kong ◽  
Jin Wang ◽  
Xuejie Zhang
2020 ◽  
Vol 10 (11) ◽  
pp. 3833 ◽  
Author(s):  
Haidar Almubarak ◽  
Yakoub Bazi ◽  
Naif Alajlan

In this paper, we propose a method for localizing the optic nerve head and segmenting the optic disc/cup in retinal fundus images. The approach is based on a simple two-stage Mask-RCNN compared to sophisticated methods that represent the state-of-the-art in the literature. In the first stage, we detect and crop around the optic nerve head then feed the cropped image as input for the second stage. The second stage network is trained using a weighted loss to produce the final segmentation. To further improve the detection in the first stage, we propose a new fine-tuning strategy by combining the cropping output of the first stage with the original training image to train a new detection network using different scales for the region proposal network anchors. We evaluate the method on Retinal Fundus Images for Glaucoma Analysis (REFUGE), Magrabi, and MESSIDOR datasets. We used the REFUGE training subset to train the models in the proposed method. Our method achieved 0.0430 mean absolute error in the vertical cup-to-disc ratio (MAE vCDR) on the REFUGE test set compared to 0.0414 obtained using complex and multiple ensemble networks methods. The models trained with the proposed method transfer well to datasets outside REFUGE, achieving a MAE vCDR of 0.0785 and 0.077 on MESSIDOR and Magrabi datasets, respectively, without being retrained. In terms of detection accuracy, the proposed new fine-tuning strategy improved the detection rate from 96.7% to 98.04% on MESSIDOR and from 93.6% to 100% on Magrabi datasets compared to the reported detection rates in the literature.


2021 ◽  
Vol 20 (5s) ◽  
pp. 1-25
Author(s):  
Zhenge Jia ◽  
Yiyu Shi ◽  
Samir Saba ◽  
Jingtong Hu

Atrial Fibrillation (AF), one of the most prevalent arrhythmias, is an irregular heart-rate rhythm causing serious health problems such as stroke and heart failure. Deep learning based methods have been exploited to provide an end-to-end AF detection by automatically extracting features from Electrocardiogram (ECG) signal and achieve state-of-the-art results. However, the pre-trained models cannot adapt to each patient’s rhythm due to the high variability of rhythm characteristics among different patients. Furthermore, the deep models are prone to overfitting when fine-tuned on the limited ECG of the specific patient for personalization. In this work, we propose a prior knowledge incorporated learning method to effectively personalize the model for patient-specific AF detection and alleviate the overfitting problems. To be more specific, a prior-incorporated portion importance mechanism is proposed to enforce the network to learn to focus on the targeted portion of the ECG, following the cardiologists’ domain knowledge in recognizing AF. A prior-incorporated regularization mechanism is further devised to alleviate model overfitting during personalization by regularizing the fine-tuning process with feature priors on typical AF rhythms of the general population. The proposed personalization method embeds the well-defined prior knowledge in diagnosing AF rhythm into the personalization procedure, which improves the personalized deep model and eliminates the workload of manually adjusting parameters in conventional AF detection method. The prior knowledge incorporated personalization is feasibly and semi-automatically conducted on the edge, device of the cardiac monitoring system. We report an average AF detection accuracy of 95.3% of three deep models over patients, surpassing the pre-trained model by a large margin of 11.5% and the fine-tuning strategy by 8.6%.


2018 ◽  
Vol 69 (1) ◽  
pp. 24-31
Author(s):  
Khaled S. Hatamleh ◽  
Qais A. Khasawneh ◽  
Adnan Al-Ghasem ◽  
Mohammad A. Jaradat ◽  
Laith Sawaqed ◽  
...  

Abstract Scanning Electron Microscopes are extensively used for accurate micro/nano images exploring. Several strategies have been proposed to fine tune those microscopes in the past few years. This work presents a new fine tuning strategy of a scanning electron microscope sample table using four bar piezoelectric actuated mechanisms. The introduced paper presents an algorithm to find all possible inverse kinematics solutions of the proposed mechanism. In addition, another algorithm is presented to search for the optimal inverse kinematic solution. Both algorithms are used simultaneously by means of a simulation study to fine tune a scanning electron microscope sample table through a pre-specified circular or linear path of motion. Results of the study shows that, proposed algorithms were able to minimize the power required to drive the piezoelectric actuated mechanism by a ratio of 97.5% for all simulated paths of motion when compared to general non-optimized solution.


Author(s):  
Xiaotong Lu ◽  
Han Huang ◽  
Weisheng Dong ◽  
Xin Li ◽  
Guangming Shi

Network pruning has been proposed as a remedy for alleviating the over-parameterization problem of deep neural networks. However, its value has been recently challenged especially from the perspective of neural architecture search (NAS). We challenge the conventional wisdom of pruning-after-training by proposing a joint search-and-training approach that directly learns a compact network from the scratch. By treating pruning as a search strategy, we present two new insights in this paper: 1) it is possible to expand the search space of networking pruning by associating each filter with a learnable weight; 2) joint search-and-training can be conducted iteratively to maximize the learning efficiency. More specifically, we propose a coarse-to-fine tuning strategy to iteratively sample and update compact sub-network to approximate the target network. The weights associated with network filters will be accordingly updated by joint search-and-training to reflect learned knowledge in NAS space. Moreover, we introduce strategies of random perturbation (inspired by Monte Carlo) and flexible thresholding (inspired by Reinforcement Learning) to adjust the weight and size of each layer. Extensive experiments on ResNet and VGGNet demonstrate the superior performance of our proposed method on popular datasets including CIFAR10, CIFAR100 and ImageNet.


2018 ◽  
pp. 283-293
Author(s):  
Craig R. Hickman ◽  
Michael A. Silva
Keyword(s):  

2020 ◽  
Vol 34 (05) ◽  
pp. 8815-8821 ◽  
Author(s):  
Sheng Shen ◽  
Zhen Dong ◽  
Jiayu Ye ◽  
Linjian Ma ◽  
Zhewei Yao ◽  
...  

Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use Hessian-based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.


2009 ◽  
Vol 86 (2) ◽  
pp. 28004 ◽  
Author(s):  
Y. Sun ◽  
B. Danila ◽  
K. Josić ◽  
K. E. Bassler

Author(s):  
Youngmin Ro ◽  
Jongwon Choi ◽  
Dae Ung Jo ◽  
Byeongho Heo ◽  
Jongin Lim ◽  
...  

In person re-identification (ReID) task, because of its shortage of trainable dataset, it is common to utilize fine-tuning method using a classification network pre-trained on a large dataset. However, it is relatively difficult to sufficiently finetune the low-level layers of the network due to the gradient vanishing problem. In this work, we propose a novel fine-tuning strategy that allows low-level layers to be sufficiently trained by rolling back the weights of high-level layers to their initial pre-trained weights. Our strategy alleviates the problem of gradient vanishing in low-level layers and robustly trains the low-level layers to fit the ReID dataset, thereby increasing the performance of ReID tasks. The improved performance of the proposed strategy is validated via several experiments. Furthermore, without any addons such as pose estimation or segmentation, our strategy exhibits state-of-the-art performance using only vanilla deep convolutional neural network architecture.


Sign in / Sign up

Export Citation Format

Share Document