Deep Attentive Ranking Networks for Learning to Order Sentences

We present an attention-based ranking framework for learning to order sentences given a paragraph. Our framework is built on a bidirectional sentence encoder and a self-attention based transformer network to obtain an input order invariant representation of paragraphs. Moreover, it allows seamless training using a variety of ranking based loss functions, such as pointwise, pairwise, and listwise ranking. We apply our framework on two tasks: Sentence Ordering and Order Discrimination. Our framework outperforms various state-of-the-art methods on these tasks on a variety of evaluation metrics. We also show that it achieves better results when using pairwise and listwise ranking losses, rather than the pointwise ranking loss, which suggests that incorporating relative positions of two or more sentences in the loss function contributes to better learning.

Download Full-text

Assessing the Impact of the Loss Function, Architecture and Image Type for Deep Learning-Based Wildfire Segmentation

Applied Sciences ◽

10.3390/app11157046 ◽

2021 ◽

Vol 11 (15) ◽

pp. 7046

Author(s):

Jorge Francisco Ciprián-Sánchez ◽

Gilberto Ochoa-Ruiz ◽

Lucile Rossi ◽

Frédéric Morandini

Keyword(s):

Deep Learning ◽

Loss Function ◽

State Of The Art ◽

Fire Detection ◽

Loss Functions ◽

Wildfire Spread ◽

Combine Information ◽

The Impact ◽

Image Type ◽

Segmentation Models

Wildfires stand as one of the most relevant natural disasters worldwide, particularly more so due to the effect of climate change and its impact on various societal and environmental levels. In this regard, a significant amount of research has been done in order to address this issue, deploying a wide variety of technologies and following a multi-disciplinary approach. Notably, computer vision has played a fundamental role in this regard. It can be used to extract and combine information from several imaging modalities in regard to fire detection, characterization and wildfire spread forecasting. In recent years, there has been work pertaining to Deep Learning (DL)-based fire segmentation, showing very promising results. However, it is currently unclear whether the architecture of a model, its loss function, or the image type employed (visible, infrared, or fused) has the most impact on the fire segmentation results. In the present work, we evaluate different combinations of state-of-the-art (SOTA) DL architectures, loss functions, and types of images to identify the parameters most relevant to improve the segmentation results. We benchmark them to identify the top-performing ones and compare them to traditional fire segmentation techniques. Finally, we evaluate if the addition of attention modules on the best performing architecture can further improve the segmentation results. To the best of our knowledge, this is the first work that evaluates the impact of the architecture, loss function, and image type in the performance of DL-based wildfire segmentation models.

Download Full-text

Learning Large Logic Programs By Going Beyond Entailment

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/287 ◽

2020 ◽

Author(s):

Andrew Cropper ◽

Sebastijan Dumančic

Keyword(s):

Logic Programming ◽

Loss Function ◽

Inductive Logic Programming ◽

State Of The Art ◽

Inductive Logic ◽

Program Synthesis ◽

Loss Functions ◽

Logic Programs ◽

Binary Decision ◽

Best First Search

A major challenge in inductive logic programming (ILP) is learning large programs. We argue that a key limitation of existing systems is that they use entailment to guide the hypothesis search. This approach is limited because entailment is a binary decision: a hypothesis either entails an example or does not, and there is no intermediate position. To address this limitation, we go beyond entailment and use 'example-dependent' loss functions to guide the search, where a hypothesis can partially cover an example. We implement our idea in Brute, a new ILP system which uses best-first search, guided by an example-dependent loss function, to incrementally build programs. Our experiments on three diverse program synthesis domains (robot planning, string transformations, and ASCII art), show that Brute can substantially outperform existing ILP systems, both in terms of predictive accuracies and learning times, and can learn programs 20 times larger than state-of-the-art systems.

Download Full-text

A New Dataset and Deep Residual Spectral Spatial Network for Hyperspectral Image Classification

Symmetry ◽

10.3390/sym12040561 ◽

2020 ◽

Vol 12 (4) ◽

pp. 561 ◽

Cited By ~ 1

Author(s):

Yiming Xue ◽

Dan Zeng ◽

Fansheng Chen ◽

Yueming Wang ◽

Zhijiang Zhang

Keyword(s):

Neural Networks ◽

Loss Function ◽

Hyperspectral Image ◽

State Of The Art ◽

Feature Learning ◽

Spatial Network ◽

Hyperspectral Image Classification ◽

Classification Framework ◽

Prediction Confidence ◽

Art Methods

Due to the limited varieties and sizes of existing public hyperspectral image (HSI) datasets, the classification accuracies are higher than 99% with convolutional neural networks (CNNs). In this paper, we presented a new HSI dataset named Shandong Feicheng, whose size and pixel quantity are much larger. It also has a larger intra-class variance and a smaller inter-class variance. State-of-the-art methods were compared on it to verify its diversity. Otherwise, to reduce overfitting caused by the imbalance between high dimension and small quantity of labeled HSI data, existing CNNs for HSI classification are relatively shallow and suffer from low capacity of feature learning. To solve this problem, we proposed an HSI classification framework named deep residual spectral spatial setwork (DRSSN). By using shortcut connection structure, which is an asymmetry structure, DRSSN can be deeper to extract features with better discrimination. In addition, to alleviate insufficient training caused by unbalanced sample sizes between easily and hard classified samples, we proposed a novel training loss function named sample balanced loss, which allocated weights to the losses of samples according to their prediction confidence. Experimental results on two popular datasets and our proposed dataset showed that our proposed network could provide competitive results compared with state-of-the-art methods.

Download Full-text

Novel loss functions for ensemble-based medical image classification

PLoS ONE ◽

10.1371/journal.pone.0261307 ◽

2021 ◽

Vol 16 (12) ◽

pp. e0261307

Author(s):

Sivaramakrishnan Rajaraman ◽

Ghada Zamzmi ◽

Sameer K. Antani

Keyword(s):

Loss Function ◽

State Of The Art ◽

Model Performance ◽

Classification Performance ◽

Loss Functions ◽

Weighted Averaging ◽

Classification Task ◽

Source Distribution ◽

Multi Class Classification ◽

The Individual

Medical images commonly exhibit multiple abnormalities. Predicting them requires multi-class classifiers whose training and desired reliable performance can be affected by a combination of factors, such as, dataset size, data source, distribution, and the loss function used to train deep neural networks. Currently, the cross-entropy loss remains the de-facto loss function for training deep learning classifiers. This loss function, however, asserts equal learning from all classes, leading to a bias toward the majority class. Although the choice of the loss function impacts model performance, to the best of our knowledge, we observed that no literature exists that performs a comprehensive analysis and selection of an appropriate loss function toward the classification task under study. In this work, we benchmark various state-of-the-art loss functions, critically analyze model performance, and propose improved loss functions for a multi-class classification task. We select a pediatric chest X-ray (CXR) dataset that includes images with no abnormality (normal), and those exhibiting manifestations consistent with bacterial and viral pneumonia. We construct prediction-level and model-level ensembles to improve classification performance. Our results show that compared to the individual models and the state-of-the-art literature, the weighted averaging of the predictions for top-3 and top-5 model-level ensembles delivered significantly superior classification performance (p < 0.05) in terms of MCC (0.9068, 95% confidence interval (0.8839, 0.9297)) metric. Finally, we performed localization studies to interpret model behavior and confirm that the individual models and ensembles learned task-specific features and highlighted disease-specific regions of interest. The code is available at https://github.com/sivaramakrishnan-rajaraman/multiloss_ensemble_models.

Download Full-text

Road crack segmentation using an attention residual U-Net with generative adversarial

Mathematical Biosciences and Engineering ◽

10.3934/mbe.2021473 ◽

2021 ◽

Vol 18 (6) ◽

pp. 9669-9684

Author(s):

Xing Hu ◽

◽

Minghui Yao ◽

Dawei Zhang

Keyword(s):

Visual Attention ◽

Loss Function ◽

State Of The Art ◽

The State ◽

Attention Mechanism ◽

Semantic Features ◽

Convolutional Network ◽

Adversarial Learning ◽

Art Methods ◽

Visual Attention Mechanism

<abstract> <p>This paper proposed an end-to-end road crack segmentation model based on attention mechanism and deep FCN with generative adversarial learning. We create a segmentation network by introducing a visual attention mechanism and residual module to a fully convolutional network(FCN) to capture richer local features and more global semantic features and get a better segment result. Besides, we use an adversarial network consisting of convolutional layers as a discrimination network. The main contributions of this work are as follows: 1) We introduce a CNN model as a discriminate network to realize adversarial learning to guide the training of the segmentation network, which is trained in a min-max way: the discrimination network is trained by maximizing the loss function, while the segmentation network is trained with the only gradient passed by the discrimination network and aim at minimizing the loss function, and finally an optimal segmentation network is obtained; 2) We add the residual modular and the visual attention mechanism to U-Net, which makes the segmentation results more robust, refined and smooth; 3) Extensive experiments are conducted on three public road crack datasets to evaluate the performance of our proposed model. Qualitative and quantitative comparisons between the proposed method and the state-of-the-art methods show that the proposed method outperforms or is comparable to the state-of-the-art methods in both F1 score and precision. In particular, compared with U-Net, the mIoU of our proposed method is increased about 3%~17% compared with the three public datasets.</p> </abstract>

Download Full-text

Double Additive Margin Softmax Loss for Face Recognition

Applied Sciences ◽

10.3390/app10010060 ◽

2019 ◽

Vol 10 (1) ◽

pp. 60 ◽

Cited By ~ 1

Author(s):

Shengwei Zhou ◽

Caikou Chen ◽

Guojiang Han ◽

Xielian Hou

Keyword(s):

Neural Networks ◽

Face Recognition ◽

Loss Function ◽

State Of The Art ◽

Feature Learning ◽

Loss Functions ◽

Deep Convolutional Neural Networks ◽

Large Margin ◽

Face Features ◽

Geometrical Explanation

Learning large-margin face features whose intra-class variance is small and inter-class diversity is one of important challenges in feature learning applying Deep Convolutional Neural Networks (DCNNs) for face recognition. Recently, an appealing line of research is to incorporate an angular margin in the original softmax loss functions for obtaining discriminative deep features during the training of DCNNs. In this paper we propose a novel loss function, termed as double additive margin Softmax loss (DAM-Softmax). The presented loss has a clearer geometrical explanation and can obtain highly discriminative features for face recognition. Extensive experimental evaluation of several recent state-of-the-art softmax loss functions are conducted on the relevant face recognition benchmarks, CASIA-Webface, LFW, CALFW, CPLFW, and CFP-FP. We show that the proposed loss function consistently outperforms the state-of-the-art.

Download Full-text

Valence electron spectroscopy of inhomogeneous media

Proceedings, annual meeting, Electron Microscopy Society of America ◽

10.1017/s0424820100130389 ◽

1992 ◽

Vol 50 (2) ◽

pp. 1150-1151

Author(s):

A. Howie ◽

D.W. McComb

Keyword(s):

Specific Gravity ◽

Loss Function ◽

Electron Spectroscopy ◽

Valence Electron ◽

Peak Height ◽

Loss Functions ◽

Loss Peak ◽

High Energies ◽

Valence Electron Density ◽

Electron Microscopist

The bulk loss function Im(-l/ε (ω)), a well established tool for the interpretation of valence loss spectra, is being progressively adapted to the wide variety of inhomogeneous samples of interest to the electron microscopist. Proportionality between n, the local valence electron density, and ε-1 (Sellmeyer's equation) has sometimes been assumed but may not be valid even in homogeneous samples. Figs. 1 and 2 show the experimentally measured bulk loss functions for three pure silicates of different specific gravity ρ - quartz (ρ = 2.66), coesite (ρ = 2.93) and a zeolite (ρ = 1.79). Clearly, despite the substantial differences in density, the shift of the prominent loss peak is very small and far less than that predicted by scaling e for quartz with Sellmeyer's equation or even the somewhat smaller shift given by the Clausius-Mossotti (CM) relation which assumes proportionality between n (or ρ in this case) and (ε - 1)/(ε + 2). Both theories overestimate the rise in the peak height for coesite and underestimate the increase at high energies.

Download Full-text

Multi-hop assortativities for network classification

Journal of Complex Networks ◽

10.1093/comnet/cny034 ◽

2018 ◽

Vol 7 (4) ◽

pp. 603-622 ◽

Cited By ~ 1

Author(s):

Leonardo Gutiérrez-Gómez ◽

Jean-Charles Delvenne

Keyword(s):

Machine Learning ◽

Scientific Collaboration ◽

State Of The Art ◽

Medical Engineering ◽

Research Field ◽

Classification Task ◽

Collaboration Network ◽

Structural Patterns ◽

Art Methods

Abstract Several social, medical, engineering and biological challenges rely on discovering the functionality of networks from their structure and node metadata, when it is available. For example, in chemoinformatics one might want to detect whether a molecule is toxic based on structure and atomic types, or discover the research field of a scientific collaboration network. Existing techniques rely on counting or measuring structural patterns that are known to show large variations from network to network, such as the number of triangles, or the assortativity of node metadata. We introduce the concept of multi-hop assortativity, that captures the similarity of the nodes situated at the extremities of a randomly selected path of a given length. We show that multi-hop assortativity unifies various existing concepts and offers a versatile family of ‘fingerprints’ to characterize networks. These fingerprints allow in turn to recover the functionalities of a network, with the help of the machine learning toolbox. Our method is evaluated empirically on established social and chemoinformatic network benchmarks. Results reveal that our assortativity based features are competitive providing highly accurate results often outperforming state of the art methods for the network classification task.

Download Full-text

Study on Radar Echo-Filling in an Occlusion Area by a Deep Learning Algorithm

Remote Sensing ◽

10.3390/rs13091779 ◽

2021 ◽

Vol 13 (9) ◽

pp. 1779

Author(s):

Xiaoyan Yin ◽

Zhiqun Hu ◽

Jiafeng Zheng ◽

Boyong Li ◽

Yuanyuan Zuo

Keyword(s):

Deep Learning ◽

Loss Function ◽

Learning Algorithm ◽

Weather Radar ◽

Loss Functions ◽

Training Dataset ◽

Echo Intensity ◽

Common Mean ◽

Deep Learning Algorithm ◽

Radar Beam

Radar beam blockage is an important error source that affects the quality of weather radar data. An echo-filling network (EFnet) is proposed based on a deep learning algorithm to correct the echo intensity under the occlusion area in the Nanjing S-band new-generation weather radar (CINRAD/SA). The training dataset is constructed by the labels, which are the echo intensity at the 0.5° elevation in the unblocked area, and by the input features, which are the intensity in the cube including multiple elevations and gates corresponding to the location of bottom labels. Two loss functions are applied to compile the network: one is the common mean square error (MSE), and the other is a self-defined loss function that increases the weight of strong echoes. Considering that the radar beam broadens with distance and height, the 0.5° elevation scan is divided into six range bands every 25 km to train different models. The models are evaluated by three indicators: explained variance (EVar), mean absolute error (MAE), and correlation coefficient (CC). Two cases are demonstrated to compare the effect of the echo-filling model by different loss functions. The results suggest that EFnet can effectively correct the echo reflectivity and improve the data quality in the occlusion area, and there are better results for strong echoes when the self-defined loss function is used.

Download Full-text

Automatic Detection of Discrimination Actions from Social Images

Electronics ◽

10.3390/electronics10030325 ◽

2021 ◽

Vol 10 (3) ◽

pp. 325

Author(s):

Zhihao Wu ◽

Baopeng Zhang ◽

Tianchen Zhou ◽

Yan Li ◽

Jianping Fan

Keyword(s):

Action Recognition ◽

State Of The Art ◽

Automatic Detection ◽

Experimental Results ◽

Practical Approach ◽

Detection And Identification ◽

Art Methods ◽

Image Set ◽

Social Images ◽

Relationship Identification

In this paper, we developed a practical approach for automatic detection of discrimination actions from social images. Firstly, an image set is established, in which various discrimination actions and relations are manually labeled. To the best of our knowledge, this is the first work to create a dataset for discrimination action recognition and relationship identification. Secondly, a practical approach is developed to achieve automatic detection and identification of discrimination actions and relationships from social images. Thirdly, the task of relationship identification is seamlessly integrated with the task of discrimination action recognition into one single network called the Co-operative Visual Translation Embedding++ network (CVTransE++). We also compared our proposed method with numerous state-of-the-art methods, and our experimental results demonstrated that our proposed methods can significantly outperform state-of-the-art approaches.

Download Full-text