An analytic theory of shallow networks dynamics for hinge loss classification*

Neural networks have been shown to perform incredibly well in classification tasks over structured high-dimensional datasets. However, the learning dynamics of such networks is still poorly understood. In this paper we study in detail the training dynamics of a simple type of neural network: a single hidden layer trained to perform a classification task. We show that in a suitable mean-field limit this case maps to a single-node learning problem with a time-dependent dataset determined self-consistently from the average nodes population. We specialize our theory to the prototypical case of a linearly separable data and a linear hinge loss, for which the dynamics can be explicitly solved in the infinite dataset limit. This allows us to address in a simple setting several phenomena appearing in modern networks such as slowing down of training dynamics, crossover between rich and lazy learning, and overfitting. Finally, we assess the limitations of mean-field theory by studying the case of large but finite number of nodes and of training samples.

Controllably Deep Supervision and Multi-Scale Feature Fusion Network for Cloud and Snow Detection Based on Medium- and High-Resolution Imagery Dataset

Clouds and snow in remote sensing imageries cover underlying surface information, reducing image availability. Moreover, they interact with each other, decreasing the cloud and snow detection accuracy. In this study, we propose a convolutional neural network for cloud and snow detection, named the cloud and snow detection network (CSD-Net). It incorporates the multi-scale feature fusion module (MFF) and the controllably deep supervision and feature fusion structure (CDSFF). MFF can capture and aggregate features at various scales, ensuring that the extracted high-level semantic features of clouds and snow are more distinctive. CDSFF can provide a deeply supervised mechanism with hinge loss and combine information from adjacent layers to gain more representative features. It ensures the gradient flow is more oriented and error-less, while retaining more effective information. Additionally, a high-resolution cloud and snow dataset based on WorldView2 (CSWV) was created and released. This dataset meets the training requirements of deep learning methods for clouds and snow in high-resolution remote sensing images. Based on the datasets with varied resolutions, CSD-Net is compared to eight state-of-the-art deep learning methods. The experiment results indicate that CSD-Net has an excellent detection accuracy and efficiency. Specifically, the mean intersection over the union (MIoU) of CSD-Net is the highest in the corresponding experiment. Furthermore, the number of parameters in our proposed network is just 7.61 million, which is the lowest of the tested methods. It only has 88.06 GFLOPs of floating point operations, which is less than the U-Net, DeepLabV3+, PSPNet, SegNet-Modified, MSCFF, and GeoInfoNet. Meanwhile, CSWV has a higher annotation quality since the same method can obtain a greater accuracy on it.

Improving robustness of a deep learning-based lung-nodule classification model of CT images with respect to image noise

Objective: Robustness is an important aspect to consider, when developing methods for medical image analysis. This study investigated robustness properties of deep neural networks (DNNs) for a lung nodule classification problem based on CT images and proposed a solution to improve robustness. Approach: We firstly constructed a class of four DNNs with different widths, each predicting an output label (benign or malignant) for an input CT image cube containing a lung nodule. These networks were trained to achieve Area Under the Curve of 0.891-0.914 on a testing dataset. We then added to the input CT image cubes noise signals generated randomly using a realistic CT image noise model based on a noise power spectrum at 100 mAs, and monitored the DNN’s output change. We defined $SAR_{5} (\%)$ to quantify the robustness of the trained DNN model, indicating that for $5\%$ of CT image cubes, the noise can change the prediction results with a chance of at least $SAR_{5} (\%)$. To understand robustness, we viewed the information processing pipeline by the DNN as a two-step process, with the first step using all but the last layers to extract representations of the input CT image cubes in a latent space, and the second step employing the last fully-connected layer as a linear classifier to determine the position of the sample representations relative to a decision plane. To improve robustness, we proposed to retrain the last layer of the DNN with a Supporting Vector Machine (SVM) hinge loss function to enforce the desired position of the decision plane. Main results: $SAR_{5}$ ranged in $47.0\sim 62.0\%$ in different DNNs. The unrobustness behavior may be ascribed to the unfavorable placement of the decision plane in the latent representation space, which made some samples be perturbed to across the decision plane and hence susceptible to noise. The DNN-SVM model improved robustness over the DNN model and reduced $SAR_{5}$ by $8.8\sim 21.0\%$. Significance: This study provided insights about the potential reason for the unrobustness behavior of DNNs and the proposed DNN-SVM model improved model robustness.

e-Commerce Online Intelligent Customer Service System Based on Fuzzy Control

In order to improve the intelligence of the e-commerce online intelligent customer service system, this paper proposes a deep rejection recognition algorithm based on the maximum interval squared hinge loss and combines the actual needs of the e-commerce online customer service system to build an intelligent customer service system with the support of the fuzzy control system. Moreover, this article chooses to build a domain ontology library for structured storage of domain knowledge needed by customer service chatbots. In addition, this article analyzes the dialogue structure based on the speech act model and combines the semantic vector model of the question sentence on the basis of the dialogue structure to understand the question sentence, which helps to improve the accuracy of the answer feedback of the Internet shopping customer service robot. Finally, this article designs experiments to verify the performance of the online customer service system constructed in this article and analyzes the experimental results through statistical methods. The experimental results show that the online intelligent customer service system constructed in this paper has certain practical effects.

Detection of Epilepsy based on EEG Signals using PCA with ANN Model

Epilepsy is a chronic neurological illness that affects millions of people throughout the world. Epilepsy affects around 50 million people globally. It is estimated that if epilepsy is correctly diagnosed and treated, up to 70% of people with the condition will be seizure-free. There is a need to detect epilepsy at the initial stages to reduce symptoms by medications and other strategies. We use Epileptic Seizure Recognition dataset to train the model which is provided by UCI Machine Learning Repository. There are 179 attributes and 11,500 unique values in this dataset. MLP, PCA with RF, QDA, LDA, and PCA with ANN were applied among them; PCA with ANN provided the better metrics. For the metrics, we received the following findings. It is 97.55% Accuracy, 94.24% Precision, 91.48% recall, 83.38% hinge loss, and 2.32% mean squared error.

A Hybrid Deep Learning and Optimized Machine Learning Approach for Rose Leaf Disease Classification

Analysis of the symptoms of rose leaves can identify up to 15 different diseases. This research aims to develop Convolutional Neural Network models for classifying the diseases on rose leaves using hybrid deep learning techniques with Support Vector Machine (SVM). The developed models were based on the VGG16 architecture and early or late fusion techniques were applied to concatenate the output from a fully connected layer. The results showed that the developed models based on early fusion performed better than the developed models on either late fusion or VGG16 alone. In addition, it was found that the models using the SVM classifier had better efficiency in classifying the diseases appearing on rose leaves than the models using the softmax function classifier. In particular, a hybrid deep learning model based on early fusion and SVM, which applied the categorical hinge loss function, yielded a validation accuracy of 88.33% and a validation loss of 0.0679, which were higher than the ones of the other models. Moreover, this model was evaluated by 10-fold cross-validation with 90.26% accuracy, 90.59% precision, 92.44% recall, and 91.50% F1-score for disease classification on rose leaves.

Stochastic Subgradient for Large-Scale Support Vector Machine Using the Generalized Pinball Loss Function

In this paper, we propose a stochastic gradient descent algorithm, called stochastic gradient descent method-based generalized pinball support vector machine (SG-GPSVM), to solve data classification problems. This approach was developed by replacing the hinge loss function in the conventional support vector machine (SVM) with a generalized pinball loss function. We show that SG-GPSVM is convergent and that it approximates the conventional generalized pinball support vector machine (GPSVM). Further, the symmetric kernel method was adopted to evaluate the performance of SG-GPSVM as a nonlinear classifier. Our suggested algorithm surpasses existing methods in terms of noise insensitivity, resampling stability, and accuracy for large-scale data scenarios, according to the experimental results.

The Novel Sequence Distance Measuring Algorithm Based on Optimal Transport and Cross-Attention Mechanism

In this paper, we propose a novel sequence distance measuring algorithm based on optimal transport (OT) and cross-attention mechanism. Given a source sequence and a target sequence, we first calculate the ground distance between each pair of source and target terms of the two sequences. The ground distance is calculated over the subsequences around the two terms. We firstly pay attention from each the source terms to each target terms with attention weights, so that we have a representative source subsequence vector regarding each term in the target subsequence. Then, we pay attention from each representative vector of the term of the target subsequence to the entire source subsequence. In this way, we construct the cross-attention weights and use them to calculate the pairwise ground distances. With the ground distances, we derive the OT distance between the two sequences and train the attention parameters and ground distance metric parameters together. The training process is conducted with training triplets of sequences, where each triplet is composed of an anchor sequence, a must-link sequence, and a cannot-link sequence. The corresponding hinge loss function of each triplet is minimized, and we develop an iterative algorithm to solve the optimal transport problem and the attention/ground distance metric parameters in an alternate way. The experiments over sequence similarity search benchmark datasets, including text, video, and rice smut protein sequence data, are conducted. The experimental results show the algorithm is effective.

Pinball Loss Twin Support Vector Clustering

Twin Support Vector Clustering (TWSVC) is a clustering algorithm inspired by the principles of Twin Support Vector Machine (TWSVM). TWSVC has already outperformed other traditional plane based clustering algorithms. However, TWSVC uses hinge loss, which maximizes shortest distance between clusters and hence suffers from noise-sensitivity and low re-sampling stability. In this article, we propose Pinball loss Twin Support Vector Clustering (pinTSVC) as a clustering algorithm. The proposed pinTSVC model incorporates the pinball loss function in the plane clustering formulation. Pinball loss function introduces favorable properties such as noise-insensitivity and re-sampling stability. The time complexity of the proposed pinTSVC remains equivalent to that of TWSVC. Extensive numerical experiments on noise-corrupted benchmark UCI and artificial datasets have been provided. Results of the proposed pinTSVC model are compared with TWSVC, Twin Bounded Support Vector Clustering (TBSVC) and Fuzzy c-means clustering (FCM). Detailed and exhaustive comparisons demonstrate the better performance and generalization of the proposed pinTSVC for noise-corrupted datasets. Further experiments and analysis on the performance of the above-mentioned clustering algorithms on structural MRI (sMRI) images taken from the ADNI database, face clustering, and facial expression clustering have been done to demonstrate the effectiveness and feasibility of the proposed pinTSVC model.