Transfer Learning for Tabular Data

Deep learning models for tabular data are restricted to a specific table format. Computer vision models, on the other hand, have a broader applicability; they work on all images and can learn universal features. This allows them to be trained on enormous corpora and have very wide transferability and applicability. Inspired by these properties, this work presents an architecture that aims to capture useful patterns across arbitrary tables. The model is trained on randomly sampled subsets of features from a table, processed by a convolutional network. This internal representation captures feature interactions that appear in the table. Experimental results show that the embeddings produced by this model are useful and transferable across many commonly used machine learning benchmarks datasets. Specifically, that using the embeddings produced by the network as additional features, improves the performance of a number of classifiers.

Transfer Learning for Tabular Data

Deep learning models for tabular data are restricted to a specific table format. Computer vision models, on the other hand, have a broader applicability; they work on all images and can learn universal features. This allows them to be trained on enormous corpora and have very wide transferability and applicability. Inspired by these properties, this work presents an architecture that aims to capture useful patterns across arbitrary tables. The model is trained on randomly sampled subsets of features from a table, processed by a convolutional network. This internal representation captures feature interactions that appear in the table. Experimental results show that the embeddings produced by this model are useful and transferable across many commonly used machine learning benchmarks datasets. Specifically, that using the embeddings produced by the network as additional features, improves the performance of a number of classifiers.

SELNN: Effective Crossover Features by Attention Mechanism and Logarithmic Conversion for CTR

With the continuous increase of web resources, information overload is becoming more and more serious. Getting the information that meets the needs of a large amount of data has become an urgent problem. One of the popular studies in click-through rate prediction is to construct feature interactions to improve prediction accuracy. Traditional models with low-order interactions cannot adequately represent the intersection information, and deep models with undifferentiated feature intersections lack relevance. In this paper, we propose a model called SELNN, which improves the accuracy of click-through rate prediction through attention mechanism and logarithmic conversion. On the one hand, improving attention mechanisms learns the importance of each feature itself. On the other hand, combining logarithmic conversion structure with feedforward neural networks to learn different orders of feature interactions. The experimental results indicated that the AUC values of SELNN on the Criteo and Movielens-1M datasets were 81.46% and 87.36%, respectively. SELNN effectively improves the prediction accuracy and reduces the number of parameters and computational effort.

Semantic Explanation for Deep Neural Networks Using Feature Interactions

Given the promising results obtained by deep-learning techniques in multimedia analysis, the explainability of predictions made by networks has become important in practical applications. We present a method to generate semantic and quantitative explanations that are easily interpretable by humans. The previous work to obtain such explanations has focused on the contributions of each feature, taking their sum to be the prediction result for a target variable; the lack of discriminative power due to this simple additive formulation led to low explanatory performance. Our method considers not only individual features but also their interactions, for a more detailed interpretation of the decisions made by networks. The algorithm is based on the factorization machine, a prediction method that calculates factor vectors for each feature. We conducted experiments on multiple datasets with different models to validate our method, achieving higher performance than the previous work. We show that including interactions not only generates explanations but also makes them richer and is able to convey more information. We show examples of produced explanations in a simple visual format and verify that they are easily interpretable and plausible.

LCHI: Low-Order Correlation and High-Order Interaction Integrated Model Oriented to Network Intrusion Detection

Network intrusion poses a severe threat to the Internet of Things (IoT). Thus, it is essential to study information security protection technology in IoT. Learning sophisticated feature interactions is critical in improving detection accuracy for network intrusion. Despite significant progress, existing methods seem to have a strong bias towards single low- or high-order feature interaction. Moreover, they always extract all possible low-order interactions indiscriminately, introducing too much noise. To address the above problems, we propose a low-order correlation and high-order interaction (LCHI) integrated feature extraction model. First, we selectively extract the beneficial low-order correlation between the same-type features by the multivariate correlation analysis (MCA) model and attention mechanism. Second, we extract the complicated high-order feature interaction by the deep neural network (DNN) model. Finally, we emphasize both the low- and high-order feature interactions and incorporate them. Our LCHI model seamlessly combines the linearity of MCA in modeling lower-order feature correlation and the nonlinearity of DNN in modeling higher-order feature interaction. Conceptually, our LCHI is more expressive than the previous models. We carry on a series of experiments on the public wireless and wired network intrusion detection datasets. The experimental results show that LCHI improves 1.06%, 2.46%, 3.74%, 0.25%, 1.17%, and 0.64% on the AWID, NSL-KDD, UNSW-NB15, CICIDS 2017, CICIDS 2018, and DAPT 2020 datasets, respectively.

Spatial Attention Enhances Crowded Stimulus Encoding across Modeled Receptive Fields by Increasing Redundancy of Feature Representations

Any visual system, biological or artificial, must make a trade-off between the number of units used to represent the visual environment and the spatial resolution of the sampling array. Humans and some other animals are able to allocate attention to spatial locations to reconfigure the sampling array of receptive fields (RFs), thereby enhancing the spatial resolution of representations without changing the overall number of sampling units. Here, we examine how representations of visual features in a fully convolutional neural network interact and interfere with each other in an eccentricity-dependent RF pooling array and how these interactions are influenced by dynamic changes in spatial resolution across the array. We study these feature interactions within the framework of visual crowding, a well-characterized perceptual phenomenon in which target objects in the visual periphery that are easily identified in isolation are much more difficult to identify when flanked by similar nearby objects. By separately simulating effects of spatial attention on RF size and on the density of the pooling array, we demonstrate that the increase in RF density due to attention is more beneficial than changes in RF size for enhancing target classification for crowded stimuli. Furthermore, by varying target and flanker spacing, as well as the spatial extent of attention, we find that feature redundancy across RFs has more influence on target classification than the fidelity of the feature representations themselves. Based on these findings, we propose a candidate mechanism by which spatial attention relieves visual crowding through enhanced feature redundancy that is mostly due to increased RF density.

A Novel Integrated Network with LightGBM for Click-Through Rate Prediction

Click-through Rate (CTR) prediction has become one of the core tasks of the recommendation system and its online advertising with the development of e-commerce. In the CTR prediction field, different features extraction schemes are used to mine the user click behavior to achieve the maximum CTR, which helps the advertisers maximize their profits. At present, achievements have been made in CTR prediction based on Deep Neural Network (DNN), but insufficiently, DNN can only learn high-order features combination. In this paper, Product & Cross supported Stacking Network with LightGBM (PCSNL) is proposed for CTR prediction to solve such problems. Firstly, the L 1 and L 2 regularizations are imposed on Light Gradient Boosting Machine (LightGBM) to prevent overfitting. Secondly, the method of vector-wise feature interactions is added to product layer in product network to learn second-order feature combinations. Lastly, feature information is fully learned through the cross network, product network and stacking network in PCSNL. The online ads CTR prediction datasets released by Huawei and Avazu on the Kaggle platform are involved for experiments. It is shown that the PCSN model and PCSNL have better performance than the traditional CTR prediction models and deep learning models.