Towards machine learning analytics for jet substructure

Abstract The past few years have seen a rapid development of machine-learning algorithms. While surely augmenting performance, these complex tools are often treated as black-boxes and may impair our understanding of the physical processes under study. The aim of this paper is to move a first step into the direction of applying expert-knowledge in particle physics to calculate the optimal decision function and test whether it is achieved by standard training, thus making the aforementioned black-box more transparent. In particular, we consider the binary classification problem of discriminating quark-initiated jets from gluon-initiated ones. We construct a new version of the widely used N-subjettiness, which features a simpler theoretical behaviour than the original one, while maintaining, if not exceeding, the discrimination power. We input these new observables to the simplest possible neural network, i.e. the one made by a single neuron, or perceptron, and we analytically study the network behaviour at leading logarithmic accuracy. We are able to determine under which circumstances the perceptron achieves optimal performance. We also compare our analytic findings to an actual implementation of a perceptron and to a more realistic neural network and find very good agreement.

Download Full-text

Near-Optimal Sparse Neural Trees for Supervised Learning

10.20944/preprints202105.0117.v2 ◽

2021 ◽

Author(s):

Tanujit Chakraborty

Keyword(s):

Neural Network ◽

Machine Learning ◽

Decision Tree ◽

Classification Tree ◽

Mathematical Formulation ◽

Machine Learning Algorithms ◽

Optimal Decision ◽

Feed Forward ◽

Feed Forward Network ◽

Tree Algorithms

Decision tree algorithms have been among the most popular algorithms for interpretable (transparent) machine learning since the early 1980s. On the other hand, deep learning methods have boosted the capacity of machine learning algorithms and are now being used for non-trivial applications in various applied domains. But training a fully-connected deep feed-forward network by gradient-descent backpropagation is slow and requires arbitrary choices regarding the number of hidden units and layers. In this paper, we propose near-optimal neural regression trees, intending to make it much faster than deep feed-forward networks and for which it is not essential to specify the number of hidden units in the hidden layers of the neural network in advance. The key idea is to construct a decision tree and then simulate the decision tree with a neural network. This work aims to build a mathematical formulation of neural trees and gain the complementary benefits of both sparse optimal decision trees and neural trees. We propose near-optimal sparse neural trees (NSNT) that is shown to be asymptotically consistent and robust in nature. Additionally, the proposed NSNT model obtain a fast rate of convergence which is near-optimal up to some logarithmic factor. We comprehensively benchmark the proposed method on a sample of 80 datasets (40 classification datasets and 40 regression datasets) from the UCI machine learning repository. We establish that the proposed method is likely to outperform the current state-of-the-art methods (random forest, XGBoost, optimal classification tree, and near-optimal nonlinear trees) for the majority of the datasets.

Download Full-text

Near-optimal Sparse Neural Trees

10.20944/preprints202105.0117.v1 ◽

2021 ◽

Author(s):

Tanujit Chakraborty ◽

Tanmoy Chakraborty

Keyword(s):

Neural Network ◽

Machine Learning ◽

Decision Tree ◽

Classification Tree ◽

Mathematical Formulation ◽

Machine Learning Algorithms ◽

Optimal Decision ◽

Feed Forward ◽

Feed Forward Network ◽

Tree Algorithms

Decision tree algorithms have been among the most popular algorithms for interpretable (transparent) machine learning since the early 1980s. On the other hand, deep learning methods have boosted the capacity of machine learning algorithms and are now being used for non-trivial applications in various applied domains. But training a fully-connected deep feed-forward network by gradient-descent backpropagation is slow and requires arbitrary choices regarding the number of hidden units and layers. In this paper, we propose near-optimal neural regression trees, intending to make it much faster than deep feed-forward networks and for which it is not essential to specify the number of hidden units in the hidden layers of the neural network in advance. The key idea is to construct a decision tree and then simulate the decision tree with a neural network. This work aims to build a mathematical formulation of neural trees and gain the complementary benefits of both sparse optimal decision trees and neural trees. We propose near-optimal sparse neural trees (NSNT) that is shown to be asymptotically consistent and robust in nature. Additionally, the proposed NSNT model obtain a fast rate of convergence which is near-optimal upto some logarithmic factor. We comprehensively benchmark the proposed method on a sample of 80 datasets (40 classification datasets and 40 regression datasets) from the UCI machine learning repository. We establish that the proposed method is likely to outperform the current state-of-the-art methods (random forest, XGBoost, optimal classification tree, and near-optimal nonlinear trees) for the majority of the datasets.

Download Full-text

Product Sentiment Analysis for Amazon Reviews

International Journal of Computer Science and Information Technology ◽

10.5121/ijcsit.2021.13302 ◽

2021 ◽

Vol 13 (3) ◽

pp. 15-30

Author(s):

Arwa S. M. AlQahtani

Keyword(s):

Machine Learning ◽

Short Term Memory ◽

Binary Classification ◽

Rapid Development ◽

Performance Model ◽

Machine Learning Algorithms ◽

Sentiment Classification ◽

Learning Approaches ◽

Customer Reviews ◽

Online Customer Reviews

Recently, Ecommerce has Witnessed Rapid Development. As A Result, Online Purchasing has grown, and that has led to Growth in Online Customer Reviews of Products. The Implied Opinions in Customer Reviews Have a Massive Influence on Customer's Decision Purchasing, Since the Customer's Opinion About the Product is Influenced by Other Consumers' Recommendations or Complaints. This Research Provides an Analysis of the Amazon Reviews Dataset and Studies Sentiment Classification with Different Machine Learning Approaches. First, the Reviews were Transformed into Vector Representation using different Techniques, I.E., Bag-Of-Words, Tf-Idf, and Glove. Then, we Trained Various Machine Learning Algorithms, I.E., Logistic Regression, Random Forest, Naïve Bayes, Bidirectional Long-Short Term Memory, and Bert. After That, We Evaluated the Models using Accuracy, F1-Score, Precision, Recall, and Cross-Entropy Loss Function. Then, We Analyized The Best Performance Model in Order to Investigate Its Sentiment Classification. The Experiment was Conducted on Multiclass Classifications, Then we Selected the Best Performing Model And Re-Trained It on the Binary Classification.

Download Full-text

Machine-Learning-Based Radiomics MRI Model for Survival Prediction of Recurrent Glioblastomas Treated with Bevacizumab

Diagnostics ◽

10.3390/diagnostics11071263 ◽

2021 ◽

Vol 11 (7) ◽

pp. 1263

Author(s):

Samy Ammari ◽

Raoul Sallé de Chou ◽

Tarek Assi ◽

Mehdi Touat ◽

Emilie Chouzenoux ◽

...

Keyword(s):

Machine Learning ◽

Therapeutic Option ◽

Binary Classification ◽

Progression Free Survival ◽

Recurrent Glioblastoma ◽

Machine Learning Algorithms ◽

Survival Prediction ◽

Classification Models ◽

Angiogenic Therapy ◽

Recurrent Gbm

Anti-angiogenic therapy with bevacizumab is a widely used therapeutic option for recurrent glioblastoma (GBM). Nevertheless, the therapeutic response remains highly heterogeneous among GBM patients with discordant outcomes. Recent data have shown that radiomics, an advanced recent imaging analysis method, can help to predict both prognosis and therapy in a multitude of solid tumours. The objective of this study was to identify novel biomarkers, extracted from MRI and clinical data, which could predict overall survival (OS) and progression-free survival (PFS) in GBM patients treated with bevacizumab using machine-learning algorithms. In a cohort of 194 recurrent GBM patients (age range 18–80), radiomics data from pre-treatment T2 FLAIR and gadolinium-injected MRI images along with clinical features were analysed. Binary classification models for OS at 9, 12, and 15 months were evaluated. Our classification models successfully stratified the OS. The AUCs were equal to 0.78, 0.85, and 0.76 on the test sets (0.79, 0.82, and 0.87 on the training sets) for the 9-, 12-, and 15-month endpoints, respectively. Regressions yielded a C-index of 0.64 (0.74) for OS and 0.57 (0.69) for PFS. These results suggest that radiomics could assist in the elaboration of a predictive model for treatment selection in recurrent GBM patients.

Download Full-text

TPOT-NN: augmenting tree-based automated machine learning with neural network estimators

Genetic Programming and Evolvable Machines ◽

10.1007/s10710-021-09401-z ◽

2021 ◽

Author(s):

Joseph D. Romano ◽

Trang T. Le ◽

Weixuan Fu ◽

Jason H. Moore

Keyword(s):

Neural Network ◽

Machine Learning ◽

Binary Classification ◽

Inductive Learning ◽

Future Directions ◽

High Performing ◽

Learning Tasks ◽

Benchmark Datasets ◽

Automated Machine Learning ◽

Standard Tree

AbstractAutomated machine learning (AutoML) and artificial neural networks (ANNs) have revolutionized the field of artificial intelligence by yielding incredibly high-performing models to solve a myriad of inductive learning tasks. In spite of their successes, little guidance exists on when to use one versus the other. Furthermore, relatively few tools exist that allow the integration of both AutoML and ANNs in the same analysis to yield results combining both of their strengths. Here, we present TPOT-NN—a new extension to the tree-based AutoML software TPOT—and use it to explore the behavior of automated machine learning augmented with neural network estimators (AutoML+NN), particularly when compared to non-NN AutoML in the context of simple binary classification on a number of public benchmark datasets. Our observations suggest that TPOT-NN is an effective tool that achieves greater classification accuracy than standard tree-based AutoML on some datasets, with no loss in accuracy on others. We also provide preliminary guidelines for performing AutoML+NN analyses, and recommend possible future directions for AutoML+NN methods research, especially in the context of TPOT.

Download Full-text

Studying the Potential of Graphcore® IPUs for Applications in Particle Physics

Computing and Software for Big Science ◽

10.1007/s41781-021-00057-z ◽

2021 ◽

Vol 5 (1) ◽

Author(s):

Samuel Maddrell-Mander ◽

Lakshan Ram Madhan Mohan ◽

Alexander Marshall ◽

Daniel O’Hanlon ◽

Konstantinos Petridis ◽

...

Keyword(s):

Neural Network ◽

Machine Learning ◽

Kalman Filter ◽

Particle Physics ◽

Processing Unit ◽

Network Architectures ◽

Track Reconstruction ◽

Event Simulation ◽

Scattering Correction ◽

New Type

AbstractThis paper presents the first study of Graphcore’s Intelligence Processing Unit (IPU) in the context of particle physics applications. The IPU is a new type of processor optimised for machine learning. Comparisons are made for neural-network-based event simulation, multiple-scattering correction, and flavour tagging, implemented on IPUs, GPUs and CPUs, using a variety of neural network architectures and hyperparameters. Additionally, a Kálmán filter for track reconstruction is implemented on IPUs and GPUs. The results indicate that IPUs hold considerable promise in addressing the rapidly increasing compute needs in particle physics.

Download Full-text

Internet financial supervision based on machine learning and improved neural network

Journal of Intelligent & Fuzzy Systems ◽

10.3233/jifs-189555 ◽

2020 ◽

pp. 1-12

Author(s):

Cao Yanli

Keyword(s):

Neural Network ◽

Machine Learning ◽

Credit Risk ◽

Machine Learning Algorithms ◽

Financial Supervision ◽

Loan Pricing ◽

Comprehensive Management ◽

Risk Pricing ◽

Basel Agreement ◽

Relevant Factors

The research on the risk pricing of Internet finance online loans not only enriches the theory and methods of online loan pricing, but also helps to improve the level of online loan risk pricing. In order to improve the efficiency of Internet financial supervision, this article builds an Internet financial supervision system based on machine learning algorithms and improved neural network algorithms. Moreover, on the basis of factor analysis and discretization of loan data, this paper selects the relatively mature Logistic regression model to evaluate the credit risk of the borrower and considers the comprehensive management of credit risk and the matching with income. In addition, according to the relevant provisions of the New Basel Agreement on expected losses and economic capital, starting from the relevant factors, this article combines the credit risk assessment results to obtain relevant factors through regional research and conduct empirical analysis. The research results show that the model constructed in this paper has certain reliability.

Download Full-text

EMOTIONS RECOGNITION IN HUMAN SPEECH USING DEEP NEURAL NETWORKS

Vestnik komp iuternykh i informatsionnykh tekhnologii ◽

10.14489/vkit.2021.01.pp.044-051 ◽

2021 ◽

pp. 44-51

Author(s):

E. Yu. Shchetinin

Keyword(s):

Neural Network ◽

Machine Learning ◽

Neural Networks ◽

Convolutional Neural Network ◽

Recurrent Neural Network ◽

Deep Neural Networks ◽

Learning Algorithms ◽

Machine Learning Algorithms ◽

Audio Recordings ◽

Computer Studies

The recognition of human emotions is one of the most relevant and dynamically developing areas of modern speech technologies, and the recognition of emotions in speech (RER) is the most demanded part of them. In this paper, we propose a computer model of emotion recognition based on an ensemble of bidirectional recurrent neural network with LSTM memory cell and deep convolutional neural network ResNet18. In this paper, computer studies of the RAVDESS database containing emotional speech of a person are carried out. RAVDESS-a data set containing 7356 files. Entries contain the following emotions: 0 – neutral, 1 – calm, 2 – happiness, 3 – sadness, 4 – anger, 5 – fear, 6 – disgust, 7 – surprise. In total, the database contains 16 classes (8 emotions divided into male and female) for a total of 1440 samples (speech only). To train machine learning algorithms and deep neural networks to recognize emotions, existing audio recordings must be pre-processed in such a way as to extract the main characteristic features of certain emotions. This was done using Mel-frequency cepstral coefficients, chroma coefficients, as well as the characteristics of the frequency spectrum of audio recordings. In this paper, computer studies of various models of neural networks for emotion recognition are carried out on the example of the data described above. In addition, machine learning algorithms were used for comparative analysis. Thus, the following models were trained during the experiments: logistic regression (LR), classifier based on the support vector machine (SVM), decision tree (DT), random forest (RF), gradient boosting over trees – XGBoost, convolutional neural network CNN, recurrent neural network RNN (ResNet18), as well as an ensemble of convolutional and recurrent networks Stacked CNN-RNN. The results show that neural networks showed much higher accuracy in recognizing and classifying emotions than the machine learning algorithms used. Of the three neural network models presented, the CNN + BLSTM ensemble showed higher accuracy.

Download Full-text

Detection of Spoofing Attack using Machine Learning based on Multi-Layer Neural Network in Single-Frequency GPS Receivers

Journal of Navigation ◽

10.1017/s0373463317000558 ◽

2017 ◽

Vol 71 (1) ◽

pp. 169-188 ◽

Cited By ~ 12

Author(s):

E. Shafiee ◽

M. R. Mosavi ◽

M. Moazedi

Keyword(s):

Neural Network ◽

Machine Learning ◽

Late Phase ◽

Machine Learning Algorithms ◽

Computational Method ◽

Single Frequency ◽

Detection Accuracy ◽

Gps Receiver ◽

Navigation Data ◽

Spoofing Detection

The importance of the Global Positioning System (GPS) and related electronic systems continues to increase in a range of environmental, engineering and navigation applications. However, civilian GPS signals are vulnerable to Radio Frequency (RF) interference. Spoofing is an intentional intervention that aims to force a GPS receiver to acquire and track invalid navigation data. Analysis of spoofing and authentic signal patterns represents the differences as phase, energy and imaginary components of the signal. In this paper, early-late phase, delta, and signal level as the three main features are extracted from the correlation output of the tracking loop. Using these features, spoofing detection can be performed by exploiting conventional machine learning algorithms such as K-Nearest Neighbourhood (KNN) and naive Bayesian classifier. A Neural Network (NN) as a learning machine is a modern computational method for collecting the required knowledge and predicting the output values in complicated systems. This paper presents a new approach for GPS spoofing detection based on multi-layer NN whose inputs are indices of features. Simulation results on a software GPS receiver showed adequate detection accuracy was obtained from NN with a short detection time.

Download Full-text

A Bayesian neural network predicts the dissolution of compact planetary systems

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.2026053118 ◽

2021 ◽

Vol 118 (40) ◽

pp. e2026053118

Author(s):

Miles Cranmer ◽

Daniel Tamayo ◽

Hanno Rein ◽

Peter Battaglia ◽

Samuel Hadden ◽

...

Keyword(s):

Neural Network ◽

Machine Learning ◽

Time Series ◽

Planetary System ◽

Machine Learning Algorithms ◽

Orbital Elements ◽

Bayesian Neural Network ◽

Inference Model ◽

Link Type ◽

Numerical Integrator

We introduce a Bayesian neural network model that can accurately predict not only if, but also when a compact planetary system with three or more planets will go unstable. Our model, trained directly from short N-body time series of raw orbital elements, is more than two orders of magnitude more accurate at predicting instability times than analytical estimators, while also reducing the bias of existing machine learning algorithms by nearly a factor of three. Despite being trained on compact resonant and near-resonant three-planet configurations, the model demonstrates robust generalization to both nonresonant and higher multiplicity configurations, in the latter case outperforming models fit to that specific set of integrations. The model computes instability estimates up to 105 times faster than a numerical integrator, and unlike previous efforts provides confidence intervals on its predictions. Our inference model is publicly available in the SPOCK (https://github.com/dtamayo/spock) package, with training code open sourced (https://github.com/MilesCranmer/bnn_chaos_model).

Download Full-text