Preparing Distributed Computing Operations for the HL-LHC Era With Operational Intelligence

As a joint effort from various communities involved in the Worldwide LHC Computing Grid, the Operational Intelligence project aims at increasing the level of automation in computing operations and reducing human interventions. The distributed computing systems currently deployed by the LHC experiments have proven to be mature and capable of meeting the experimental goals, by allowing timely delivery of scientific results. However, a substantial number of interventions from software developers, shifters, and operational teams is needed to efficiently manage such heterogenous infrastructures. Under the scope of the Operational Intelligence project, experts from several areas have gathered to propose and work on “smart” solutions. Machine learning, data mining, log analysis, and anomaly detection are only some of the tools we have evaluated for our use cases. In this community study contribution, we report on the development of a suite of operational intelligence services to cover various use cases: workload management, data management, and site operations.

Real-time Visibility, Operational Insights and Actions with SAP Operational Process Intelligence, powered by HANA

Over the past decade, business operations in organizations are increasingly generating vast amounts of data. This steep increase in operational data – often known as big data – is a result of organizations capturing data at more fine-granular levels in business operations. Leveraging operational big data to improve business operations can create a sustainable competitive advantage to organizations. In this paper, we argue that operational big data is ”dead data” unless it is contextualized and made actionable for line-of-business users. Moreover, we explain how organizations can use SAP Operational Process Intelligence and the power of HANA to first, contextualize operational big data for real-time operational intelligence, second, provide line-of-business users visibility into business operations and business situations as they evolve and third, propose appropriate actions to respond to these business situations and help them reach business outcomes.

Tailoring Mission Effectiveness and Efficiency of a Ground Vehicle Using Exergy-Based Model Predictive Control (MPC)

To ensure dominance over a multi-domain battlespace, energy and power utilization must be accurately characterized for the dissimilar operational conditions. Using MATLAB/Simulink in combination with multiple neural networks, we created a methodology which was simulated the energy dynamics of a ground vehicle in parallel to running predictive neural network (NN) based predictive algorithms to address two separate research questions: (1) can energy and exergy flow characterization be developed at a future point in time, and (2) can we use the predictive algorithms to extend the energy and exergy flow characterization and derive operational intelligence, used to inform our control based algorithms or provide optimized recommendations to a battlefield commander in real-time. Using our predictive algorithms we confirmed that the future energy and exergy flow characterizations could be generated using the NNs, which was validated through simulation using two separately created datasets, one for training and one for testing. We then used the NNs to implement a model predictive control (MPC) framework to flexibly operate the vehicles thermal coolant loop (TCL), subject to exergy destruction. In this way we could tailor the performance of the vehicle to accommodate a more mission effective solution or a less energy intensive solution. The MPC resulted in a more effective solution when compared to six other simulated conditions, which consumed less exergy than two of the six cases. Our results indicate that we can derive operational intelligence from the predictive algorithms and use it to inform a model predictive control (MPC) framework to reduce wasted energy and exergy destruction subject to the variable operating conditions.

Real-Time High-Load Infrastructure Transaction Status Output Prediction Using Operational Intelligence and Big Data Technologies

An approach to use Operational Intelligence with mathematical modeling and Machine Learning to solve industrial technology projects problems are very crucial for today’s IT (information technology) processes and operations, taking into account the exponential growth of information and the growing trend of Big Data-based projects. Monitoring and managing high-load data projects require new approaches to infrastructure, risk management, and data-driven decision support. Key difficulties that might arise when performing IT Operations are high error rates, unplanned downtimes, poor infrastructure KPIs and metrics. The methods used in the study include machine learning models, data preprocessing, missing data imputation, SRE (site reliability engineering) indicators computation, quantitative research, and a qualitative study of data project demands. A requirements analysis for the implementation of an Operational Intelligence solution with Machine learning capabilities has been conducted and represented in the study. A model based on machine learning algorithms for transaction status code and output predictions, in order to execute system load testing, risks identification and, to avoid downtimes, is developed. Metrics and indicators for determining infrastructure load are given in the paper to obtain Operational intelligence and Site reliability insights. It turned out that data mining among the set of Operational Big Data simplifies the task of getting an understanding of what is happening with requests within the data acquisition pipeline and helps identify errors before a user faces them. Transaction tracing in a distributed environment has been enhanced using machine learning and mathematical modelling. Additionally, a step-by-step algorithm for applying the application monitoring solution in a data-based project, especially when it is dealing with Big Data is described and proposed within the study.

Operational Intelligence for Distributed Computing Systems for Exascale Science

In the near future, large scientific collaborations will face unprecedented computing challenges. Processing and storing exabyte datasets require a federated infrastructure of distributed computing resources. The current systems have proven to be mature and capable of meeting the experiment goals, by allowing timely delivery of scientific results. However, a substantial amount of interventions from software developers, shifters and operational teams is needed to efficiently manage such heterogeneous infrastructures. A wealth of operational data can be exploited to increase the level of automation in computing operations by using adequate techniques, such as machine learning (ML), tailored to solve specific problems. The Operational Intelligence project is a joint effort from various WLCG communities aimed at increasing the level of automation in computing operations. We discuss how state-of-the-art technologies can be used to build general solutions to common problems and to reduce the operational cost of the experiment computing infrastructure.