Towards autoscaling of Apache Flink jobs

Data stream processing has been gaining attention in the past decade. Apache Flink is an open-source distributed stream processing engine that is able to process a large amount of data in real time with low latency. Computations are distributed among a cluster of nodes. Currently, provisioning the appropriate amount of cloud resources must be done manually ahead of time. A dynamically varying workload may exceed the capacity of the cluster, or leave resources underutilized. In our paper, we describe an architecture that enables the automatic scaling of Flink jobs on Kubernetes based on custom metrics, and describe a simple scaling policy. We also measure the e ects of state size and target parallelism on the duration of the scaling operation, which must be considered when designing an autoscaling policy, so that the Flink job respects a Service Level Agreement.

Optimizing Convolutional Neural Network Accelerator on Low-Cost FPGA

Convolutional neural network (CNN) is one of the most promising algorithms that outweighs other traditional methods in terms of accuracy in classification tasks. However, several CNNs, such as VGG, demand a huge computation in convolutional layers. Many accelerators implemented on powerful FPGAs have been introduced to address the problems. In this paper, we present a VGG-based accelerator which is optimized for a low-cost FPGA. In order to optimize the FPGA resource of logic element and memory, we propose a dedicated input buffer that maximizes the data reuse. In addition, we design a low resource processing engine with the optimal number of Multiply Accumulate (MAC) units. In the experiments, we use VGG16 model for inference to evaluate the performance of our accelerator and achieve a throughput of 38.8[Formula: see text]GOPS at a clock speed of 150[Formula: see text]MHz on Intel Cyclone V SX SoC. The experimental results show that our design is better than previous works in terms of resource efficiency.

Laurelin: Java-native ROOT I/O for Apache Spark

Apache Spark[1] is one of the predominant frameworks in the big data space, providing a fully-functional query processing engine, vendor support for hardware accelerators, and performant integrations with scientific computing libraries. One difficulty in adopting conventional big data frameworks to HEP workflows is the lack of support for the ROOT file format in these frameworks. Laurelin[6] implements ROOT I/O with a pure Java library, with no bindings to the C++ ROOT[2] implementation, and is readily installable via standard Java packaging tools. It provides a performant interface enabling Spark to read (and soon write) ROOT TTrees, enabling users to process these data without a pre-processing phase converting to an intermediate format.

Traditional village protection based on big data under the impact of COVID-19

Under the influence of novel corona virus pneumonia epidemic, the protection of traditional villages is put forward higher request. The spread of the epidemic among villages will make the situation of epidemic prevention and control more difficult. As an important part of culture, traditional villages have high historical value. In this paper, the traditional village protection method, a new geographical data algorithm IData storage method. Compared with the traditional ArcGIS method, it improves the efficiency and accuracy of topographic map entry. IData’s data factory can use the symbolic technology of skeleton lines to represent all the figures in the national standard mode, and any complex figure can only be represented by one element. Idate can quickly load data and render symbols in a drawing. With the powerful data processing engine of IData data factory, we can check out the errors that other software can’t find and process the data automatically. Records of the loss of traditional villages can be recorded quickly. The establishment and protection of traditional villages have had a beneficial impact.