On the Reproducibility and Replicability of Deep Learning in Software Engineering

Context: Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years. This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge. Objective: Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors: (1) reproducibility —whether the reported experimental results can be obtained by other researchers using authors’ artifacts (i.e., source code and datasets) with the same experimental setup; and (2) replicability —whether the reported experimental result can be obtained by other researchers using their re-implemented artifacts with a different experimental setup. We observed that DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work. This is mainly due to high model complexity with many manually set parameters and the time-consuming optimization process, unlike classical supervised machine learning (ML) methods (e.g., random forest). This study aims to investigate the urgency and importance of reproducibility and replicability for DL studies on SE tasks. Method: In this study, we conducted a literature review on 147 DL studies recently published in 20 SE venues and 20 AI (Artificial Intelligence) venues to investigate these issues. We also re-ran four representative DL models in SE to investigate important factors that may strongly affect the reproducibility and replicability of a study. Results: Our statistics show the urgency of investigating these two factors in SE, where only 10.2% of the studies investigate any research question to show that their models can address at least one issue of replicability and/or reproducibility. More than 62.6% of the studies do not even share high-quality source code or complete data to support the reproducibility of their complex models. Meanwhile, our experimental results show the importance of reproducibility and replicability, where the reported performance of a DL model could not be reproduced for an unstable optimization process. Replicability could be substantially compromised if the model training is not convergent, or if performance is sensitive to the size of vocabulary and testing data. Conclusion: It is urgent for the SE community to provide a long-lasting link to a high-quality reproduction package, enhance DL-based solution stability and convergence, and avoid performance sensitivity on different sampled data.

A Modular End-to-End Framework for Secure Firmware Updates on Embedded Systems

Firmware refers to device read-only resident code which includes microcode and macro-instruction-level routines. For Internet-of-Things (IoT) devices without an operating system, firmware includes all the necessary instructions on how such embedded systems operate and communicate. Thus, firmware updates are essential parts of device functionality. They provide the ability to patch vulnerabilities, address operational issues, and improve device reliability and performance during the lifetime of the system. This process, however, is often exploited by attackers in order to inject malicious firmware code into the embedded device. In this article, we present a framework for secure firmware updates on embedded systems. This approach is based on hardware primitives and cryptographic modules, and it can be deployed in environments where communication channels might be insecure. The implementation of the framework is flexible, as it can be adapted in regards to the IoT device’s available hardware resources and constraints. Our security analysis shows that our framework is resilient to a variety of attack vectors. The experimental setup demonstrates the feasibility of the approach. By implementing a variety of test cases on FPGA, we demonstrate the adaptability and performance of the framework. Experiments indicate that the update procedure for a 1183-kB firmware image could be achieved, in a secure manner, under 1.73 seconds.

Flipping food during grilling tasks, a dataset of utensils kinematics and dynamics, food pose and subject gaze

This paper presents a multivariate dataset of 2866 food flipping movements, performed by 4 chefs and 5 home cooks, with different grilled food and two utensils (spatula and tweezers). The 3D trajectories of strategic points in the utensils were tracked using optoelectronic motion capture. The pinching force of the tweezers, the bending force and torsion torque of the spatula were also recorded, as well as videos and the subject gaze. These data were collected using a custom experimental setup that allowed the execution of flipping movements with freshly cooked food, without having the sensors near the dangerous cooking area. Complementary, the 2D position of food was computed from the videos. The action of flipping food is, indeed, gaining the attention of both researchers and manufacturers of foodservice technology. The reported dataset contains valuable measurements (1) to characterize and model flipping movements as performed by humans, (2) to develop bio-inspired methods to control a cooking robot, or (3) to study new algorithms for human actions recognition.

Supplemental Material: History of earthquakes along the creeping section of the San Andreas fault, California, USA

Supplemental figures related to biomarker analysis, thermal modeling, and experimental setup; more detailed methodology, and tables containing model parameters and biomarker data.<br>

The Effects of Group differentiation by students’ learning strategies

This study analyses the effects of group differentiation by students’ learning strategies of around 1200 students in 46 classes from eight secondary schools in the Netherlands. In an experimental setup with randomization at the class level, division of students over three groups per class (an instruction-independent group, an average group, and an instruction-dependent group) is based on learning strategies, measures using the Motivated Strategies for Learning Questionnaire (MSLQ). Each group is offered instruction fitting their own learning strategy. The results show that student performance is higher in classes where the differentiation was applied, and that these students score higher at some scales of the posttest of the questionnaire on motivation, metacognition and self-regulation. However, there are differences between classrooms from different teachers. Additional teacher questionnaires confirm the discrepancy in teacher attitudes towards the intervention.

Supplemental Material: History of earthquakes along the creeping section of the San Andreas fault, California, USA

Supplemental figures related to biomarker analysis, thermal modeling, and experimental setup; more detailed methodology, and tables containing model parameters and biomarker data.<br>

Some Theoretical and Experimental Extensions Based on the Properties of the Intrinsic Transfer Matrix

The work approaches new theoretical and experimental studies in the elastic characterization of materials, based on the properties of the intrinsic transfer matrix. The term ‘intrinsic transfer matrix’ was firstly introduced by us in order to characterize the system in standing wave case, when the stationary wave is confined inside the sample. An important property of the intrinsic transfer matrix is that at resonance, and in absence of attenuation, the eigenvalues are real. This property underlies a numerical method which permits to find the phase velocity for the longitudinal wave in a sample. This modal approach is a numerical method which takes into account the eigenvalues, which are analytically estimated for simple elastic systems. Such elastic systems are characterized by a simple distribution of eigenmodes, which may be easily highlighted by experiment. The paper generalizes the intrinsic transfer matrix method by including the attenuation and a study of the influence of inhomogeneity. The condition for real eigenvalues in that case shows that the frequencies of eigenmodes are not affected by attenuation. For the influence of inhomogeneity, we consider a case when the sound speed is varying along the layer’s length in the medium of interest, with an accompanying dispersion. The paper also studies the accuracy of the method in estimating the wave velocity and determines an optimal experimental setup in order to reduce the influence of frequency errors.

Spline-based specimen shape optimization for robust material model calibration

Identification from field measurements allows several parameters to be identified from a single test, provided that the measurements are sensitive enough to the parameters to be identified. To do this, authors use empirically defined geometries (with holes, notches...). The first attempts to optimize the specimen to maximize the sensitivity of the measurement are linked to a design space that is either very small (parametric optimization), which does not allow the exploration of very different designs, or, conversely, very large (topology optimization), which sometimes leads to designs that are not regular and cannot be manufactured. In this paper, we propose an intermediate approach based on a non-invasive CAD-inspired optimization strategy which relies on the definition of univariate spline Free-Form Deformation boxes to reduce the design space and thus regularize the problem. Then, from the modeling point of view, we propose a new objective function that takes into account the experimental setup and we add constraint functions that ensure that the gain is real and the shape physically sound. Several examples show that with this method and at low cost, one can significantly improve the identification of constitutive parameters without changing the experimental setup.