Agricultural Machinery Virtual Assembly System Using Dynamic Gesture Recognitive Interaction Based on a CNN and LSTM Network

The agricultural machinery experiment is restricted by the crop production season. Missing the crop growth cycle will extend the machine development period. The use of virtual reality technology to complete preassembly and preliminary experiments can reduce the loss caused by this problem. To improve the intelligence and stability of virtual assembly, this paper proposed a more stable dynamic gesture cognition framework: the TCP/IP protocol constituted the network communication terminal, the leap motion-based vision system constituted the gesture data collection terminal, and the CNN-LSTM network constituted the dynamic gesture recognition classification terminal. The dynamic gesture recognition framework and the harvester virtual assembly platform formed a virtual assembly system to achieve gesture interaction. Through experimental analysis, the improved CNN-LSTM network had a small volume and could quickly establish a stable and accurate gesture recognition model with an average accuracy of 98.0% (±0.894). The assembly efficiency of the virtual assembly system with the framework was improved by approximately 15%. The results showed that the accuracy and stability of this model met the requirements, the corresponding assembly parts were robust in the virtual simulation environment of the whole machine, and the harvesting behaviour in the virtual reality scene was close to the real scene. The virtual assembly system under this framework provided technical support for unmanned farms and virtual experiments on agricultural machinery.

Multiple perspectives integration for virtual reality-aided assemblability assessment in complex virtual assembly environment

Virtual Reality (VR) has been proved as a promising tool for industrial design, but the traditional VR interface of first-person perspective (1PP) is not useful enough to support assemblability assessment in complex virtual assembly environments. In this paper, we proposed the multi-perspectives interface (MPI) which integrates the 1PP and the third-person perspective (3PP) using handheld World-in-Miniature (WIM). The MPI allows users to simulate the assembly operations in a natural manner similar to 1PP, while providing users with an overview of the assembly status through the WIM to assess the assemblability with superior spatial awareness. Two studies were included in this paper. The first study tested MPI in a general interaction task, which reveals stronger spatial awareness in MPI than in 1PP without the cost of losing natural interaction. The second study tested the performance, usability, and workload of MPI in an assemblability assessment task. The results show the advantages of MPI in the reachability evaluation. The contribution of this paper is exploring a novel interface for VR-aided assembly design and evaluation system.

Measurement uncertainty assessment for virtual assembly

Virtual assembly (VA) is a method for datum definition and quality prediction of assemblies considering local form deviations of relevant geometries. Point clouds of measured objects are registered in order to recreate the objects' hypothetical physical assembly state. By VA, the geometrical verification becomes more accurate and, thus, increasingly function oriented. The VA algorithm is a nonlinear, constrained derivate of the Gaussian best fit algorithm, where outlier points strongly influence the registration result. In order to assess the robustness of the developed algorithm, the propagation of measurement uncertainties through the nonlinear transformation due to VA is studied. The work compares selected propagation methods distinguished from their levels of abstraction. The results reveal larger propagated uncertainties by VA compared to the unconstrained Gaussian best fit.