Virtual reality (VR) is becoming increasingly popular in educational applications, but insufficient users’ feel of immersion often slows the further adoption of VR. Many solutions with a focus on the results rather than the details of the interactions between the objects in the real and virtual worlds have been developed. Therefore, the real procedures are distorted and the users lose their perception of in-person participation.
In order to improve the users’ feel of immersion further and to simulate more realistic operations in VR, a procedure-oriented approach for the combination of real and virtual environments is proposed here. As its name implies, this approach emphasizes the details of the procedures, namely how to capture, track, operate and interoperate the real and virtual objects in a mixed environment. In order to illustrate this idea, a prototype of mixed real and virtual assembly, in con-junction with object recognition and rigid-object tracking functions based on robotic vision techniques, is presented as an example. This prototype is designed based on a game-based virtual laoratory system, and the specific implementation is a planetary gear train experiment. In this experiment, all models of the parts with the information required for the assembly are created, labeled and added to the database of the virtual laboratory system. The physical parts are marked in order to facilitate object recognition and object tracking. During the experiment, the main assembly with one missing planetary gear is accomplished in a purely virtual environment. In the real world, the missing planetary gear is tracked by a Kinect while the user is manipulating this gear. Then, the system recognizes this gear based on the markers and couples the corresponding virtual model of that gear with the avatar’s hand in the virtual environment. Afterward, the cam-era tracks the real part, and the user can adjust its pose and location to finish the final assembly. The main benefit of this implementation is that the user can take advantage of some simple real parts in conjunction with virtual models of sophisticated parts in order to get realistic experience with the assembly process.
An Efficient and Robust Multi-Object Recognition and Tracking Algorithm using Mask Region based Convolution Neural Network (R-CNN)