The Vicarios Virtual Reality Interface for Remote Robotic Teleoperation

Intuitive interaction is the cornerstone of accurate and effective performance in remote robotic teleoperation. It requires high-fidelity in control actions as well as perception (vision, haptic, and other sensory feedback) of the remote environment. This paper presents Vicarios, a Virtual Reality (VR) based interface with the aim of facilitating intuitive real-time remote teleoperation, while utilizing the inherent benefits of VR, including immersive visualization, freedom of user viewpoint selection, and fluidity of interaction through natural action interfaces. Vicarios aims to enhance the situational awareness, using the concept of viewpoint-independent mapping between the operator and the remote scene, thereby giving the operator better control in the perception-action loop. The article describes the overall system of Vicarios, with its software, hardware, and communication framework. A comparative user study quantifies the impact of the interface and its features, including immersion and instantaneous user viewpoint changes, termed “teleporting”, on users’ performance. The results show that users’ performance with the VR-based interface was either similar to or better than the baseline condition of traditional stereo video feedback, approving the realistic nature of the Vicarios interface. Furthermore, including the teleporting feature in VR significantly improved participants’ performance and their appreciation for it, which was evident in the post-questionnaire results. Vicarios capitalizes on the intuitiveness and flexibility of VR to improve accuracy in remote teleoperation.

Research on Determining the Inspection Point of Multirotor UAV Power Tower

During the inspection process of power pole towers, manual positioning is mainly used to select the shooting viewpoints of the drone, which leads to erroneous viewpoint selection and inaccurate shootings of inspected objects. Also, neglecting the effect of the sun’s backlight on photographs contributes to poor photo quality that does not meet inspection requirements. Aiming at the selection of shooting viewpoints during multirotor unmanned aerial vehicles’ inspection on power poles, this paper proposes an automatic positioning method that determines the shooting viewpoints by considering UAV performance, airborne camera parameters, and the size of objects to be measured. Considering the factors of sun illumination, we optimize the method to ensure the positions of the viewpoints and to ensure that the images can be clearly generated so that the observers can check the power pole towers through the images when shooting is also taken into consideration. Finally, the automatic calculation method of the related viewpoints is implemented in the Java language. Experiments show that the method can accurately obtain the positions of the drones’ viewpoints and reduce the number of viewpoints, which significantly improves the efficiency and quality of inspection shooting.

SALIENCY OF SUBTLE ENTITIES WITHIN 3-D POINT CLOUDS

Visual saliency is defined by regions of the scene that stand out from their neighbors and attract immediate attention. In image processing, visual saliency is frequently used to focus local analysis of key features. Though their advantage is largely acknowledged, little research has been carried concerning 3-D data, and even less in relation to data acquired by laser scanners for mapping. In this paper, we propose a new saliency measure for laser scanned point-clouds, governed by the neurological concepts of center-surround and low-level features. Adjusted to large point sets, we propose a fast geometric descriptor, which quantifies the distance of a point from its surrounding. We show that the proposed model highlights not only salient details in watertight models, but also in airborne and terrestrially scanned scenes that may hold subtle entities embedded within the topography. The detection of such regions paves the way to a myriad of applications, such as feature and pattern extraction, registration, classification, viewpoint selection, point-cloud simplification, landmark detection, etc.

Sorting Objects from a Conveyor Belt Using POMDPs with Multiple-Object Observations and Information-Gain Rewards

We consider a robot that must sort objects transported by a conveyor belt into different classes. Multiple observations must be performed before taking a decision on the class of each object, because the imperfect sensing sometimes detects the incorrect object class. The objective is to sort the sequence of objects in a minimal number of observation and decision steps. We describe this task in the framework of partially observable Markov decision processes, and we propose a reward function that explicitly takes into account the information gain of the viewpoint selection actions applied. The DESPOT algorithm is applied to solve the problem, automatically obtaining a sequence of observation viewpoints and class decision actions. Observations are made either only for the object on the first position of the conveyor belt or for multiple adjacent positions at once. The performance of the single- and multiple-position variants is compared, and the impact of including the information gain is analyzed. Real-life experiments with a Baxter robot and an industrial conveyor belt are provided.

Multi-View Visual Question Answering with Active Viewpoint Selection

This paper proposes a framework that allows the observation of a scene iteratively to answer a given question about the scene. Conventional visual question answering (VQA) methods are designed to answer given questions based on single-view images. However, in real-world applications, such as human–robot interaction (HRI), in which camera angles and occluded scenes must be considered, answering questions based on single-view images might be difficult. Since HRI applications make it possible to observe a scene from multiple viewpoints, it is reasonable to discuss the VQA task in multi-view settings. In addition, because it is usually challenging to observe a scene from arbitrary viewpoints, we designed a framework that allows the observation of a scene actively until the necessary scene information to answer a given question is obtained. The proposed framework achieves comparable performance to a state-of-the-art method in question answering and simultaneously decreases the number of required observation viewpoints by a significant margin. Additionally, we found our framework plausibly learned to choose better viewpoints for answering questions, lowering the required number of camera movements. Moreover, we built a multi-view VQA dataset based on real images. The proposed framework shows high accuracy (94.01%) for the unseen real image dataset.