scholarly journals Automatic Reconstruction of Building-Scale Indoor 3D Environment with a Deep-Reinforcement-Learning-Based Mobile Robot

2021 ◽  
Vol 6 ◽  
pp. 11-23
Author(s):  
Menglong Yang ◽  
Katashi Nagao
Keyword(s):  
Reinforcement Learning ◽  
Mobile Robot ◽  
Real World ◽  
Low Cost ◽  
Three Dimensional ◽  
Point Clouds ◽  
Autonomous Agent ◽  
Entire Area ◽  
The Real ◽  
3D Point Clouds

The aim of this paper is to digitize the environments in which humans live, at low cost, and reconstruct highly accurate three-dimensional environments that are based on those in the real world. This three-dimensional content can be used such as for virtual reality environments and three-dimensional maps for automatic driving systems. In general, however, a three-dimensional environment must be carefully reconstructed by manually moving the sensors used to first scan the real environment on which the three-dimensional one is based. This is done so that every corner of an entire area can be measured, but time and costs increase as the area expands. Therefore, a system that creates three-dimensional content that is based on real-world large-scale buildings at low cost is proposed. This involves automatically scanning the indoors with a mobile robot that uses low-cost sensors and generating 3D point clouds. When the robot reaches an appropriate measurement position, it collects the three-dimensional data of shapes observable from that position by using a 3D sensor and 360-degree panoramic camera. The problem of determining an appropriate measurement position is called the “next best view problem,” and it is difficult to solve in a complicated indoor environment. To deal with this problem, a deep reinforcement learning method is employed. It combines reinforcement learning, with which an autonomous agent learns strategies for selecting behavior, and deep learning done using a neural network. As a result, 3D point cloud data can be generated with better quality than the conventional rule-based approach.

scholarly journals Estimating tree stem diameters and volume from smartphone photogrammetric point clouds

2019 ◽  
Vol 93 (3) ◽  
pp. 411-429 ◽  
Author(s):  
Maria Immacolata Marzulli ◽  
Pasi Raumonen ◽  
Roberto Greco ◽  
Manuela Persia ◽  
Patrizia Tartarino
Keyword(s):  
Low Cost ◽  
Three Dimensional ◽  
Laser Scanner ◽  
Point Clouds ◽  
Stem Volume ◽  
Forest Plot ◽  
3D Point Clouds ◽  
Dense Point ◽  
Minimum Number ◽  
Scale Point

Abstract Methods for the three-dimensional (3D) reconstruction of forest trees have been suggested for data from active and passive sensors. Laser scanner technologies have become popular in the last few years, despite their high costs. Since the improvements in photogrammetric algorithms (e.g. structure from motion—SfM), photographs have become a new low-cost source of 3D point clouds. In this study, we use images captured by a smartphone camera to calculate dense point clouds of a forest plot using SfM. Eighteen point clouds were produced by changing the densification parameters (Image scale, Point density, Minimum number of matches) in order to investigate their influence on the quality of the point clouds produced. In order to estimate diameter at breast height (d.b.h.) and stem volumes, we developed an automatic method that extracts the stems from the point cloud and then models them with cylinders. The results show that Image scale is the most influential parameter in terms of identifying and extracting trees from the point clouds. The best performance with cylinder modelling from point clouds compared to field data had an RMSE of 1.9 cm and 0.094 m3, for d.b.h. and volume, respectively. Thus, for forest management and planning purposes, it is possible to use our photogrammetric and modelling methods to measure d.b.h., stem volume and possibly other forest inventory metrics, rapidly and without felling trees. The proposed methodology significantly reduces working time in the field, using ‘non-professional’ instruments and automating estimates of dendrometric parameters.

scholarly journals ACCURACY ASSESSMENT OF UNDERWATER PHOTOGRAMMETRIC THREE DIMENSIONAL MODELLING FOR CORAL REEFS

2016 ◽  
Vol XLI-B5 ◽  
pp. 821-828 ◽  
Author(s):  
T. Guo ◽  
A. Capra ◽  
M. Troyer ◽  
A. Gruen ◽  
A. J. Brooks ◽  
...  
Keyword(s):  
Coral Cover ◽  
Accuracy Assessment ◽  
Low Cost ◽  
Three Dimensional ◽  
Point Clouds ◽  
3D Modelling ◽  
Relative Accuracy ◽  
3D Point Clouds ◽  
The Individual ◽  
Working Distance

Recent advances in automation of photogrammetric 3D modelling software packages have stimulated interest in reconstructing highly accurate 3D object geometry in unconventional environments such as underwater utilizing simple and low-cost camera systems. The accuracy of underwater 3D modelling is affected by more parameters than in single media cases. This study is part of a larger project on 3D measurements of temporal change of coral cover in tropical waters. It compares the accuracies of 3D point clouds generated by using images acquired from a system camera mounted in an underwater housing and the popular GoPro cameras respectively. A precisely measured calibration frame was placed in the target scene in order to provide accurate control information and also quantify the errors of the modelling procedure. In addition, several objects (cinder blocks) with various shapes were arranged in the air and underwater and 3D point clouds were generated by automated image matching. These were further used to examine the relative accuracy of the point cloud generation by comparing the point clouds of the individual objects with the objects measured by the system camera in air (the best possible values). Given a working distance of about 1.5 m, the GoPro camera can achieve a relative accuracy of 1.3 mm in air and 2.0 mm in water. The system camera achieved an accuracy of 1.8 mm in water, which meets our requirements for coral measurement in this system.

scholarly journals A Self-Assembly Portable Mobile Mapping System for Archeological Reconstruction Based on VSLAM-Photogrammetric Algorithm

Sensors ◽  
2019 ◽  
Vol 19 (18) ◽  
pp. 3952 ◽  
Author(s):  
* ◽  
*
Keyword(s):  
Self Assembly ◽  
Average Distance ◽  
Low Cost ◽  
Three Dimensional ◽  
Point Clouds ◽  
3D Models ◽  
Data Capture ◽  
Information Modeling ◽  
3D Point Clouds ◽  
Mean Square Errors

Three Dimensional (3D) models are widely used in clinical applications, geosciences, cultural heritage preservation, and engineering; this, together with new emerging needs such as building information modeling (BIM) develop new data capture techniques and devices with a low cost and reduced learning curve that allow for non-specialized users to employ it. This paper presents a simple, self-assembly device for 3D point clouds data capture with an estimated base price under €2500; furthermore, a workflow for the calculations is described that includes a Visual SLAM-photogrammetric threaded algorithm that has been implemented in C++. Another purpose of this work is to validate the proposed system in BIM working environments. To achieve it, in outdoor tests, several 3D point clouds were obtained and the coordinates of 40 points were obtained by means of this device, with data capture distances ranging between 5 to 20 m. Subsequently, those were compared to the coordinates of the same targets measured by a total station. The Euclidean average distance errors and root mean square errors (RMSEs) ranging between 12–46 mm and 8–33 mm respectively, depending on the data capture distance (5–20 m). Furthermore, the proposed system was compared with a commonly used photogrammetric methodology based on Agisoft Metashape software. The results obtained demonstrate that the proposed system satisfies (in each case) the tolerances of ‘level 1’ (51 mm) and ‘level 2’ (13 mm) for point cloud acquisition in urban design and historic documentation, according to the BIM Guide for 3D Imaging (U.S. General Services).

scholarly journals A study on recognizing multi-real world object and estimating 3D position in augmented reality

2021 ◽  
Author(s):  
Taemin Lee ◽  
Changhun Jung ◽  
Kyungtaek Lee ◽  
Sanghyun Seo
Keyword(s):  
Augmented Reality ◽  
Real World ◽  
Visual Information ◽  
Point Cloud ◽  
Three Dimensional ◽  
Point Clouds ◽  
Location Estimation ◽  
Virtual Space ◽  
Camera Tracking ◽  
The Real

AbstractAs augmented reality technologies develop, real-time interactions between objects present in the real world and virtual space are required. Generally, recognition and location estimation in augmented reality are carried out using tracking techniques, typically markers. However, using markers creates spatial constraints in simultaneous tracking of space and objects. Therefore, we propose a system that enables camera tracking in the real world and visualizes virtual visual information through the recognition and positioning of objects. We scanned the space using an RGB-D camera. A three-dimensional (3D) dense point cloud map is created using point clouds generated through video images. Among the generated point cloud information, objects are detected and retrieved based on the pre-learned data. Finally, using the predicted pose of the detected objects, other information may be augmented. Our system estimates object recognition and 3D pose based on simple camera information, enabling the viewing of virtual visual information based on object location.

scholarly journals Enhancing Image-Based Multiscale Heritage Recording with Near-Infrared Data

2020 ◽  
Vol 9 (4) ◽  
pp. 269 ◽  
Author(s):  
Efstathios Adamopoulos ◽  
Fulvio Rinaudo
Keyword(s):  
Near Infrared ◽  
Low Cost ◽  
Three Dimensional ◽  
Point Clouds ◽  
Archaeological Site ◽  
Short Review ◽  
Additional Information ◽  
3D Point Clouds ◽  
Heritage Science ◽  
Imaging Sensors

Passive sensors, operating in the visible (VIS) spectrum, have widely been used towards the trans-disciplinary documentation, understanding, and protection of tangible cultural heritage (CH). Although, many heritage science fields benefit significantly from additional information that can be acquired in the near-infrared (NIR) spectrum. NIR imagery, captured for heritage applications, has been mostly investigated with two-dimensional (2D) approaches or by 2D-to-three-dimensional (3D) integrations following complicated techniques, including expensive imaging sensors and setups. The availability of high-resolution digital modified cameras and software implementations of Structure-from-Motion (SfM) and Multiple-View-Stereo (MVS) algorithms, has made the production of models with spectral textures more feasible than ever. In this research, a short review of image-based 3D modeling with NIR data is attempted. The authors aim to investigate the use of near-infrared imagery from relatively low-cost modified sensors for heritage digitization, alongside the usefulness of spectral textures produced, oriented towards heritage science. Therefore, thorough experimentation and assessment with different software are conducted and presented, utilizing NIR imagery and SfM/MVS methods. Dense 3D point clouds and textured meshes have been produced and evaluated for their metric validity and radiometric quality, comparing to results produced from VIS imagery. The datasets employed come from heritage assets of different dimensions, from an archaeological site to a medium-sized artwork, to evaluate implementation on different levels of accuracy and specifications of texture resolution.

scholarly journals ACCURACY ASSESSMENT OF UNDERWATER PHOTOGRAMMETRIC THREE DIMENSIONAL MODELLING FOR CORAL REEFS

2016 ◽  
Vol XLI-B5 ◽  
pp. 821-828 ◽  
Author(s):  
T. Guo ◽  
A. Capra ◽  
M. Troyer ◽  
A. Gruen ◽  
A. J. Brooks ◽  
...  
Keyword(s):  
Coral Cover ◽  
Accuracy Assessment ◽  
Low Cost ◽  
Three Dimensional ◽  
Point Clouds ◽  
3D Modelling ◽  
Relative Accuracy ◽  
3D Point Clouds ◽  
The Individual ◽  
Working Distance

Recent advances in automation of photogrammetric 3D modelling software packages have stimulated interest in reconstructing highly accurate 3D object geometry in unconventional environments such as underwater utilizing simple and low-cost camera systems. The accuracy of underwater 3D modelling is affected by more parameters than in single media cases. This study is part of a larger project on 3D measurements of temporal change of coral cover in tropical waters. It compares the accuracies of 3D point clouds generated by using images acquired from a system camera mounted in an underwater housing and the popular GoPro cameras respectively. A precisely measured calibration frame was placed in the target scene in order to provide accurate control information and also quantify the errors of the modelling procedure. In addition, several objects (cinder blocks) with various shapes were arranged in the air and underwater and 3D point clouds were generated by automated image matching. These were further used to examine the relative accuracy of the point cloud generation by comparing the point clouds of the individual objects with the objects measured by the system camera in air (the best possible values). Given a working distance of about 1.5 m, the GoPro camera can achieve a relative accuracy of 1.3 mm in air and 2.0 mm in water. The system camera achieved an accuracy of 1.8 mm in water, which meets our requirements for coral measurement in this system.

scholarly journals Monitoring Selective Logging in a Pine-Dominated Forest in Central Germany with Repeated Drone Flights Utilizing A Low Cost RTK Quadcopter

Drones ◽  
2020 ◽  
Vol 4 (2) ◽  
pp. 11 ◽  
Author(s):  
Christian Thiel ◽  
Marlin M. Müller ◽  
Christian Berger ◽  
Felix Cremer ◽  
Clémence Dubois ◽  
...  
Keyword(s):  
Low Cost ◽  
Three Dimensional ◽  
Point Clouds ◽  
Selective Logging ◽  
Ease Of Use ◽  
Tree Level ◽  
Unmanned Aerial Systems ◽  
3D Point Clouds ◽  
Individual Trees ◽  
New Generation

There is no doubt that unmanned aerial systems (UAS) will play an increasing role in Earth observation in the near future. The field of application is very broad and includes aspects of environmental monitoring, security, humanitarian aid, or engineering. In particular, drones with camera systems are already widely used. The capability to compute ultra-high-resolution orthomosaics and three-dimensional (3D) point clouds from UAS imagery generates a wide interest in such systems, not only in the science community, but also in industry and agencies. In particular, forestry sciences benefit from ultra-high-structural and spectral information as regular tree level-based monitoring becomes feasible. There is a great need for this kind of information as, for example, due to the spring and summer droughts in Europe in the years 2018/2019, large quantities of individual trees were damaged or even died. This study focuses on selective logging at the level of individual trees using repeated drone flights. Using the new generation of UAS, which allows for sub-decimeter-level positioning accuracies, a change detection approach based on bi-temporal UAS acquisitions was implemented. In comparison to conventional UAS, the effort of implementing repeated drone flights in the field was low, because no ground control points needed to be surveyed. As shown in this study, the geometrical offset between the two collected datasets was below 10 cm across the site, which enabled a direct comparison of both datasets without the need for post-processing (e.g., image matching). For the detection of logged trees, we utilized the spectral and height differences between both acquisitions. For their delineation, an object-based approach was employed, which was proven to be highly accurate (precision = 97.5%; recall = 91.6%). Due to the ease of use of such new generation, off-the-shelf consumer drones, their decreasing purchase costs, the quality of available workflows for data processing, and the convincing results presented here, UAS-based data can and should complement conventional forest inventory practices.

scholarly journals How to train your robot with deep reinforcement learning: lessons we have learned

2021 ◽  
pp. 027836492098785
Author(s):  
Julian Ibarz ◽  
Jie Tan ◽  
Chelsea Finn ◽  
Mrinal Kalakrishnan ◽  
Peter Pastor ◽  
...  
Keyword(s):  
Machine Learning ◽  
Reinforcement Learning ◽  
Case Studies ◽  
Real World ◽  
Review Article ◽  
The Real ◽  
Complex Skills ◽  
Real World Learning ◽  
Level Sensor ◽  
Embodied Agent

Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low-level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time, real-world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn: as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.

Robotic motion compensation for bone movement, using ultrasound images

2015 ◽  
Vol 42 (5) ◽  
pp. 466-474 ◽  
Author(s):  
P.M.B. Torres ◽  
P. J. S. Gonçalves ◽  
J.M.M. Martins
Keyword(s):  
Motion Compensation ◽  
Optical Measurement ◽  
Three Dimensional ◽  
Point Clouds ◽  
Robotic System ◽  
Ultrasound Images ◽  
Bone Drilling ◽  
Content Type ◽  
3D Point Clouds ◽  
Robotic Motion

Purpose – The purpose of this paper is to present a robotic motion compensation system, using ultrasound images, to assist orthopedic surgery. The robotic system can compensate for femur movements during bone drilling procedures. Although it may have other applications, the system was thought to be used in hip resurfacing (HR) prosthesis surgery to implant the initial guide tool. The system requires no fiducial markers implanted in the patient, by using only non-invasive ultrasound images. Design/methodology/approach – The femur location in the operating room is obtained by processing ultrasound (USA) and computer tomography (CT) images, obtained, respectively, in the intra-operative and pre-operative scenarios. During surgery, the bone position and orientation is obtained by registration of USA and CT three-dimensional (3D) point clouds, using an optical measurement system and also passive markers attached to the USA probe and to the drill. The system description, image processing, calibration procedures and results with simulated and real experiments are presented and described to illustrate the system in operation. Findings – The robotic system can compensate for femur movements, during bone drilling procedures. In most experiments, the update was always validated, with errors of 2 mm/4°. Originality/value – The navigation system is based entirely on the information extracted from images obtained from CT pre-operatively and USA intra-operatively. Contrary to current surgical systems, it does not use any type of implant in the bone to track the femur movements.

scholarly journals Comparison of Depth Camera and Terrestrial Laser Scanner in Monitoring Structural Deflections

Sensors ◽  
2020 ◽  
Vol 21 (1) ◽  
pp. 201
Author(s):  
Michael Bekele Maru ◽  
Donghwan Lee ◽  
Kassahun Demissie Tola ◽  
Seunghee Park
Keyword(s):  
Optical Sensors ◽  
Point Cloud ◽  
Three Dimensional ◽  
Laser Scanner ◽  
Point Clouds ◽  
Depth Camera ◽  
Terrestrial Laser Scanner ◽  
Cloud Data ◽  
3D Point Clouds ◽  
3D Information

Modeling a structure in the virtual world using three-dimensional (3D) information enhances our understanding, while also aiding in the visualization, of how a structure reacts to any disturbance. Generally, 3D point clouds are used for determining structural behavioral changes. Light detection and ranging (LiDAR) is one of the crucial ways by which a 3D point cloud dataset can be generated. Additionally, 3D cameras are commonly used to develop a point cloud containing many points on the external surface of an object around it. The main objective of this study was to compare the performance of optical sensors, namely a depth camera (DC) and terrestrial laser scanner (TLS) in estimating structural deflection. We also utilized bilateral filtering techniques, which are commonly used in image processing, on the point cloud data for enhancing their accuracy and increasing the application prospects of these sensors in structure health monitoring. The results from these sensors were validated by comparing them with the outputs from a linear variable differential transformer sensor, which was mounted on the beam during an indoor experiment. The results showed that the datasets obtained from both the sensors were acceptable for nominal deflections of 3 mm and above because the error range was less than ±10%. However, the result obtained from the TLS were better than those obtained from the DC.

Sign in / Sign up

Export Citation Format

Share Document