Pantomime

We introduce Pantomime, a novel mid-air gesture recognition system exploiting spatio-temporal properties of millimeter-wave radio frequency (RF) signals. Pantomime is positioned in a unique region of the RF landscape: mid-resolution mid-range high-frequency sensing, which makes it ideal for motion gesture interaction. We configure a commercial frequency-modulated continuous-wave radar device to promote spatial information over the temporal resolution by means of sparse 3D point clouds and contribute a deep learning architecture that directly consumes the point cloud, enabling real-time performance with low computational demands. Pantomime achieves 95% accuracy and 99% AUC in a challenging set of 21 gestures articulated by 41 participants in two indoor environments, outperforming four state-of-the-art 3D point cloud recognizers. We further analyze the effect of the environment in 5 different indoor environments, the effect of articulation speed, angle, and the distance of the person up to 5m. We have publicly made available the collected mmWave gesture dataset consisting of nearly 22,000 gesture instances along with our radar sensor configuration, trained models, and source code for reproducibility. We conclude that pantomime is resilient to various input conditions and that it may enable novel applications in industrial, vehicular, and smart home scenarios.

Download Full-text

Fast and Robust Flight Altitude Estimation of Multirotor UAVs in Dynamic Unstructured Environments Using 3D Point Cloud Sensors

Aerospace ◽

10.3390/aerospace5030094 ◽

2018 ◽

Vol 5 (3) ◽

pp. 94 ◽

Cited By ~ 5

Author(s):

Hriday Bavle ◽

Jose Sanchez-Lopez ◽

Paloma Puente ◽

Alejandro Rodriguez-Ramos ◽

Carlos Sampedro ◽

...

Keyword(s):

Point Cloud ◽

Point Clouds ◽

Estimation Algorithm ◽

Indoor Environments ◽

3D Point Cloud ◽

Flight Altitude ◽

Cloud Data ◽

3D Point Clouds ◽

Laser Altimeters ◽

Altitude Estimation

This paper presents a fast and robust approach for estimating the flight altitude of multirotor Unmanned Aerial Vehicles (UAVs) using 3D point cloud sensors in cluttered, unstructured, and dynamic indoor environments. The objective is to present a flight altitude estimation algorithm, replacing the conventional sensors such as laser altimeters, barometers, or accelerometers, which have several limitations when used individually. Our proposed algorithm includes two stages: in the first stage, a fast clustering of the measured 3D point cloud data is performed, along with the segmentation of the clustered data into horizontal planes. In the second stage, these segmented horizontal planes are mapped based on the vertical distance with respect to the point cloud sensor frame of reference, in order to provide a robust flight altitude estimation even in presence of several static as well as dynamic ground obstacles. We validate our approach using the IROS 2011 Kinect dataset available in the literature, estimating the altitude of the RGB-D camera using the provided 3D point clouds. We further validate our approach using a point cloud sensor on board a UAV, by means of several autonomous real flights, closing its altitude control loop using the flight altitude estimated by our proposed method, in presence of several different static as well as dynamic ground obstacles. In addition, the implementation of our approach has been integrated in our open-source software framework for aerial robotics called Aerostack.

Download Full-text

INDOOR PHOTOGRAMMETRY AIDED WITH UWB NAVIGATION

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-2-683-2018 ◽

2018 ◽

Vol XLII-2 ◽

pp. 683-690 ◽

Cited By ~ 1

Author(s):

A. Masiero ◽

F. Fissore ◽

A. Guarnieri ◽

A. Vettore

Keyword(s):

Point Cloud ◽

Inertial Sensors ◽

Point Clouds ◽

External Information ◽

Indoor Environments ◽

Metric Reconstruction ◽

North East ◽

3D Point Clouds ◽

Significant Interest ◽

The Subject

The subject of photogrammetric surveying with mobile devices, in particular smartphones, is becoming of significant interest in the research community. Nowadays, the process of providing 3D point clouds with photogrammetric procedures is well known. However, external information is still typically needed in order to move from the point cloud obtained from images to a 3D metric reconstruction. This paper investigates the integration of information provided by an UWB positioning system with visual based reconstruction to produce a metric reconstruction. Furthermore, the orientation (with respect to North-East directions) of the obtained model is assessed thanks to the use of inertial sensors included in the considered UWB devices. Results of this integration are shown on two case studies in indoor environments.

Download Full-text

GENERATION OF GROUND TRUTH DATASETS FOR THE ANALYSIS OF 3D POINT CLOUDS IN URBAN SCENES ACQUIRED VIA DIFFERENT SENSORS

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xlii-3-2009-2018 ◽

2018 ◽

Vol XLII-3 ◽

pp. 2009-2015 ◽

Cited By ~ 1

Author(s):

Y. Xu ◽

Z. Sun ◽

R. Boerner ◽

T. Koch ◽

L. Hoegner ◽

...

Keyword(s):

Point Cloud ◽

Spatial Information ◽

Ground Truth ◽

Semantic Segmentation ◽

Point Clouds ◽

Urban Scenes ◽

3D Space ◽

3D Point Clouds ◽

Testing Site ◽

Voxel Grid

In this work, we report a novel way of generating ground truth dataset for analyzing point cloud from different sensors and the validation of algorithms. Instead of directly labeling large amount of 3D points requiring time consuming manual work, a multi-resolution 3D voxel grid for the testing site is generated. Then, with the help of a set of basic labeled points from the reference dataset, we can generate a 3D labeled space of the entire testing site with different resolutions. Specifically, an octree-based voxel structure is applied to voxelize the annotated reference point cloud, by which all the points are organized by 3D grids of multi-resolutions. When automatically annotating the new testing point clouds, a voting based approach is adopted to the labeled points within multiple resolution voxels, in order to assign a semantic label to the 3D space represented by the voxel. Lastly, robust line- and plane-based fast registration methods are developed for aligning point clouds obtained via various sensors. Benefiting from the labeled 3D spatial information, we can easily create new annotated 3D point clouds of different sensors of the same scene directly by considering the corresponding labels of 3D space the points located, which would be convenient for the validation and evaluation of algorithms related to point cloud interpretation and semantic segmentation.

Download Full-text

BRUTE FORCE MATCHING BETWEEN CAMERA SHOTS AND SYNTHETIC IMAGES FROM POINT CLOUDS

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprsarchives-xli-b5-771-2016 ◽

2016 ◽

Vol XLI-B5 ◽

pp. 771-777 ◽

Cited By ~ 1

Author(s):

R. Boerner ◽

M. Kröhnert

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Spatial Information ◽

Point Clouds ◽

Synthetic Image ◽

3D Point Cloud ◽

Point Cloud Data ◽

Cloud Data ◽

Force Matching ◽

3D Point Clouds

3D point clouds, acquired by state-of-the-art terrestrial laser scanning techniques (TLS), provide spatial information about accuracies up to several millimetres. Unfortunately, common TLS data has no spectral information about the covered scene. However, the matching of TLS data with images is important for monoplotting purposes and point cloud colouration. Well-established methods solve this issue by matching of close range images and point cloud data by fitting optical camera systems on top of laser scanners or rather using ground control points. <br><br> The approach addressed in this paper aims for the matching of 2D image and 3D point cloud data from a freely moving camera within an environment covered by a large 3D point cloud, e.g. a 3D city model. The key advantage of the free movement affects augmented reality applications or real time measurements. Therefore, a so-called real image, captured by a smartphone camera, has to be matched with a so-called synthetic image which consists of reverse projected 3D point cloud data to a synthetic projection centre whose exterior orientation parameters match the parameters of the image, assuming an ideal distortion free camera.

Download Full-text

INTEGRATION OF 3D POINT CLOUDS WITH SEMANTIC 3D CITY MODELS – PROVIDING SEMANTIC INFORMATION BEYOND CLASSIFICATION

ISPRS Annals of Photogrammetry Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-annals-viii-4-w2-2021-105-2021 ◽

2021 ◽

Vol VIII-4/W2-2021 ◽

pp. 105-112

Author(s):

C. Beil ◽

T. Kutzner ◽

B. Schwab ◽

B. Willenborg ◽

A. Gawronski ◽

...

Keyword(s):

Point Cloud ◽

Semantic Information ◽

Spatial Information ◽

Point Clouds ◽

Semantic Knowledge ◽

Point Cloud Data ◽

Cloud Data ◽

3D City Models ◽

3D Point Clouds ◽

City Models

Abstract. A range of different and increasingly accessible acquisition methods, the possibility for frequent data updates of large areas, and a simple data structure are some of the reasons for the popularity of three-dimensional (3D) point cloud data. While there are multiple techniques for segmenting and classifying point clouds, capabilities of common data formats such as LAS for providing semantic information are mostly limited to assigning points to a certain category (classification). However, several fields of application, such as digital urban twins used for simulations and analyses, require more detailed semantic knowledge. This can be provided by semantic 3D city models containing hierarchically structured semantic and spatial information. Although semantic models are often reconstructed from point clouds, they are usually geometrically less accurate due to generalization processes. First, point cloud data structures / formats are discussed with respect to their semantic capabilities. Then, a new approach for integrating point clouds with semantic 3D city models is presented, consequently combining respective advantages of both data types. In addition to elaborate (and established) semantic concepts for several thematic areas, the new version 3.0 of the international Open Geospatial Consortium (OGC) standard CityGML also provides a PointCloud module. In this paper a scheme is shown, how CityGML 3.0 can be used to provide semantic structures for point clouds (directly or stored in a separate LAS file). Methods and metrics to automatically assign points to corresponding Level of Detail (LoD)2 or LoD3 models are presented. Subsequently, dataset examples implementing these concepts are provided for download.

Download Full-text

Geometric Modelling for 3D Point Clouds of Elbow Joints in Piping Systems

Sensors ◽

10.3390/s20164594 ◽

2020 ◽

Vol 20 (16) ◽

pp. 4594

Author(s):

Ting On Chan ◽

Linyuan Xia ◽

Derek D. Lichti ◽

Yeran Sun ◽

Jun Wang ◽

...

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Spatial Information ◽

Geometric Model ◽

Point Clouds ◽

Piping System ◽

Random Errors ◽

Registration Accuracy ◽

Cloud Data ◽

3D Point Clouds

Pipe elbow joints exist in almost every piping system supporting many important applications such as clean water supply. However, spatial information of the elbow joints is rarely extracted and analyzed from observations such as point cloud data obtained from laser scanning due to lack of a complete geometric model that can be applied to different types of joints. In this paper, we proposed a novel geometric model and several model adaptions for typical elbow joints including the 90° and 45° types, which facilitates the use of 3D point clouds of the elbow joints collected from laser scanning. The model comprises translational, rotational, and dimensional parameters, which can be used not only for monitoring the joints’ geometry but also other applications such as point cloud registrations. Both simulated and real datasets were used to verify the model, and two applications derived from the proposed model (point cloud registration and mounting bracket detection) were shown. The results of the geometric fitting of the simulated datasets suggest that the model can accurately recover the geometry of the joint with very low translational (0.3 mm) and rotational (0.064°) errors when ±0.02 m random errors were introduced to coordinates of a simulated 90° joint (with diameter equal to 0.2 m). The fitting of the real datasets suggests that the accuracy of the diameter estimate reaches 97.2%. The joint-based registration accuracy reaches sub-decimeter and sub-degree levels for the translational and rotational parameters, respectively.

Download Full-text

BRUTE FORCE MATCHING BETWEEN CAMERA SHOTS AND SYNTHETIC IMAGES FROM POINT CLOUDS

ISPRS - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences ◽

10.5194/isprs-archives-xli-b5-771-2016 ◽

2016 ◽

Vol XLI-B5 ◽

pp. 771-777

Author(s):

R. Boerner ◽

M. Kröhnert

Keyword(s):

Point Cloud ◽

Laser Scanning ◽

Spatial Information ◽

Point Clouds ◽

Synthetic Image ◽

3D Point Cloud ◽

Point Cloud Data ◽

Cloud Data ◽

Force Matching ◽

3D Point Clouds

Download Full-text

Symmetry Analysis of Oriental Polygonal Pagodas Using 3D Point Clouds for Cultural Heritage

Sensors ◽

10.3390/s21041228 ◽

2021 ◽

Vol 21 (4) ◽

pp. 1228

Author(s):

Ting On Chan ◽

Linyuan Xia ◽

Yimin Chen ◽

Wei Lang ◽

Tingting Chen ◽

...

Keyword(s):

Cultural Heritage ◽

Unmanned Aerial Vehicle ◽

Point Cloud ◽

Geometric Model ◽

Digital Camera ◽

Point Clouds ◽

Symmetry Analysis ◽

3D Point Clouds ◽

Transmission Towers ◽

Aerial Vehicle

Ancient pagodas are usually parts of hot tourist spots in many oriental countries due to their unique historical backgrounds. They are usually polygonal structures comprised by multiple floors, which are separated by eaves. In this paper, we propose a new method to investigate both the rotational and reflectional symmetry of such polygonal pagodas through developing novel geometric models to fit to the 3D point clouds obtained from photogrammetric reconstruction. The geometric model consists of multiple polygonal pyramid/prism models but has a common central axis. The method was verified by four datasets collected by an unmanned aerial vehicle (UAV) and a hand-held digital camera. The results indicate that the models fit accurately to the pagodas’ point clouds. The symmetry was realized by rotating and reflecting the pagodas’ point clouds after a complete leveling of the point cloud was achieved using the estimated central axes. The results show that there are RMSEs of 5.04 cm and 5.20 cm deviated from the perfect (theoretical) rotational and reflectional symmetries, respectively. This concludes that the examined pagodas are highly symmetric, both rotationally and reflectionally. The concept presented in the paper not only work for polygonal pagodas, but it can also be readily transformed and implemented for other applications for other pagoda-like objects such as transmission towers.

Download Full-text

MATLAB Virtual Toolbox for Retrospective Rockfall Source Detection and Volume Estimation Using 3D Point Clouds: A Case Study of a Subalpine Molasse Cliff

Geosciences ◽

10.3390/geosciences11020075 ◽

2021 ◽

Vol 11 (2) ◽

pp. 75

Author(s):

Dario Carrea ◽

Antonio Abellan ◽

Marc-Henri Derron ◽

Neal Gauvin ◽

Michel Jaboyedoff

Keyword(s):

Point Cloud ◽

Natural Hazard ◽

Point Clouds ◽

Volume Estimation ◽

Natural Phenomena ◽

Frequency Distributions ◽

Erosion Rates ◽

3D Point Clouds

The use of 3D point clouds to improve the understanding of natural phenomena is currently applied in natural hazard investigations, including the quantification of rockfall activity. However, 3D point cloud treatment is typically accomplished using nondedicated (and not optimal) software. To fill this gap, we present an open-source, specific rockfall package in an object-oriented toolbox developed in the MATLAB® environment. The proposed package offers a complete and semiautomatic 3D solution that spans from extraction to identification and volume estimations of rockfall sources using state-of-the-art methods and newly implemented algorithms. To illustrate the capabilities of this package, we acquired a series of high-quality point clouds in a pilot study area referred to as the La Cornalle cliff (West Switzerland), obtained robust volume estimations at different volumetric scales, and derived rockfall magnitude–frequency distributions, which assisted in the assessment of rockfall activity and long-term erosion rates. An outcome of the case study shows the influence of the volume computation on the magnitude–frequency distribution and ensuing erosion process interpretation.

Download Full-text

Comparison of Depth Camera and Terrestrial Laser Scanner in Monitoring Structural Deflections

Sensors ◽

10.3390/s21010201 ◽

2020 ◽

Vol 21 (1) ◽

pp. 201

Author(s):

Michael Bekele Maru ◽

Donghwan Lee ◽

Kassahun Demissie Tola ◽

Seunghee Park

Keyword(s):

Optical Sensors ◽

Point Cloud ◽

Three Dimensional ◽

Laser Scanner ◽

Point Clouds ◽

Depth Camera ◽

Terrestrial Laser Scanner ◽

Cloud Data ◽

3D Point Clouds ◽

3D Information

Modeling a structure in the virtual world using three-dimensional (3D) information enhances our understanding, while also aiding in the visualization, of how a structure reacts to any disturbance. Generally, 3D point clouds are used for determining structural behavioral changes. Light detection and ranging (LiDAR) is one of the crucial ways by which a 3D point cloud dataset can be generated. Additionally, 3D cameras are commonly used to develop a point cloud containing many points on the external surface of an object around it. The main objective of this study was to compare the performance of optical sensors, namely a depth camera (DC) and terrestrial laser scanner (TLS) in estimating structural deflection. We also utilized bilateral filtering techniques, which are commonly used in image processing, on the point cloud data for enhancing their accuracy and increasing the application prospects of these sensors in structure health monitoring. The results from these sensors were validated by comparing them with the outputs from a linear variable differential transformer sensor, which was mounted on the beam during an indoor experiment. The results showed that the datasets obtained from both the sensors were acceptable for nominal deflections of 3 mm and above because the error range was less than ±10%. However, the result obtained from the TLS were better than those obtained from the DC.

Download Full-text