scholarly journals Robust Loop Closure Detection Integrating Visual–Spatial–Semantic Information via Topological Graphs and CNN Features

2020 ◽  
Vol 12 (23) ◽  
pp. 3890
Author(s):  
Yuwei Wang ◽  
Yuanying Qiu ◽  
Peitao Cheng ◽  
Xuechao Duan
Keyword(s):  
Semantic Information ◽  
Graph Matching ◽  
Spatial Relationships ◽  
Dynamic Scenes ◽  
Topological Graphs ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Visual Spatial ◽  
Localization And Mapping ◽  
The Impact

Loop closure detection is a key module for visual simultaneous localization and mapping (SLAM). Most previous methods for this module have not made full use of the information provided by images, i.e., they have only used the visual appearance or have only considered the spatial relationships of landmarks; the visual, spatial and semantic information have not been fully integrated. In this paper, a robust loop closure detection approach integrating visual–spatial–semantic information is proposed by employing topological graphs and convolutional neural network (CNN) features. Firstly, to reduce mismatches under different viewpoints, semantic topological graphs are introduced to encode the spatial relationships of landmarks, and random walk descriptors are employed to characterize the topological graphs for graph matching. Secondly, dynamic landmarks are eliminated by using semantic information, and distinctive landmarks are selected for loop closure detection, thus alleviating the impact of dynamic scenes. Finally, to ease the effect of appearance changes, the appearance-invariant descriptor of the landmark region is extracted by a pre-trained CNN without the specially designed manual features. The proposed approach weakens the influence of viewpoint changes and dynamic scenes, and extensive experiments conducted on open datasets and a mobile robot demonstrated that the proposed method has more satisfactory performance compared to state-of-the-art methods.

scholarly journals Long-Term Loop Closure Detection through Visual-Spatial Information Preserving Multi-Order Graph Matching

2020 ◽  
Vol 34 (06) ◽  
pp. 10369-10376
Author(s):  
Peng Gao ◽  
Hao Zhang
Keyword(s):  
Spatial Information ◽  
Graph Matching ◽  
Fundamental Problem ◽  
Graph Representation ◽  
Spatial Relationships ◽  
Matching Problem ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Visual Spatial

Loop closure detection is a fundamental problem for simultaneous localization and mapping (SLAM) in robotics. Most of the previous methods only consider one type of information, based on either visual appearances or spatial relationships of landmarks. In this paper, we introduce a novel visual-spatial information preserving multi-order graph matching approach for long-term loop closure detection. Our approach constructs a graph representation of a place from an input image to integrate visual-spatial information, including visual appearances of the landmarks and the background environment, as well as the second and third-order spatial relationships between two and three landmarks, respectively. Furthermore, we introduce a new formulation that formulates loop closure detection as a multi-order graph matching problem to compute a similarity score directly from the graph representations of the query and template images, instead of performing conventional vector-based image matching. We evaluate the proposed multi-order graph matching approach based on two public long-term loop closure detection benchmark datasets, including the St. Lucia and CMU-VL datasets. Experimental results have shown that our approach is effective for long-term loop closure detection and it outperforms the previous state-of-the-art methods.

scholarly journals Incremental Loop Closure Verification by Guided Sampling

2017 ◽  
Vol 21 (1) ◽  
pp. 59-66
Author(s):  
Tanaka Kanji ◽  
Keyword(s):  
Image Retrieval ◽  
Good Precision ◽  
Sampling Strategies ◽  
Detection Problem ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Robot Trajectory ◽  
Localization And Mapping ◽  
Random Sample Consensus ◽  
Closure Constraints

Loop closure detection, which is the task of identifying locations revisited by a robot in a sequence of odometry and perceptual observations, is typically formulated as a combination of two subtasks: (1) bag-of-words image retrieval and (2) post-verification using random sample consensus (RANSAC) geometric verification. The main contribution of this study is the proposal of a novel post-verification framework that achieves good precision recall trade-off in loop closure detection. This study is motivated by the fact that not all loop closure hypotheses are equally plausible (e.g., owing to mutual consistency between loop closure constraints) and that if we have evidence that one hypothesis is more plausible than the others, then it should be verified more frequently. We demonstrate that the loop closure detection problem can be viewed as an instance of a multi-model hypothesize-and-verify framework. Thus, we can build guided sampling strategies on this framework where loop closures proposed using image retrieval are verified in a planned order (rather than in a conventional uniform order) to operate in a constant time. Experimental results using a stereo simultaneous localization and mapping (SLAM) system confirm that the proposed strategy, the use of loop closure constraints and robot trajectory hypotheses as a guide, achieves promising results despite the fact that there exists a significant number of false positive constraints and hypotheses.

scholarly journals LIO-CSI: LiDAR inertial odometry with loop closure combined with semantic information

PLoS ONE ◽  
2021 ◽  
Vol 16 (12) ◽  
pp. e0261053
Author(s):  
Gang Wang ◽  
Saihang Gao ◽  
Han Ding ◽  
Hao Zhang ◽  
Hongmin Cai
Keyword(s):  
Point Cloud ◽  
Semantic Information ◽  
Feature Matching ◽  
Autonomous Driving ◽  
Geometric Feature ◽  
Semantic Features ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Front End ◽  
Dynamic Objects

Accurate and reliable state estimation and mapping are the foundation of most autonomous driving systems. In recent years, researchers have focused on pose estimation through geometric feature matching. However, most of the works in the literature assume a static scenario. Moreover, a registration based on a geometric feature is vulnerable to the interference of a dynamic object, resulting in a decline of accuracy. With the development of a deep semantic segmentation network, we can conveniently obtain the semantic information from the point cloud in addition to geometric information. Semantic features can be used as an accessory to geometric features that can improve the performance of odometry and loop closure detection. In a more realistic environment, semantic information can filter out dynamic objects in the data, such as pedestrians and vehicles, which lead to information redundancy in generated map and map-based localization failure. In this paper, we propose a method called LiDAR inertial odometry (LIO) with loop closure combined with semantic information (LIO-CSI), which integrates semantic information to facilitate the front-end process as well as loop closure detection. First, we made a local optimization on the semantic labels provided by the Sparse Point-Voxel Neural Architecture Search (SPVNAS) network. The optimized semantic information is combined into the front-end process of tightly-coupled light detection and ranging (LiDAR) inertial odometry via smoothing and mapping (LIO-SAM), which allows us to filter dynamic objects and improve the accuracy of the point cloud registration. Then, we proposed a semantic assisted scan-context method to improve the accuracy and robustness of loop closure detection. The experiments were conducted on an extensively used dataset KITTI and a self-collected dataset on the Jilin University (JLU) campus. The experimental results demonstrate that our method is better than the purely geometric method, especially in dynamic scenarios, and it has a good generalization ability.

scholarly journals SVG-Loop: Semantic–Visual–Geometric Information-Based Loop Closure Detection

2021 ◽  
Vol 13 (17) ◽  
pp. 3520
Author(s):  
Zhian Yuan ◽  
Ke Xu ◽  
Xiaoyu Zhou ◽  
Bin Deng ◽  
Yanxin Ma
Keyword(s):  
Detection Algorithm ◽  
Detection Methods ◽  
Vector Model ◽  
Limited Information ◽  
Complex Environments ◽  
Dynamic Features ◽  
Geometric Information ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Localization And Mapping

Loop closure detection is an important component of visual simultaneous localization and mapping (SLAM). However, most existing loop closure detection methods are vulnerable to complex environments and use limited information from images. As higher-level image information and multi-information fusion can improve the robustness of place recognition, a semantic–visual–geometric information-based loop closure detection algorithm (SVG-Loop) is proposed in this paper. In detail, to reduce the interference of dynamic features, a semantic bag-of-words model was firstly constructed by connecting visual features with semantic labels. Secondly, in order to improve detection robustness in different scenes, a semantic landmark vector model was designed by encoding the geometric relationship of the semantic graph. Finally, semantic, visual, and geometric information was integrated by fuse calculation of the two modules. Compared with art-of-the-state methods, experiments on the TUM RBG-D dataset, KITTI odometry dataset, and practical environment show that SVG-Loop has advantages in complex environments with varying light, changeable weather, and dynamic interference.

scholarly journals Multi-Objective Optimization of Loop Closure Detection Parameters for Indoor 2D Simultaneous Localization and Mapping

Sensors ◽  
2020 ◽  
Vol 20 (7) ◽  
pp. 1906
Author(s):  
Dongxiao Han ◽  
Yuwen Li ◽  
Tao Song ◽  
Zhenyang Liu
Keyword(s):  
Simultaneous Localization And Mapping ◽  
Ground Truth ◽  
Optimization Method ◽  
Evaluation Metrics ◽  
Multi Objective Optimization ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Multi Objective ◽  
Evaluation Approach ◽  
Localization And Mapping

Aiming at addressing the issues related to the tuning of loop closure detection parameters for indoor 2D graph-based simultaneous localization and mapping (SLAM), this article proposes a multi-objective optimization method for these parameters. The proposed method unifies the Karto SLAM algorithm, an efficient evaluation approach for map quality with three quantitative metrics, and a multi-objective optimization algorithm. More particularly, the evaluation metrics, i.e., the proportion of occupied grids, the number of corners and the amount of enclosed areas, can reflect the errors such as overlaps, blurring and misalignment when mapping nested loops, even in the absence of ground truth. The proposed method has been implemented and validated by testing on four datasets and two real-world environments. For all these tests, the map quality can be improved using the proposed method. Only loop closure detection parameters have been considered in this article, but the proposed evaluation metrics and optimization method have potential applications in the automatic tuning of other SLAM parameters to improve the map quality.

scholarly journals Dynamic-DSO: Direct Sparse Odometry Using Objects Semantic Information for Dynamic Environments

2020 ◽  
Vol 10 (4) ◽  
pp. 1467
Author(s):  
Chao Sheng ◽  
Shuguo Pan ◽  
Wang Gao ◽  
Yong Tan ◽  
Tao Zhao
Keyword(s):  
Semantic Information ◽  
Direct Method ◽  
Visual Odometry ◽  
Dynamic Environments ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Camera Pose ◽  
Photometric Error ◽  
Tracking Model ◽  
Dynamic Objects

Traditional Simultaneous Localization and Mapping (SLAM) (with loop closure detection), or Visual Odometry (VO) (without loop closure detection), are based on the static environment assumption. When working in dynamic environments, they perform poorly whether using direct methods or indirect methods (feature points methods). In this paper, Dynamic-DSO which is a semantic monocular direct visual odometry based on DSO (Direct Sparse Odometry) is proposed. The proposed system is completely implemented with the direct method, which is different from the most current dynamic systems combining the indirect method with deep learning. Firstly, convolutional neural networks (CNNs) are applied to the original RGB image to generate the pixel-wise semantic information of dynamic objects. Then, based on the semantic information of the dynamic objects, dynamic candidate points are filtered out in keyframes candidate points extraction; only static candidate points are reserved in the tracking and optimization module, to achieve accurate camera pose estimation in dynamic environments. The photometric error calculated by the projection points in dynamic region of subsequent frames are removed from the whole photometric error in pyramid motion tracking model. Finally, the sliding window optimization which neglects the photometric error calculated in the dynamic region of each keyframe is applied to obtain the precise camera pose. Experiments on the public TUM dynamic dataset and the modified Euroc dataset show that the positioning accuracy and robustness of the proposed Dynamic-DSO is significantly higher than the state-of-the-art direct method in dynamic environments, and the semi-dense cloud map constructed by Dynamic-DSO is clearer and more detailed.

scholarly journals A Light Visual Mapping and Navigation Framework for Low-Cost Robots

2015 ◽  
Vol 24 (4) ◽  
pp. 505-524 ◽  
Author(s):  
Stephane Bazeille ◽  
Emmanuel Battesti ◽  
David Filliat
Keyword(s):  
Large Range ◽  
Low Cost ◽  
Indoor Environments ◽  
Noise Levels ◽  
Visual Sensor ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Localization And Mapping ◽  
Computational Resources

AbstractWe address the problems of localization, mapping, and guidance for robots with limited computational resources by combining vision with the metrical information given by the robot odometry. We propose in this article a novel light and robust topometric simultaneous localization and mapping framework using appearance-based visual loop-closure detection enhanced with the odometry. The main advantage of this combination is that the odometry makes the loop-closure detection more accurate and reactive, while the loop-closure detection enables the long-term use of odometry for guidance by correcting the drift. The guidance approach is based on qualitative localization using vision and odometry, and is robust to visual sensor occlusions or changes in the scene. The resulting framework is incremental, real-time, and based on cheap sensors provided on many robots (a camera and odometry encoders). This approach is, moreover, particularly well suited for low-power robots as it is not dependent on the image processing frequency and latency, and thus it can be applied using remote processing. The algorithm has been validated on a Pioneer P3DX mobile robot in indoor environments, and its robustness is demonstrated experimentally for a large range of odometry noise levels.

scholarly journals Role of Deep Learning in Loop Closure Detection for Visual and Lidar SLAM: A Survey

Sensors ◽  
2021 ◽  
Vol 21 (4) ◽  
pp. 1243
Author(s):  
Saba Arshad ◽  
Gon-Woo Kim
Keyword(s):  
Deep Learning ◽  
Detailed Comparison ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Detection Algorithms ◽  
Mobile Objects ◽  
Localization And Mapping ◽  
Loop Detection ◽  
Scene Representation ◽  
Matching Strategy

Loop closure detection is of vital importance in the process of simultaneous localization and mapping (SLAM), as it helps to reduce the cumulative error of the robot’s estimated pose and generate a consistent global map. Many variations of this problem have been considered in the past and the existing methods differ in the acquisition approach of query and reference views, the choice of scene representation, and associated matching strategy. Contributions of this survey are many-fold. It provides a thorough study of existing literature on loop closure detection algorithms for visual and Lidar SLAM and discusses their insight along with their limitations. It presents a taxonomy of state-of-the-art deep learning-based loop detection algorithms with detailed comparison metrics. Also, the major challenges of conventional approaches are identified. Based on those challenges, deep learning-based methods were reviewed where the identified challenges are tackled focusing on the methods providing long-term autonomy in various conditions such as changing weather, light, seasons, viewpoint, and occlusion due to the presence of mobile objects. Furthermore, open challenges and future directions were also discussed.

scholarly journals A LiDAR/Visual SLAM Backend with Loop Closure Detection and Graph Optimization

2021 ◽  
Vol 13 (14) ◽  
pp. 2720
Author(s):  
Shoubin Chen ◽  
Baoding Zhou ◽  
Changhui Jiang ◽  
Weixing Xue ◽  
Qingquan Li
Keyword(s):  
Point Clouds ◽  
Visual Features ◽  
Visual Slam ◽  
Key Factors ◽  
Loop Closure ◽  
Loop Closure Detection ◽  
Active Sensor ◽  
Localization And Mapping ◽  
Graph Optimization ◽  
Pose Graph Optimization

LiDAR (light detection and ranging), as an active sensor, is investigated in the simultaneous localization and mapping (SLAM) system. Typically, a LiDAR SLAM system consists of front-end odometry and back-end optimization modules. Loop closure detection and pose graph optimization are the key factors determining the performance of the LiDAR SLAM system. However, the LiDAR works at a single wavelength (905 nm), and few textures or visual features are extracted, which restricts the performance of point clouds matching based loop closure detection and graph optimization. With the aim of improving LiDAR SLAM performance, in this paper, we proposed a LiDAR and visual SLAM backend, which utilizes LiDAR geometry features and visual features to accomplish loop closure detection. Firstly, the bag of word (BoW) model, describing the visual similarities, was constructed to assist in the loop closure detection and, secondly, point clouds re-matching was conducted to verify the loop closure detection and accomplish graph optimization. Experiments with different datasets were carried out for assessing the proposed method, and the results demonstrated that the inclusion of the visual features effectively helped with the loop closure detection and improved LiDAR SLAM performance. In addition, the source code, which is open source, is available for download once you contact the corresponding author.

scholarly journals LAYOUT SLAM WITH MODEL BASED LOOP CLOSURE FOR 3D INDOOR CORRIDOR RECONSTRUCTION

2018 ◽  
Vol IV-2 ◽  
pp. 25-32
Author(s):  
A. Baligh Jahromi ◽  
G. Sohn ◽  
J. Jung ◽  
M. Shahbazi ◽  
J. Kang
Keyword(s):  
Model Matching ◽  
Loop Closure ◽  
Related Information ◽  
Localization And Mapping ◽  
Abrupt Changes ◽  
Camera Orientation ◽  
Mapping Scale ◽  
The Impact ◽  
Loop Closing ◽  
Do So

In this paper, we extend a recently proposed visual Simultaneous Localization and Mapping (SLAM) techniques, known as Layout SLAM, to make it robust against error accumulations, abrupt changes of camera orientation and miss-association of newly visited parts of the scene to the previously visited landmarks. To do so, we present a novel technique of loop closing based on layout model matching; i.e., both model information (topology and geometry of reconstructed models) and image information (photometric features) are used to address a loop-closure detection. The advantages of using the layout-related information in the proposed loop-closing technique are twofold. First, it imposes a metric constraint on the global map consistency and, thus, adjusts the mapping scale drifts. Second, it can reduce matching ambiguity in the context of indoor corridors, where the scene is homogenously textured and extracting sufficient amount of distinguishable point features is a challenging task. To test the impact of the proposed technique on the performance of Layout SLAM, we have performed the experiments on wide-angle videos captured by a handheld camera. This dataset was collected from the indoor corridors of a building at York University. The obtained results demonstrate that the proposed method successfully detects the instances of loops while producing very limited trajectory errors.

Sign in / Sign up

Export Citation Format

Share Document