A new function of stereo matching algorithm based on hybrid convolutional neural network

This paper proposes a new hybrid method between the learning-based and handcrafted methods for a stereo matching algorithm. The main purpose of the stereo matching algorithm is to produce a disparity map. This map is essential for many applications, including three-dimensional (3D) reconstruction. The raw disparity map computed by a convolutional neural network (CNN) is still prone to errors in the low texture region. The algorithm is set to improve the matching cost computation stage with hybrid CNN-based combined with truncated directional intensity computation. The difference in truncated directional intensity value is employed to decrease radiometric errors. The proposed method’s raw matching cost went through the cost aggregation step using the bilateral filter (BF) to improve accuracy. The winner-take-all (WTA) optimization uses the aggregated cost volume to produce an initial disparity map. Finally, a series of refinement processes enhance the initial disparity map for a more accurate final disparity map. This paper verified the performance of the algorithm using the Middlebury online stereo benchmarking system. The proposed algorithm achieves the objective of generating a more accurate and smooth disparity map with different depths at low texture regions through better matching cost quality.

Download Full-text

Stereo Matching Algorithm Based on Hybrid Convolutional Neural Network and Directional Intensity Difference

International Journal of Emerging Technology and Advanced Engineering ◽

10.46338/ijetae0621_10 ◽

2021 ◽

Vol 11 (6) ◽

pp. 86-96

Author(s):

Mohd Saad Hamid ◽

◽

Nurulfajar Abd Manap ◽

Rostam Affendi Hamzah ◽

Ahmad Fauzan Kadmin

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Stereo Matching ◽

Good Accuracy ◽

Intensity Difference ◽

Disparity Map ◽

Matching Algorithm ◽

Matching Cost ◽

Autonomous Vehicle Navigation ◽

Repetitive Pattern

Fundamentally, a stereo matching algorithm produces a disparity map or depth map. This map contains valuable information for many applications, such as range estimation, autonomous vehicle navigation and 3D surface reconstruction. The stereo matching process faces various challenges to get an accurate result for example low texture area, repetitive pattern and discontinuity regions. The proposed algorithm must be robust and viable with all of these challenges and is capable to deliver good accuracy. Hence, this article proposes a new stereo matching algorithm based on a hybrid Convolutional Neural Network (CNN) combined with directional intensity differences at the matching cost stage. The proposed algorithm contains a deep learning-based method and a handcrafted method. Then, the bilateral filter is used to aggregate the matching cost volume while preserving the object edges. The Winner-Take-All (WTA) is utilized at the optimization stage which the WTA normalizes the disparity values. At the last stage, a series of refinement processes will be applied to enhance the final disparity map. A standard benchmarking evaluation system from the Middlebury Stereo dataset is used to measure the algorithm performance. This dataset provides images with the characteristics of low texture area, repetitive pattern and discontinuity regions. The average error produced for all pixel regions is 8.51%, while the nonoccluded region is 5.77%. Based on the experimental results, the proposed algorithm produces good accuracy and robustness against the stereo matching challenges. It is also competitive with other published methods and can be used as a complete algorithm

Download Full-text

Stereo Matching Algorithm Based on Three-Dimensional Convolutional Neural Network

Acta Optica Sinica ◽

10.3788/aos201939.1115001 ◽

2019 ◽

Vol 39 (11) ◽

pp. 1115001

Author(s):

王玉锋 Wang Yufeng ◽

王宏伟 Wang Hongwei ◽

于光 Yu Guang ◽

杨明权 Yang Mingquan ◽

袁昱纬 Yuan Yuwei ◽

...

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Stereo Matching ◽

Three Dimensional ◽

Matching Algorithm

Download Full-text

Improved stereo matching algorithm based on multi-scale fusion

Xibei Gongye Daxue Xuebao/Journal of Northwestern Polytechnical University ◽

10.1051/jnwpu/20213940876 ◽

2021 ◽

Vol 39 (4) ◽

pp. 876-882

Author(s):

Xing Chen ◽

Wenhai Zhang ◽

Yu Hou ◽

Lin Yang

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Stereo Matching ◽

Dynamic Programming Algorithm ◽

Good Effect ◽

Disparity Map ◽

Data Set ◽

Matching Algorithm ◽

Multi Scale ◽

Feature Pyramid

Aiming at the low matching accuracy of local stereo matching algorithm in weak texture or discontinuous disparity areas, a stereo matching algorithm combining multi-scale fusion of convolutional neural network (CNN) and feature pyramid structure (FPN) is proposed. The feature pyramid is applied on the basis of the convolutional neural network to realize the multi-scale feature extraction and fusion of the image, which improves the matching similarity of the image blocks. The guide graph filter is used to quickly and effectively complete the cost aggregation. The disparity selection stage adapts the improvement dynamic programming algorithm to obtain the initial disparity map. The initial disparity map is refined so as to obtain the final disparity map. The algorithm is trained and tested on the image provided by Middlebury data set, and the result shows that the disparity map obtained by the algorithm has good effect.

Download Full-text

Matching Large Baseline Oblique Stereo Images Using an End-to-End Convolutional Neural Network

Remote Sensing ◽

10.3390/rs13020274 ◽

2021 ◽

Vol 13 (2) ◽

pp. 274

Author(s):

Guobiao Yao ◽

Alper Yilmaz ◽

Li Zhang ◽

Fei Meng ◽

Haibin Ai ◽

...

Keyword(s):

Neural Network ◽

Deep Learning ◽

Convolutional Neural Network ◽

Stereo Matching ◽

Least Square ◽

Affine Invariant ◽

Stereo Images ◽

Distance Ratio ◽

Matching Algorithm ◽

End To End

The available stereo matching algorithms produce large number of false positive matches or only produce a few true-positives across oblique stereo images with large baseline. This undesired result happens due to the complex perspective deformation and radiometric distortion across the images. To address this problem, we propose a novel affine invariant feature matching algorithm with subpixel accuracy based on an end-to-end convolutional neural network (CNN). In our method, we adopt and modify a Hessian affine network, which we refer to as IHesAffNet, to obtain affine invariant Hessian regions using deep learning framework. To improve the correlation between corresponding features, we introduce an empirical weighted loss function (EWLF) based on the negative samples using K nearest neighbors, and then generate deep learning-based descriptors with high discrimination that is realized with our multiple hard network structure (MTHardNets). Following this step, the conjugate features are produced by using the Euclidean distance ratio as the matching metric, and the accuracy of matches are optimized through the deep learning transform based least square matching (DLT-LSM). Finally, experiments on Large baseline oblique stereo images acquired by ground close-range and unmanned aerial vehicle (UAV) verify the effectiveness of the proposed approach, and comprehensive comparisons demonstrate that our matching algorithm outperforms the state-of-art methods in terms of accuracy, distribution and correct ratio. The main contributions of this article are: (i) our proposed MTHardNets can generate high quality descriptors; and (ii) the IHesAffNet can produce substantial affine invariant corresponding features with reliable transform parameters.

Download Full-text

Convolutional neural network using multi-scale information for stereo matching cost computation

2016 IEEE International Conference on Image Processing (ICIP) ◽

10.1109/icip.2016.7532995 ◽

2016 ◽

Cited By ~ 3

Author(s):

Jiahui Chen ◽

Chun Yuan

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Stereo Matching ◽

Multi Scale ◽

Matching Cost

Download Full-text

Stereo Matching Based on Immune Neural Network in Abdomen Reconstruction

Mathematical Problems in Engineering ◽

10.1155/2015/242794 ◽

2015 ◽

Vol 2015 ◽

pp. 1-15

Author(s):

Huan Liu ◽

Kuangrong Hao ◽

Yongsheng Ding ◽

Chunjuan Ouyang

Keyword(s):

Neural Network ◽

Stereo Matching ◽

Feature Matching ◽

Three Dimensional ◽

Target Point ◽

Feature Points ◽

Matching Algorithm ◽

Optimization Task ◽

Feasible Alternative ◽

Single Neural Network

Stereo feature matching is a technique that finds an optimal match in two images from the same entity in the three-dimensional world. The stereo correspondence problem is formulated as an optimization task where an energy function, which represents the constraints on the solution, is to be minimized. A novel intelligent biological network (Bio-Net), which involves the human B-T cells immune system into neural network, is proposed in this study in order to learn the robust relationship between the input feature points and the output matched points. A model from input-output data (left reference point-right target point) is established. In the experiments, the abdomen reconstructions for different-shape mannequins are then performed by means of the proposed method. The final results are compared and analyzed, which demonstrate that the proposed approach greatly outperforms the single neural network and the conventional matching algorithm in precise. Particularly, as far as time cost and efficiency, the proposed method exhibits its significant promising and potential for improvement. Hence, it is entirely considered as an effective and feasible alternative option for stereo matching.

Download Full-text

mproved Stereo Matching Algorithm based on Census Transform and Dynamic Histogram Cost Computation

International Journal of Emerging Technology and Advanced Engineering ◽

10.46338/ijetae0821_07 ◽

2021 ◽

Vol 11 (8) ◽

pp. 48-57

Author(s):

A. F. Kadmin ◽

◽

R. A. Hamzah ◽

M. N. Abd Manap ◽

M. S. Hamid ◽

...

Keyword(s):

Stereo Matching ◽

Disparity Map ◽

Local Method ◽

Matching Algorithm ◽

Census Transform ◽

Traditional Taxonomy ◽

Matching Cost ◽

Winner Takes All ◽

The Cost ◽

Vision Algorithm

Stereo matching is a significant subject in the stereo vision algorithm. Traditional taxonomy composition consists of several issues in the stereo correspondences process such as radiometric distortion, discontinuity, and low accuracy at the low texture regions. This new taxonomy improves the local method of stereo matching algorithm based on the dynamic cost computation for disparity map measurement. This method utilised modified dynamic cost computation in the matching cost stage. A modified Census Transform with dynamic histogram is used to provide the cost volume. An adaptive bilateral filtering is applied to retain the image depth and edge information in the cost aggregation stage. A Winner Takes All (WTA) optimisation is applied in the disparity selection and a left-right check with an adaptive bilateral median filtering are employed for final refinement. Based on the dataset of standard Middlebury, the taxonomy has better accuracy and outperformed several other state-ofthe-art algorithms. Keywords—Stereo matching, disparity map, dynamic cost, census transform, local method

Download Full-text

Computing the stereo matching cost with a convolutional neural network

2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) ◽

10.1109/cvpr.2015.7298767 ◽

2015 ◽

Cited By ~ 210

Author(s):

Jure Zbontar ◽

Yann LeCun

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Stereo Matching ◽

Matching Cost

Download Full-text

3D-Modeling Depth Optimization Using a Neural Network and Image Segments

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.333-335.1096 ◽

2013 ◽

Vol 333-335 ◽

pp. 1096-1105 ◽

Cited By ~ 1

Author(s):

Fan Jun Liu ◽

Bin Gang Cao

Keyword(s):

Neural Network ◽

Optimization Algorithm ◽

Fourier Transformation ◽

Stereo Matching ◽

Three Dimensional ◽

Reference Image ◽

Fast Fourier Transformation ◽

Stereo Pair ◽

Disparity Map ◽

Ground Truth Data

We present a 3D(three-dimensional)-modeling disparity-map optimization algorithm using a neural network and image segments for stereo navigation. We decompose the optimization algorithm problem into two sub-problems: initial stereo matching and depth optimization. A two-step procedure is proposed to solve the sub-problems sequentially. The first step is a region based NCC(normalized cross-correlation) matching process. But we use fast Fourier transformation and inverse fast Fourier transformation to eliminate redundant calculations in NCC, and we create a high-confidence disparity map by cross checking. In the second step, the reference image (the left image of the inputted stereo pair) is segmented into regions according to homogeneous color. A neural network is then built to model the three dimensional surface and applied to refine disparities in each image segment. The experimental results obtained for Middlebury test datasets and real stereo road images indicate that our method is competitive with the best stereo matching algorithms currently available. In particular, the approach has significantly improved performance for road images used in navigation and the disparity maps recovered by our algorithm are similar to ground truth data.

Download Full-text