scholarly journals A Benchmark Environment for Neuromorphic Stereo Vision

2021 ◽  
Vol 8 ◽  
Author(s):  
L. Steffen ◽  
M. Elfgen ◽  
S. Ulbrich ◽  
A. Roennau ◽  
R. Dillmann
Keyword(s):  
Stereo Vision ◽  
Stereo Matching ◽  
Depth Sensor ◽  
Processing Times ◽  
Redundant Data ◽  
High Resource ◽  
Meaningful Comparison ◽  
Neuromorphic Hardware ◽  
Depth Reconstruction ◽  
Event Based

Without neuromorphic hardware, artificial stereo vision suffers from high resource demands and processing times impeding real-time capability. This is mainly caused by high frame rates, a quality feature for conventional cameras, generating large amounts of redundant data. Neuromorphic visual sensors generate less redundant and more relevant data solving the issue of over- and undersampling at the same time. However, they require a rethinking of processing as established techniques in conventional stereo vision do not exploit the potential of their event-based operation principle. Many alternatives have been recently proposed which have yet to be evaluated on a common data basis. We propose a benchmark environment offering the methods and tools to compare different algorithms for depth reconstruction from two event-based sensors. To this end, an experimental setup consisting of two event-based and one depth sensor as well as a framework enabling synchronized, calibrated data recording is presented. Furthermore, we define metrics enabling a meaningful comparison of the examined algorithms, covering aspects such as performance, precision and applicability. To evaluate the benchmark, a stereo matching algorithm was implemented as a testing candidate and multiple experiments with different settings and camera parameters have been carried out. This work is a foundation for a robust and flexible evaluation of the multitude of new techniques for event-based stereo vision, allowing a meaningful comparison.

scholarly journals A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems

2017 ◽  
Vol 7 (1) ◽  
Author(s):  
Marc Osswald ◽  
Sio-Hoi Ieng ◽  
Ryad Benosman ◽  
Giacomo Indiveri
Keyword(s):  
Neural Network ◽  
Machine Vision ◽  
Stereo Vision ◽  
Correspondence Problem ◽  
Spiking Neural Network ◽  
Neuromorphic Engineering ◽  
Stereo Correspondence ◽  
Vision Systems ◽  
Neuromorphic Hardware ◽  
Event Based

Abstract Stereo vision is an important feature that enables machine vision systems to perceive their environment in 3D. While machine vision has spawned a variety of software algorithms to solve the stereo-correspondence problem, their implementation and integration in small, fast, and efficient hardware vision systems remains a difficult challenge. Recent advances made in neuromorphic engineering offer a possible solution to this problem, with the use of a new class of event-based vision sensors and neural processing devices inspired by the organizing principles of the brain. Here we propose a radically novel model that solves the stereo-correspondence problem with a spiking neural network that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. We validate the model with experimental results, highlighting features that are in agreement with both computational neuroscience stereo vision theories and experimental findings. We demonstrate its features with a prototype neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological stereo vision processing systems.

scholarly journals High-Accuracy Recognition and Localization of Moving Targets in an Indoor Environment Using Binocular Stereo Vision

2021 ◽  
Vol 10 (4) ◽  
pp. 234
Author(s):  
Jing Ding ◽  
Zhigang Yan ◽  
Xuchen We
Keyword(s):  
Stereo Vision ◽  
Stereo Matching ◽  
Three Dimensional ◽  
Target Localization ◽  
Parallel Structure ◽  
Moving Target ◽  
Target Area ◽  
Moving Targets ◽  
Binocular Stereo Vision ◽  
Binocular Stereo

To obtain effective indoor moving target localization, a reliable and stable moving target localization method based on binocular stereo vision is proposed in this paper. A moving target recognition extraction algorithm, which integrates displacement pyramid Horn–Schunck (HS) optical flow, Delaunay triangulation and Otsu threshold segmentation, is presented to separate a moving target from a complex background, called the Otsu Delaunay HS (O-DHS) method. Additionally, a stereo matching algorithm based on deep matching and stereo vision is presented to obtain dense stereo matching points pairs, called stereo deep matching (S-DM). The stereo matching point pairs of the moving target were extracted with the moving target area and stereo deep matching point pairs, then the three dimensional coordinates of the points in the moving target area were reconstructed according to the principle of binocular vision’s parallel structure. Finally, the moving target was located by the centroid method. The experimental results showed that this method can better resist image noise and repeated texture, can effectively detect and separate moving targets, and can match stereo image points in repeated textured areas more accurately and stability. This method can effectively improve the effectiveness, accuracy and robustness of three-dimensional moving target coordinates.

scholarly journals Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽  
2021 ◽  
Vol 21 (9) ◽  
pp. 3240
Author(s):  
Tehreem Syed ◽  
Vijay Kakani ◽  
Xuenan Cui ◽  
Hakil Kim
Keyword(s):  
Neural Networks ◽  
Gradient Descent ◽  
Spiking Neural Networks ◽  
License Plate ◽  
Training Techniques ◽  
Neuromorphic Hardware ◽  
Private And Public ◽  
Embedded Platforms ◽  
Public Datasets ◽  
Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

scholarly journals A Review of Algorithms and Hardware Implementations for Spiking Neural Networks

2021 ◽  
Vol 11 (2) ◽  
pp. 23
Author(s):  
Duy-Anh Nguyen ◽  
Xuan-Tu Tran ◽  
Francesca Iacopi
Keyword(s):  
Neural Networks ◽  
Computational Cost ◽  
Superior Performance ◽  
Training Algorithms ◽  
Current State ◽  
Hardware Implementations ◽  
Neuromorphic Hardware ◽  
High Level ◽  
Event Based ◽  
Hardware Platforms

Deep Learning (DL) has contributed to the success of many applications in recent years. The applications range from simple ones such as recognizing tiny images or simple speech patterns to ones with a high level of complexity such as playing the game of Go. However, this superior performance comes at a high computational cost, which made porting DL applications to conventional hardware platforms a challenging task. Many approaches have been investigated, and Spiking Neural Network (SNN) is one of the promising candidates. SNN is the third generation of Artificial Neural Networks (ANNs), where each neuron in the network uses discrete spikes to communicate in an event-based manner. SNNs have the potential advantage of achieving better energy efficiency than their ANN counterparts. While generally there will be a loss of accuracy on SNN models, new algorithms have helped to close the accuracy gap. For hardware implementations, SNNs have attracted much attention in the neuromorphic hardware research community. In this work, we review the basic background of SNNs, the current state and challenges of the training algorithms for SNNs and the current implementations of SNNs on various hardware platforms.

Symbol recognition system based on 3D stereo vision

2020 ◽  
pp. 1-10
Author(s):  
Linlin Wang
Keyword(s):  
Camera Calibration ◽  
Stereo Vision ◽  
Stereo Matching ◽  
Dimensional Space ◽  
Three Dimensional ◽  
Recognition System ◽  
Symbol Recognition ◽  
Two Dimensional ◽  
Measuring Machine ◽  
Left And Right

With the continuous development of computer science and technology, symbol recognition systems may be converted from two-dimensional space to three-dimensional space. Therefore, this article mainly introduces the symbol recognition system based on 3D stereo vision. The three-dimensional image is taken by the visual coordinate measuring machine in two places on the left and right. Perform binocular stereo matching on the edge of the feature points of the two images. A corner detection algorithm combining SUSAN and Harris is used to detect the left and right camera calibration templates. The two-dimensional coordinate points of the object are determined by the image stereo matching module, and the three-dimensional discrete coordinate points of the object space can be obtained according to the transformation relationship between the image coordinates and the actual object coordinates. Then draw the three-dimensional model of the object through the three-dimensional drawing software. Experimental data shows that the logic resources and memory resources occupied by image preprocessing account for 30.4% and 27.4% of the entire system, respectively. The results show that the system can calibrate the internal and external parameters of the camera. In this way, the camera calibration result will be more accurate and the range will be wider. At the same time, it can effectively make up for the shortcomings of traditional modeling techniques to ensure the measurement accuracy of the detection system.

PRECISE SHAPE MEASUREMENT OF DYNAMIC SURFACE VIA SINGLE CAMERA STEREO

2006 ◽  
Vol 03 (01) ◽  
pp. 53-60
Author(s):  
LUPING AN ◽  
YUNDE JIA ◽  
MINGTAO PEI ◽  
HONGBIN DENG
Keyword(s):  
Stereo Vision ◽  
Stereo Matching ◽  
Vision System ◽  
Least Square ◽  
Shape Measurement ◽  
Single Camera ◽  
Dynamic Surface ◽  
Moving Least Square ◽  
Stereo Vision System ◽  
Precise Shape

In this article, a method of the precise shape measurement of dynamic surfaces via a single camera stereo vision system is presented, a cross-curve pattern is painted on the surface of an object, and the intersections of cross-curves which represent the shape of the object are measured by the stereo vision system. The system with a single camera is modeled as a virtual binocular stereo by strong calibration technique. Binocular epipolar rectification is used to make the stereo matching efficient, and principal curves theory is employed to extract curves in images for stereo matching. Under the framework of RANSAC, the curves are interpolated robustly with cubic spline based on moving-least-square (MLS). Experimental results on both static and dynamic deforming surfaces illustrate the effectiveness of the proposed method.

scholarly journals Stereo Imaging Using Hardwired Self-Organizing Object Segmentation

Sensors ◽  
2020 ◽  
Vol 20 (20) ◽  
pp. 5833
Author(s):  
Ching-Han Chen ◽  
Guan-Wei Lan ◽  
Ching-Yi Chen ◽  
Yen-Hsiang Huang
Keyword(s):  
Neural Network ◽  
Stereo Vision ◽  
Stereo Matching ◽  
Imaging System ◽  
Object Segmentation ◽  
Hierarchical Architecture ◽  
Absolute Difference ◽  
Stereo Imaging ◽  
Som Neural Network ◽  
Self Organizing

Stereo vision utilizes two cameras to acquire two respective images, and then determines the depth map by calculating the disparity between two images. In general, object segmentation and stereo matching are some of the important technologies that are often used in establishing stereo vision systems. In this study, we implement a highly efficient self-organizing map (SOM) neural network hardware accelerator as unsupervised color segmentation for real-time stereo imaging. The stereo imaging system is established by pipelined, hierarchical architecture, which includes an SOM neural network module, a connected component labeling module, and a sum-of-absolute-difference-based stereo matching module. The experiment is conducted on a hardware resources-constrained embedded system. The performance of stereo imaging system is able to achieve 13.8 frames per second of 640 × 480 resolution color images.

scholarly journals A Real-Time Range Finding System with Binocular Stereo Vision

10.5772/50921 ◽  
2012 ◽  
Vol 9 (1) ◽  
pp. 26 ◽  
Author(s):  
Xiao-Bo Lai ◽  
Hai-Shun Wang ◽  
Yue-Hong Xu
Keyword(s):  
Real Time ◽  
Stereo Vision ◽  
Stereo Matching ◽  
Single Point ◽  
Time Range ◽  
Real Time Control ◽  
Matching Algorithm ◽  
Range Finding ◽  
Binocular Stereo Vision ◽  
Binocular Stereo

To acquire range information for mobile robots, a TMS320DM642 DSP-based range finding system with binocular stereo vision is proposed. Firstly, paired images of the target are captured and a Gaussian filter, as well as improved Sobel kernels, are achieved. Secondly, a feature-based local stereo matching algorithm is performed so that the space location of the target can be determined. Finally, in order to improve the reliability and robustness of the stereo matching algorithm under complex conditions, the confidence filter and the left-right consistency filter are investigated to eliminate the mismatching points. In addition, the range finding algorithm is implemented in the DSP/BIOS operating system to gain real-time control. Experimental results show that the average accuracy of range finding is more than 99% for measuring single-point distances equal to 120cm in the simple scenario and the algorithm takes about 39ms for ranging a time in a complex scenario. The effectivity, as well as the feasibility, of the proposed range finding system are verified.

Sign in / Sign up

Export Citation Format

Share Document