scholarly journals A Residual Network and FPGA Based Real-Time Depth Map Enhancement System

Entropy ◽  
2021 ◽  
Vol 23 (5) ◽  
pp. 546
Author(s):  
Zhenni Li ◽  
Haoyi Sun ◽  
Yuliang Gao ◽  
Jiao Wang
Keyword(s):  
Real Time ◽  
Super Resolution ◽  
Depth Map ◽  
Acquisition System ◽  
Depth Image ◽  
Fpga Design ◽  
Depth Sensing ◽  
Residual Network ◽  
Real Time Processing ◽  
Depth Maps

Depth maps obtained through sensors are often unsatisfactory because of their low-resolution and noise interference. In this paper, we propose a real-time depth map enhancement system based on a residual network which uses dual channels to process depth maps and intensity maps respectively and cancels the preprocessing process, and the algorithm proposed can achieve real-time processing speed at more than 30 fps. Furthermore, the FPGA design and implementation for depth sensing is also introduced. In this FPGA design, intensity image and depth image are captured by the dual-camera synchronous acquisition system as the input of neural network. Experiments on various depth map restoration shows our algorithms has better performance than existing LRMC, DE-CNN and DDTF algorithms on standard datasets and has a better depth map super-resolution, and our FPGA completed the test of the system to ensure that the data throughput of the USB 3.0 interface of the acquisition system is stable at 226 Mbps, and support dual-camera to work at full speed, that is, 54 fps@ (1280 × 960 + 328 × 248 × 3).

scholarly journals No-Reference Depth Map Quality Evaluation Model Based on Depth Map Edge Confidence Measurement in Immersive Video Applications

2019 ◽  
Vol 11 (10) ◽  
pp. 204 ◽  
Author(s):  
Dogan ◽  
Haddad ◽  
Ekmekcioglu ◽  
Kondoz
Keyword(s):  
Quality Evaluation ◽  
Evaluation Model ◽  
Depth Map ◽  
Depth Image ◽  
Depth Maps ◽  
Model Based ◽  
Objective Quality ◽  
Quality Evaluation Model ◽  
Confidence Measurement

When it comes to evaluating perceptual quality of digital media for overall quality of experience assessment in immersive video applications, typically two main approaches stand out: Subjective and objective quality evaluation. On one hand, subjective quality evaluation offers the best representation of perceived video quality assessed by the real viewers. On the other hand, it consumes a significant amount of time and effort, due to the involvement of real users with lengthy and laborious assessment procedures. Thus, it is essential that an objective quality evaluation model is developed. The speed-up advantage offered by an objective quality evaluation model, which can predict the quality of rendered virtual views based on the depth maps used in the rendering process, allows for faster quality assessments for immersive video applications. This is particularly important given the lack of a suitable reference or ground truth for comparing the available depth maps, especially when live content services are offered in those applications. This paper presents a no-reference depth map quality evaluation model based on a proposed depth map edge confidence measurement technique to assist with accurately estimating the quality of rendered (virtual) views in immersive multi-view video content. The model is applied for depth image-based rendering in multi-view video format, providing comparable evaluation results to those existing in the literature, and often exceeding their performance.

Spatial Distance-based Interpolation Algorithm for Computer Generated 2D+Z Images

2020 ◽  
Vol 2020 (2) ◽  
pp. 140-1-140-6
Author(s):  
Yuzhong Jiao ◽  
Kayton Wai Keung Cheung ◽  
Mark Ping Chan Mok ◽  
Yiu Kei Li
Keyword(s):  
Depth Map ◽  
Linear Interpolation ◽  
Spatial Distance ◽  
Depth Image ◽  
Interpolation Algorithm ◽  
3D Display ◽  
Common Input ◽  
Depth Maps ◽  
High Resolution Images ◽  
Rgb Images

Computer generated 2D plus Depth (2D+Z) images are common input data for 3D display with depth image-based rendering (DIBR) technique. Due to their simplicity, linear interpolation methods are usually used to convert low-resolution images into high-resolution images for not only depth maps but also 2D RGB images. However linear methods suffer from zigzag artifacts in both depth map and RGB images, which severely affects the 3D visual experience. In this paper, spatial distance-based interpolation algorithm for computer generated 2D+Z images is proposed. The method interpolates RGB images with the help of depth and edge information from depth maps. Spatial distance from interpolated pixel to surrounding available pixels is utilized to obtain the weight factors of surrounding pixels. Experiment results show that such spatial distance-based interpolation can achieve sharp edges and less artifacts for 2D RGB images. Naturally, it can improve the performance of 3D display. Since bilinear interpolation is used in homogenous areas, the proposed algorithm keeps low computational complexity.

scholarly journals Exploring RGBDepth Fusion for Real-Time Object Detection

Sensors ◽  
2019 ◽  
Vol 19 (4) ◽  
pp. 866 ◽  
Author(s):  
Tanguy Ophoff ◽  
Kristof Van Beeck ◽  
Toon Goedemé
Keyword(s):  
Object Detection ◽  
Real Time ◽  
Network Architecture ◽  
Depth Information ◽  
Special Focus ◽  
Single Shot ◽  
Depth Sensing ◽  
Real Time Processing ◽  
Depth Cameras ◽  
Optimal Fusion

In this paper, we investigate whether fusing depth information on top of normal RGB data for camera-based object detection can help to increase the performance of current state-of-the-art single-shot detection networks. Indeed, depth sensing is easily acquired using depth cameras such as a Kinect or stereo setups. We investigate the optimal manner to perform this sensor fusion with a special focus on lightweight single-pass convolutional neural network (CNN) architectures, enabling real-time processing on limited hardware. For this, we implement a network architecture allowing us to parameterize at which network layer both information sources are fused together. We performed exhaustive experiments to determine the optimal fusion point in the network, from which we can conclude that fusing towards the mid to late layers provides the best results. Our best fusion models significantly outperform the baseline RGB network in both accuracy and localization of the detections.

scholarly journals CAREDAS: a Comprehensive Architecture for a Redundant and Evolutive Data Acquisition System for JHR reactor

2021 ◽  
Vol 253 ◽  
pp. 04008
Author(s):  
Fabrice Leroux ◽  
Lionel Ducobu ◽  
Frédéric Milleville
Keyword(s):  
Data Acquisition ◽  
Real Time ◽  
Data Acquisition System ◽  
High Availability ◽  
Acquisition System ◽  
Real Time Processing ◽  
New Material ◽  
Under Construction ◽  
And Control ◽  
Measuring Bench

A new material testing reactor Jules Horowitz Reactor is under construction at CEA Cadarache. The materials to be irradiated will be placed into experimental devices around the reactor. Process and measurements of experimental devices will be carried out by command control. A data acquisition system having processing performances will be associated to the programmable logic controller. The challenge is to design and realize for twenty experiment devices a high availability data acquisition system architecture for 50 years of sustainability. The real time target will achieve 24/7 data acquisition and real time processing. This scalable architecture could be use as well for JHR experimental devices with high availability as for testbed. This architecture could be run on a standalone station for a measuring bench or deployed on cluster for high availability. CAREDAS’s design is modular and use proven widely used open source solutions. All parts are independent from each other and can be replaced with another technology with the same functionalities. This ensures sustainability and control of software sources.

Multi-Scale Frequency Reconstruction for Guided Depth Map Super-Resolution via Deep Residual Network

2020 ◽  
Vol 30 (2) ◽  
pp. 297-306 ◽  
Author(s):  
Yifan Zuo ◽  
Qiang Wu ◽  
Yuming Fang ◽  
Ping An ◽  
Liqin Huang ◽  
...  
Keyword(s):  
Super Resolution ◽  
Depth Map ◽  
Residual Network ◽  
Multi Scale

scholarly journals Semi-Automatic Depth Map Generation In Unconstrained Images And Video Sequences For 2D To Stereoscopic 3D Conversion

2021 ◽  
Author(s):  
Raymond Phan
Keyword(s):  
Random Walks ◽  
Human Perception ◽  
Depth Map ◽  
Graph Cuts ◽  
Depth Image ◽  
Video Sequences ◽  
Edge Preserving ◽  
Stereoscopic 3D ◽  
Depth Maps ◽  
3D Content

In this work, we describe a system for accurately estimating depth through synthetic depth maps in unconstrained conventional monocular images and video sequences, to semi-automatically convert these into their stereoscopic 3D counterparts. With current accepted industry efforts, this conversion process is performed automatically in a black box fashion, or manually converted using human operators to extract features and objects on a frame by frame basis, known as rotoscopers. Automatic conversion is the least labour intensive, but allows little to no user intervention, and error correction can be difficult. Manual is the most accurate, providing the most control, but very time consuming, and is prohibitive for use to all but the largest production studios. Noting the merits and disadvantages between these two methods, a semi-automatic method blends the two together, allowing for faster and accurate conversion, while decreasing time for releasing 3D content for user digest. Semi-automatic methods require the user to place user-defined strokes over the image, or over several keyframes in the case of video, corresponding to a rough estimate of the depths in the scene at these strokes. After, the rest of the depths are determined, creating depth maps to generate stereoscopic 3D content, and Depth Image Based Rendering is employed to generate the artificial views. Here, depth map estimation can be considered as a multi-label image segmentation problem: each class is a depth value. Additionally, for video, we allow the option of labeling only the first frame, and the strokes are propagated using one of two techniques: A modified computer vision object tracking algorithm, and edge-aware temporally consistent optical flow./p pFundamentally, this work combines the merits of two well-respected segmentation algorithms: Graph Cuts and Random Walks. The diffusion of depths, with smooth gradients from Random Walks, combined with the edge preserving properties from Graph Cuts can create the best possible result. To demonstrate that the proposed framework generates good quality stereoscopic content with minimal effort, we create results and compare to the current best known semi-automatic conversion framework. We also show that our results are more suitable for human perception in comparison to this framework.

scholarly journals REAL-TIME DEPTH MAP ESTIMATION FROM INFRARED STEREO IMAGES OF RGB-D CAMERAS

2021 ◽  
Vol V-2-2021 ◽  
pp. 107-112
Author(s):  
J. Zhong ◽  
M. Li ◽  
X. Liao ◽  
J. Qin ◽  
H. Zhang ◽  
...  
Keyword(s):  
Real Time ◽  
Block Size ◽  
Depth Map ◽  
Measurement Range ◽  
Block Matching ◽  
Depth Information ◽  
Depth Maps ◽  
Sensing Systems ◽  
Global Matching ◽  
Efficient Matching

Abstract. RGB-D cameras are novel sensing systems that can rapidly provide accurate depth information for 3D perception, among which the type based on active stereo vision has been widely used. However, there are some problems exiting in use, such as the short measurement range and incomplete depth maps. This paper presents a robust and efficient matching algorithm based on semi-global matching to obtain more complete and accurate depth maps in real time. Considering characteristics of captured infrared speckle images, the Gaussian filter is performed firstly to restrain noise and enhance the relativity. It also adopts the idea of block matching for reliability, and a dynamic threshold selection of the block size is used to adapt to various situation. Moreover, several optimizations are applied to improve precision and reduce error. Through experiments on the Intel Realsense R200, the excellent capability of our proposed method is verified.

Design of Data Acquisition System Based on FPGA

2011 ◽  
Vol 403-408 ◽  
pp. 1592-1595
Author(s):  
Guo Sheng Xu
Keyword(s):  
Data Processing ◽  
Data Acquisition ◽  
Real Time ◽  
High Speed ◽  
Image Data ◽  
Data Acquisition System ◽  
Acquisition System ◽  
Time Data ◽  
Real Time Processing ◽  
Field Programmable

A new kind of data acquisition system is introduced in this paper, in which the multi-channel synchronized real-time data acquisition under the coordinate control of field-programmable gate array(FPGA) is realized. The design uses field programmable gate arrays(FPGA) for the data processing and logic control. For high speed CCD image data processing, the paper adopts regional parallel processing based on FPGA. The FPGA inner block RAM is used to build high speed image data buffer is put into operation to achieve high speed image data integration and real-time processing. The proposed data acquisition system has characteristics of stable performance, flexible expansion, high real-timeness and integration

Sign in / Sign up

Export Citation Format

Share Document