A deep-shallow and global–local multi-feature fusion network for photometric stereo

AbstractPhotometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination. Traditional methods normally adopt simplified reflectance models to make the surface orientation computable. However, the real reflectances of surfaces greatly limit applicability of such methods to real-world objects. While deep neural networks have been employed to handle non-Lambertian surfaces, these methods are subject to blurring and errors, especially in high-frequency regions (such as crinkles and edges), caused by spectral bias: neural networks favor low-frequency representations so exhibit a bias towards smooth functions. In this paper, therefore, we propose a self-learning conditional network with multi-scale features for photometric stereo, avoiding blurred reconstruction in such regions. Our explorations include: (i) a multi-scale feature fusion architecture, which keeps high-resolution representations and deep feature extraction, simultaneously, and (ii) an improved gradient-motivated conditionally parameterized convolution (GM-CondConv) in our photometric stereo network, with different combinations of convolution kernels for varying surfaces. Extensive experiments on public benchmark datasets show that our calibrated photometric stereo method outperforms the state-of-the-art.

Download Full-text

Multi-Feature Fusion Identification of Important Nodes in Traffic Network

International Conference on Transportation and Development 2020 ◽

10.1061/9780784483152.015 ◽

2020 ◽

Author(s):

Yuxin Xiao ◽

Jianming Hu ◽

Zuo Zhang ◽

Yi Zhang

Keyword(s):

Feature Fusion ◽

Traffic Network ◽

Important Nodes

Download Full-text

3D Scanning Solution for Textured Object using Photometric Stereo with Multiple Known Light Sources

Archiving Conference ◽

10.2352/issn.2168-3204.2018.1.0.3 ◽

2018 ◽

Vol 2018 (1) ◽

pp. 6-9 ◽

Cited By ~ 2

Author(s):

Arnold Cheveau

Keyword(s):

Photometric Stereo ◽

3D Scanning ◽

Light Sources

Download Full-text

A Study on Utilization of Three-Dimensional Sensor Lip Image for Developing a Pronunciation Recognition System

Journal of Imaging Science and Technology ◽

10.2352/j.imagingsci.technol.2019.63.5.050402 ◽

2019 ◽

Vol 63 (5) ◽

pp. 50402-1-50402-9 ◽

Cited By ~ 1

Author(s):

Ing-Jr Ding ◽

Chong-Min Ruan

Keyword(s):

Principal Component Analysis ◽

Automatic Speech Recognition ◽

Feature Fusion ◽

Three Dimensional ◽

Principal Component ◽

Recognition System ◽

Geometrical Characteristics ◽

3D Geometry ◽

Different Types ◽

The Disabled

Abstract The acoustic-based automatic speech recognition (ASR) technique has been a matured technique and widely seen to be used in numerous applications. However, acoustic-based ASR will not maintain a standard performance for the disabled group with an abnormal face, that is atypical eye or mouth geometrical characteristics. For governing this problem, this article develops a three-dimensional (3D) sensor lip image based pronunciation recognition system where the 3D sensor is efficiently used to acquire the action variations of the lip shapes of the pronunciation action from a speaker. In this work, two different types of 3D lip features for pronunciation recognition are presented, 3D-(x, y, z) coordinate lip feature and 3D geometry lip feature parameters. For the 3D-(x, y, z) coordinate lip feature design, 18 location points, each of which has 3D-sized coordinates, around the outer and inner lips are properly defined. In the design of 3D geometry lip features, eight types of features considering the geometrical space characteristics of the inner lip are developed. In addition, feature fusion to combine both 3D-(x, y, z) coordinate and 3D geometry lip features is further considered. The presented 3D sensor lip image based feature evaluated the performance and effectiveness using the principal component analysis based classification calculation approach. Experimental results on pronunciation recognition of two different datasets, Mandarin syllables and Mandarin phrases, demonstrate the competitive performance of the presented 3D sensor lip image based pronunciation recognition system.

Download Full-text