Rich Structural Index for Stereoscopic Image Quality Assessment

The human visual system (HVS), affected by viewing distance when perceiving the stereo image information, is of great significance to study of stereoscopic image quality assessment. Many methods of stereoscopic image quality assessment do not have comprehensive consideration for human visual perception characteristics. In accordance with this, we propose a Rich Structural Index (RSI) for Stereoscopic Image objective Quality Assessment (SIQA) method based on multi-scale perception characteristics. To begin with, we put the stereo pair into the image pyramid based on Contrast Sensitivity Function (CSF) to obtain sensitive images of different resolution . Then, we obtain local Luminance and Structural Index (LSI) in a locally adaptive manner on gradient maps which consider the luminance masking and contrast masking. At the same time we use Singular Value Decomposition (SVD) to obtain the Sharpness and Intrinsic Structural Index (SISI) to effectively capture the changes introduced in the image (due to distortion). Meanwhile, considering the disparity edge structures, we use gradient cross-mapping algorithm to obtain Depth Texture Structural Index (DTSI). After that, we apply the standard deviation method for the above results to obtain contrast index of reference and distortion components. Finally, for the loss caused by the randomness of the parameters, we use Support Vector Machine Regression based on Genetic Algorithm (GA-SVR) training to obtain the final quality score. We conducted a comprehensive evaluation with state-of-the-art methods on four open databases. The experimental results show that the proposed method has stable performance and strong competitive advantage.

High-Precision Digital Surface Model Extraction from Satellite Stereo Images Fused with ICESat-2 Data

The lack of ground control points (GCPs) affects the elevation accuracy of digital surface models (DSMs) generated by optical satellite stereo images and limits the application of high-resolution DSMs. It is a feasible idea to use ICESat-2 (Ice, Cloud, and land Elevation Satellite-2) laser altimetry data to improve the elevation accuracy of optical stereo images, but it is necessary to accurately match the two types of data. This paper proposes a DSM registration strategy based on terrain similarity (BOTS), which integrates ICESat-2 laser altimetry data without GCPs and improves the DSM elevation accuracy generation from optical satellite stereo pairs. Under different terrain conditions, Worldview-2, SV-1, GF-7, and ZY-3 stereo pairs were used to verify the effectiveness of this method. The experimental results show that the BOTS method proposed in this paper is more robust when there are a large number of abnormal points in the ICESat-2 data or there is a large elevation gap between DSMs. After fusion of ICESat-2 data, the DSM elevation accuracy extracted from the satellite stereo pair is improved by 73~92%, and the root mean square error (RMSE) of Worldview-2 DSM reaches 0.71 m.

Stereo Vision System for Vision-Based Control of Inspection-Class ROVs

The inspection-class Remotely Operated Vehicles (ROVs) are crucial in underwater inspections. Their prime function is to allow the replacing of humans during risky subaquatic operations. These vehicles gather videos from underwater scenes that are sent online to a human operator who provides control. Furthermore, these videos are used for analysis. This demands an RGB camera operating at a close distance to the observed objects. Thus, to obtain a detailed depiction, the vehicle should move with a constant speed and a measured distance from the bottom. As very few inspection-class ROVs possess navigation systems that facilitate these requirements, this study had the objective of designing a vision-based control method to compensate for this limitation. To this end, a stereo vision system and image-feature matching and tracking techniques were employed. As these tasks are challenging in the underwater environment, we carried out analyses aimed at finding fast and reliable image-processing techniques. The analyses, through a sequence of experiments designed to test effectiveness, were carried out in a swimming pool using a VideoRay Pro 4 vehicle. The results indicate that the method under consideration enables automatic control of the vehicle, given that the image features are present in stereo-pair images as well as in consecutive frames captured by the left camera.

CALIBRATION OF SMARTPHONE’S REAR DUAL CAMERA SYSTEM

This paper aims to calibrate smartphone’s rear dual camera system which is composed of two lenses, namely; wide-angle lens and telephoto lens. The proposed approach handles large sized images. Calibration was done by capturing 13 photos for a chessboard pattern from different exposure positions. First, photos were captured in dual camera mode. Then, for both wide-angle and telephoto lenses, image coordinates for node points of the chessboard were extracted. Afterwards, intrinsic, extrinsic, and lens distortion parameters for each lens were calculated. In order to enhance the accuracy of the calibration model, a constrained least-squares solution was applied. The applied constraint was that the relative extrinsic parameters of both wide-angle and telephoto lenses were set as constant regardless of the exposure position. Moreover, photos were rectified in order to eliminate the effect of lens distortion. For results evaluation, two oriented photos were chosen to perform a stereo-pair intersection. Then, the node points of the chessboard pattern were used as check points.

Group-Based Atrous Convolution Stereo Matching Network

Stereo matching is the key technology in stereo vision. Given a pair of rectified images, stereo matching determines correspondences between the pair images and estimate depth by obtaining disparity between corresponding pixels. The current work has shown that depth estimation from a stereo pair of images can be formulated as a supervised learning task with an end-to-end frame based on convolutional neural networks (CNNs). However, 3D CNN puts a great burden on memory storage and computation, which further leads to the significantly increased computation time. To alleviate this issue, atrous convolution was proposed to reduce the number of convolutional operations via a relatively sparse receptive field. However, this sparse receptive field makes it difficult to find reliable corresponding points in fuzzy areas, e.g., occluded areas and untextured areas, owing to the loss of rich contextual information. To address this problem, we propose the Group-based Atrous Convolution Spatial Pyramid Pooling (GASPP) to robustly segment objects at multiple scales with affordable computing resources. The main feature of the GASPP module is to set convolutional layers with continuous dilation rate in each group, so that it can reduce the impact of holes introduced by atrous convolution on network performance. Moreover, we introduce a tailored cascade cost volume in a pyramid form to reduce memory, so as to meet real-time performance. The group-based atrous convolution stereo matching network is evaluated on the street scene benchmark KITTI 2015 and Scene Flow and achieves state-of-the-art performance.

Automated Coastline Extraction Using the Very High Resolution WorldView (WV) Satellite Imagery and Developed Coastline Extraction Tool (CET)

The accurate extraction of a coastline is necessary for various studies of coastal processes, as well as for the management and protection of coastal areas. Very high-resolution satellite imagery has great potential for coastline extraction; however, noises in spectral data can cause significant errors. Here, we present a newly developed Coastal Extraction Tool (CET) that overcomes such errors and allows accurate and time-efficient automated coastline extraction based on a combination of WorldView-2 (WV-2) multispectral imagery and stereo-pair-derived digital surface model (DSM). Coastline extraction is performed and tested on the Iž-Rava island group, situated within the Northern Dalmatian archipelago (Croatia). Extracted coastlines were compared to (a) coastlines extracted from state topographic map (1:25,000), and (b) coastline extracted by another available tool. The accuracy of the extracted coastline was validated with centimeter accuracy reference data acquired using a UAV system (Matrice 600 Pro + MicaSense RedEdge-MX). Within the study area, two small islets were detected that have not been mapped during the earlier coastline mapping efforts. CET proved to be a highly accurate coastline mapping technique that successfully overcomes spectral-induced errors. In future research, we are planning to integrate data obtained by UAVs infrared thermography (IRT) and in situ sensors, measuring sea and land surface temperatures (SST and LST), into the CET, given that this has shown promising results. Considering its accuracy and ease of use, we suggest that CET can be applied for automated coastline extraction in other large and indented coastal areas. Additionally, we suggest that CET could be applied in longitudinal geomorphological coastal erosion studies for the automated detection of spatio-temporal coastline displacement.

Assessment of digital image correlation vibrometry in the presence of thermal flow disturbance

Digital Image Correlation (DIC) is an image-based method for measuring displacement and/or stain on the surface of a structure. When coupled with a stereo pair of highspeed cameras, DIC can also capture three-dimensional dynamic deformation of a structure under vibratory loading. However, high frequency and small amplitude displacement typically associated with structural vibrations mean that extra care is required during measurement and data processing. It becomes more challenging when thermal disturbances are present in the optical path, for example from a heated air flow, which introduces extraneous noise due to disturbances in the refractive index. In the present study, a simple composite plate was vibrated under a shaker excitation and stereo DIC measurements were performed. The obtained vibratory displacement results were compared against accelerometers and a laser Doppler vibrometer. Heated air flow was introduced in front of the plate to observe the effects of thermal disturbances on the DIC measurements. Although the contributions from the thermal disturbances were clearly visible in the DIC displacement data, it was shown that the vibratory deflections of the structure could still be extracted by post processing of the DIC data.

Enhanced Lunar Topographic Mapping Using Multiple Stereo Images Taken by Yutu-2 Rover with Changing Illumination Conditions

In a planetary-rover exploration mission, stereovision-based 3D reconstruction has been widely applied to topographic mapping of the planetary surface using stereo cameras onboard the rover. In this study, we propose an enhanced topographic mapping method based on multiple stereo images taken at the same rover location with changing illumination conditions. Key steps of the method include dense matching of stereo images, 3D point-cloud generation, point-cloud co-registration, and fusion. The final point cloud has more complete coverage and more details of the terrain than that conventionally generated from a single stereo pair. The effectiveness of the proposed method is verified by experiments using the Yutu-2 rover, in which two data sets were acquired by the navigation cameras at two locations and under changing illumination conditions. This method, which does not involve complex operations, has great potential for application in planetary-rover and lander missions.

PanCam: the ‘science eyes’ of the Rosalind Franklin (ExoMars 2022) rover

<p>The scientific objectives of the ExoMars Rosalind Franklin rover [1] are designed to answer several key questions in the search for life on Mars. In particular, the unique subsurface drill will address some of these questions for the first time, such as the possible existence and stability of sub-surface organics. PanCam [2] will establish the surface geological and morphological context for the mission, working in collaboration with other context instruments. Here, we describe the PanCam scientific objectives in geology, atmospheric science and 3D vision. We discuss the design of PanCam, which includes a stereo pair of Wide Angle Cameras (WACs), each of which has an 11 position filter wheel, and a High Resolution Camera (HRC) for high resolution investigations of rock texture at a distance. The cameras and electronics are housed in an optical bench that provides the mechanical interface to the rover mast and a planetary protection barrier.  The electronic interface is via the PanCam Interface Unit (PIU), and power conditioning is via a DC-DC converter. PanCam also includes a calibration target mounted on the rover deck for radiometric calibration, fiducial markers for geometric calibration and a rover inspection mirror. Recent simulations [3] show the view from PanCam, the ‘science eyes’ of the Rosalind Franklin rover.</p> <p><strong>References:</strong></p> <p>[1] Vago, J.L., F. Westall, A.J. Coates, et al., Habitability on Early Mars and the Search for Biosignatures with the ExoMars Rover, <em>Astrobiology</em>, 17(6-7), 471-510, doi:10.1089/ast.2016.1533, Jul 2017.</p> <p>[2] Coates, A.J., R. Jaumann, A.D. Griffiths, et al., The PanCam instrument for the ExoMars rover, <em>Astrobiology</em>, 17 (6-7), 511-541, doi: 10.1089/ast.2016.1548, Jul 2017.</p> <p>[3] Miles, H.C., M.D. Gunn and A.J. Coates, Seeing through the ‘Science Eyes’ of the ExoMars Rover, IEEE Computer Graphics & Applications, Applications Department, 40, 71-81, doi: 10.1109/MCG.2020.2970796, Mar-Apr 2020.</p>