Lab Scale Model Experiment of Smart Hopper System to Remove Blockages Using Machine Vision and Collaborative Robot

In this study, we propose a smart hopper system that automatically unblocks obstructions caused by rocks dropped into hoppers at mining sites. The proposed system captures RGB (red green blue) and D (depth) images of the upper surfaces of hopper models using an RGB-D camera and transmits them to a computer. Then, a virtual hopper system is used to identify rocks via machine vision-based image processing techniques, and an appropriate motion is simulated in a robot arm. Based on the simulation, the robot arm moves to the location of the rock in the real world and removes it from the actual hopper. The recognition accuracy of the proposed model is evaluated in terms of the quantity and location of rocks. The results confirm that rocks are accurately recognized at all positions in the hopper by the proposed system.

Improvement of generalization capability of two-dimensional salt segmentation via iterative semi-supervised learning

Accurate interpretations of subsurface salts are vital to oil and gas exploration. Manually interpreting them from seismic depth images, however, is labor-intensive. Consequently, use of deep learning tools such as a convolutional neural network for automatic salt interpretation recently became popular. Because of poor generalization capabilities, interpreting salt boundaries using these tools is difficult when labeled data are available from one geological region and we like to make predictions for other nearby regions with varied geological features. At the same time, due to vast amount of the data involved and the associated computational complexities needed for training, such generalization is necessary for solving practical salt interpretation problems. In this work, we propose a semi-supervised training, which allows the predicted model to iteratively improve as more and more information is distilled from the unlabeled data into the model. In addition, by performing mixup between labeled and unlabeled data during training, we encourage the predicted models to linearly behave across training samples; thereby improving the generalization capability of the method. For each iteration, we use the model obtained from previous iteration to generate pseudo labels for the unlabeled data. This automated consecutive data distillation allows our model prediction to improve with iteration, without any need for human intervention. To demonstrate the effectiveness and efficiency, we apply the method on two-dimensional images extracted from a real three-dimensional seismic data volume. By comparing our predictions and fully supervised baseline predictions with those that were manually interpreted and we consider as “ground truth”, we find than the prediction quality our new method surpasses the baseline prediction. We therefore conclude that our new method is a viable tool for automated salt delineation from seismic depth images.

Blending Colored and Depth CNN Pipelines in an Ensemble Learning Classification Approach for Warehouse Application Using Synthetic and Real Data

Electric companies face flow control and inventory obstacles such as reliability, outlays, and time-consuming tasks. Convolutional Neural Networks (CNNs) combined with computational vision approaches can process image classification in warehouse management applications to tackle this problem. This study uses synthetic and real images applied to CNNs to deal with classification of inventory items. The results are compared to seek the neural networks that better suit this application. The methodology consists of fine-tuning several CNNs on Red–Green–Blue (RBG) and Red–Green–Blue-Depth (RGB-D) synthetic and real datasets, using the best architecture of each domain in a blended ensemble approach. The proposed blended ensemble approach was not yet explored in such an application, using RGB and RGB-D data, from synthetic and real domains. The use of a synthetic dataset improved accuracy, precision, recall and f1-score in comparison with models trained only on the real domain. Moreover, the use of a blend of DenseNet and Resnet pipelines for colored and depth images proved to outperform accuracy, precision and f1-score performance indicators over single CNNs, achieving an accuracy measurement of 95.23%. The classification task is a real logistics engineering problem handled by computer vision and artificial intelligence, making full use of RGB and RGB-D images of synthetic and real domains, applied in an approach of blended CNN pipelines.

Loop Closure Detection in RGB-D SLAM by Utilizing Siamese ConvNet Features

Loop closure detection is a key challenge in visual simultaneous localization and mapping (SLAM) systems, which has attracted significant research interest in recent years. It entails correctly determining whether a scene has previously been visited by a mobile robot and completely establishing the consistent maps of motion. There are many loop closure detection methods that have been proposed, but most of these algorithms are handcrafted features-based and perform weak robustness to illumination variations. In this paper, we investigate a Siamese Convolutional Neural Network (SCNN) to solve the task of loop closure detection in RGB-D SLAM. Firstly, we use a pre-trained SCNN model to extract features as image descriptors; then, the L2 norm distance is adopted as a similarity metric between descriptors. In terms of the learned features for matching, there are two key issues for discussion: (1) how to define an appropriate loss as supervision (utilizing the cross-entropy loss, the contrastive loss, or the combination of two); and (2) how to combine the appearance information in RGB images and position information in depth images (utilizing early fusion, mid-level fusion or late fusion). We compare our proposed method of different baseline by experiments carried out on two public datasets (New College and NYU), and our performance outperforms the state-of-the-art.

The Constraints between Edge Depth and Uncertainty for Monocular Depth Estimation

The self-supervised monocular depth estimation paradigm has become an important branch of computer vision depth-estimation tasks. However, the depth estimation problem arising from object edge depth pulling or occlusion is still unsolved. The grayscale discontinuity of object edges leads to a relatively high depth uncertainty of pixels in these regions. We improve the geometric edge prediction results by taking uncertainty into account in the depth-estimation task. To this end, we explore how uncertainty affects this task and propose a new self-supervised monocular depth estimation technique based on multi-scale uncertainty. In addition, we introduce a teacher–student architecture in models and investigate the impact of different teacher networks on the depth and uncertainty results. We evaluate the performance of our paradigm in detail on the standard KITTI dataset. The experimental results show that the accuracy of our method increased from 87.7% to 88.2%, the AbsRel error rate decreased from 0.115 to 0.11, the SqRel error rate decreased from 0.903 to 0.822, and the RMSE error rate decreased from 4.863 to 4.686 compared with the benchmark Monodepth2. Our approach has a positive impact on the problem of texture replication or inaccurate object boundaries, producing sharper and smoother depth images.

Segmentation of Tomato Growing Truss a Depth Image Conversion Model Based on CycleGAN

Background: The truss on tomato plants is a group or cluster of smaller stems where flowers and fruit develop, while a growing truss is the most extended part of the stem. Because the state of the growing truss reacts sensitively to the surrounding environment, it is essential to control the growth in the early stages. With the recent development of IT and artificial intelligence technology in agriculture, a previous study developed a real-time acquisition and evaluation method for images using robots. Further, we used image processing to locate the growing truss and flowering rooms to extract growth information such as the height of the flower room and hard crab. Among the different vision algorithms, the CycleGAN algorithm was used to generate and transform unpaired images using generatively learning images. In this study, we developed a robot-based system for simultaneously acquiring RGB and depth images of the tomato growing truss and flower room groups.Results: The segmentation performance for approximately 35 samples was compared through the false negative (FN) and false positive (FP) indicators. For the depth camera image, we obtained FN as 17.55±3.01% and FP as 17.76±3.55%. Similarly, for CycleGAN, we obtained FN as approximately 19.24±1.45% and FP as 18.24±1.54%. As a result of image processing through depth image, IoU was 63.56 ± 8.44%, and when segmentation was performed through CycelGAN, IoU was 69.25 ± 4.42%, indicating that CycleGAN is advantageous in extracting the desired growing truss. Conclusions: The scannability was confirmed when the image scanning robot drove in a straight line through the plantation in the tomato greenhouse, which confirmed the on-site possibility of the image extraction technique using CycleGAN. In the future, the proposed approach is expected to be used in vision technology to scan the tomato growth indicators in greenhouses using an unmanned robot platform.