scholarly journals Y2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences

2019 ◽  
Vol 33 ◽  
pp. 126-133 ◽  
Author(s):  
Zhizhong Han ◽  
Mingyang Shang ◽  
Xiyang Wang ◽  
Yu-Shen Liu ◽  
Matthias Zwicker
Keyword(s):  
Network Architecture ◽  
State Of The Art ◽  
Computational Cost ◽  
Representation Learning ◽  
Semantic Meaning ◽  
3D Shape ◽  
Joint Reconstruction ◽  
Recent Method ◽  
Hierarchical Constraints ◽  
3D Shapes

Jointly learning representations of 3D shapes and text is crucial to support tasks such as cross-modal retrieval or shape captioning. A recent method employs 3D voxels to represent 3D shapes, but this limits the approach to low resolutions due to the computational cost caused by the cubic complexity of 3D voxels. Hence the method suffers from a lack of detailed geometry. To resolve this issue, we propose Y2Seq2Seq, a view-based model, to learn cross-modal representations by joint reconstruction and prediction of view and word sequences. Specifically, the network architecture of Y2Seq2Seq bridges the semantic meaning embedded in the two modalities by two coupled “Y” like sequence-tosequence (Seq2Seq) structures. In addition, our novel hierarchical constraints further increase the discriminability of the cross-modal representations by employing more detailed discriminative information. Experimental results on cross-modal retrieval and 3D shape captioning show that Y2Seq2Seq outperforms the state-of-the-art methods.

scholarly journals MeshNet: Mesh Neural Network for 3D Shape Representation

2019 ◽  
Vol 33 ◽  
pp. 8279-8286 ◽  
Author(s):  
Yutong Feng ◽  
Yifan Feng ◽  
Haoxuan You ◽  
Xibin Zhao ◽  
Yue Gao
Keyword(s):  
Neural Network ◽  
Computer Vision ◽  
Point Cloud ◽  
State Of The Art ◽  
Shape Representation ◽  
Shape Classification ◽  
Retrieval Performance ◽  
3D Shape ◽  
General Architecture ◽  
3D Shapes

Mesh is an important and powerful type of data for 3D shapes and widely studied in the field of computer vision and computer graphics. Regarding the task of 3D shape representation, there have been extensive research efforts concentrating on how to represent 3D shapes well using volumetric grid, multi-view and point cloud. However, there is little effort on using mesh data in recent years, due to the complexity and irregularity of mesh data. In this paper, we propose a mesh neural network, named MeshNet, to learn 3D shape representation from mesh data. In this method, face-unit and feature splitting are introduced, and a general architecture with available and effective blocks are proposed. In this way, MeshNet is able to solve the complexity and irregularity problem of mesh and conduct 3D shape representation well. We have applied the proposed MeshNet method in the applications of 3D shape classification and retrieval. Experimental results and comparisons with the state-of-the-art methods demonstrate that the proposed MeshNet can achieve satisfying 3D shape classification and retrieval performance, which indicates the effectiveness of the proposed method on 3D shape representation.

scholarly journals View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions

2019 ◽  
Vol 33 ◽  
pp. 8376-8384 ◽  
Author(s):  
Zhizhong Han ◽  
Mingyang Shang ◽  
Yu-Shen Liu ◽  
Matthias Zwicker
Keyword(s):  
Network Architecture ◽  
Large Scale ◽  
Shape Representation ◽  
Feature Space ◽  
Representation Learning ◽  
Global Memory ◽  
Inter Prediction ◽  
Aggregate Information ◽  
Local View ◽  
3D Shapes

In this paper, we present a novel unsupervised representation learning approach for 3D shapes, which is an important research challenge as it avoids the manual effort required for collecting supervised data. Our method trains an RNNbased neural network architecture to solve multiple view inter-prediction tasks for each shape. Given several nearby views of a shape, we define view inter-prediction as the task of predicting the center view between the input views, and reconstructing the input views in a low-level feature space. The key idea of our approach is to implement the shape representation as a shape-specific global memory that is shared between all local view inter-predictions for each shape. Intuitively, this memory enables the system to aggregate information that is useful to better solve the view inter-prediction tasks for each shape, and to leverage the memory as a viewindependent shape representation. Our approach obtains the best results using a combination of L2 and adversarial losses for the view inter-prediction task. We show that VIP-GAN outperforms state-of-the-art methods in unsupervised 3D feature learning on three large-scale 3D shape benchmarks.

scholarly journals An Analysis and Application of Fast Nonnegative Orthogonal Matching Pursuit for Image Categorization in Deep Networks

2015 ◽  
Vol 2015 ◽  
pp. 1-9
Author(s):  
Bo Wang ◽  
Jichang Guo ◽  
Yan Zhang
Keyword(s):  
Large Scale ◽  
State Of The Art ◽  
Matching Pursuit ◽  
Computational Cost ◽  
Representation Learning ◽  
Orthogonal Matching Pursuit ◽  
Image Categorization ◽  
Image Patches ◽  
Deep Networks ◽  
Shape Vector

Nonnegative orthogonal matching pursuit (NOMP) has been proven to be a more stable encoder for unsupervised sparse representation learning. However, previous research has shown that NOMP is suboptimal in terms of computational cost, as the coefficients selection and refinement using nonnegative least squares (NNLS) have been divided into two separate steps. It is found that this problem severely reduces the efficiency of encoding for large-scale image patches. In this work, we study fast nonnegative OMP (FNOMP) as an efficient encoder which can be accelerated by the implementation ofQRfactorization and iterations of coefficients in deep networks for full-size image categorization task. It is analyzed and demonstrated that using relatively simple gain-shape vector quantization for training dictionary, FNOMP not only performs more efficiently than NOMP for encoding but also significantly improves the classification accuracy compared to OMP based algorithm. In addition, FNOMP based algorithm is superior to other state-of-the-art methods on several publicly available benchmarks, that is, Oxford Flowers, UIUC-Sports, and Caltech101.

scholarly journals EfficientPose: Scalable single-person pose estimation

2020 ◽  
Author(s):  
Daniel Groos ◽  
Heri Ramampiaro ◽  
Espen AF Ihlen
Keyword(s):  
Pose Estimation ◽  
Network Architecture ◽  
State Of The Art ◽  
Computational Cost ◽  
Real Life ◽  
Low Complexity ◽  
Human Pose Estimation ◽  
Efficient Detection ◽  
Single Person ◽  
Human Pose

Abstract Single-person human pose estimation facilitates markerless movement analysis in sports, as well as in clinical applications. Still, state-of-the-art models for human pose estimation generally do not meet the requirements of real-life applications. The proliferation of deep learning techniques has resulted in the development of many advanced approaches. However, with the progresses in the field, more complex and inefficient models have also been introduced, which have caused tremendous increases in computational demands. To cope with these complexity and inefficiency challenges, we propose a novel convolutional neural network architecture, called EfficientPose, which exploits recently proposed EfficientNets in order to deliver efficient and scalable single-person pose estimation. EfficientPose is a family of models harnessing an effective multi-scale feature extractor and computationally efficient detection blocks using mobile inverted bottleneck convolutions, while at the same time ensuring that the precision of the pose configurations is still improved. Due to its low complexity and efficiency, EfficientPose enables real-world applications on edge devices by limiting the memory footprint and computational cost. The results from our experiments, using the challenging MPII single-person benchmark, show that the proposed EfficientPose models substantially outperform the widely-used OpenPose model both in terms of accuracy and computational efficiency. In particular, our top-performing model achieves state-of-the-art accuracy on single-person MPII, with low-complexity ConvNets.

scholarly journals Siamese CNN-BiLSTM Architecture for 3D Shape Representation Learning

2018 ◽  
Author(s):  
Guoxian Dai ◽  
Jin Xie ◽  
Yi Fang
Keyword(s):  
Neural Network ◽  
Loss Function ◽  
Short Term Memory ◽  
Shape Representation ◽  
Feature Space ◽  
Representation Learning ◽  
3D Shape ◽  
Aggregate Information ◽  
3D Shapes ◽  
2D Images

Learning a 3D shape representation from a collection of its rendered 2D images has been extensively studied. However, existing view-based techniques have not yet fully exploited the information among all the views of projections. In this paper, by employing recurrent neural network to efficiently capture features across different views, we propose a siamese CNN-BiLSTM network for 3D shape representation learning. The proposed method minimizes a discriminative loss function to learn a deep nonlinear transformation, mapping 3D shapes from the original space into a nonlinear feature space. In the transformed space, the distance of 3D shapes with the same label is minimized, otherwise the distance is maximized to a large margin. Specifically, the 3D shapes are first projected into a group of 2D images from different views. Then convolutional neural network (CNN) is adopted to extract features from different view images, followed by a bidirectional long short-term memory (LSTM) to aggregate information across different views. Finally, we construct the whole CNN-BiLSTM network into a siamese structure with contrastive loss function. Our proposed method is evaluated on two benchmarks, ModelNet40 and SHREC 2014, demonstrating superiority over the state-of-the-art methods.

scholarly journals LightOCT: Exploring the depth for Retinal disease detection

2021 ◽  
Author(s):  
Amandeep Kaur ◽  
Vinayak Singh ◽  
Gargi Chakraverty
Keyword(s):  
Neural Network ◽  
Network Architecture ◽  
State Of The Art ◽  
Computational Cost ◽  
Retinal Disease ◽  
Retinal Damage ◽  
Training Time ◽  
Deep Model ◽  
Precise Diagnosis ◽  
Future Work

With the advancement in technology and computation capabilities, identifying retinal damage through state-of-the-art CNNs architectures has led to the speedy and precise diagnosis, thus inhibiting further disease development. In this study, we focus on the classification of retinal damage caused by detecting choroidal neovascularization (CNV), diabetic macular edema (DME), DRUSEN, and NORMAL in optical coherence tomography (OCT) images. The emphasis of our experiment is to investigate the component of depth in the neural network architecture. We introduce a shallow convolution neural network - LightOCT, outperforming the other deep model configurations, with the lowest value of LVCEL and highest accuracy (+98\% in each class). Next, we experimented to find the best fit optimizer for LightOCT. The results proved that the combination of LightOCT and Adam gave the most optimal results. Finally, we compare our approach with transfer learning models, and LightOCT outperforms the state-of-the-art models in terms of computational cost, least training time and gives comparable results in the criteria of accuracy. We would direct our future work to improve the accuracy metrics with shallow models such that the trade-off between training time and accuracy is reduced.

scholarly journals Unsupervised Discovery, Control, and Disentanglement of Semantic Attributes With Applications to Anomaly Detection

2021 ◽  
Vol 33 (3) ◽  
pp. 802-826
Author(s):  
William Paul ◽  
I-Jeng Wang ◽  
Fady Alajaji ◽  
Philippe Burlina
Keyword(s):  
Mutual Information ◽  
Anomaly Detection ◽  
Network Architecture ◽  
State Of The Art ◽  
Representation Learning ◽  
Generative Models ◽  
Superior Performance ◽  
Detection Methods ◽  
Latent Factors ◽  
Semantic Attributes

Our work focuses on unsupervised and generative methods that address the following goals: (1) learning unsupervised generative representations that discover latent factors controlling image semantic attributes, (2) studying how this ability to control attributes formally relates to the issue of latent factor disentanglement, clarifying related but dissimilar concepts that had been confounded in the past, and (3) developing anomaly detection methods that leverage representations learned in the first goal. For goal 1, we propose a network architecture that exploits the combination of multiscale generative models with mutual information (MI) maximization. For goal 2, we derive an analytical result, lemma 1 , that brings clarity to two related but distinct concepts: the ability of generative networks to control semantic attributes of images they generate, resulting from MI maximization, and the ability to disentangle latent space representations, obtained via total correlation minimization. More specifically, we demonstrate that maximizing semantic attribute control encourages disentanglement of latent factors. Using lemma 1 and adopting MI in our loss function, we then show empirically that for image generation tasks, the proposed approach exhibits superior performance as measured in the quality and disentanglement of the generated images when compared to other state-of-the-art methods, with quality assessed via the Fréchet inception distance (FID) and disentanglement via mutual information gap. For goal 3, we design several systems for anomaly detection exploiting representations learned in goal 1 and demonstrate their performance benefits when compared to state-of-the-art generative and discriminative algorithms. Our contributions in representation learning have potential applications in addressing other important problems in computer vision, such as bias and privacy in AI.

scholarly journals Predicting materials properties without crystal structure: deep representation learning from stoichiometry

2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Rhys E. A. Goodall ◽  
Alpha A. Lee
Keyword(s):  
Crystal Structure ◽  
Machine Learning ◽  
State Of The Art ◽  
Computational Cost ◽  
Weighted Graph ◽  
Representation Learning ◽  
Learning Approach ◽  
Materials Properties ◽  
Machine Learning Approach ◽  
Low Computational Cost

AbstractMachine learning has the potential to accelerate materials discovery by accurately predicting materials properties at a low computational cost. However, the model inputs remain a key stumbling block. Current methods typically use descriptors constructed from knowledge of either the full crystal structure — therefore only applicable to materials with already characterised structures — or structure-agnostic fixed-length representations hand-engineered from the stoichiometry. We develop a machine learning approach that takes only the stoichiometry as input and automatically learns appropriate and systematically improvable descriptors from data. Our key insight is to treat the stoichiometric formula as a dense weighted graph between elements. Compared to the state of the art for structure-agnostic methods, our approach achieves lower errors with less data.

scholarly journals Fast 3D Shape Recovery from Shadows Projected on Arbitrary Curved Surfaces

2010 ◽  
Vol 9 (1) ◽  
pp. 1-6
Author(s):  
Yukihiro Yamashita ◽  
Fumihiko Sakaue ◽  
Jun Sato
Keyword(s):  
Shape Recovery ◽  
Computational Cost ◽  
Curved Surfaces ◽  
The Real ◽  
3D Shape ◽  
Planar Surfaces ◽  
Reconstruction Methods ◽  
Real Scene ◽  
3D Shape Recovery ◽  
3D Shapes

The shadow based 3D surface reconstruction methods usually assume that shadows are projected on planar surfaces. However, shadows are often projected on curved surfaces in the real scene. Recently, the shadow graph has been proposed for representing shadow information efficiently, and for recovering 3D shapes from shadows projected on curved surfaces. Unfortunately, the method requires a large computational cost and is weak to the image intensity noises. In this paper, we introduce 1D shadow graphs which can represent shadow information quite efficiently, and can be used for recovering 3D shapes with much smaller computational costs than before. We also extend our method, so that we can recover 3D shape quite accurately by using shading information as well as shadow information. The proposed method is tested by using the real and synthetic images.

scholarly journals DASOT: A Unified Framework Integrating Data Association and Single Object Tracking for Online Multi-Object Tracking

2020 ◽  
Vol 34 (07) ◽  
pp. 10672-10679
Author(s):  
Qi Chu ◽  
Wanli Ouyang ◽  
Bin Liu ◽  
Feng Zhu ◽  
Nenghai Yu
Keyword(s):  
Object Tracking ◽  
Network Architecture ◽  
State Of The Art ◽  
Data Association ◽  
Computational Cost ◽  
Single Object ◽  
Unified Framework ◽  
Convolutional Network ◽  
Comparable Performance ◽  
Feature Sharing

In this paper, we propose an online multi-object tracking (MOT) approach that integrates data association and single object tracking (SOT) with a unified convolutional network (ConvNet), named DASOTNet. The intuition behind integrating data association and SOT is that they can complement each other. Following Siamese network architecture, DASOTNet consists of the shared feature ConvNet, the data association branch and the SOT branch. Data association is treated as a special re-identification task and solved by learning discriminative features for different targets in the data association branch. To handle the problem that the computational cost of SOT grows intolerably as the number of tracked objects increases, we propose an efficient two-stage tracking method in the SOT branch, which utilizes the merits of correlation features and can simultaneously track all the existing targets within one forward propagation. With feature sharing and the interaction between them, data association branch and the SOT branch learn to better complement each other. Using a multi-task objective, the whole network can be trained end-to-end. Compared with state-of-the-art online MOT methods, our method is much faster while maintaining a comparable performance.

Sign in / Sign up

Export Citation Format

Share Document