scholarly journals MULE: Multimodal Universal Language Embedding

2020 ◽  
Vol 34 (07) ◽  
pp. 11254-11261 ◽  
Author(s):  
Donghyun Kim ◽  
Kuniaki Saito ◽  
Kate Saenko ◽  
Stan Sclaroff ◽  
Bryan Plummer
Keyword(s):  
Data Augmentation ◽  
Good Representation ◽  
Modular Approach ◽  
Prior Work ◽  
Single Model ◽  
Retrieval Task ◽  
Universal Language ◽  
Sentence Retrieval ◽  
Multilingual Learning ◽  
Language Data

Existing vision-language methods typically support two languages at a time at most. In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages. We accomplish this by learning a single shared Multimodal Universal Language Embedding (MULE) which has been visually-semantically aligned across all languages. Then we learn to relate MULE to visual data as if it were a single language. Our method is not architecture specific, unlike prior work which typically learned separate branches for each language, enabling our approach to easily be adapted to many vision-language methods and tasks. Since MULE learns a single language branch in the multimodal model, we can also scale to support many languages, and languages with fewer annotations can take advantage of the good representation learned from other (more abundant) language data. We demonstrate the effectiveness of our embeddings on the bidirectional image-sentence retrieval task, supporting up to four languages in a single model. In addition, we show that Machine Translation can be used for data augmentation in multilingual learning, which, combined with MULE, improves mean recall by up to 20.2% on a single language compared to prior work, with the most significant gains seen on languages with relatively few annotations. Our code is publicly available1.

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

2021 ◽  
Vol 5 (2) ◽  
pp. 1-30
Author(s):  
HyeonJung Park ◽  
Youngki Lee ◽  
JeongGil Ko
Keyword(s):  
Real Time ◽  
Sign Language ◽  
Data Augmentation ◽  
Language Translation ◽  
Mobile Platforms ◽  
Depth Cameras ◽  
Language Data ◽  
In The Wild ◽  
Environmental Robustness ◽  
Cloud Servers

In this work we present SUGO, a depth video-based system for translating sign language to text using a smartphone's front camera. While exploiting depth-only videos offer benefits such as being less privacy-invasive compared to using RGB videos, it introduces new challenges which include dealing with low video resolutions and the sensors' sensitiveness towards user motion. We overcome these challenges by diversifying our sign language video dataset to be robust to various usage scenarios via data augmentation and design a set of schemes to emphasize human gestures from the input images for effective sign detection. The inference engine of SUGO is based on a 3-dimensional convolutional neural network (3DCNN) to classify a sequence of video frames as a pre-trained word. Furthermore, the overall operations are designed to be light-weight so that sign language translation takes place in real-time using only the resources available on a smartphone, with no help from cloud servers nor external sensing components. Specifically, to train and test SUGO, we collect sign language data from 20 individuals for 50 Korean Sign Language words, summing up to a dataset of ~5,000 sign gestures and collect additional in-the-wild data to evaluate the performance of SUGO in real-world usage scenarios with different lighting conditions and daily activities. Comprehensively, our extensive evaluations show that SUGO can properly classify sign words with an accuracy of up to 91% and also suggest that the system is suitable (in terms of resource usage, latency, and environmental robustness) to enable a fully mobile solution for sign language translation.

A multimodel inference approach to categorical variant choice: construction, priming and frequency effects on the choice between full and contracted forms of am, are and is

2017 ◽  
Vol 13 (2) ◽  
pp. 203-260 ◽  
Author(s):  
Danielle Barth ◽  
Vsevolod Kapatsinski
Keyword(s):  
Model Selection ◽  
Prior Work ◽  
Single Model ◽  
Frequency Effects ◽  
Multimodel Inference ◽  
Classic Problem ◽  
Model Selection Uncertainty ◽  
Corpus Data ◽  
Selection For ◽  
Auxiliary Contraction

AbstractThe present paper presents a multimodel inference approach to linguistic variation, expanding on prior work by Kuperman and Bresnan (2012). We argue that corpus data often present the analyst with high model selection uncertainty. This uncertainty is inevitable given that language is highly redundant: every feature is predictable from multiple other features. However, uncertainty involved in model selection is ignored by the standard method of selecting the single best model and inferring the effects of the predictors under the assumption that the best model is true. Multimodel inference avoids committing to a single model. Rather, we make predictions based on the entire set of plausible models, with contributions of models weighted by the models' predictive value. We argue that multimodel inference is superior to model selection for both the I-Language goal of inferring the mental grammars that generated the corpus, and the E-Language goal of predicting characteristics of future speech samples from the community represented by the corpus. Applying multimodel inference to the classic problem of English auxiliary contraction, we show that the choice between multimodel inference and model selection matters in practice: the best model may contain predictors that are not significant when the full set of plausible models is considered, and may omit predictors that are significant considering the full set of models. We also contribute to the study of English auxiliary contraction. We document the effects of priming, contextual predictability, and specific syntactic constructions and provide evidence against effects of phonological context.

scholarly journals Text-to-speech system for low-resource language using cross-lingual transfer learning and data augmentation

2021 ◽  
Vol 2021 (1) ◽  
Author(s):  
Zolzaya Byambadorj ◽  
Ryota Nishimura ◽  
Altangerel Ayush ◽  
Kengo Ohta ◽  
Norihide Kitaoka
Keyword(s):  
Transfer Learning ◽  
Data Augmentation ◽  
Prediction Models ◽  
Target Language ◽  
Text To Speech ◽  
Paired Data ◽  
Low Resource ◽  
Language Data ◽  
Single Speaker ◽  
Cross Lingual

AbstractDeep learning techniques are currently being applied in automated text-to-speech (TTS) systems, resulting in significant improvements in performance. However, these methods require large amounts of text-speech paired data for model training, and collecting this data is costly. Therefore, in this paper, we propose a single-speaker TTS system containing both a spectrogram prediction network and a neural vocoder for the target language, using only 30 min of target language text-speech paired data for training. We evaluate three approaches for training the spectrogram prediction models of our TTS system, which produce mel-spectrograms from the input phoneme sequence: (1) cross-lingual transfer learning, (2) data augmentation, and (3) a combination of the previous two methods. In the cross-lingual transfer learning method, we used two high-resource language datasets, English (24 h) and Japanese (10 h). We also used 30 min of target language data for training in all three approaches, and for generating the augmented data used for training in methods 2 and 3. We found that using both cross-lingual transfer learning and augmented data during training resulted in the most natural synthesized target speech output. We also compare single-speaker and multi-speaker training methods, using sequential and simultaneous training, respectively. The multi-speaker models were found to be more effective for constructing a single-speaker, low-resource TTS model. In addition, we trained two Parallel WaveGAN (PWG) neural vocoders, one using 13 h of our augmented data with 30 min of target language data and one using the entire 12 h of the original target language dataset. Our subjective AB preference test indicated that the neural vocoder trained with augmented data achieved almost the same perceived speech quality as the vocoder trained with the entire target language dataset. Overall, we found that our proposed TTS system consisting of a spectrogram prediction network and a PWG neural vocoder was able to achieve reasonable performance using only 30 min of target language training data. We also found that by using 3 h of target language data, for training the model and for generating augmented data, our proposed TTS model was able to achieve performance very similar to that of the baseline model, which was trained with 12 h of target language data.

scholarly journals Splits or waves? Trees or webs? How divergence measures and network analysis can unravel language histories

2010 ◽  
Vol 365 (1559) ◽  
pp. 3829-3843 ◽  
Author(s):  
Paul Heggarty ◽  
Warren Maguire ◽  
April McMahon
Keyword(s):  
Phylogenetic Analyses ◽  
Wave Model ◽  
Distance Measures ◽  
Family Tree ◽  
Language Family ◽  
Single Model ◽  
Language Data ◽  
Principled Explanation ◽  
Lexical Data ◽  
Phylogenetic Method

Linguists have traditionally represented patterns of divergence within a language family in terms of either a ‘splits’ model, corresponding to a branching family tree structure, or the wave model, resulting in a (dialect) continuum. Recent phylogenetic analyses, however, have tended to assume the former as a viable idealization also for the latter. But the contrast matters, for it typically reflects different processes in the real world: speaker populations either separated by migrations, or expanding over continuous territory. Since history often leaves a complex of both patterns within the same language family, ideally we need a single model to capture both, and tease apart the respective contributions of each. The ‘network’ type of phylogenetic method offers this, so we review recent applications to language data. Most have used lexical data, encoded as binary or multi-state characters. We look instead at continuous distance measures of divergence in phonetics. Our output networks combine branch- and continuum-like signals in ways that correspond well to known histories (illustrated for Germanic, and particularly English). We thus challenge the traditional insistence on shared innovations, setting out a new, principled explanation for why complex language histories can emerge correctly from distance measures, despite shared retentions and parallel innovations.

scholarly journals National Scale 3D Mapping of Soil pH Using a Data Augmentation Approach

2020 ◽  
Vol 12 (18) ◽  
pp. 2872 ◽  
Author(s):  
Pierre Roudier ◽  
Olivia R. Burge ◽  
Sarah J. Richardson ◽  
James K. McCarthy ◽  
Gerard J. Grealish ◽  
...  
Keyword(s):  
New Zealand ◽  
Soil Ph ◽  
Data Augmentation ◽  
National Scale ◽  
Single Model ◽  
Final Model ◽  
New Approach ◽  
Wide Range ◽  
Regression Framework ◽  
Bottom Depth

Understanding the spatial variation of soil pH is critical for many different stakeholders across different fields of science, because it is a master variable that plays a central role in many soil processes. This study documents the first attempt to map soil pH (1:5 H2O) at high resolution (100 m) in New Zealand. The regression framework used follows the paradigm of digital soil mapping, and a limited number of environmental covariates were selected using variable selection, before calibration of a quantile regression forest model. In order to adapt the outcomes of this work to a wide range of different depth supports, a new approach, which includes depth of sampling as a covariate, is proposed. It relies on data augmentation, a process where virtual observations are drawn from statistical populations constructed using the observed data, based on the top and bottom depth of sampling, and including the uncertainty surrounding the soil pH measurement. A single model can then be calibrated and deployed to estimate pH a various depths. Results showed that the data augmentation routine had a beneficial effect on prediction uncertainties, in particular when reference measurement uncertainties are taken into account. Further testing found that the optimal rate of augmentation for this dataset was 3-fold. Inspection of the final model revealed that the most important variables for predicting soil pH distribution in New Zealand were related to land cover and climate, in particular to soil water balance. The evaluation of this approach on those validation sites set aside before modelling showed very good results (R2=0.65, CCC=0.79, RMSE=0.54), that significantly out-performed existing soil pH information for the country.

scholarly journals Skin Lesion Classification Using Additional Patient Information

2021 ◽  
Vol 2021 ◽  
pp. 1-6
Author(s):  
Qilin Sun ◽  
Chao Huang ◽  
Minjie Chen ◽  
Hui Xu ◽  
Yali Yang
Keyword(s):  
Skin Lesion ◽  
Patient Information ◽  
Data Augmentation ◽  
Challenge Test ◽  
Skin Lesions ◽  
Melanocytic Nevus ◽  
Additional Patient ◽  
Test Time ◽  
Single Model ◽  
Lesion Classification

In this paper, we describe our method for skin lesion classification. The goal is to classify skin lesions based on dermoscopic images to several diagnoses’ classes presented in the HAM (Human Against Machine) dataset: melanoma (MEL), melanocytic nevus (NV), basal cell carcinoma (BCC), actinic keratosis (AK), benign keratosis (BKL), dermatofibroma (DF), and vascular lesion (VASC). We propose a simplified solution which has a better accuracy than previous methods, but only predicted on a single model that is practical for a real-world scenario. Our results show that using a network with additional metadata as input achieves a better classification performance. This metadata includes both the patient information and the extra information during the data augmentation process. On the international skin imaging collaboration (ISIC) 2018 skin lesion classification challenge test set, our algorithm yields a balanced multiclass accuracy of 88.7% on a single model and 89.5% for the embedding solution, which makes it the currently first ranked algorithm on the live leaderboard. To improve the inference accuracy. Test time augmentation (TTA) is applied. We also demonstrate how Grad-CAM is applied in TTA. Therefore, TTA and Grad-CAM can be integrated in heat map generation, which can be very helpful to assist the clinician for diagnosis.

scholarly journals Data Augmentation for Human Keypoint Estimation Deep Learning based Sign Language Translation

Electronics ◽  
2020 ◽  
Vol 9 (8) ◽  
pp. 1257
Author(s):  
Chan-Il Park ◽  
Chae-Bong Sohn
Keyword(s):  
Deep Learning ◽  
Sign Language ◽  
Data Augmentation ◽  
Language Translation ◽  
Video Data ◽  
Training Data ◽  
Learning Performance ◽  
Learning Technology ◽  
Learning Techniques ◽  
Language Data

Deep learning technology has developed constantly and is applied in many fields. In order to correctly apply deep learning techniques, sufficient learning must be preceded. Various conditions are necessary for sufficient learning. One of the most important conditions is training data. Collecting sufficient training data is fundamental, because if the training data are insufficient, deep learning will not be done properly. Many types of training data are collected, but not all of them. So, we may have to collect them directly. Collecting takes a lot of time and hard work. To reduce this effort, the data augmentation method is used to increase the training data. Data augmentation has some common methods, but often requires different methods for specific data. For example, in order to recognize sign language, video data processed with openpose are used. In this paper, we propose a new data augmentation method for sign language data used for learning translation, and we expect to improve the learning performance, according to the proposed method.

scholarly journals Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders

2021 ◽  
Vol 17 (4) ◽  
pp. 1-23
Author(s):  
Nicola Messina ◽  
Giuseppe Amato ◽  
Andrea Esuli ◽  
Fabrizio Falchi ◽  
Claudio Gennaro ◽  
...  
Keyword(s):  
Information Retrieval ◽  
Large Scale ◽  
State Of The Art ◽  
Retrieval Task ◽  
Fine Grained ◽  
Retrieval Systems ◽  
Sentence Retrieval ◽  
Novel Approach ◽  
Sentence Level ◽  
Sentence Matching

Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the task of cross-modal retrieval through image-sentence matching based on word-region alignments, using supervision only at the global image-sentence level. Specifically, we present a novel approach called Transformer Encoder Reasoning and Alignment Network (TERAN). TERAN enforces a fine-grained match between the underlying components of images and sentences (i.e., image regions and words, respectively) to preserve the informative richness of both modalities. TERAN obtains state-of-the-art results on the image retrieval task on both MS-COCO and Flickr30k datasets. Moreover, on MS-COCO, it also outperforms current approaches on the sentence retrieval task. Focusing on scalable cross-modal information retrieval, TERAN is designed to keep the visual and textual data pipelines well separated. Cross-attention links invalidate any chance to separately extract visual and textual features needed for the online search and the offline indexing steps in large-scale retrieval systems. In this respect, TERAN merges the information from the two domains only during the final alignment phase, immediately before the loss computation. We argue that the fine-grained alignments produced by TERAN pave the way toward the research for effective and efficient methods for large-scale cross-modal information retrieval. We compare the effectiveness of our approach against relevant state-of-the-art methods. On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks on the Recall@1 metric. The code used for the experiments is publicly available on GitHub at https://github.com/mesnico/TERAN .

scholarly journals Static hand gesture recognition of Arabic sign language by using deep CNNs

2021 ◽  
Vol 24 (1) ◽  
pp. 178
Author(s):  
Mohammad H. Ismail ◽  
Shefa A. Dawwd ◽  
Fakhradeen H. Ali
Keyword(s):  
Feature Extraction ◽  
Sign Language ◽  
Network Models ◽  
Language Recognition ◽  
Sign Language Recognition ◽  
Single Model ◽  
Neural Network Models ◽  
Testing Dataset ◽  
Language Data ◽  
Arabic Sign Language

An Arabic sign language recognition using two concatenated deep convolution neural network models DenseNet121 & VGG16 is presented. The pre-trained models are fed with images, and then the system can automatically recognize the Arabic sign language. To evaluate the performance of concatenated two models in the Arabic sign language recognition, the red-green-blue (RGB) images for various static signs are collected in a dataset. The dataset comprises 220,000 images for 44 categories: 32 letters, 11 numbers (0:10), and 1 for none. For each of the static signs, there are 5000 images collected from different volunteers. The pre-trained models were used and trained on prepared Arabic sign language data. These models were used after some modification. Also, an attempt has been made to adopt two models from the previously trained models, where they are trained in parallel deep feature extractions. Then they are combined and prepared for the classification stage. The results demonstrate the comparison between the performance of the single model and multi-model. It appears that most of the multi-model is better in feature extraction and classification than the single models. And also show that when depending on the total number of incorrect recognize sign image in training, validation and testing dataset, the best convolutional neural networks (CNN) model in feature extraction and classification Arabic sign language is the DenseNet121 for a single model using and DenseNet121 & VGG16 for multi-model using.

Mind wandering as data augmentation: How mental travel supports abstraction

2020 ◽  
Vol 43 ◽  
Author(s):  
Myrthe Faber
Keyword(s):  
Machine Learning ◽  
Data Augmentation ◽  
Mental Content ◽  
Mind Wandering ◽  
Theoretical Framework ◽  
Important Addition

Abstract Gilead et al. state that abstraction supports mental travel, and that mental travel critically relies on abstraction. I propose an important addition to this theoretical framework, namely that mental travel might also support abstraction. Specifically, I argue that spontaneous mental travel (mind wandering), much like data augmentation in machine learning, provides variability in mental content and context necessary for abstraction.

Sign in / Sign up

Export Citation Format

Share Document