scholarly journals Context-Aware Self-Attention Networks

2019 ◽  
Vol 33 ◽  
pp. 387-394 ◽  
Author(s):  
Baosong Yang ◽  
Jian Li ◽  
Derek F. Wong ◽  
Lidia S. Chao ◽  
Xing Wang ◽  
...  
Keyword(s):  
English Translation ◽  
Contextual Information ◽  
The Self ◽  
Context Aware ◽  
Short Term ◽  
Internal Representations ◽  
Attention Networks ◽  
External Resources ◽  
Attention Model ◽  
Neural Representations

Self-attention model has shown its flexibility in parallel computation and the effectiveness on modeling both long- and short-term dependencies. However, it calculates the dependencies between representations without considering the contextual information, which has proven useful for modeling dependencies among neural representations in various natural language tasks. In this work, we focus on improving self-attention networks through capturing the richness of context. To maintain the simplicity and flexibility of the self-attention networks, we propose to contextualize the transformations of the query and key layers, which are used to calculate the relevance between elements. Specifically, we leverage the internal representations that embed both global and deep contexts, thus avoid relying on external resources. Experimental results on WMT14 English⇒German and WMT17 Chinese⇒English translation tasks demonstrate the effectiveness and universality of the proposed methods. Furthermore, we conducted extensive analyses to quantify how the context vectors participate in the self-attention model.

Stable neural representations of mental states across target people and stimulus modalities

2020 ◽  
Author(s):  
Miriam E. Weaverdyck ◽  
Mark Allen Thornton ◽  
Diana Tamir
Keyword(s):  
Mental State ◽  
Mental States ◽  
Brain Regions ◽  
The Self ◽  
Similarity Analysis ◽  
Multiple Modalities ◽  
Neural Representations ◽  
Conceptual Core ◽  
Stimulus Modalities ◽  
Huge Variety

Each individual experiences mental states in their own idiosyncratic way, yet perceivers are able to accurately understand a huge variety of states across unique individuals. How do they accomplish this feat? Do people think about their own anger in the same ways as another person’s? Is reading about someone’s anxiety the same as seeing it? Here, we test the hypothesis that a common conceptual core unites mental state representations across contexts. Across three studies, participants judged the mental states of multiple targets, including a generic other, the self, a socially close other, and a socially distant other. Participants viewed mental state stimuli in multiple modalities, including written scenarios and images. Using representational similarity analysis, we found that brain regions associated with social cognition expressed stable neural representations of mental states across both targets and modalities. This suggests that people use stable models of mental states across different people and contexts.

scholarly journals Context-Aware Neural Machine Translation for Korean Honorific Expressions

Electronics ◽  
2021 ◽  
Vol 10 (13) ◽  
pp. 1589
Author(s):  
Yongkeun Hwang ◽  
Yanghoon Kim ◽  
Kyomin Jung
Keyword(s):  
Machine Translation ◽  
Deep Neural Networks ◽  
Contextual Information ◽  
Context Aware ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
Sentence Level ◽  
Proposed Model ◽  
The Given ◽  
The Relationship

Neural machine translation (NMT) is one of the text generation tasks which has achieved significant improvement with the rise of deep neural networks. However, language-specific problems such as handling the translation of honorifics received little attention. In this paper, we propose a context-aware NMT to promote translation improvements of Korean honorifics. By exploiting the information such as the relationship between speakers from the surrounding sentences, our proposed model effectively manages the use of honorific expressions. Specifically, we utilize a novel encoder architecture that can represent the contextual information of the given input sentences. Furthermore, a context-aware post-editing (CAPE) technique is adopted to refine a set of inconsistent sentence-level honorific translations. To demonstrate the efficacy of the proposed method, honorific-labeled test data is required. Thus, we also design a heuristic that labels Korean sentences to distinguish between honorific and non-honorific styles. Experimental results show that our proposed method outperforms sentence-level NMT baselines both in overall translation quality and honorific translations.

scholarly journals Prediction of Head Movement in 360-Degree Videos Using Attention Model

Sensors ◽  
2021 ◽  
Vol 21 (11) ◽  
pp. 3678
Author(s):  
Dongwon Lee ◽  
Minji Choi ◽  
Joohyun Lee
Keyword(s):  
Machine Learning ◽  
Short Term Memory ◽  
Moving Average ◽  
The Other ◽  
Learning Models ◽  
Short Term ◽  
Term Memory ◽  
Attention Model ◽  
Long Short Term Memory ◽  
Machine Learning Models

In this paper, we propose a prediction algorithm, the combination of Long Short-Term Memory (LSTM) and attention model, based on machine learning models to predict the vision coordinates when watching 360-degree videos in a Virtual Reality (VR) or Augmented Reality (AR) system. Predicting the vision coordinates while video streaming is important when the network condition is degraded. However, the traditional prediction models such as Moving Average (MA) and Autoregression Moving Average (ARMA) are linear so they cannot consider the nonlinear relationship. Therefore, machine learning models based on deep learning are recently used for nonlinear predictions. We use the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) neural network methods, originated in Recurrent Neural Networks (RNN), and predict the head position in the 360-degree videos. Therefore, we adopt the attention model to LSTM to make more accurate results. We also compare the performance of the proposed model with the other machine learning models such as Multi-Layer Perceptron (MLP) and RNN using the root mean squared error (RMSE) of predicted and real coordinates. We demonstrate that our model can predict the vision coordinates more accurately than the other models in various videos.

Grouping in short-term verbal memory: Is position coded temporally?

2002 ◽  
Vol 55 (2) ◽  
pp. 391-424 ◽  
Author(s):  
Honey L.H. Ng ◽  
Murray T. Maybery
Keyword(s):  
Serial Position ◽  
Verbal Memory ◽  
Contextual Information ◽  
Positional Information ◽  
Stimulus Onset ◽  
Time Dependent ◽  
Phonological Loop ◽  
Loop Model ◽  
Short Term ◽  
Temporal Grouping

The nature of the mechanisms that code item position in serial short-term verbal recall was investigated with reference to temporal grouping phenomena—effects that arise when additional pauses are inserted in a presented list to form groups of items. Several recent models attempt to explain these phenomena by assuming that positional information is retained by associating items with contextual information. According to two of the models—the Phonological Loop model (Hitch, Burgess, Towse, & Culpin, 1996) and the OSCAR model (Brown, Preece, & Hulme, 2000)—contextual information depends critically on the timing of item presentation with reference to group onset. By contrast, according to the Start-End model (Henson, 1998) and a development from it, which we label the Oscillator-Revised Start-End model (Henson & Burgess, 1997), contextual information is independent of time from group onset. Three experiments examined whether coding of position is time dependent. The critical manipulation was to vary stimulus-onset asynchrony from one group to the next in the same list. Lists of consonants were presented visually, but with vocalization in Experiment 1, auditorily in Experiment 2, and auditorily with articulatory suppression in Experiment 3. The pattern of order errors consistently favoured the predictions of the time-independent models over those of the time-dependent models in that across-group transpositions reflected within-group serial position rather than time from group onset. Errors involving intrusions from previous lists also reflected within-group serial position, thereby extending support for the time-independent models.

scholarly journals Study on the Preferred Application-Oriented Index for Mental Fatigue Detection

2018 ◽  
Vol 15 (11) ◽  
pp. 2555 ◽  
Author(s):  
Tianhong Duan ◽  
Nong Zhang ◽  
Kaiway Li ◽  
Xuelin Hou ◽  
Jun Pei
Keyword(s):  
Short Term Memory ◽  
Fatigue Testing ◽  
Mental Fatigue ◽  
The Self ◽  
Short Term ◽  
Flicker Fusion Frequency ◽  
Term Memory ◽  
Severe Fatigue ◽  
Speed Perception ◽  
Fatigue Detection

Most of the research on mental fatigue evaluation has mainly concentrated on some indexes that require sophisticated and large instruments that make the detection of mental fatigue cumbersome, time-consuming, and difficult to apply on a large scale. A quick and sensitive mental fatigue detection index is necessary so that mentally fatigued workers can be alerted in time and take corresponding countermeasures. However, to date, no studies have compared the sensitivity of common objective evaluation indexes. To solve these problems, this study recruited 56 human subjects. These subjects were evaluated using six fatigue indexes: the Stanford sleepiness scale, digital span, digital decoding, short-term memory, critical flicker fusion frequency (CFF), and speed perception deviation. The results of the fatigue tests before and after mental fatigue were compared, and a one-way analysis of variance (ANOVA) was performed on the speed perception deviation. The results indicated the significance of this index. Considering individual differences, the relative fatigue index (RFI) was proposed to compare the sensitivity of the indexes. The results showed that when the self-rated fatigue grade changed from non-fatigue to mild fatigue, the ranges of RFI values for digital span, digital decoding, short-term memory, and CFF were 0.175–0.258, 0.194–0.316, 0.068–0.139, and 0.055–0.075, respectively. Correspondingly, when the self-rated fatigue grade changed to severe fatigue, the ranges of RFI values for the above indexes were 0.415–0.577, 0.482–0.669, 0.329–0.396, and 0.114–0.218, respectively. These results suggest that the sensitivity of the digital decoding, digital span, short-term memory, and CFF decreased sequentially when the self-evaluated fatigue grade changed from no fatigue to mild or severe fatigue. The RFI individuality of the speed perception deviation is highly variable and is not suitable as an evaluation index. In mental fatigue testing, digital decoding testing can provide faster, more convenient, and more accurate results.

scholarly journals A Context-Aware Accurate Wellness Determination (CAAWD) Model for Elderly People Using Lazy Associative Classification

Sensors ◽  
2019 ◽  
Vol 19 (7) ◽  
pp. 1613 ◽  
Author(s):  
Farhan Sabir Ujager ◽  
Azhar Mahmood
Keyword(s):  
Elderly People ◽  
Data Extraction ◽  
Contextual Information ◽  
Elderly Person ◽  
The Elderly ◽  
Sensor Nodes ◽  
Behavior Patterns ◽  
Context Aware ◽  
Classification Rules ◽  
Training Time

Wireless Sensor Network (WSN) based smart homes are proving to be an ideal candidate to provide better healthcare facilities to elderly people in their living areas. Several currently proposed techniques have implementation and usage complexities (such as wearable devices and the charging of these devices) which make these proposed techniques less acceptable for elderly people, while the behavioral analysis based on visual techniques lacks privacy. In this paper, a context-aware accurate wellness determination (CAAWD) model for elderly people is presented, where behavior monitoring information is extracted by using simple sensor nodes attached to household objects and appliances for the analysis of daily, frequent behavior patterns of elderly people in a simple and non-obtrusive manner. A contextual data extraction algorithm (CDEA) is proposed for the generation of contextually comprehensive behavior-training instances for accurate wellness classification. The CDEA presents an activity’s spatial–temporal information along with behavioral contextual correlation aspects (such as the object/appliance of usage and sub-activities of an activity) which are vital for accurate wellness analysis and determination. As a result, the classifier is trained in a more logical manner in the context of behavior parameters which are more relevant for wellness determination. The frequent behavioral patterns are classified using the lazy associative classifier (LAC) for wellness determination. The associative nature of LAC helps to integrate spatial–temporal and related contextual attributes (provided by CDEA) of elderly behavior to generate behavior-focused classification rules. Similarly, LAC provides high accuracy with less training time of the classifier, includes minimum-support behavior patterns, and selects highly accurate classification rules for the classification of a test instance. CAAWD further introduces the ability to contextually validate the authenticity of the already classified instance by taking behavioral contextual information (of the elderly person) from the caregiver. Due to the consideration of spatial–temporal behavior contextual attributes, the use of an efficient classifier, and the ability to contextually validate the classified instances, it has been observed that the CAAWD model out-performs currently proposed techniques in terms of accuracy, precision, and f-measure.

scholarly journals IMPLEMENTASI CONTEXT AWARE PADA SISTEM TRAVEL PLANNER BERBASIS MOBILE

2021 ◽  
Vol 5 (1) ◽  
pp. 23-27
Author(s):  
Viery Darmawan ◽  
◽  
Rengga Asmara ◽  
Ira Prasetyaningrum
Keyword(s):  
Contextual Information ◽  
The Internet ◽  
Context Aware ◽  
New Approach ◽  
Tourist Attractions ◽  
Technological Advances ◽  
Long Time ◽  
Travel Efficiency ◽  
Opening And Closing

In the era of technological advances, tourists will first seek information about the tourist object to be addressed, even tourists often don't have a destination, so they have to search one by one via the internet. In determining travel plans, it is often to see one by one the review of tourist attractions and conclude the results will take a long time, while tourists need actual and fast information to determine the travel plans. In this study, the authors take a new approach, namely by creating a mobile-based travel planner system that compiles travel plans automatically by considering contextual information related to tourist location points, whether of tourist locations during travel days, travel opening and closing hours, so that it will increase travel efficiency without having to do the research manually which takes a long time. The system can also provide travel recommendations based on visitor comments sentiment on Google Places and is equipped with a trip route that will be generated automatically. This research is useful for helping tourists plan their trip actually because of the consideration of contextual information so that it will make it easier and save tourists time.

scholarly journals Efficient Context-Aware Neural Machine Translation with Layer-Wise Weighting and Input-Aware Gating

2020 ◽  
Author(s):  
Hongfei Xu ◽  
Deyi Xiong ◽  
Josef van Genabith ◽  
Qiuhui Liu
Keyword(s):  
Machine Translation ◽  
Contextual Information ◽  
Computational Cost ◽  
Representation Learning ◽  
Vital Role ◽  
Context Aware ◽  
Neural Machine Translation ◽  
Gating Mechanism ◽  
Sentence Level ◽  
Parallel Data

Existing Neural Machine Translation (NMT) systems are generally trained on a large amount of sentence-level parallel data, and during prediction sentences are independently translated, ignoring cross-sentence contextual information. This leads to inconsistency between translated sentences. In order to address this issue, context-aware models have been proposed. However, document-level parallel data constitutes only a small part of the parallel data available, and many approaches build context-aware models based on a pre-trained frozen sentence-level translation model in a two-step training manner. The computational cost of these approaches is usually high. In this paper, we propose to make the most of layers pre-trained on sentence-level data in contextual representation learning, reusing representations from the sentence-level Transformer and significantly reducing the cost of incorporating contexts in translation. We find that representations from shallow layers of a pre-trained sentence-level encoder play a vital role in source context encoding, and propose to perform source context encoding upon weighted combinations of pre-trained encoder layers' outputs. Instead of separately performing source context and input encoding, we propose to iteratively and jointly encode the source input and its contexts and to generate input-aware context representations with a cross-attention layer and a gating mechanism, which resets irrelevant information in context encoding. Our context-aware Transformer model outperforms the recent CADec [Voita et al., 2019c] on the English-Russian subtitle data and is about twice as fast in training and decoding.

Sign in / Sign up

Export Citation Format

Share Document