AttendAffectNet–Emotion Prediction of Movie Viewers Using Multimodal Fusion with Self-Attention

In this paper, we tackle the problem of predicting the affective responses of movie viewers, based on the content of the movies. Current studies on this topic focus on video representation learning and fusion techniques to combine the extracted features for predicting affect. Yet, these typically, while ignoring the correlation between multiple modality inputs, ignore the correlation between temporal inputs (i.e., sequential features). To explore these correlations, a neural network architecture—namely AttendAffectNet (AAN)—uses the self-attention mechanism for predicting the emotions of movie viewers from different input modalities. Particularly, visual, audio, and text features are considered for predicting emotions (and expressed in terms of valence and arousal). We analyze three variants of our proposed AAN: Feature AAN, Temporal AAN, and Mixed AAN. The Feature AAN applies the self-attention mechanism in an innovative way on the features extracted from the different modalities (including video, audio, and movie subtitles) of a whole movie to, thereby, capture the relationships between them. The Temporal AAN takes the time domain of the movies and the sequential dependency of affective responses into account. In the Temporal AAN, self-attention is applied on the concatenated (multimodal) feature vectors representing different subsequent movie segments. In the Mixed AAN, we combine the strong points of the Feature AAN and the Temporal AAN, by applying self-attention first on vectors of features obtained from different modalities in each movie segment and then on the feature representations of all subsequent (temporal) movie segments. We extensively trained and validated our proposed AAN on both the MediaEval 2016 dataset for the Emotional Impact of Movies Task and the extended COGNIMUSE dataset. Our experiments demonstrate that audio features play a more influential role than those extracted from video and movie subtitles when predicting the emotions of movie viewers on these datasets. The models that use all visual, audio, and text features simultaneously as their inputs performed better than those using features extracted from each modality separately. In addition, the Feature AAN outperformed other AAN variants on the above-mentioned datasets, highlighting the importance of taking different features as context to one another when fusing them. The Feature AAN also performed better than the baseline models when predicting the valence dimension.

Platinum and Taxane Based Adjuvant and Neoadjuvant Chemotherapy in Early Triple-Negative Breast Cancer: A Narrative Review

Platinum (Pt) derivatives such as cisplatin and carboplatin are the class of drugs with proven activity against triple-negative breast cancer (TNBC). This is due to the ability of Pt compounds to interfere with the DNA repair mechanisms of the neoplastic cells. Taxanes have been efficacious against estrogen receptor-negative tumors and act by disruption of microtubule function. Due to their distinct mechanisms of action and routes of metabolism, the combination of the Pt agents and taxanes results in reduced systemic toxicity, which is ideal for treating TNBC. Also, the sensitivity of BRCA1-mutated cells to taxanes remains unsolved as in vitro evidence indicates resistance against taxanes due to BRCA1 mutations. Recent evidence suggests that the combination of carboplatin and paclitaxel resulted in better pathological complete response (pCR) in patients with TNBC, both in neoadjuvant and adjuvant settings. In vitro studies showed sequential dependency and optimal time scheduling of Pt- and taxane-based chemotherapy. Also, combining carboplatin with docetaxel in the NAC regimen yields an excellent pCR in patients with BRCA-associated and wild-type TNBC. TNBC is a therapeutic challenge that can be tackled by identifying new therapeutic sub-targets and specific cross-sections that can be benefitted from the addition of Pt- and taxane-based chemotherapy. This review summarizes the merits as well as the mechanism of Pt- and taxane-based adjuvant and neoadjuvant chemotherapies in early TNBC from the available and ongoing clinical studies.

Sequential dependency for affective appraisal of food images

How we perceive the world is not solely determined by our experiences at a given moment in time, but also by what we have experienced in our immediate past. Here, we investigated whether such sequential effects influence the affective appraisal of food images. Participants from 16 different countries (N = 1278) watched a randomly presented sequence of 60 different food images and reported their affective appraisal of each image in terms of valence and arousal. For both measures, we conducted an inter-trial analysis, based on whether the rating on the preceding trial(s) was low or high. The analyses showed that valence and arousal ratings for a given food image are both assimilated towards the ratings on the previous trial (i.e., a positive serial dependence). For a given trial, the arousal rating depends on the arousal ratings up to three trials back. For valence, we observed a positive dependence for the immediately preceding trial only, while a negative (repulsive) dependence was present up to four trials back. These inter-trial effects were larger for males than for females, but independent of the participants’ BMI, age, and cultural background. The results of this exploratory study may be relevant for the design of websites of food delivery services and restaurant menus.

Efficient Energy Management Based on Convolutional Long Short-Term Memory Network for Smart Power Distribution System

An efficient energy management system is integrated with the power grid to collect information about the energy consumption and provide the appropriate control to optimize the supply–demand pattern. Therefore, there is a need for intelligent decisions for the generation and distribution of energy, which is only possible by making the correct future predictions. In the energy market, future knowledge of the energy consumption pattern helps the end-user to decide when to buy or sell the energy to reduce the energy cost and decrease the peak consumption. The Internet of things (IoT) and energy data analytic techniques have provided the convenience to collect the data from the end devices on a large scale and to manipulate all the recorded data. Forecasting an electric load is fairly challenging due to the high uncertainty and dynamic nature involved due to spatiotemporal pattern consumption. Existing conventional forecasting models lack the ability to deal with the spatio-temporally varying data. To overcome the above-mentioned challenges, this work proposes an encoder–decoder model based on convolutional long short-term memory networks (ConvLSTM) for energy load forecasting. The proposed architecture uses encode consisting of multiple ConvLSTM layers to extract the salient features in the data and to learn the sequential dependency and then passes the output to the decoder, having LSTM layers to make forecasting. The forecasting results produced by the proposed approach are favorably comparable to the existing state-of-the-art and better than the conventional methods with the least error rate. Quantitative analyses show that a mean absolute percentage error (MAPE) of 6.966% for household energy consumption and 16.81% for city-wide energy consumption is obtained for the proposed forecasting model in comparison with existing encoder–decoder-based deep learning models for two real-world datasets.

Farey Trees Explain Sequential Effects in Choice Response Time

The latencies of successive two-alternative, forced-choice response times display intricately patterned sequential effects, or dependencies. They vary as a function of particular trial-histories, and in terms of the order and identity of previously presented stimuli and registered responses. This article tests a novel hypothesis that sequential effects are governed by dynamic principles, such as those entailed by a discrete sine-circle map adaptation of the Haken Kelso Bunz (HKB) bimanual coordination model. The model explained the sequential effects expressed in two classic sequential dependency data sets. It explained the rise of a repetition advantage, the acceleration of repeated affirmative responses, in tasks with faster paces. Likewise, the model successfully predicted an alternation advantage, the acceleration of interleaved affirmative and negative responses, when a task’s pace slows and becomes more variable. Detailed analyses of five studies established oscillatory influences on sequential effects in the context of balanced and biased trial presentation rates, variable pacing, progressive and differential cognitive loads, and dyadic performance. Overall, the empirical patterns revealed lawful oscillatory constraints governing sequential effects in the time-course and accuracy of performance across a broad continuum of recognition and decision activities.

Multi-Temporal Unmanned Aerial Vehicle Remote Sensing for Vegetable Mapping Using an Attention-Based Recurrent Convolutional Neural Network

Vegetable mapping from remote sensing imagery is important for precision agricultural activities such as automated pesticide spraying. Multi-temporal unmanned aerial vehicle (UAV) data has the merits of both very high spatial resolution and useful phenological information, which shows great potential for accurate vegetable classification, especially under complex and fragmented agricultural landscapes. In this study, an attention-based recurrent convolutional neural network (ARCNN) has been proposed for accurate vegetable mapping from multi-temporal UAV red-green-blue (RGB) imagery. The proposed model firstly utilizes a multi-scale deformable CNN to learn and extract rich spatial features from UAV data. Afterwards, the extracted features are fed into an attention-based recurrent neural network (RNN), from which the sequential dependency between multi-temporal features could be established. Finally, the aggregated spatial-temporal features are used to predict the vegetable category. Experimental results show that the proposed ARCNN yields a high performance with an overall accuracy of 92.80%. When compared with mono-temporal classification, the incorporation of multi-temporal UAV imagery could significantly boost the accuracy by 24.49% on average, which justifies the hypothesis that the low spectral resolution of RGB imagery could be compensated by the inclusion of multi-temporal observations. In addition, the attention-based RNN in this study outperforms other feature fusion methods such as feature-stacking. The deformable convolution operation also yields higher classification accuracy than that of a standard convolution unit. Results demonstrate that the ARCNN could provide an effective way for extracting and aggregating discriminative spatial-temporal features for vegetable mapping from multi-temporal UAV RGB imagery.

Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation

Non-Autoregressive Neural Machine Translation (NAT) achieves significant decoding speedup through generating target words independently and simultaneously. However, in the context of non-autoregressive translation, the word-level cross-entropy loss cannot model the target-side sequential dependency properly, leading to its weak correlation with the translation quality. As a result, NAT tends to generate influent translations with over-translation and under-translation errors. In this paper, we propose to train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The bag-of-ngrams training objective is differentiable and can be efficiently calculated, which encourages NAT to capture the target-side sequential dependency and correlates well with the translation quality. We validate our approach on three translation tasks and show that our approach largely outperforms the NAT baseline by about 5.0 BLEU scores on WMT14 En↔De and about 2.5 BLEU scores on WMT16 En↔Ro.