scholarly journals Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey

2020 ◽  
Author(s):  
Benyamin Ghojogh ◽  
Ali Ghodsi
Keyword(s):  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Attention Mechanism ◽  
Survey Paper

This is a tutorial and survey paper on the attention mechanism, transformers, BERT, and GPT. We first explain attention mechanism, sequence-to-sequence model without and with attention, self-attention, and attention in different areas such as natural language processing and computer vision. Then, we explain transformers which do not use any recurrence. We explain all the parts of encoder and decoder in the transformer, including positional encoding, multihead self-attention and cross-attention, and masked multihead attention. Thereafter, we introduce the Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT) as the stacks of encoders and decoders of transformer, respectively. We explain their characteristics and how they work.

Efficient Channel Attention Based Encoder–Decoder Approach for Image Captioning in Hindi

2022 ◽  
Vol 21 (3) ◽  
pp. 1-17
Author(s):  
Santosh Kumar Mishra ◽  
Gaurav Rai ◽  
Sriparna Saha ◽  
Pushpak Bhattacharyya
Keyword(s):  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
English Language ◽  
Image Understanding ◽  
Attention Mechanism ◽  
Image Captioning ◽  
Textual Description ◽  
Hindi Language

Image captioning refers to the process of generating a textual description that describes objects and activities present in a given image. It connects two fields of artificial intelligence, computer vision, and natural language processing. Computer vision and natural language processing deal with image understanding and language modeling, respectively. In the existing literature, most of the works have been carried out for image captioning in the English language. This article presents a novel method for image captioning in the Hindi language using encoder–decoder based deep learning architecture with efficient channel attention. The key contribution of this work is the deployment of an efficient channel attention mechanism with bahdanau attention and a gated recurrent unit for developing an image captioning model in the Hindi language. Color images usually consist of three channels, namely red, green, and blue. The channel attention mechanism focuses on an image’s important channel while performing the convolution, which is basically to assign higher importance to specific channels over others. The channel attention mechanism has been shown to have great potential for improving the efficiency of deep convolution neural networks (CNNs). The proposed encoder–decoder architecture utilizes the recently introduced ECA-NET CNN to integrate the channel attention mechanism. Hindi is the fourth most spoken language globally, widely spoken in India and South Asia; it is India’s official language. By translating the well-known MSCOCO dataset from English to Hindi, a dataset for image captioning in Hindi is manually created. The efficiency of the proposed method is compared with other baselines in terms of Bilingual Evaluation Understudy (BLEU) scores, and the results obtained illustrate that the method proposed outperforms other baselines. The proposed method has attained improvements of 0.59%, 2.51%, 4.38%, and 3.30% in terms of BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores, respectively, with respect to the state-of-the-art. Qualities of the generated captions are further assessed manually in terms of adequacy and fluency to illustrate the proposed method’s efficacy.

A Hindi Image Caption Generation Framework Using Deep Learning

2021 ◽  
Vol 20 (2) ◽  
pp. 1-19
Author(s):  
Santosh Kumar Mishra ◽  
Rijul Dhir ◽  
Sriparna Saha ◽  
Pushpak Bhattacharyya
Keyword(s):  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
English Language ◽  
Image Captioning ◽  
Textual Description ◽  
Proposed Model ◽  
Hindi Language ◽  
The Given

Image captioning is the process of generating a textual description of an image that aims to describe the salient parts of the given image. It is an important problem, as it involves computer vision and natural language processing, where computer vision is used for understanding images, and natural language processing is used for language modeling. A lot of works have been done for image captioning for the English language. In this article, we have developed a model for image captioning in the Hindi language. Hindi is the official language of India, and it is the fourth most spoken language in the world, spoken in India and South Asia. To the best of our knowledge, this is the first attempt to generate image captions in the Hindi language. A dataset is manually created by translating well known MSCOCO dataset from English to Hindi. Finally, different types of attention-based architectures are developed for image captioning in the Hindi language. These attention mechanisms are new for the Hindi language, as those have never been used for the Hindi language. The obtained results of the proposed model are compared with several baselines in terms of BLEU scores, and the results show that our model performs better than others. Manual evaluation of the obtained captions in terms of adequacy and fluency also reveals the effectiveness of our proposed approach. Availability of resources : The codes of the article are available at https://github.com/santosh1821cs03/Image_Captioning_Hindi_Language ; The dataset will be made available: http://www.iitp.ac.in/∼ai-nlp-ml/resources.html .

Computer Vision and Natural Language Processing

2017 ◽  
Vol 49 (4) ◽  
pp. 1-44 ◽  
Author(s):  
Peratham Wiriyathammabhum ◽  
Douglas Summers-Stay ◽  
Cornelia Fermüller ◽  
Yiannis Aloimonos
Keyword(s):  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing

The Concept of Integrating Artificial Intelligence Technologies Into Human Resources in a Digital Paradigm

2020 ◽  
Vol 9 (2) ◽  
pp. 5-9
Author(s):  
Oksana Chulanova
Keyword(s):  
Artificial Intelligence ◽  
Computer Vision ◽  
Natural Language Processing ◽  
Decision Support ◽  
Speech Recognition ◽  
Human Resources ◽  
Natural Language ◽  
Language Processing

The article discusses the capabilities of artificial intelligence technologies - technologies based on the use of artificial intelligence, including natural language processing, intellectual decision support, computer vision, speech recognition and synthesis, and promising methods of artificial intelligence. The results of the author's study and the analysis of artificial intelligence technologies and their capabilities for optimizing work with staff are presented. A study conducted by the author allowed us to develop an author's concept of integrating artificial intelligence technologies into work with personnel in the digital paradigm.

scholarly journals Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering

2021 ◽  
pp. 111-127
Author(s):  
Rajat Koner ◽  
Hang Li ◽  
Marcel Hildebrandt ◽  
Deepan Das ◽  
Volker Tresp ◽  
...  
Keyword(s):  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Human Performance ◽  
Question Answering ◽  
Scene Graph ◽  
Visual Question Answering ◽  
Learning Agent ◽  
Modal Reasoning

AbstractVisual Question Answering (VQA) is concerned with answering free-form questions about an image. Since it requires a deep semantic and linguistic understanding of the question and the ability to associate it with various objects that are present in the image, it is an ambitious task and requires multi-modal reasoning from both computer vision and natural language processing. We propose Graphhopper, a novel method that approaches the task by integrating knowledge graph reasoning, computer vision, and natural language processing techniques. Concretely, our method is based on performing context-driven, sequential reasoning based on the scene entities and their semantic and spatial relationships. As a first step, we derive a scene graph that describes the objects in the image, as well as their attributes and their mutual relationships. Subsequently, a reinforcement learning agent is trained to autonomously navigate in a multi-hop manner over the extracted scene graph to generate reasoning paths, which are the basis for deriving answers. We conduct an experimental study on the challenging dataset GQA, based on both manually curated and automatically generated scene graphs. Our results show that we keep up with human performance on manually curated scene graphs. Moreover, we find that Graphhopper outperforms another state-of-the-art scene graph reasoning model on both manually curated and automatically generated scene graphs by a significant margin.

scholarly journals An Overview of Image Caption Generation Methods

2020 ◽  
Vol 2020 ◽  
pp. 1-13 ◽  
Author(s):  
Haoran Wang ◽  
Yue Zhang ◽  
Xiaosheng Yu
Keyword(s):  
Artificial Intelligence ◽  
Computer Vision ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Rapid Development ◽  
Evaluation Criteria ◽  
Arduous Task ◽  
Image Caption Generation ◽  
Image Caption

In recent years, with the rapid development of artificial intelligence, image caption has gradually attracted the attention of many researchers in the field of artificial intelligence and has become an interesting and arduous task. Image caption, automatically generating natural language descriptions according to the content observed in an image, is an important part of scene understanding, which combines the knowledge of computer vision and natural language processing. The application of image caption is extensive and significant, for example, the realization of human-computer interaction. This paper summarizes the related methods and focuses on the attention mechanism, which plays an important role in computer vision and is recently widely used in image caption generation tasks. Furthermore, the advantages and the shortcomings of these methods are discussed, providing the commonly used datasets and evaluation criteria in this field. Finally, this paper highlights some open challenges in the image caption task.

PhraseAttn: Dynamic Slot Capsule Networks for phrase representation in Neural Machine Translation

2021 ◽  
pp. 1-8
Author(s):  
Binh Nguyen ◽  
Binh Le ◽  
Long H.B. Nguyen ◽  
Dien Dinh
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Machine Translation ◽  
Language Processing ◽  
Vital Role ◽  
Attention Mechanism ◽  
Neural Machine Translation ◽  
Translation Model ◽  
Word Representation

 Word representation plays a vital role in most Natural Language Processing systems, especially for Neural Machine Translation. It tends to capture semantic and similarity between individual words well, but struggle to represent the meaning of phrases or multi-word expressions. In this paper, we investigate a method to generate and use phrase information in a translation model. To generate phrase representations, a Primary Phrase Capsule network is first employed, then iteratively enhancing with a Slot Attention mechanism. Experiments on the IWSLT English to Vietnamese, French, and German datasets show that our proposed method consistently outperforms the baseline Transformer, and attains competitive results over the scaled Transformer with two times lower parameters.

High accuracy offering attention mechanisms based deep learning approach using CNN/bi-LSTM for sentiment analysis

2021 ◽  
Vol ahead-of-print (ahead-of-print) ◽  
Author(s):  
Venkateswara Rao Kota ◽  
Shyamala Devi Munisamy
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Sentiment Analysis ◽  
Language Processing ◽  
Attention Mechanism ◽  
Supervised Machine Learning ◽  
Method Performance ◽  
Content Type

PurposeNeural network (NN)-based deep learning (DL) approach is considered for sentiment analysis (SA) by incorporating convolutional neural network (CNN), bi-directional long short-term memory (Bi-LSTM) and attention methods. Unlike the conventional supervised machine learning natural language processing algorithms, the authors have used unsupervised deep learning algorithms.Design/methodology/approachThe method presented for sentiment analysis is designed using CNN, Bi-LSTM and the attention mechanism. Word2vec word embedding is used for natural language processing (NLP). The discussed approach is designed for sentence-level SA which consists of one embedding layer, two convolutional layers with max-pooling, one LSTM layer and two fully connected (FC) layers. Overall the system training time is 30 min.FindingsThe method performance is analyzed using metrics like precision, recall, F1 score, and accuracy. CNN is helped to reduce the complexity and Bi-LSTM is helped to process the long sequence input text.Originality/valueThe attention mechanism is adopted to decide the significance of every hidden state and give a weighted sum of all the features fed as input.

scholarly journals Enabling Intelligence through Deep Learning using IoT in a Classroom Environment based on a multimodal approach

2021 ◽  
Vol 12 (2) ◽  
pp. 381-393
Author(s):  
Lakshaga Jyothi M, Et. al.
Keyword(s):  
Computer Vision ◽  
Deep Learning ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Classroom Environment ◽  
Educational Institution ◽  
Daily Lives ◽  
Learning Principles ◽  
Intelligent Classroom

Smart Classrooms are becoming very popular nowadays. The boom of recent technologies such as the Internet of Things, thanks to those technologies that are tremendously equipping every corner of a diverse set of fields. Every educational institution has set some benchmark on adopting these technologies in their daily lives. But due to some constraints and setbacks, these IoT technological embodiments in the educational sector is still in the premature stage. The major success of any technological evolution is based on its full-fledged implementation to fit the society in the broader concern. The breakthrough in recent years by Deep Learning principles as it outperforms traditional machine learning models to solve any tasks especially, Computer Vision and Natural language processing problems.  A fusion of Computer Vision and Natural Language Processing as a new astonishing field that have shown its existence in the recent years. Using such mixtures with the IoT platforms is a challenging task and and has not reached the eyes of many researchers across the globe.  Many researchers of the past have shown interest in designing an intelligent classroom on a different context. Hence to fill this gap, we have proposed an approach or a conceptual model through which Deep Learning architectures fused in the IoT systems results in an Intelligent Classroom via such hybrid systems. Apart from this, we have also discussed the major challenges, limitations as well as opportunities that can arise with Deep Learning-based IoT Solutions. In this paper, we have summarized the available applications of these technologies to suit our solution.  Thus, this paper can be taken as a kickstart for our research to have a glimpse of the available papers for the success of our proposed approach.

Sign in / Sign up

Export Citation Format

Share Document