scholarly journals Transformers in Vision: A Survey

2022 ◽  
Author(s):  
Salman Khan ◽  
Muzammal Naseer ◽  
Munawar Hayat ◽  
Syed Waqas Zamir ◽  
Fahad Shahbaz Khan ◽  
...  
Keyword(s):  
Computer Vision ◽  
Video Processing ◽  
Architectural Design ◽  
Large Scale ◽  
Question Answering ◽  
Short Term Memory ◽  
Super Resolution ◽  
Input Sequence ◽  
3D Analysis ◽  
Comprehensive Overview

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g. , Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities ( e.g. , images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks ( e.g. , image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks ( e.g. , visual-question answering, visual reasoning, and visual grounding), video processing ( e.g. , activity recognition, video forecasting), low-level vision ( e.g. , image super-resolution, image enhancement, and colorization) and 3D analysis ( e.g. , point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges towards the application of transformer models in computer vision.

scholarly journals Improved ACD-Based Financial Trade Durations Prediction Leveraging LSTM Networks and Attention Mechanism

2021 ◽  
Vol 2021 ◽  
pp. 1-11
Author(s):  
Yong Shi ◽  
Wei Dai ◽  
Wen Long ◽  
Bo Li
Keyword(s):  
Large Scale ◽  
Short Term Memory ◽  
Input Sequence ◽  
Parameter Tuning ◽  
Attention Mechanism ◽  
Market Liquidity ◽  
Security Market ◽  
Modelling Framework ◽  
Acd Model ◽  
The Impact

The liquidity risk factor of security market plays an important role in the formulation of trading strategies. A more liquid stock market means that the securities can be bought or sold more easily. As a sound indicator of market liquidity, the transaction duration is the focus of this study. We concentrate on estimating the probability density function p Δ t i + 1 | G i , where Δ t i + 1 represents the duration of the (i + 1)-th transaction and G i represents the historical information at the time when the (i + 1)-th transaction occurs. In this paper, we propose a new ultrahigh-frequency (UHF) duration modelling framework by utilizing long short-term memory (LSTM) networks to extend the conditional mean equation of classic autoregressive conditional duration (ACD) model while retaining the probabilistic inference ability. And then, the attention mechanism is leveraged to unveil the internal mechanism of the constructed model. In order to minimize the impact of manual parameter tuning, we adopt fixed hyperparameters during the training process. The experiments applied to a large-scale dataset prove the superiority of the proposed hybrid models. In the input sequence, the temporal positions which are more important for predicting the next duration can be efficiently highlighted via the added attention mechanism layer.

scholarly journals Explicit Knowledge-based Reasoning for Visual Question Answering

2017 ◽  
Author(s):  
Peng Wang ◽  
Qi Wu ◽  
Chunhua Shen ◽  
Anthony Dick ◽  
Anton van den Hengel
Keyword(s):  
Natural Language ◽  
Large Scale ◽  
Question Answering ◽  
Explicit Knowledge ◽  
Short Term Memory ◽  
Short Term ◽  
Term Memory ◽  
Visual Question Answering ◽  
Knowledge Based ◽  
Long Short Term Memory

We describe a method for visual question answering which is capable of reasoning about an image on the basis of information extracted from a large-scale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can explain the reasoning by which it developed its answer. It is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in testing. We also provide a dataset and a protocol by which to evaluate general visual question answering methods.

scholarly journals A review of collective robotic construction

2019 ◽  
Vol 4 (28) ◽  
pp. eaau8479 ◽  
Author(s):  
Kirstin H. Petersen ◽  
Nils Napp ◽  
Robert Stuart-Smith ◽  
Daniela Rus ◽  
Mirko Kovac
Keyword(s):  
Architectural Design ◽  
Large Scale ◽  
Performance Metrics ◽  
Research Area ◽  
Sustainable Construction ◽  
Shared Environment ◽  
Comprehensive Overview ◽  
Active Research ◽  
High Level ◽  
Robotic Construction

The increasing need for safe, inexpensive, and sustainable construction, combined with novel technological enablers, has made large-scale construction by robot teams an active research area. Collective robotic construction (CRC) specifically concerns embodied, autonomous, multirobot systems that modify a shared environment according to high-level user-specified goals. CRC tightly integrates architectural design, the construction process, mechanisms, and control to achieve scalability and adaptability. This review gives a comprehensive overview of research trends, open questions, and performance metrics.

scholarly journals Artificial Intelligence & Machine Learning in Computer Vision Applications

2020 ◽  
Vol 7 (1) ◽  
pp. 2-3
Author(s):  
Shadi Saleh
Keyword(s):  
Artificial Intelligence ◽  
Machine Learning ◽  
Neural Networks ◽  
Computer Vision ◽  
Deep Learning ◽  
Large Scale ◽  
Short Term Memory ◽  
Graphics Processing Unit ◽  
Multimedia Data ◽  
Processing Unit

Deep learning and machine learning innovations are at the core of the ongoing revolution in Artificial Intelligence for the interpretation and analysis of multimedia data. The convergence of large-scale datasets and more affordable Graphics Processing Unit (GPU) hardware has enabled the development of neural networks for data analysis problems that were previously handled by traditional handcrafted features. Several deep learning architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short Term Memory (LSTM)/Gated Recurrent Unit (GRU), Deep Believe Networks (DBN), and Deep Stacking Networks (DSNs) have been used with new open source software and libraries options to shape an entirely new scenario in computer vision processing.

Modern Technology for Image processing and Computer vision -A Review

2018 ◽  
Vol 1 (2) ◽  
pp. 17-23
Author(s):  
Takialddin Al Smadi
Keyword(s):  
Image Processing ◽  
Computer Vision ◽  
Augmented Reality ◽  
Video Compression ◽  
Video Processing ◽  
Vision System ◽  
Citrus Fruit ◽  
Modern Technology ◽  
Image And Video Processing ◽  
Depth Video

This survey outlines the use of computer vision in Image and video processing in multidisciplinary applications; either in academia or industry, which are active in this field.The scope of this paper covers the theoretical and practical aspects in image and video processing in addition of computer vision, from essential research to evolution of application.In this paper a various subjects of image processing and computer vision will be demonstrated ,these subjects are spanned from the evolution of mobile augmented reality (MAR) applications, to augmented reality under 3D modeling and real time depth imaging, video processing algorithms will be discussed to get higher depth video compression, beside that in the field of mobile platform an automatic computer vision system for citrus fruit has been implemented ,where the Bayesian classification with Boundary Growing to detect the text in the video scene. Also the paper illustrates the usability of the handed interactive method to the portable projector based on augmented reality.   © 2018 JASET, International Scholars and Researchers Association

Dehydroepiandrosterone (DHEA) and its Sulphate (DHEAS) in Alzheimer’s Disease

2020 ◽  
Vol 17 (2) ◽  
pp. 141-157 ◽  
Author(s):  
Dubravka S. Strac ◽  
Marcela Konjevod ◽  
Matea N. Perkovic ◽  
Lucija Tudor ◽  
Gordana N. Erjavec ◽  
...  
Keyword(s):  
Alzheimer’S Disease ◽  
Alzheimer's Disease ◽  
Large Scale ◽  
Therapeutic Potential ◽  
Relevant Literature ◽  
Comprehensive Overview ◽  
Brain Functions ◽  
Specific Effects ◽  
And Behavior

Background: Neurosteroids Dehydroepiandrosterone (DHEA) and Dehydroepiandrosterone Sulphate (DHEAS) are involved in many important brain functions, including neuronal plasticity and survival, cognition and behavior, demonstrating preventive and therapeutic potential in different neuropsychiatric and neurodegenerative disorders, including Alzheimer’s disease. Objective: The aim of the article was to provide a comprehensive overview of the literature on the involvement of DHEA and DHEAS in Alzheimer’s disease. Method: PubMed and MEDLINE databases were searched for relevant literature. The articles were selected considering their titles and abstracts. In the selected full texts, lists of references were searched manually for additional articles. Results: We performed a systematic review of the studies investigating the role of DHEA and DHEAS in various in vitro and animal models, as well as in patients with Alzheimer’s disease, and provided a comprehensive discussion on their potential preventive and therapeutic applications. Conclusion: Despite mixed results, the findings of various preclinical studies are generally supportive of the involvement of DHEA and DHEAS in the pathophysiology of Alzheimer’s disease, showing some promise for potential benefits of these neurosteroids in the prevention and treatment. However, so far small clinical trials brought little evidence to support their therapy in AD. Therefore, large-scale human studies are needed to elucidate the specific effects of DHEA and DHEAS and their mechanisms of action, prior to their applications in clinical practice.

scholarly journals NPU RGB+D Dataset and a Feature-Enhanced LSTM-DGCN Method for Action Recognition of Basketball Players

2021 ◽  
Vol 11 (10) ◽  
pp. 4426
Author(s):  
Chunyan Ma ◽  
Ji Fan ◽  
Jinghao Yao ◽  
Tao Zhang
Keyword(s):  
Action Recognition ◽  
Large Scale ◽  
Short Term Memory ◽  
Evaluation Criteria ◽  
Image Data ◽  
Basketball Player ◽  
Basketball Players ◽  
Convolutional Network ◽  
Atomic Actions ◽  
New Feature

Computer vision-based action recognition of basketball players in basketball training and competition has gradually become a research hotspot. However, owing to the complex technical action, diverse background, and limb occlusion, it remains a challenging task without effective solutions or public dataset benchmarks. In this study, we defined 32 kinds of atomic actions covering most of the complex actions for basketball players and built the dataset NPU RGB+D (a large scale dataset of basketball action recognition with RGB image data and Depth data captured in Northwestern Polytechnical University) for 12 kinds of actions of 10 professional basketball players with 2169 RGB+D videos and 75 thousand frames, including RGB frame sequences, depth maps, and skeleton coordinates. Through extracting the spatial features of the distances and angles between the joint points of basketball players, we created a new feature-enhanced skeleton-based method called LSTM-DGCN for basketball player action recognition based on the deep graph convolutional network (DGCN) and long short-term memory (LSTM) methods. Many advanced action recognition methods were evaluated on our dataset and compared with our proposed method. The experimental results show that the NPU RGB+D dataset is very competitive with the current action recognition algorithms and that our LSTM-DGCN outperforms the state-of-the-art action recognition methods in various evaluation criteria on our dataset. Our action classifications and this NPU RGB+D dataset are valuable for basketball player action recognition techniques. The feature-enhanced LSTM-DGCN has a more accurate action recognition effect, which improves the motion expression ability of the skeleton data.

scholarly journals An Attention-Based Multilayer GRU Model for Multistep-Ahead Short-Term Load Forecasting

Sensors ◽  
2021 ◽  
Vol 21 (5) ◽  
pp. 1639
Author(s):  
Seungmin Jung ◽  
Jihoon Moon ◽  
Sungwoo Park ◽  
Eenjun Hwang
Keyword(s):  
Power Consumption ◽  
Prediction Models ◽  
Short Term Memory ◽  
Load Forecasting ◽  
Input Sequence ◽  
Short Term ◽  
Performance Improvements ◽  
Short Term Load Forecasting ◽  
Significant Performance ◽  
Input Variables

Recently, multistep-ahead prediction has attracted much attention in electric load forecasting because it can deal with sudden changes in power consumption caused by various events such as fire and heat wave for a day from the present time. On the other hand, recurrent neural networks (RNNs), including long short-term memory and gated recurrent unit (GRU) networks, can reflect the previous point well to predict the current point. Due to this property, they have been widely used for multistep-ahead prediction. The GRU model is simple and easy to implement; however, its prediction performance is limited because it considers all input variables equally. In this paper, we propose a short-term load forecasting model using an attention based GRU to focus more on the crucial variables and demonstrate that this can achieve significant performance improvements, especially when the input sequence of RNN is long. Through extensive experiments, we show that the proposed model outperforms other recent multistep-ahead prediction models in the building-level power consumption forecasting.

scholarly journals Global soil moisture data derived through machine learning trained with in-situ measurements

2021 ◽  
Vol 8 (1) ◽  
Author(s):  
Sungmin O. ◽  
Rene Orth
Keyword(s):  
Machine Learning ◽  
Soil Moisture ◽  
Large Scale ◽  
Short Term Memory ◽  
Temporal Dynamics ◽  
Soil Moisture Data ◽  
Wide Range ◽  
Global Soil

AbstractWhile soil moisture information is essential for a wide range of hydrologic and climate applications, spatially-continuous soil moisture data is only available from satellite observations or model simulations. Here we present a global, long-term dataset of soil moisture derived through machine learning trained with in-situ measurements, SoMo.ml. We train a Long Short-Term Memory (LSTM) model to extrapolate daily soil moisture dynamics in space and in time, based on in-situ data collected from more than 1,000 stations across the globe. SoMo.ml provides multi-layer soil moisture data (0–10 cm, 10–30 cm, and 30–50 cm) at 0.25° spatial and daily temporal resolution over the period 2000–2019. The performance of the resulting dataset is evaluated through cross validation and inter-comparison with existing soil moisture datasets. SoMo.ml performs especially well in terms of temporal dynamics, making it particularly useful for applications requiring time-varying soil moisture, such as anomaly detection and memory analyses. SoMo.ml complements the existing suite of modelled and satellite-based datasets given its distinct derivation, to support large-scale hydrological, meteorological, and ecological analyses.

scholarly journals A Survey on Contrastive Self-Supervised Learning

Technologies ◽  
2020 ◽  
Vol 9 (1) ◽  
pp. 2
Author(s):  
Ashish Jaiswal ◽  
Ashwin Ramesh Babu ◽  
Mohammad Zaki Zadeh ◽  
Debapriya Banerjee ◽  
Fillia Makedon
Keyword(s):  
Computer Vision ◽  
Supervised Learning ◽  
Language Processing ◽  
Large Scale ◽  
Performance Comparison ◽  
Extensive Review ◽  
Future Directions ◽  
Dominant Component ◽  
Supervised Methods ◽  
The Cost

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating large-scale datasets. It is capable of adopting self-defined pseudolabels as supervision and use the learned representations for several downstream tasks. Specifically, contrastive learning has recently become a dominant component in self-supervised learning for computer vision, natural language processing (NLP), and other domains. It aims at embedding augmented versions of the same sample close to each other while trying to push away embeddings from different samples. This paper provides an extensive review of self-supervised methods that follow the contrastive approach. The work explains commonly used pretext tasks in a contrastive learning setup, followed by different architectures that have been proposed so far. Next, we present a performance comparison of different methods for multiple downstream tasks such as image classification, object detection, and action recognition. Finally, we conclude with the limitations of the current methods and the need for further techniques and future directions to make meaningful progress.

Sign in / Sign up

Export Citation Format

Share Document