scholarly journals Preliminary Results on Different Text Processing Tasks Using Encoder-Decoder Networks and the Causal Feature Extractor

2020 ◽  
Vol 10 (17) ◽  
pp. 5772
Author(s):  
Adrián Javaloy ◽  
Ginés García-Mateos
Keyword(s):  
Language Processing ◽  
Network Architecture ◽  
State Of The Art ◽  
Text Processing ◽  
Short Term ◽  
Training Time ◽  
Preliminary Results ◽  
Feature Extractor ◽  
Decoder Architecture ◽  
Different Types

Deep learning methods are gaining popularity in different application domains, and especially in natural language processing. It is commonly believed that using a large enough dataset and an adequate network architecture, almost any processing problem can be solved. A frequent and widely used typology is the encoder-decoder architecture, where the input data is transformed into an intermediate code by means of an encoder, and then a decoder takes this code to produce its output. Different types of networks can be used in the encoder and the decoder, depending on the problem of interest, such as convolutional neural networks (CNN) or long-short term memories (LSTM). This paper uses for the encoder a method recently proposed, called Causal Feature Extractor (CFE). It is based on causal convolutions (i.e., convolutions that depend only on one direction of the input), dilatation (i.e., increasing the aperture size of the convolutions) and bidirectionality (i.e., independent networks in both directions). Some preliminary results are presented on three different tasks and compared with state-of-the-art methods: bilingual translation, LaTeX decompilation and audio transcription. The proposed method achieves promising results, showing its ubiquity to work with text, audio and images. Moreover, it has a shorter training time, requiring less time per iteration, and a good use of the attention mechanisms based on attention matrices.

scholarly journals Text Normalization Using Encoder–Decoder Networks Based on the Causal Feature Extractor

2020 ◽  
Vol 10 (13) ◽  
pp. 4551 ◽  
Author(s):  
Adrián Javaloy ◽  
Ginés García-Mateos
Keyword(s):  
Language Processing ◽  
Convergence Time ◽  
Intermediate Form ◽  
Fine Grained ◽  
Word Level ◽  
Feature Extractor ◽  
Decoder Architecture ◽  
Different Types ◽  
Main Ideas ◽  
Text Normalization

The encoder–decoder architecture is a well-established, effective and widely used approach in many tasks of natural language processing (NLP), among other domains. It consists of two closely-collaborating components: An encoder that transforms the input into an intermediate form, and a decoder producing the output. This paper proposes a new method for the encoder, named Causal Feature Extractor (CFE), based on three main ideas: Causal convolutions, dilatations and bidirectionality. We apply this method to text normalization, which is a ubiquitous problem that appears as the first step of many text-to-speech (TTS) systems. Given a text with symbols, the problem consists in writing the text exactly as it should be read by the TTS system. We make use of an attention-based encoder–decoder architecture using a fine-grained character-level approach rather than the usual word-level one. The proposed CFE is compared to other common encoders, such as convolutional neural networks (CNN) and long-short term memories (LSTM). Experimental results show the feasibility of CFE, achieving better results in terms of accuracy, number of parameters, convergence time, and use of an attention mechanism based on attention matrices. The obtained accuracy ranges from 83.5% to 96.8% correctly normalized sentences, depending on the dataset. Moreover, the proposed method is generic and can be applied to different types of input such as text, audio and images.

scholarly journals Fighting Together against the Pandemic: Learning Multiple Models on Tomography Images for COVID-19 Diagnosis

AI ◽  
2021 ◽  
Vol 2 (2) ◽  
pp. 261-273
Author(s):  
Mario Manzo ◽  
Simone Pellino
Keyword(s):  
Network Architecture ◽  
State Of The Art ◽  
Ensemble Classification ◽  
Effective Vaccine ◽  
Rt Pcr ◽  
Neural Network Architecture ◽  
Experimental Phase ◽  
Different Types ◽  
Polymerase Chain ◽  
Better Than

COVID-19 has been a great challenge for humanity since the year 2020. The whole world has made a huge effort to find an effective vaccine in order to save those not yet infected. The alternative solution is early diagnosis, carried out through real-time polymerase chain reaction (RT-PCR) tests or thorax Computer Tomography (CT) scan images. Deep learning algorithms, specifically convolutional neural networks, represent a methodology for image analysis. They optimize the classification design task, which is essential for an automatic approach with different types of images, including medical. In this paper, we adopt a pretrained deep convolutional neural network architecture in order to diagnose COVID-19 disease from CT images. Our idea is inspired by what the whole of humanity is achieving, as the set of multiple contributions is better than any single one for the fight against the pandemic. First, we adapt, and subsequently retrain for our assumption, some neural architectures that have been adopted in other application domains. Secondly, we combine the knowledge extracted from images by the neural architectures in an ensemble classification context. Our experimental phase is performed on a CT image dataset, and the results obtained show the effectiveness of the proposed approach with respect to the state-of-the-art competitors.

scholarly journals An Optimized Abstractive Text Summarization Model Using Peephole Convolutional LSTM

Symmetry ◽  
2019 ◽  
Vol 11 (10) ◽  
pp. 1290 ◽  
Author(s):  
Rahman ◽  
Siddiqui
Keyword(s):  
Language Processing ◽  
Short Term Memory ◽  
State Of The Art ◽  
Text Summarization ◽  
Short Term ◽  
Term Memory ◽  
Semantic Coherence ◽  
Long Short Term Memory ◽  
Central Composite ◽  
Convolutional Lstm

Abstractive text summarization that generates a summary by paraphrasing a long text remains an open significant problem for natural language processing. In this paper, we present an abstractive text summarization model, multi-layered attentional peephole convolutional LSTM (long short-term memory) (MAPCoL) that automatically generates a summary from a long text. We optimize parameters of MAPCoL using central composite design (CCD) in combination with the response surface methodology (RSM), which gives the highest accuracy in terms of summary generation. We record the accuracy of our model (MAPCoL) on a CNN/DailyMail dataset. We perform a comparative analysis of the accuracy of MAPCoL with that of the state-of-the-art models in different experimental settings. The MAPCoL also outperforms the traditional LSTM-based models in respect of semantic coherence in the output summary.

scholarly journals Measuring the Evolution of a Scientific Field through Citation Frames

2018 ◽  
Vol 6 ◽  
pp. 391-406 ◽  
Author(s):  
David Jurgens ◽  
Srijan Kumar ◽  
Raine Hoover ◽  
Dan McFarland ◽  
Dan Jurafsky
Keyword(s):  
Natural Language Processing ◽  
Citation Analysis ◽  
Language Processing ◽  
State Of The Art ◽  
Citation Count ◽  
Scientific Field ◽  
Discourse Structure ◽  
Small Scale ◽  
Behavioral Study ◽  
Different Types

Citations have long been used to characterize the state of a scientific field and to identify influential works. However, writers use citations for different purposes, and this varied purpose influences uptake by future scholars. Unfortunately, our understanding of how scholars use and frame citations has been limited to small-scale manual citation analysis of individual papers. We perform the largest behavioral study of citations to date, analyzing how scientific works frame their contributions through different types of citations and how this framing affects the field as a whole. We introduce a new dataset of nearly 2,000 citations annotated for their function, and use it to develop a state-of-the-art classifier and label the papers of an entire field: Natural Language Processing. We then show how differences in framing affect scientific uptake and reveal the evolution of the publication venues and the field as a whole. We demonstrate that authors are sensitive to discourse structure and publication venue when citing, and that how a paper frames its work through citations is predictive of the citation count it will receive. Finally, we use changes in citation framing to show that the field of NLP is undergoing a significant increase in consensus.

scholarly journals A transformer-based approach to irony and sarcasm detection

2020 ◽  
Vol 32 (23) ◽  
pp. 17309-17320
Author(s):  
Rolandos Alexandros Potamias ◽  
Georgios Siolas ◽  
Andreas - Georgios Stafylopatis
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Network Architecture ◽  
Figurative Language ◽  
State Of The Art ◽  
Unresolved Issue ◽  
Discussion Forums ◽  
Large Margin ◽  
Neural Architecture ◽  
Benchmark Datasets

AbstractFigurative language (FL) seems ubiquitous in all social media discussion forums and chats, posing extra challenges to sentiment analysis endeavors. Identification of FL schemas in short texts remains largely an unresolved issue in the broader field of natural language processing, mainly due to their contradictory and metaphorical meaning content. The main FL expression forms are sarcasm, irony and metaphor. In the present paper, we employ advanced deep learning methodologies to tackle the problem of identifying the aforementioned FL forms. Significantly extending our previous work (Potamias et al., in: International conference on engineering applications of neural networks, Springer, Berlin, pp 164–175, 2019), we propose a neural network methodology that builds on a recently proposed pre-trained transformer-based network architecture which is further enhanced with the employment and devise of a recurrent convolutional neural network. With this setup, data preprocessing is kept in minimum. The performance of the devised hybrid neural architecture is tested on four benchmark datasets, and contrasted with other relevant state-of-the-art methodologies and systems. Results demonstrate that the proposed methodology achieves state-of-the-art performance under all benchmark datasets, outperforming, even by a large margin, all other methodologies and published studies.

A text-based multi-span network for reading comprehension

2021 ◽  
pp. 1-13
Author(s):  
Deguang Chen ◽  
Ziping Ma ◽  
Lin Wei ◽  
Yanbin Zhu ◽  
Jinlin Ma ◽  
...  
Keyword(s):  
Reading Comprehension ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
State Of The Art ◽  
Market Value ◽  
The State ◽  
Experimental Results ◽  
Training Time ◽  
The Public

Text-based reading comprehension models have great research significance and market value and are one of the main directions of natural language processing. Reading comprehension models of single-span answers have recently attracted more attention and achieved significant results. In contrast, multi-span answer models for reading comprehension have been less investigated and their performances need improvement. To address this issue, in this paper, we propose a text-based multi-span network for reading comprehension, ALBERT_SBoundary, and build a multi-span answer corpus, MultiSpan_NMU. We also conduct extensive experiments on the public multi-span corpus, MultiSpan_DROP, and our multi-span answer corpus, MultiSpan_NMU, and compare the proposed method with the state-of-the-art. The experimental results show that our proposed method achieves F1 scores of 84.10 and 92.88 on MultiSpan_DROP and MultiSpan_NMU datasets, respectively, while it also has fewer parameters and a shorter training time.

scholarly journals Stacked Boosters Network Architecture for Short-Term Load Forecasting in Buildings

Energies ◽  
2020 ◽  
Vol 13 (9) ◽  
pp. 2370 ◽  
Author(s):  
Tuukka Salmi ◽  
Jussi Kiljander ◽  
Daniel Pakkala
Keyword(s):  
Time Series ◽  
Network Architecture ◽  
State Of The Art ◽  
Multivariate Time Series ◽  
Load Forecasting ◽  
Building Energy ◽  
Short Term ◽  
Univariate Time Series ◽  
Short Term Load Forecasting ◽  
Base Learner

This paper presents a novel deep learning architecture for short-term load forecasting of building energy loads. The architecture is based on a simple base learner and multiple boosting systems that are modelled as a single deep neural network. The architecture transforms the original multivariate time series into multiple cascading univariate time series. Together with sparse interactions, parameter sharing and equivariant representations, this approach makes it possible to combat against overfitting while still achieving good presentation power with a deep network architecture. The architecture is evaluated in several short-term load forecasting tasks with energy data from an office building in Finland. The proposed architecture outperforms state-of-the-art load forecasting model in all the tasks.

scholarly journals Vietnamese Text Classification Algorithm using Long Short Term Memory and Word2Vec

2020 ◽  
Vol 19 (6) ◽  
pp. 1255-1279
Author(s):  
Huu Nguyen Phat ◽  
Nguyen Thi Minh Anh
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Industrial Revolution ◽  
Short Term Memory ◽  
Text Processing ◽  
Research Effort ◽  
Short Term ◽  
Term Memory ◽  
Long Short Term Memory

In the context of the ongoing forth industrial revolution and fast computer science development the amount of textual information becomes huge. So, prior to applying the seemingly appropriate methodologies and techniques to the above data processing their nature and characteristics should be thoroughly analyzed and understood. At that, automatic text processing incorporated in the existing systems may facilitate many procedures. So far, text classification is one of the basic applications to natural language processing accounting for such factors as emotions’ analysis, subject labeling etc. In particular, the existing advancements in deep learning networks demonstrate that the proposed methods may fit the documents’ classifying, since they possess certain extra efficiency; for instance, they appeared to be effective for classifying texts in English. The thorough study revealed that practically no research effort was put into an expertise of the documents in Vietnamese language. In the scope of our study, there is not much research for documents in Vietnamese. The development of deep learning models for document classification has demonstrated certain improvements for texts in Vietnamese. Therefore, the use of long short term memory network with Word2vec is proposed to classify text that improves both performance and accuracy. The here developed approach when compared with other traditional methods demonstrated somewhat better results at classifying texts in Vietnamese language. The evaluation made over datasets in Vietnamese shows an accuracy of over 90%; also the proposed approach looks quite promising for real applications.

scholarly journals Audio Captioning with Composition of Acoustic and Semantic Information

2021 ◽  
Vol 15 (02) ◽  
pp. 143-160
Author(s):  
Ayşegül Özkaya Eren ◽  
Mustafa Sert
Keyword(s):  
Language Processing ◽  
Semantic Information ◽  
State Of The Art ◽  
Research Area ◽  
Audio Features ◽  
Audio Clip ◽  
Proposed Model ◽  
Decoder Architecture ◽  
Gated Recurrent Units ◽  
New Research

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder–decoder-based models without considering semantic information. To fill this gap, we present a novel encoder–decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-based encoder–decoder model. To enable semantic embeddings for the test audios, we introduce a Multilayer Perceptron classifier to predict the semantic embeddings of those clips. We also present exhaustive experiments to show the efficiency of different features and datasets for our proposed model the audio captioning task. To extract audio features, we use the log Mel energy features, VGGish embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive experiments on two audio captioning datasets Clotho and AudioCaps show that our proposed model outperforms state-of-the-art audio captioning models across different evaluation metrics and using the semantic information improves the captioning performance.

scholarly journals DBDNet: Learning Bi-directional Dynamics for Early Action Prediction

2019 ◽  
Author(s):  
Guoliang Pang ◽  
Xionghui Wang ◽  
Jian-Fang Hu ◽  
Qing Zhang ◽  
Wei-Shi Zheng
Keyword(s):  
Short Term Memory ◽  
State Of The Art ◽  
Action Prediction ◽  
Unified Framework ◽  
Short Term ◽  
Term Memory ◽  
Early Action ◽  
Decoder Architecture ◽  
Long Short Term Memory ◽  
Future Direction

Predicting future actions from observed partial videos is very challenging as the missing future is uncertain and sometimes has multiple possibilities. To obtain a reliable future estimation, a novel encoder-decoder architecture is proposed for integrating the tasks of synthesizing future motions from observed videos and reconstructing observed motions from synthesized future motions in an unified framework, which can capture the bi-directional dynamics depicted in partial videos along the temporal (past-to-future) direction and reverse chronological (future-back-to-past) direction. We then employ a bi-directional long short-term memory (Bi-LSTM) architecture to exploit the learned bi-directional dynamics for predicting early actions. Our experiments on two benchmark action datasets show that learning bi-directional dynamics benefits the early action prediction and our system clearly outperforms the state-of-the-art methods.

Sign in / Sign up

Export Citation Format

Share Document