scholarly journals Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

2021 ◽  
Author(s):  
Björn Hommel ◽  
Franz-Josef Wollang ◽  
Veronika Kotova ◽  
Hannes Zacher ◽  
Stefan C. Schmukle
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Automatic Generation ◽  
Short Term ◽  
Item Generation ◽  
Aptitude Testing ◽  
Long Short Term Memory ◽  
Automatic Item Generation

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

scholarly journals Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

Psychometrika ◽  
2021 ◽  
Author(s):  
Björn E. Hommel ◽  
Franz-Josef M. Wollang ◽  
Veronika Kotova ◽  
Hannes Zacher ◽  
Stefan C. Schmukle
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Automatic Generation ◽  
Short Term ◽  
Item Generation ◽  
Aptitude Testing ◽  
Long Short Term Memory ◽  
Automatic Item Generation

AbstractAlgorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

scholarly journals Learning molecular dynamics with simple language model built upon long short-term memory neural network

2020 ◽  
Vol 11 (1) ◽  
Author(s):  
Sun-Ting Tsai ◽  
En-Jui Kuo ◽  
Pratyush Tiwary
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Language Model ◽  
Short Term ◽  
Term Memory ◽  
Molecular Systems ◽  
Simple Language ◽  
Long Short Term Memory

Abstract Recurrent neural networks have led to breakthroughs in natural language processing and speech recognition. Here we show that recurrent networks, specifically long short-term memory networks can also capture the temporal evolution of chemical/biophysical trajectories. Our character-level language model learns a probabilistic model of 1-dimensional stochastic trajectories generated from higher-dimensional dynamics. The model captures Boltzmann statistics and also reproduces kinetics across a spectrum of timescales. We demonstrate how training the long short-term memory network is equivalent to learning a path entropy, and that its embedding layer, instead of representing contextual meaning of characters, here exhibits a nontrivial connectivity between different metastable states in the underlying physical system. We demonstrate our model’s reliability through different benchmark systems and a force spectroscopy trajectory for multi-state riboswitch. We anticipate that our work represents a stepping stone in the understanding and use of recurrent neural networks for understanding the dynamics of complex stochastic molecular systems.

scholarly journals Persian sentences to phoneme sequences conversion based on recurrent neural networks

2016 ◽  
Vol 6 (1) ◽  
pp. 219-225 ◽  
Author(s):  
Yasser Mohseni Behbahani ◽  
Bagher Babaali ◽  
Mussa Turdalyuly
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Short Term Memory ◽  
Contextual Information ◽  
Short Term ◽  
Persian Language ◽  
Short Vowels ◽  
Sequential Labeling ◽  
Long Short Term Memory

AbstractGrapheme to phoneme conversion is one of the main subsystems of Text-to-Speech (TTS) systems. Converting sequence of written words to their corresponding phoneme sequences for the Persian language is more challenging than other languages; because in the standard orthography of this language the short vowels are omitted and the pronunciation ofwords depends on their positions in a sentence. Common approaches used in the Persian commercial TTS systems have several modules and complicated models for natural language processing and homograph disambiguation that make the implementation harder as well as reducing the overall precision of system. In this paper we define the grapheme-to-phoneme conversion as a sequential labeling problem; and use the modified Recurrent Neural Networks (RNN) to create a smart and integrated model for this purpose. The recurrent networks are modified to be bidirectional and equipped with Long-Short Term Memory (LSTM) blocks to acquire most of the past and future contextual information for decision making. The experiments conducted in this paper show that in addition to having a unified structure the bidirectional RNN-LSTM has a good performance in recognizing the pronunciation of the Persian sentences with the precision more than 98 percent.

scholarly journals Using Machine Learning Methods in the Financial Market for Technical Analysis Based on Hybrid Models

2020 ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Hybrid Model ◽  
Deep Neural Network ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Short Term ◽  
Term Memory ◽  
Input Structure ◽  
Long Short Term Memory

In this study, it is presented a new hybrid model based on deep neural networks to predict the direction and magnitude of the Forex market movement in the short term. The overall model presented is based on the scalping strategy and is provided for high frequency transactions. The proposed hybrid model is based on a combination of three models based on deep neural networks. The first model is a deep neural network with a multi-input structure consisting of a combination of Long Short Term Memory layers. The second model is a deep neural network with a multi-input structure made of a combination of one-dimensional Convolutional Neural network layers. The third model has a simpler structure and is a multi-input model of the Multi-Layer Perceptron layers. The overall model was also a model based on the majority vote of three top models. This study showed that models based on Long Short-Term Memory layers provided better results than the other models and even hybrid models with more than 70% accurate.

Comparative Assessment of Image Captioning Models

2020 ◽  
Vol 17 (1) ◽  
pp. 473-478
Author(s):  
Mayank ◽  
Naveen Kumar Gondhi
Keyword(s):  
Neural Networks ◽  
Language Processing ◽  
English Language ◽  
Short Term Memory ◽  
Comparative Assessment ◽  
Image Feature ◽  
Image Captioning ◽  
Short Term ◽  
Long Short Term Memory ◽  
Simple Sentences

Image Captioning is the combination of Computer Vision and Natural Language Processing (NLP) in which simple sentences have been automatically generated describing the content of the image. This paper presents the comparative analysis of different models used for the generation of descriptive English captions for a given image. Feature extractions of the images are done using Convolutional Neural Networks (CNN). These features are then, passed onto Recurrent Neural Networks (RNN) or Long Short-term Memory (LSTM) to generate captions in English language. The evaluation metrics used to appraise the conduct of the models are BLEU score, CIDEr and METEOR.

Sentiment Analysis in the Light of LSTM Recurrent Neural Networks

2018 ◽  
Vol 9 (1) ◽  
pp. 33-39 ◽  
Author(s):  
Subarno Pal ◽  
Soumadip Ghosh ◽  
Amitava Nag
Keyword(s):  
Neural Networks ◽  
Sentiment Analysis ◽  
Recurrent Neural Networks ◽  
Deep Neural Networks ◽  
Short Term Memory ◽  
Short Term ◽  
Different Types ◽  
Long Short Term Memory ◽  
Increase In Accuracy ◽  
Temporal Sequences

Long short-term memory (LSTM) is a special type of recurrent neural network (RNN) architecture that was designed over simple RNNs for modeling temporal sequences and their long-range dependencies more accurately. In this article, the authors work with different types of LSTM architectures for sentiment analysis of movie reviews. It has been showed that LSTM RNNs are more effective than deep neural networks and conventional RNNs for sentiment analysis. Here, the authors explore different architectures associated with LSTM models to study their relative performance on sentiment analysis. A simple LSTM is first constructed and its performance is studied. On subsequent stages, the LSTM layer is stacked one upon another which shows an increase in accuracy. Later the LSTM layers were made bidirectional to convey data both forward and backward in the network. The authors hereby show that a layered deep LSTM with bidirectional connections has better performance in terms of accuracy compared to the simpler versions of LSTM used here.

Sign in / Sign up

Export Citation Format

Share Document