Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation

Algorithmic automatic item generation can be used to obtain large quantities of cognitive items in the domains of knowledge and aptitude testing. However, conventional item models used by template-based automatic item generation techniques are not ideal for the creation of items for non-cognitive constructs. Progress in this area has been made recently by employing long short-term memory recurrent neural networks to produce word sequences that syntactically resemble items typically found in personality questionnaires. To date, such items have been produced unconditionally, without the possibility of selectively targeting personality domains. In this article, we offer a brief synopsis on past developments in natural language processing and explain why the automatic generation of construct-specific items has become attainable only due to recent technological progress. We propose that pre-trained causal transformer models can be fine-tuned to achieve this task using implicit parameterization in conjunction with conditional generation. We demonstrate this method in a tutorial-like fashion and finally compare aspects of validity in human- and machine-authored items using empirical data. Our study finds that approximately two-thirds of the automatically generated items show good psychometric properties (factor loadings above .40) and that one-third even have properties equivalent to established and highly curated human-authored items. Our work thus demonstrates the practical use of deep neural networks for non-cognitive automatic item generation.

Standardized Achievement Testing, Aptitude Testing, and Attitude Testing: How Similar or Different are these Concepts in Educational Assessment?

Testing is an inevitable issue in educational and psychological measurement and assessment. Over the years, several tests and testing mechanisms have been developed to assess different latent traits of learners or examinees. Of these testing forms, standardized achievement, aptitude and attitude testing have taken the forefronts in education, psychology, and research. Whereas these concepts are distinct, they, however, play similar roles which have been misconceived by many, especially those without expertise in educational assessment and measurement. The aim of this paper is to compare standardized achievement testing, aptitude testing, and attitude testing, using a narrative literature review approach. The paper provides a synthesis of gathered information that delineates conditions necessary for the utilization of each testing approach.

Stages of Acquisition and the P/E Model of Working Memory: Complementary or contrasting approaches to foreign language aptitude?

This paper explores the roles of both working memory (WM) and more traditional aptitude components, such as input processing and language analytic ability in the context of foreign language learning aptitude. More specifically, the paper compares two current perspectives on language aptitude: the Stages Approach (Skehan, 2016, 2019) and the P/E Model (Wen, 2016, 2019). Input processing and noticing, pattern identification and complexification, and feedback are examined as they relate to both perspectives and are then used to discuss existing aptitude testing, recent research, and broader theoretical issues. It is argued that WM and language aptitude play different but complementary roles at each of these stages, reflecting the various linguistic and psycholinguistic processes that are most prominent in other aspects of language learning. Overall, though both perspectives posit that WM and language aptitude have equal importance at the input processing stage, they exert greater influence at each of the remaining stages. More traditional views of aptitude dominate at the pattern identification and complexification stage and WM with the feedback stage.

Testing Hearing

This introduction frames Testing Hearing: The Making of Modern Aurality by showing how the modern cultural practices of hearing and testing have emerged from a long interrelationship. Since the early nineteenth century, auditory test tools and the results of hearing tests have fed back into instrument calibration, human training, architecture, and new musical sounds. Hearing tests received a further boost around 1900 as a result of injury compensation laws and professional demands for aptitude testing. Applied on a large scale, tests of seemingly small measure—of auditory acuity, of hearing range—helped redefine the modern concept of hearing as such. During the twentieth and twenty-first centuries, the epistemic function of hearing expanded. Hearing took on the dual role of test object and test instrument; in the latter case, human hearing became a gauge by which to evaluate or regulate materials, nonhuman organisms, equipment, and technological systems. Testing hearing has been an enduring cultural technique in the modern period, situated between histories of scientific experimentation and fields of application.

Spin-Offs: A Better World Through Military Psychology

Military research routinely yields spin-offs that are useful in the civilian domain. In the hard sciences, World War I spun off advances in chemistry, and World War II produced advances in physics that changed the world. Military psychological science is no different. Aptitude testing sprung from the efforts of psychologists during World War I to help the military better select and classify incoming personnel. Clinical psychology and human factors engineering were boosted as a result of World War II. The Vietnam conflict led to a better understanding of combat stress and contributed to the including of posttraumatic stress disorder as a diagnostic label. All had direct application to the civilian sector. This chapter considers spin-offs from contemporary military psychological research that will benefit general society including better ways to treat stress and promote resilience, select and train employees, and enhance leadership strategies and cultural skills.

Psychological Science and the Art of War

This chapter explores the history of military psychology and its influence on war. Beginning with World War I and continuing to today’s military operations, psychology has provided the military with better ways to select, train, develop, and lead soldiers in combat. Notable contributions of military psychology include aptitude testing, human factors engineering, clinical psychology, cyber technology, and positive psychology. Military psychologists may be civilians or uniformed members of all branches of service. They are employed in universities, government laboratories, hospitals, and nongovernment organizations including corporations and private consulting firms. The Society for Military Psychology is a founding division of the American Psychological Association. Given that the human element is the most important factor in warfare, military psychology is an essential science for winning the wars of today and tomorrow.

Education and the Future

How can educators prepare their students for the many changes brought about by the information explosion? It is taking longer for students to finish their undergraduate degrees; more women are successful with earning their degrees that are men. Occupations are disappearing and new ones appearing at increasing speed. Education is one approach, but a changed and more responsive education system will be necessary. Students must learn how to be lifelong learners and discover their interests and talents through aptitude testing followed by counseling. Though of great age, the lecture remains one of the best ways to transmit information and will likely last, but in a modified form. These changes will profoundly affect politics, economics, the environment, and other areas of our lives.

Standardized Entrance/Aptitude Testing in the Era of Foucault’s Panopticon: The Bilingual Student and the Minority Student

Standardized testing is an applauded system of testing due to the uniformity that it offers. The idea is that in standardized testing, because every student is being asked exactly the same question and each question has only one specific answer, standardized examinations are neutral, value free, and exonerated from the subjectivity that an examiner or teacher may inhibit. The reality is far from it. Using a Foucauldian panoptic perspective and focusing on what is known as the aptitude or entrance examination, I argue that standardized examinations are designed in such a way that bilingual and minority students shall not score on par with their monolingual majority counterparts. The questions are designed in such a way that those students who code switch (due to bilingualism) are placed at a disadvantage. Similarly, the culture represented in the examination is White middle class, hence making the examination relatively more difficult for minority students.