From byproduct to design factor: on validating the interpretation of process indicators based on log data

AbstractInternational large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers’ interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker’s work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation.

Download Full-text

A Diagnostic Framework for the Empirical Evaluation of Learning Maps

Frontiers in Education ◽

10.3389/feduc.2021.714736 ◽

2022 ◽

Vol 6 ◽

Author(s):

W. Jake Thompson ◽

Brooke Nash

Keyword(s):

Large Scale ◽

Empirical Evaluation ◽

Essential Elements ◽

Research Literature ◽

Assessment System ◽

Content Standards ◽

Dynamic Learning ◽

Diagnostic Classification Models ◽

Diagnostic Framework ◽

Using Data

Learning progressions and learning map structures are increasingly being used as the basis for the design of large-scale assessments. Of critical importance to these designs is the validity of the map structure used to build the assessments. Most commonly, evidence for the validity of a map structure comes from procedural evidence gathered during the learning map creation process (e.g., research literature, external reviews). However, it is also important to provide support for the validity of the map structure with empirical evidence by using data gathered from the assessment. In this paper, we propose a framework for the empirical validation of learning maps and progressions using diagnostic classification models. Three methods are proposed within this framework that provide different levels of model assumptions and types of inferences. The framework is then applied to the Dynamic Learning Maps® alternate assessment system to illustrate the utility and limitations of each method. Results show that each of the proposed methods have some limitations, but they are able to provide complementary information for the evaluation of the proposed structure of content standards (Essential Elements) in the Dynamic Learning Maps assessment.

Download Full-text

Effects of Trust and Threat Messaging on Academic Cheating: A Field Study

Psychological Science ◽

10.1177/0956797620977513 ◽

2021 ◽

pp. 095679762097751

Author(s):

Li Zhao ◽

Jiaxin Zheng ◽

Haiying Mao ◽

Xinyi Yu ◽

Jiacheng Ye ◽

...

Keyword(s):

Field Study ◽

Academic Integrity ◽

Large Scale ◽

Theoretical Foundation ◽

Empirical Evaluation ◽

Psychological Theory ◽

Field Studies ◽

Educational Institutions ◽

Academic Cheating ◽

College Classrooms

Morality-based interventions designed to promote academic integrity are being used by educational institutions around the world. Although many such approaches have a strong theoretical foundation and are supported by laboratory-based evidence, they often have not been subjected to rigorous empirical evaluation in real-world contexts. In a naturalistic field study ( N = 296), we evaluated a recent research-inspired classroom innovation in which students are told, just prior to taking an unproctored exam, that they are trusted to act with integrity. Four university classes were assigned to a proctored exam or one of three types of unproctored exam. Students who took unproctored exams cheated significantly more, which suggests that it may be premature to implement this approach in college classrooms. These findings point to the importance of conducting ecologically valid and well-controlled field studies that translate psychological theory into practice when introducing large-scale educational reforms.

Download Full-text

Dynamic Loads and Response of a Spar Buoy Wind Turbine with Pitch-Controlled Rotating Blades: An Experimental Study

Energies ◽

10.3390/en14123598 ◽

2021 ◽

Vol 14 (12) ◽

pp. 3598

Author(s):

Sara Russo ◽

Pasquale Contestabile ◽

Andrea Bardazzi ◽

Elisa Leone ◽

Gregorio Iglesias ◽

...

Keyword(s):

Wind Speed ◽

Wind Turbine ◽

Degrees Of Freedom ◽

Large Scale ◽

Laboratory Data ◽

Nonlinear Behavior ◽

Offshore Wind ◽

Small Scale ◽

Wave Steepness ◽

Design Factor

New large-scale laboratory data are presented on a physical model of a spar buoy wind turbine with angular motion of control surfaces implemented (pitch control). The peculiarity of this type of rotating blade represents an essential aspect when studying floating offshore wind structures. Experiments were designed specifically to compare different operational environmental conditions in terms of wave steepness and wind speed. Results discussed here were derived from an analysis of only a part of the whole dataset. Consistent with recent small-scale experiments, data clearly show that the waves contributed to most of the model motions and mooring loads. A significant nonlinear behavior for sway, roll and yaw has been detected, whereas an increase in the wave period makes the wind speed less influential for surge, heave and pitch. In general, as the steepness increases, the oscillations decrease. However, higher wind speed does not mean greater platform motions. Data also indicate a significant role of the blade rotation in the turbine thrust, nacelle dynamic forces and power in six degrees of freedom. Certain pairs of wind speed-wave steepness are particularly unfavorable, since the first harmonic of the rotor (coupled to the first wave harmonic) causes the thrust force to be larger than that in more energetic sea states. The experiments suggest that the inclusion of pitch-controlled, variable-speed blades in physical (and numerical) tests on such types of structures is crucial, highlighting the importance of pitch motion as an important design factor.

Download Full-text

Development and Validation of a Post-Earthquake Safety Assessment System for High-Rise Buildings Using Acceleration Measurements

Mathematics ◽

10.3390/math9151758 ◽

2021 ◽

Vol 9 (15) ◽

pp. 1758

Author(s):

Koji Tsuchimoto ◽

Yasutaka Narazaki ◽

Billie F. Spencer

Keyword(s):

Safety Assessment ◽

Large Scale ◽

Model Updating ◽

Safety Evaluation ◽

Assessment System ◽

Shaking Table ◽

Structural Safety ◽

High Rise ◽

Steel Building ◽

Earthquake Safety

After a major seismic event, structural safety inspections by qualified experts are required prior to reoccupying a building and resuming operation. Such manual inspections are generally performed by teams of two or more experts and are time consuming, labor intensive, subjective in nature, and potentially put the lives of the inspectors in danger. The authors reported previously on the system for a rapid post-earthquake safety assessment of buildings using sparse acceleration data. The proposed framework was demonstrated using simulation of a five-story steel building modeled with three-dimensional nonlinear analysis subjected to historical earthquakes. The results confirmed the potential of the proposed approach for rapid safety evaluation of buildings after seismic events. However, experimental validation on large-scale structures is required prior to field implementation. Moreover, an extension to the assessment of high-rise buildings, such as those commonly used for residences and offices in modern cities, is needed. To this end, a 1/3-scale 18-story experimental steel building tested on the shaking table at E-Defense in Japan is considered. The importance of online model updating of the linear building model used to calculate the Damage Sensitive Features (DSFs) during the operation is also discussed. Experimental results confirm the efficacy of the proposed approach for rapid post-earthquake safety evaluation for high-rise buildings. Finally, a cost-benefit analysis with respect to the number of sensors used is presented.

Download Full-text

Tor Bridge Discovery: Extensive Analysis and Large-scale Empirical Evaluation

IEEE Transactions on Parallel and Distributed Systems ◽

10.1109/tpds.2013.249 ◽

2015 ◽

Vol 26 (7) ◽

pp. 1887-1899 ◽

Cited By ~ 5

Author(s):

Zhen Ling ◽

Junzhou Luo ◽

Wei Yu ◽

Ming Yang ◽

Xinwen Fu

Keyword(s):

Large Scale ◽

Empirical Evaluation ◽

Extensive Analysis

Download Full-text

EVALUATION AND IMPLEMENTATION OF REMOTE BUILDING DAMAGE ASSESSMENT SYSTEM DURING LARGE SCALE EARTHQUAKE DISASTER

Journal of Japan Society of Civil Engineers Ser A1 (Structural Engineering & Earthquake Engineering (SE/EE)) ◽

10.2208/jscejseee.70.i_263 ◽

2014 ◽

Vol 70 (4) ◽

pp. I_263-I_272

Author(s):

Makoto FUJIU ◽

Miho OHARA ◽

Kimiro MEGURO

Keyword(s):

Damage Assessment ◽

Large Scale ◽

Building Damage ◽

Assessment System ◽

Earthquake Disaster

Download Full-text

Landscape sketches

Geological Field Sketches and Illustrations ◽

10.1093/oso/9780198835929.003.0007 ◽

2019 ◽

pp. 103-116

Author(s):

Matthew J. Genge

Keyword(s):

Large Scale ◽

Earth Science ◽

Worked Examples ◽

Tectonic Structures ◽

Geological Structures ◽

Topographic Features ◽

Geological Features ◽

Additional Value

Drawings of landscapes are often used to record large-scale geological structures in Earth Science. This chapter describes how to draw geological features exposed within landscapes with a focus on the methods used to produce accurate and detailed field sketches. The tactics in drawing large fields of view, landscapes with significant topography, and vegetated landscapes with sparse outcrops are described. Three worked examples of landscape field sketches are provided to illustrate how best to record tectonic structures and volcanic features exposed over topography. Topographic features such as ravines and ridges can be added as lines to add additional value. Common mistakes considered include insufficient detail and schematic vegetation.

Download Full-text

Target Evaluation and Correlation Method (TECM) as an Assessment Approach to Global Earth Observation System of Systems (GEOSS)

International Journal of Applied Geospatial Research ◽

10.4018/jagr.2011010103 ◽

2011 ◽

Vol 2 (1) ◽

pp. 36-62 ◽

Cited By ~ 2

Author(s):

Samuel Epelbaum ◽

Mo Mansouri ◽

Alex Gorod ◽

Brian Sauser ◽

Alexander Fridman

Keyword(s):

Large Scale ◽

Correlation Method ◽

System Of Systems ◽

Earth Observation ◽

Observation System ◽

Societal Benefit ◽

Large Scale Network ◽

Assessment Approach ◽

Earth Observation System ◽

Scale Network

The Global Earth Observation System of Systems (GEOSS) is a ten-year-long Implementation Plan, which commenced in 2005 as a group effort by numerous participating countries and organizations to build a large-scale network to effectively monitor and respond to the increasingly complex web of global environmental and socio-economic issues. This paper proposes the Target Evaluation and Correlation Method (TECM) as an assessment approach to GEOSS with its 241 Targets across the nine “Societal Benefit Areas,” along with a method to identify Target Correlation Levels (TCL). Applying TECM allows concluding whether the chosen targets within the GEOSS fall into the domain of System of Systems (SoS), while using TCLs delineates the extent of inclusion for these targets in the form of a system. Furthermore, this research investigates the possible ways of raising the correlation levels of the targets for the cases in which TCLs are low.

Download Full-text

Exploring Biogas and Biofertilizer Production from Abattoir Wastes in Nigeria Using a Multi-Criteria Assessment Approach

Recycling ◽

10.3390/recycling5030018 ◽

2020 ◽

Vol 5 (3) ◽

pp. 18 ◽

Cited By ~ 1

Author(s):

Idi Guga Audu ◽

Abraham Barde ◽

Othniel Mintang Yila ◽

Peter Azikiwe Onwualu ◽

Buga Mohammed Lawal

Keyword(s):

Large Scale ◽

Fossil Fuels ◽

Swot Analysis ◽

National Policy ◽

Environmental Benefits ◽

Limiting Factors ◽

Carbon Markets ◽

Policy Reforms ◽

The North ◽

Assessment Approach

Management of waste streams from abattoirs is a major challenge in developing countries. Harnessing these wastes as resources for the production of biogas and biofertilizer could contribute to curbing the environmental menace and to addressing the problems of energy and food deficits in Nigeria. However, large scale uptake of the technology is faced with techno-socio-economic and the lack of data required for effective investment decisions. In this study, the potential use of waste generated in the north central region of Nigerian abattoirs, representing approximately 12% of the land and 6% of the population, were evaluated for suitability for biogas and biofertilizer production. Data acquired from the study sites were used for computational estimation and integrated into strengths, weaknesses, opportunities, and threats (SWOT) analysis to give a detailed overview of the prospects and the limiting factors. The study revealed that high investment costs and public subsidies for fossil fuels are the key limiting factors while the prospects of tapping into the unexploited carbon markets and multiple socio-economic and environmental benefits favors investment. Public supports in the form of national policy reforms leading to intervention programs are required for progress.

Download Full-text