Analyzing Process Data from Problem-Solving Items with N-Grams

This chapter draws on process data recorded in a computer-based large-scale program, the Programme for International Assessment of Adult Competencies (PIAAC), to address how sequences of actions recorded in problem-solving tasks are related to task performance. The purpose of this study is twofold: first, to extract and detect robust sequential action patterns that are associated with success or failure on a problem-solving item, and second, to compare the extracted sequence patterns among selected countries. Motivated by the methodologies of natural language processing and text mining, we utilized feature selection models in analyzing the process data at a variety of aggregate levels and evaluated the different methodologies in terms of predictive power of the evidence extracted from process data. It was found that action sequence patterns significantly differed by performance groups and were consistent across countries. This study also demonstrated that the process data were useful in detecting missing data and potential mistakes in item development.

Download Full-text

Using process data to understand problem-solving strategies and processes for drag-and-drop items in a large-scale mathematics assessment

Large-scale Assessments in Education ◽

10.1186/s40536-021-00095-4 ◽

2021 ◽

Vol 9 (1) ◽

Author(s):

Yang Jiang ◽

Tao Gong ◽

Luis E. Saldivia ◽

Gabrielle Cayton-Hodges ◽

Christopher Agard

Keyword(s):

Problem Solving ◽

Time Allocation ◽

Large Scale ◽

Solution Process ◽

Process Data ◽

Mathematics Assessments ◽

Assessment Design ◽

Eighth Grade Students ◽

Problem Solving Strategies ◽

Drag And Drop

AbstractIn 2017, the mathematics assessments that are part of the National Assessment of Educational Progress (NAEP) program underwent a transformation shifting the administration from paper-and-pencil formats to digitally-based assessments (DBA). This shift introduced new interactive item types that bring rich process data and tremendous opportunities to study the cognitive and behavioral processes that underlie test-takers’ performances in ways that are not otherwise possible with the response data alone. In this exploratory study, we investigated the problem-solving processes and strategies applied by the nation’s fourth and eighth graders by analyzing the process data collected during their interactions with two technology-enhanced drag-and-drop items (one item for each grade) included in the first digital operational administration of the NAEP’s mathematics assessments. Results from this research revealed how test-takers who achieved different levels of accuracy on the items engaged in various cognitive and metacognitive processes (e.g., in terms of their time allocation, answer change behaviors, and problem-solving strategies), providing insights into the common mathematical misconceptions that fourth- and eighth-grade students held and the steps where they may have struggled during their solution process. Implications of the findings for educational assessment design and limitations of this research are also discussed.

Download Full-text

The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment.

Journal of Educational Psychology ◽

10.1037/a0034716 ◽

2014 ◽

Vol 106 (3) ◽

pp. 608-626 ◽

Cited By ~ 111

Author(s):

Frank Goldhammer ◽

Johannes Naumann ◽

Annette Stelter ◽

Krisztina Tóth ◽

Heiko Rölke ◽

...

Keyword(s):

Problem Solving ◽

Task Difficulty ◽

Large Scale ◽

Time On Task ◽

Large Scale Assessment ◽

Scale Assessment ◽

Task Effect ◽

Computer Based

Download Full-text

TRENDS IN COMPUTER-BASED SECOND LANGUAGE ASSESSMENT

Annual Review of Applied Linguistics ◽

10.1017/s0267190505000127 ◽

2005 ◽

Vol 25 ◽

pp. 228-242 ◽

Cited By ~ 21

Author(s):

Joan Jamieson

Keyword(s):

Second Language ◽

Language Processing ◽

Corpus Linguistics ◽

Large Scale ◽

Language Testing ◽

Language Assessment ◽

Adaptive Tests ◽

Technological Advances ◽

Computer Based ◽

Computer Adaptive Tests

In the last 20 years, several authors have described the possible changes that computers may effect in language testing. Since ARAL's last review of general language testing trends (Clapham, 2000), authors in the Cambridge Language Assessment Series have offered various visions of how computer technology could alter the testing of second language skills. This chapter reflects these perspectives as it charts the paths recently taken in the field. Initial steps were made in the conversion of existing item types and constructs already known from paper-and- pencil testing into formats suitable for computer delivery. This conversion was closely followed by the introduction of computer-adaptive tests, which aim to make more, and perhaps, better, use of computer capabilities to tailor tests more closely to individual abilities and interests. Movement toward greater use of computers in assessment has been coupled with an assumption that computer-based tests should be better than their traditional predecessors, and some related steps have been taken. Corpus linguistics has provided tools to create more authentic assessments; the quest for authenticity has also motivated inclusion of more complex tasks and constructs. Both these innovations have begun to be incorporated into computer-based language tests. Natural language processing has also provided some tools for computerized scoring of essays, particularly relevant in large-scale language testing programs. Although computer use has not revolutionized all aspects of language testing, recent efforts have produced some of the research, technological advances, and improved pedagogical understanding needed to support progress.

Download Full-text

A Diagnostic Classification Analysis of Problem-Solving Competence using Process Data

10.31234/osf.io/wtyae ◽

2020 ◽

Author(s):

Peida Zhan ◽

Xin Qiao

Keyword(s):

Problem Solving ◽

Student Assessment ◽

Outcome Data ◽

Diagnostic Classification ◽

Process Data ◽

Action Sequence ◽

Problem Solving Skills ◽

International Student Assessment ◽

Latent Ability ◽

Problem Solving Strategies

Process data refers to data recorded by computer-based assessments (CBA) that reflect respondents’ problem-solving processes and provide greater insight into how students solve problems, instead of merely how well they solve them. Using the rich information contained in process data, this study proposed an item-specific psychometric method for analyzing process data in order to comprehensively understand respondents’ problem-solving competence. By incorporating diagnostic classification into process data analysis, the proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but can also classify respondents according to their problem-solving strategies. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving task was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving strategies than the observed score classes; (b) although only one item was used, estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the estimated unidimensional latent ability taken from the outcome data; and (c) the interactions between problem-solving skills may follow the conjunctive condensation rule, which assumes that only when a respondent has mastered all the required problem-solving skills can the specific action sequence appear. Overall, the main conclusion drawn from this study was that using diagnostic classification is a feasible and promising method for analyzing process data.

Download Full-text

Interactive computer based assessment tasks: How problem-solving process data can inform instruction

Australasian Journal of Educational Technology ◽

10.14742/ajet.1053 ◽

2010 ◽

Vol 26 (5) ◽

Cited By ~ 24

Author(s):

Nathan Zoanetti

Keyword(s):

Problem Solving ◽

A Priori ◽

Individual Student ◽

Process Data ◽

Evidence Map ◽

Traditional Assessment ◽

Targeted Instruction ◽

Computer Based ◽

Problem Solving Process ◽

Key Steps

This article presents key steps in the design and analysis of a computer based problem-solving assessment featuring interactive tasks. The purpose of the assessment is to support targeted instruction for students by diagnosing strengths and weaknesses at different stages of problem-solving. The first focus of this article is the task piloting methodology, which demonstrates the relationship between process data and a priori documented problem-solving behaviours. This work culminated in the design of a Microsoft Excel template for data transcription named a Temporal Evidence Map. The second focus of this article is to illustrate how evidence from process data can be accumulated to produce and report instructionally useful information not available through traditional assessment approaches. This is demonstrated through the production of reports profiling individual student outcomes against important aspects of problem solving.

Download Full-text

Analysis of Process Data of PISA 2012 Computer-Based Problem Solving: Application of the Modified Multilevel Mixture IRT Model

Frontiers in Psychology ◽

10.3389/fpsyg.2018.01372 ◽

2018 ◽

Vol 9 ◽

Cited By ~ 3

Author(s):

Hongyun Liu ◽

Yue Liu ◽

Meijuan Li

Keyword(s):

Problem Solving ◽

Process Data ◽

Irt Model ◽

Pisa 2012 ◽

Mixture Irt ◽

Computer Based

Download Full-text

Scientific Problem Solving in a Virtual Laboratory: A Comparison Between Individuals and Pairs

Swiss Journal of Psychology ◽

10.1024/1421-0185.67.2.71 ◽

2008 ◽

Vol 67 (2) ◽

pp. 71-83 ◽

Cited By ~ 5

Author(s):

Yolanda A. Métrailler ◽

Ester Reijnen ◽

Cornelia Kneser ◽

Klaus Opwis

Keyword(s):

Visual Search ◽

Problem Solving ◽

Process Data ◽

Scientific Problem ◽

Domain Specific ◽

Scientific Problem Solving ◽

Before And After ◽

Verbal Data ◽

Psychological Knowledge ◽

Problem Solving Task

This study compared individuals with pairs in a scientific problem-solving task. Participants interacted with a virtual psychological laboratory called Virtue to reason about a visual search theory. To this end, they created hypotheses, designed experiments, and analyzed and interpreted the results of their experiments in order to discover which of five possible factors affected the visual search process. Before and after their interaction with Virtue, participants took a test measuring theoretical and methodological knowledge. In addition, process data reflecting participants’ experimental activities and verbal data were collected. The results showed a significant but equal increase in knowledge for both groups. We found differences between individuals and pairs in the evaluation of hypotheses in the process data, and in descriptive and explanatory statements in the verbal data. Interacting with Virtue helped all students improve their domain-specific and domain-general psychological knowledge.

Download Full-text