Investigating the Adequacy of Response Time Outlier Definitions in Computer-Based Web Surveys Using Paradata SurveyFocus

2017 ◽  
Vol 36 (3) ◽  
pp. 369-378 ◽  
Author(s):  
Jan Karem Höhne ◽  
Stephan Schlosser

Web surveys are commonly used in social research because they are usually cheaper, faster, and simpler to conduct than other modes. They also enable researchers to capture paradata such as response times. Particularly, the determination of proper values to define outliers in response time analyses has proven to be an intricate challenge. In fact, to a certain degree, researchers determine them arbitrarily. In this study, we use “SurveyFocus (SF)”—a paradata tool that records the activity of the web-survey pages—to assess outlier definitions based on response time distributions. Our analyses reveal that these common procedures provide relatively sufficient results. However, they are unable to detect all respondents who temporarily leave the survey, causing bias in the response times. Therefore, we recommend a two-step procedure consisting of the utilization of SF and a common outlier definition to attain a more appropriate analysis and interpretation of response times.

2021 ◽  
Author(s):  
Jian-Qiao Zhu ◽  
Pablo Leon-Villagra ◽  
Nick Chater ◽  
Adam N Sanborn

Human cognition is fundamentally noisy. While routinely regarded as a nuisance in experimental investigation, the few studies investigating properties of cognitive noise have found surprising structure. A first line of research has shown that inter-response-time distributions are heavy-tailed. That is, response times between subsequent trials usually change only a small amount, but with occasional large changes. A second, separate, line of research has found that participants’ estimates and response times both exhibit long-range autocorrelations (i.e., 1/f noise). Thus, each judgment and response time not only depends on its immediate predecessor but also on many previous responses. These two lines of research use different tasks and have distinct theoretical explanations: models that account for heavy-tailed response times do not predict 1/f autocorrelations and vice versa. Here, we find that 1/f noise and heavy-tailed response distributions co-occur in both types of tasks. We also show that a statistical sampling algorithm, developed to deal with patchy environments, generates both heavy-tailed distributions and 1/f noise, suggesting that cognitive noise may be a functional adaptation to dealing with a complex world.


Field Methods ◽  
2017 ◽  
Vol 29 (4) ◽  
pp. 365-382 ◽  
Author(s):  
Jan Karem Höhne ◽  
Stephan Schlosser ◽  
Dagmar Krebs

Measuring attitudes and opinions employing agree/disagree (A/D) questions is a common method in social research because it appears to be possible to measure different constructs with identical response scales. However, theoretical considerations suggest that A/D questions require a considerable cognitive processing. Item-specific (IS) questions, in contrast, offer content-related response categories, implying less cognitive processing. To investigate the respective cognitive effort and response quality associated with A/D and IS questions, we conducted a web-based experiment with 1,005 students. Cognitive effort was assessed by response times and answer changes. Response quality, in contrast, was assessed by different indicators such as dropouts. According to our results, single IS questions require higher cognitive effort than single A/D questions in terms of response times. Moreover, our findings show substantial differences in processing single and grid questions.


2021 ◽  
Author(s):  
Roksana Markiewicz ◽  
Ali Mazaheri ◽  
Andrea Krott

Performance differences between bilingual and monolingual participants on conflict tasks can be affected by the balance of various sub-processes such as monitoring and stimulus categorisation. Here we investigated the effect of bilingualism on these sub-processes during a conflict task with medium monitoring demand. We examined the behavioural and evoked potentials from a group of bilingual and monolingual speakers during a flanker task with 25% incongruent trials. We analysed behavioural differences by means of averaged response times and ex-Gaussian analyses of response time distributions. For the evoked potentials we focused on the N2 (implicated to be involved in monitoring) and P300 (implicated to be involved in categorisation) responses. We found that bilinguals had significantly longer response distribution tails compared to monolinguals. Additionally, bilinguals exhibited a more pronounced N2 and smaller P3 components compared to their monolingual counterparts, independent of experimental condition, suggesting enhanced monitoring processes and reduced categorisation effort. Importantly, N2 amplitudes were positively and P3 amplitudes were negatively related to the length of response distribution tails. We postulate that these results reflect an overactive monitoring system in bilinguals in a task of medium monitoring demand. This enhanced monitoring leads to less effortful categorisation, but also occasionally to slow responses. These results suggest that changes of the cognitive control system due to bilingual experience changes the balance of processes during conflict tasks, potentially leading to a small behavioural disadvantage.


1983 ◽  
Vol 27 (2) ◽  
pp. 165-169
Author(s):  
Donald L. Fisher ◽  
Bruce Papazian

In many situations a rapid, accurate response to a traffic sign is required. The design of traffic signs which keeps response time at a minimum is an important human factors problem. An understanding of the network of cognitive processes that govern response times can be very useful to the human factors engineer, especially when it is possible to determine which processes, if shortened in duration, would lead to an overall decrease in response time. The design activity can then be focused on keeping at a minimum the duration of these critical processes. Two recently developed methods of response time analysis which make possible the determination of the critical processes are discussed: the critical path method (Schweickert, 1978) and the OP method (Fisher and Goldstein, in press).


2021 ◽  
Author(s):  
Morgan L Rosendahl ◽  
Jonathan Cohen

Tools from quantum theory have been effectively leveraged in modeling otherwise poorly understood effects in decision-making such as apparent fallacies in probability judgments and context effects. This approach has described the dynamics of two alternative forced choice (2AFC) decisions in terms of the path of a single quantum particle evolving in a single potential well. Here, we present a variant on that approach, which we name the Multi-Particle and Multi-Well (MPMW) quantum cognitive framework, in which decisions among N alternatives are treated by the sum of positional measurements of many independent quantum particles representing stimulus information, acted on by an N-well landscape that defines the decision alternatives. In this article, we apply the MPMW model to the simplest and most common case of N-alternative decision making, 2AFC dynamics. This application calls for a multi-particle double-well implementation, which allows us to construct a simple, analytically tractable discrete drift diffusion model (DDM), in the form of a Markov chain, wherein the parameters of the attractor wells reflect bottom-up (automatic) and top-down (control-dependent) influences on the integration of external information. We first analyze this Markov chain in its simplest form, as a single integrator with a generative process arising from a static quantum landscape and fixed thresholds, and then consider the case of multi-integrator processing under the same conditions. Within this system, stochasticity arises directly from the double-well quantum attractor landscape as a function of the dimensions of its wells, rather than as an external parameter requiring independent fitting. The simplicity of the Markov chain component of this model allows for easy analytical computation of closed forms for response time distributions and response probabilities that match qualitative properties of the accuracies and reaction times of humans performing 2AFC tasks. The MPMW framework produces response time distributions following inverse gaussian curves familiar from previous DDM models and empirical data, including the common observation that mean response times are faster for incorrect than for correct responses. The work presented in this paper serves as a proof of concept, based on which the MPMW framework can be extended to address more complex decision-making processes, (e.g., N-alternative forced choice, dynamic control allocation, and nesting quantum landscapes to allow for modeling at both the task and stimulus levels of processing) that we discuss as future directions.


2021 ◽  
Vol 37 (4) ◽  
pp. 907-930
Author(s):  
Georg-Christoph Haas ◽  
Stephanie Eckman ◽  
Ruben Bach

Abstract Previous research is inconclusive regarding the effects of paper and web surveys on response burdens. We conducted an establishment survey with random assignment to paper and web modes to examine this issue. We compare how the actual and perceived response burdens differ when respondents complete a survey in the paper mode, in the web mode and when they are allowed to choose between the two modes. Our results show that in the web mode, respondents have a lower estimated time to complete the questionnaire, while we do not find differences between paper and the web on the perceived response time and perceived burden. Even though the response burden in the web mode is lower, our study finds no evidence of an increased response burden when moving an establishment survey from paper to the web.


2021 ◽  
Author(s):  
Julia L.A. Knapp ◽  
Wouter R. Berghuijs ◽  
Jana von Freyberg ◽  
James W. Kirchner

<p>The time a molecule of rain takes to reach the stream is normally substantially longer than the time for discharge to respond to rainfall. This difference arises because hydraulic potentials propagate through landscapes much faster than water itself does; in other words, the celerity of wave propagation is faster than the velocity of water flow. Although these concepts are well established, most catchment studies are restricted to the calculation of the celerity or response time from hydrometric information. However, to understand the storage, release, and transport of water, as well as identify flow paths through the catchment, one needs to estimate both response and travel times, requiring both hydrometric and tracer data.</p><p>We analyzed hydrometric and tracer data from two contrasting sites, the pre-Alpine Erlenbach catchment in Switzerland and the Upper Hafren catchment at Plynlimon in Wales. For both sites, hydrometric data and sub-daily isotopic tracer time series are available, enabling the calculation of response times as well as travel time distributions and new water fractions. To gain a deeper understanding of the functioning of the two catchments, we quantified these metrics and distributions for different ranges of antecedent wetness and precipitation intensity. Generally, wetter catchment conditions and higher precipitation intensities yielded faster runoff responses and shorter travel times.  Contrasts between travel and response time distributions under varying catchment conditions also facilitated more nuanced insights into catchment functioning and the effects of catchment wetness and precipitation intensity on water storage and release.</p>


2018 ◽  
Vol 15 (1) ◽  
Author(s):  
Miha Matjašič ◽  
Vasja Vehovar ◽  
Katja Lozar Manfreda

In the last two decades, survey researchers have intensively used computerised methods for the collection of different types of paradata, such as keystrokes, mouse clicks and response times, to evaluate and improve survey instruments as well as to understand the survey response process. With the growing popularity of web surveys, the importance of paradata has further increased. Within this context, response time measurement is the prevailing paradata approach. Papers typically analyse the time (measured in milliseconds or seconds) a respondent needs to answer a certain item, question, page or questionnaire. One of the key challenges when analysing the response time is to identify and separate units that are answering too quickly or too slowly. These units can have a poor response quality and are typically labelled as response time outliers. This paper focuses on approaches for identifying and processing response time outliers. It presents a systematic overview of scientific papers on response time outliers in web surveys. The key observed characteristics of the papers are the approaches used, the level of time measurement, the processing of response time outliers and the relationship between response time and response quality. The results show that knowledge on response time outliers is scattered, inconsistent and lacking systematic comparisons of approaches. Consequently, there is a need to improve and upgrade the knowledge on this issue and to develop new approaches that will overcome existing deficiencies and inconsistencies in identifying and dealing with response time outliers.


2013 ◽  
Vol 26 (1-2) ◽  
pp. 95-122 ◽  
Author(s):  
Matthias Gondan ◽  
Steven P. Blurton

In redundant signals tasks, participants respond in the same way to two different stimuli which are presented either alone or in combination (redundant stimuli). Responses to redundant stimuli are typically faster than responses to single stimuli. Different explanations account for such redundancy gains, including race models and coactivation models. Race models predict that the cumulative response time distribution for the redundant stimuli never exceeds the summed distributions of the single stimuli (race model inequality, RMI, Miller, 1982). Based on work by Townsend and Nozawa (1995) we demonstrate that the RMI is a special case of a more general interaction contrast of response time distributions for stimuli of different intensity, or stimuli presented with onset asynchrony. The generalization of the RMI is, thus, suited for a much wider class of experiments than the standard setup in which response times for single stimuli are compared to those for double stimuli. Moreover, predictions can be derived not only for the race model, but for serial, parallel, and coactive processing modes with different stopping rules. Compared to the standard RMI, statistical power of these interaction contrasts is satisfactory, even for small onset asynchronies.


Author(s):  
Andreas Voss ◽  
Markus Nagler ◽  
Veronika Lerche

Stochastic diffusion models ( Ratcliff, 1978 ) can be used to analyze response time data from binary decision tasks. They provide detailed information about cognitive processes underlying the performance in such tasks. Most importantly, different parameters are estimated from the response time distributions of correct responses and errors that map (1) the speed of information uptake, (2) the amount of information used to make a decision, (3) possible decision biases, and (4) the duration of nondecisional processes. Although this kind of model can be applied to many experimental paradigms and provides much more insight than the analysis of mean response times can, it is still rarely used in cognitive psychology. In the present paper, we provide comprehensive information on the theory of the diffusion model, as well as on practical issues that have to be considered for implementing the model.


Sign in / Sign up

Export Citation Format

Share Document