Time to Shine: Reliable Response-Timing Using R-Shiny for Online Experiments
Rising interest in online experiments for cognitive science research lies in the ability to reacha large number of participants in a short time at a relatively low cost. However, compared tocontrolled laboratory studies, online data is far more noisy. This is especially relevant whenreliable response-timing at a millisecond-level is paramount, as it is the case for manydecision-making tasks. In this paper we sought to replicate a well-validated cognitive effect-the distance effect in number comparisons- using an online mobile-friendly app developedwith open-source tools in R-Shiny. In this task, adapted from (Dehaene et al., 1990), participantshave to decide whether a number on the screen is larger or smaller than a standard (65 in ourstudy). The distance effect stands for the fact that response time (RT) is significantly larger asthe presented number is closer to the standard. A total of N=170 participants (110 with amobile device, 60 on a desktop computer) completed 116 trials over a ~7-minute session.Using generalized linear mixed models estimated with Bayesian inference methods, we founda numerical distance effect strikingly consistent with the original study. Furthermore, wereport systematic offsets in RTs that different OS, browsers and devices introduced. Ourresults demonstrate the reliability of R-Shiny for RT data collection. By doing so, our workpaves the ground for a seamless and robust implementation of simple cognitive tasks inonline studies over desktop and mobile devices using only R, a widely popular programmingframework among cognitive scientists.