Conducting behavioral research online allows researchers to gather more data in less time than conducting studies in person. But this efficiency may sometimes have a cost. Specifically, when researchers gather data within just a few hours, their study may be subject to a time of day bias. Because participants in online platforms are generally free to complete studies whenever they want, people who take studies in the morning may be different in important ways than those who take studies at night. We explored this possibility in two studies conducted on Amazon’s Mechanical Turk. In both studies, we sampled participants at different times of the day and examined whether morning and evening active people differed on a variety of psychological and behavioral characteristics known to correlate with a preference for either morningness or eveningness. We found that participants active in the morning and the evening reported different circadian typologies. Additionally, we found that participants active in the morning reported more conscientiousness and less anxiety, depression, procrastination, internet compulsion, disruptive sleep behaviors, disordered eating, and neuroticism than those sampled in the evening. Study 2 demonstrated that many signs of sub-clinical behavior were uniquely high among evening oriented people and that differences between morning and evening oriented people remained robust after controlling for local time zones and day of the week. Overall, our findings have important implications for online sampling methods and indicate that time of day differences in the composition of online samples represent both an opportunity and a challenge for research.
STULL: Unbiased Online Sampling for Visual Exploration of Large Spatiotemporal Data
