This study investigates how online dating profiles, consisting of both pictures and texts, are visually processed, and how both components affect impression formation. The attractiveness of the profile picture was varied systematically, and texts either included language errors or not. By collecting eye tracking and perception data, we investigated whether picture attractiveness determines attention to the profile text and if the text plays a secondary role. Eye tracking results revealed that pictures are more likely to attract initial attention and that more attractive pictures receive more attention. Texts received attention regardless of the picture’s attractiveness. Moreover, perception data showed that both the pictorial and textual cues affect impression formation, but that they affect different dimensions of perceived attraction differently. Based on our results, a new multimodal information processing model is proposed, which suggests that pictures and texts are processed independently and lead to separate assessments of cue attractiveness before impression formation.
Hierarchical Multimodal Attention for Deep Video Summarization