Intersubjective interactions are highly complex processes that integrate a variety of contextual aspects—physical, pragmatic, social, cultural, normative, institutional—into which embodied individuals, with varying emotions, intents, desires, and motivations, enter. We elucidate the role of context in different varieties of social understanding. We defend a pluralist approach to social cognition and in that framework consider the limited role of mindreading understood as a form of theoretical inference or simulation, as well as the importance of embodied interaction. We argue that all of these practices need to be considered in order to comprehend the rich effects of context on social cognition. We exemplify the bidirectional influence between social understanding and context specifically by focusing on communicative practices and material engagements.