Attention flexibility is a fundamental ability, which has been explored extensively in the past. However, neurocognitive mechanisms underlying switches of attention between working memory (WM) and perceptual stimuli are still poorly understood. Previous research has found that when participants occasionally switch attention either between two perception-based tasks (within-domain switches), or between a WM- and a perception-based task (between-domain switches), a substantial and similar processing cost is elicited in both cases compared to their mere repetition (Verschooren, Schindler, De Raedt, & Pourtois, 2019). These behavioural results, however, did not inform directly about potentially different mechanisms giving rise to the similar cost observed. In this study, we addressed this question by recording 64-channel EEG while participants carried out within- versus between-domain switches of attention. ERP results showed that during early sensory processing a marked P1 attenuation was associated with both switch types, suggesting that switching influenced an early stage of information processing in this situation. Complementing source localization results confirmed that this attention effect had an extrastriate origin. Crucially, this early gating effect associated with task switching was stronger for the between compared to the within-domain switch, despite their similar behavioural cost. These new findings add to the literature by demonstrating that, even though between- and within-domain switches are associated with a similar behavioural cost, different neurocognitive mechanisms give rise to them. As such, they can inform existing cognitive and neuro-anatomical models of selective attention and flexibility, where in the past the focus has often been on within-domain switches.