Context guides perception by influencing the saliency of sensory stimuli. Accordingly, in visual cortex, responses to a stimulus are modulated by context, the visual scene surrounding the stimulus. Responses are suppressed when stimulus and surround are similar but not when they differ. The mechanisms that remove suppression when stimulus and surround differ remain unclear. Here we use optical recordings, manipulations, and computational modelling to show that a disinhibitory circuit consisting of vasoactive-intestinal-peptide-expressing (VIP) and somatostatin-expressing (SOM) inhibitory neurons modulates responses in mouse visual cortex depending on the similarity between stimulus and surround. When the stimulus and the surround are similar, VIP neurons are inactive and SOM neurons suppress excitatory neurons. However, when the stimulus and the surround differ, VIP neurons are active, thereby inhibiting SOM neurons and relieving excitatory neurons from suppression. We have identified a canonical cortical disinhibitory circuit which contributes to contextual modulation and may regulate perceptual saliency.