One of the most consistent predictors of fear of crime is gender, whereby women are more fearful of crime than men, despite their lower rates of victimization. An often-cited explanation for this apparent paradox is the “shadow” of sexual assault, which has received overwhelming support in studies among college students. However, this work has been largely quantitative and rarely comparative in nature. Using in-depth qualitative interviews with 70 undergraduates attending a suburban and an urban university, the focus of the current study was to understand if and how gender shapes students’ fear of crime and perceptions of risk on campus, as well as whether the influence of gender varies across contexts. Consistent with the shadow hypothesis, this study found that students attending university in the relatively affluent suburban setting were most concerned about the sexual assault and rape of their female peers, especially by non-student outsiders. Yet in the less advantaged urban context, the shadow was missing from students’ remarks. Instead, participants believed they were most likely to be robbed and generally knew someone who had been a victim of the crime, had they not been robbed themselves. The findings from this study have theoretical implications for the role of context in the shadow of sexual assault hypothesis, as well as methodological implications for how scholars examine gender and students’ fear of crime.