When seeing a face, people form judgments of perceptually ambiguous social categories (PASCs), for example, gun-owners, gay people, or alcoholics. Previous research has assumed that PASC judgments arise from the statistical learning of facial features in social encounters. We propose, instead, that perceivers associate facial features with traits (e.g., extroverted) and then infer PASC membership via learned stereotype associations with those traits. Across three studies, we show that when any PASC is more stereotypically associated with a trait (e.g., alcoholics = extroverted), perceivers are more likely to infer PASC membership from faces conveying that trait (Study 1). Furthermore, we demonstrate that individual differences in trait–PASC stereotypes predict face-based judgments of PASC membership (Study 2) and have a causal role in these judgments (Study 3). Together, our findings imply that people can form any number of PASC judgments from facial appearance alone by drawing on their learned social–conceptual associations.