Public housing has been an important site for empirical research on concentrated poverty, social isolation, and social organization. Scholars have demonstrated that public housing was disproportionately built in high poverty neighborhoods, thereby exacerbating the physical and social isolation of residents. They have also hypothesized that physical features of public housing may contribute to a breakdown of social organization. These hypotheses motivated the demolition of large and physically deteriorated public housing structures throughout the United States. I use the New York City Housing and Vacancy Survey to test the hypotheses that large building size and visible building disorder are associated with mistrust among neighbors, as would be expected by theories linking the built environment to social organization. Although I find some evidence that trust is less common in large buildings with higher levels of disorder, I argue that critics of public housing overstate the social effects of the built environment.