Humans have the unique ability to represent and manipulate symbols. It is widely believed that this ability is rooted in an evolutionarily ancient system used to process nonsymbolic quantities in the human brain. In the current study, we used an fMRI adaptation paradigm to isolate the representations of symbols, quantities, and physical size in forty-five human adults. Results indicate that the neural correlates supporting symbolic number processing are entirely distinct from those supporting nonsymbolic magnitude processing. At the univariate level, symbolic number processing is associated with activation in the left inferior parietal lobule, whereas the processing of nonsymbolic magnitudes (both quantity and physical size), relates to activation in the right intraparietal sulcus. At the multivariate level, normalized patterns of activation for symbolic number processing exhibit a dissimilar pattern of activation compared to nonsymbolic magnitude processing in both the left and right parietal lobes. Additionally, the patterns of activation that associate with quantity and physical size are practically indistinguishable from one another. These findings challenge the longstanding belief that the culturally acquired ability to conceptualize symbolic numbers is rooted in an evolutionarily ancient system for nonsymbolic magnitude processing. Moreover, these data reveal that the system used to process nonsymbolic numbers may actually be a general magnitude processing system used to process numerical and non-numerical magnitudes. These findings highlight the need for the field to shift away from exploring how symbols are grounded in analog nonsymbolic representations, and toward more complex questions related to the neural consequences of learning symbolic numbers.