How brains have evolved in response to particular selection pressures is illuminated by ecological correlates of differences in brain structure among contemporary species. The focus of most comparative studies has been on the overall size of brains relative to body size, hence ignoring the ways in which selection operates on specific neural systems. Here we investigate evolutionary radiations in the size of visual and olfactory brain structures within three orders of mammals: primates, bats and insectivores. The comparative relationships within these three orders show both similarities and differences. After removal of the allometric effect of overall brain size, the sizes of different structures within each sensory modality are positively correlated in all three orders. Correlations between visual and olfactory structures, however, are negative in primates, negative but non-significant in insectivores, and positive in bats. In both primates and insectivores, nocturnal lineages tend to have larger olfactory structures than do diurnal or partly diurnal lineages, and among the primates diurnal lineages have larger striate visual cortexes. Hence the apparent trade-off between vision and olfaction in primates seems to be related to the divergence of nocturnal and diurnal forms. However, negative correlations between visual and olfactory structures were also found when nocturnal strepsirhines and diurnal haplorhines were analysed separately, suggesting that ecological variables in addition to activity timing may be significant. Indeed, there were also associations with diet: frugivory was associated with enlargements of the geniculostriate visual system in diurnal primates, enlargements of olfactory structures in nocturnal primates, and possibly enlargements of both in bats. Further ecological associations were found within insectivores: aquatic lineages had smaller olfactory structures than in their non-aquatic counterparts, and fossorial lineages had smaller optic nerves than in non-fossorial forms. We conclude that activity timing, diet and habitat have each played a role in the evolutionary radiation of mammalian sensory systems, but with varying effects in the different taxa. Some of the associations between ecology and sensory systems suggest alternative explanations for correlates of overall brain size, which have in the past commonly been interpreted in terms of selection on intelligence.
