Effects of Relaxed Natural Selection on the Evolution of Behavior

Theoretical discussion of the role of natural selection in shaping behavioral variation in different habitats has been an integral part of the study of animal behavior since the late 19th century. Herbert Spencer (1888) was among the first to argue that migrating populations that fail to adjust to environmental circumstances “are the first to disappear.” A common rationale for comparing populations or related species is the desire to identify behavioral differences that correspond with habitat properties providing different patterns of selection (Tuomi 1981, Riechert 1993, this volume). Behavioral similarities are often ignored or are treated as less interesting because the thrust of the research program emphasizes behavioral differences as an empirical test of the theory of natural selection. Nevertheless, these similarities can be as revealing of evolutionary process as are differences when they reflect behavioral convergence or slow disintegration of behavior under relaxed selection (Coss and Goldthwaite 1995). When populations invade novel habitats, they not only experience new selective regimes; they can also experience relaxed selection on specific behavioral phenotypes. This is particularly common when the new habitat is missing a class of predators that was abundant in the ancestral habitat (e.g., Curio 1975, Pressley 1981). Under relaxed selection, characters may disintegrate, presumably because mutations that result in loss of the phenotype are not at a selective disadvantage. Disintegration is not always observed, however. Instead, behavioral characters are sometimes retained for long periods of time after selection has been relaxed (Coss 1991b, Kaneshiro 1989). Inferring relaxed selection requires that the history of the contrasted populations be relatively well known. Both ancestral selective regimes and behavioral characters must be known if character polarity is to be established. Character polarity must be established to distinguish disintegration from parallel evolution of novel behavior patterns. This often is a problem in population contrasts because differentiation is usually too recent to have resulted in the evolution of enough derived characters for the use of standard cladistic methods of phylogenetic reconstruction, although recent advances in statistical and molecular techniques are promising (Foster 1994, Foster and Cameron 1996). Instead, inference of character polarity has typically relied on geological evidence and comparison with closely related species.