Think about a population that is on its way to becoming specialized to fill a particular ecological niche. What is the effect of cross breeding with a population that is, perhaps, on an adaptive path to another ecological niche? Most likely the offspring will be poorly adapted for either niche. This leads to a new selective pressure, selection against cross-breeding between individuals of the two populations. Even small changes in a particular trait or behavior can lead to significant changes in mating preferences and outcomes. Consider Darwin’s finches or the Hawaiian honeycreepers mentioned previously. A major feature that distinguishes these various types of birds is the size and shapes of their beaks. These adaptations represent both the development of a behavior – that is the preference of birds to seek food from particular sources, for example, particular types of flowers or particular size seeds – and the traits needed to successfully harvest that food source, such as bill shape and size. Clearly the organism has to display the behavior, even if it is in a primitive form, that makes selection of the physical trait beneficial. This is a type of loop, where behavioral and physical traits are closely linked. You can ask yourself, could a long neck have evolved in a species that did not eat the leaves of trees?
Back to finches and honeycreepers. Mate selection in birds is often mediated by song, generally males sing and females respond (or not). As beak size and shape change, the song produced also changes102. This change is, at least originally, an unselected trait that accompanies the change in beak shape, but it can become useful if females recognize and respond to songs more like their own. This would lead to preferential mating between organisms with the same trait (beak shape). Over time, this preference could evolve into a stronger and stronger preference, until it becomes a reproductive barrier between organisms adapted to different ecological niches. Similarly, imagine that the flowers a particular subpopulation feeds on open and close at different times of the day. This could influence when an organism that feeds on a particular type of flower is sexually receptive. You can probably generate your own scenarios in which one behavioral trait has an influence on reproductive preferences. If a population is isolated from others, such effects may develop but are relatively irrelevant. They become important when two closely related but phenotypically distinct populations come back into contact. Now matings between individuals in two different populations, sometimes termed hybridization, can lead to offspring poorly adapted to either niche. This creates a selective pressure to minimize hybridization. Again, this can arise spontaneously, such as the two populations mate at different times of the day or year or respond to different behavioral queues, such as mating songs. Traits that enhance reproductive success by reducing the chance of detrimental hybridization will be preferentially chosen. The end result is what is known as reproductive isolation103. Once reproductive isolation occurs, what was one species has become two. A number of different mechanisms ranging from the behavioral to the structural and the molecular are involved in generating reproductive isolation. Behaviors may not be “attractive,” genitalia may not fit together, gametes might not fuse with one another, or embryos might not be viable - there are many possibilities.
102 Beaks, Adaptation, and Vocal Evolution in Darwin's Finches: http://bioscience.oxfordjournals.org...54/6/501.short and
Vocal mechanics in Darwin's finches: correlation of beak gape and song frequency:http://jeb.biologists.org/content/207/4/607.short
103 Beak size matters for finches' song: news.nationalgeographic.com/n...ins_finch.html