On the effects of selection and mutation on species tree inference

A common question that arises when inferring species-level phylogenies from genome-scale data is whether selection acting on certain parts of the genome could create a bias in the inferred phylogeny. While most methods for species tree inference currently assume the multispecies coalescent (MSC), all methods that we are aware of utilize only the neutral coalescent process. If selection is in fact present, failure to adequately model it could introduce substantial bias. We work toward rigorously addressing this question using mathematical theory by deriving a version of the coalescent including selection and mutation as a limiting approximation of the Wright-Fisher model with selection and mutation, and showing that it can be used to closely approximate the distribution of coalescent times in the presence of selection and mutation. We confirm the adequacy of the approximation with a simulation study, and discuss its implications for species tree inference. Our results show that in a general class containing many cases of interest, selection has only a small impact on the coalescent process, and ignoring selection when it is present does not have a substantial negative impact on inference of the species tree topology.

Variation in recombination rate affects detection of outliers in genome scans under neutrality

Genome scans can potentially identify genetic loci involved in evolutionary processes such as local adaptation and gene flow. Here, we show that recombination rate variation across a neutrally evolving genome gives rise to mixed sampling distributions of mean FST, a common population genetic summary statistic. In particular, we show that in regions of low recombination the distribution of estimates have more variance and a longer tail than in more highly recombining regions. Determining outliers from the genome-wide distribution without taking local recombination rate into consideration may therefore increase the frequency of false positives in low recombination regions and be overly conservative in more highly recombining ones. We perform genome-scans on simulated and empirical Drosophila melanogaster datasets and, in both cases, find patterns consistent with this neutral model. Similar patterns are observed for other summary statistics used to capture variation in the coalescent process. Linked selection, particularly background selection, is often invoked to explain heterogeneity in across the genome, but here we point out that even under neutrality, statistical artefacts can arise due to variation in recombination rate. Our results highlight a flaw in the design of genome scan studies and suggest that without estimates of local recombination rate, interpreting the genomic landscape of any summary statistic that captures variation in the coalescent process will be very difficult.

On a coalescence process and its branching genealogy

We define and analyze a coalescent process as a recursive box-filling process whose genealogy is given by an ancestral time-reversed, time-inhomogeneous Bienyamé‒Galton‒Watson process. Special interest is on the expected size of a typical box and its probability of being empty. Special cases leading to exact asymptotic computations are investigated when the coalescing mechanisms are either linear fractional or quadratic.

The impact of ancestral population size and incomplete lineage sorting on Bayesian estimation of species divergence times

Although the effects of the coalescent process on sequence divergence and genealogies are well understood, the virtual majority of studies that use molecular sequences to estimate times of divergence among species have failed to account for the coalescent process. Here we study the impact of ancestral population size and incomplete lineage sorting on Bayesian estimates of species divergence times under the molecular clock when the inference model ignores the coalescent process. Using a combination of mathematical analysis, computer simulations and analysis of real data, we find that the errors on estimates of times and the molecular rate can be substantial when ancestral populations are large and when there is substantial incomplete lineage sorting. For example, in a simple three-species case, we find that if the most precise fossil calibration is placed on the root of the phylogeny, the age of the internal node is overestimated, while if the most precise calibration is placed on the internal node, then the age of the root is underestimated. In both cases, the molecular rate is overestimated. Using simulations on a phylogeny of nine species, we show that substantial errors in time and rate estimates can be obtained even when dating ancient divergence events. We analyse the hominoid phylogeny and show that estimates of the neutral mutation rate obtained while ignoring the coalescent are too high. Using a coalescent-based technique to obtain geological times of divergence, we obtain estimates of the mutation rate that are within experimental estimates and we also obtain substantially older divergence times within the phylogeny.

Optimal Point Process Filtering and Estimation of the Coalescent Process

The coalescent process is an important and widely used model for inferring the dynamics of biological populations from samples of genetic diversity. Coalescent analysis typically involves applying statistical methods to either samples of genetic sequences or an estimated genealogy in order to estimate the demographic history of the population from which the samples originated. Several parametric and non-parametric estimation techniques, employing diverse methods, such as Gaussian processes and Monte Carlo particle filtering, already exist. However, these techniques often trade estimation accuracy and sophistication for methodological flexibility and ease of use. Thus, there is room for new coalescent estimation techniques that can be easily implemented for a range of inference problems while still maintaining some sense of statistical optimality. Here we introduce the Bayesian Snyder filter as a natural, easily implementable and flexible minimum mean square error estimator for parametric demographic functions. By reinterpreting the coalescent as a self-correcting inhomogeneous Poisson process, we show that the Snyder filter can be applied to both isochronous (sampled at one time point) and heterochronous (serially sampled) estimation problems. We test the estimation performance of the filter on both standard, simulated demographic models and on a well-studied empirical dataset comprising hepatitis C virus sequences from Egypt. Additionally, we provide some analytical insight into the relationship between the Snyder filter and popular maximum likelihood and skyline plot techniques for coalescent inference. The Snyder filter is an exact and direct Bayesian estimation method that provides optimal mean square error estimates. It has the potential to become as a useful, alternative technique for coalescent inference.

A Construction of a β-Coalescent via the Pruning of Binary Trees

Considering a random binary tree with n labelled leaves, we use a pruning procedure on this tree in order to construct a β(3/2,1/2)-coalescent process. We also use the continuous analogue of this construction, i.e. a pruning procedure on Aldous's continuum random tree, to construct a continuous state space process that has the same structure as the β-coalescent process up to some time change. These two constructions enable us to obtain results on the coalescent process, such as the asymptotics on the number of coalescent events or the law of the blocks involved in the last coalescent event.