Aging is accompanied by declines in white matter integrity (e.g., demyelination, decreased fiber density) that contribute to cognitive deficits. Diffusion tensor imaging (DTI) studies have observed these integrity declines in vivo separately in younger-old (ages 65-89) and oldest-old (ages 90+) adults. But it remains unclear whether the effect of age on integrity is magnified in advanced age groups and whether this may result from normal aging or dementia-related pathology. Here, we tested whether age-related differences in white matter integrity followed linear or nonlinear patterns when considering the entire older adult lifespan (n = 108; 65-98 years) and whether these patterns were influenced by oldest-old adults at increased risk of dementia (cognitive impairment no dementia, CIND). To assess the functional impact of white matter aging, we then examined the extent to which it explained age effects on episodic memory performance (delayed recall, recognition). Results revealed significant nonlinear declines in the integrity of medial temporal, callosal, and association fiber classes, with linear declines observed for the projection/thalamic fiber class. These patterns remained after excluding the oldest-old participants with CIND, indicating that larger differences in white matter integrity with increased age cannot solely be explained by pathology associated with early cognitive impairment. We also found that the effect of age on episodic memory was partially mediated by integrity of medial temporal fibers, suggesting that they are essential for facilitating memory-related neural signals across the older adult lifespan.