Basic number processing skills in individuals with reading difficulties (RD) are intact. However, children and adults with RD show clear difficulties in arithmetic, in particular in retrieving known answers from long-term memory (fact retrieval). Fact retrieval deficits are associated with weaknesses in phonological awareness, the ability to segment and manipulate speech sounds. The left angular gyrus has been suggested as a site of neurological overlap between RD and fact retrieval deficits. While there is evidence for an involvement of the angular gyrus in fact retrieval in adults, the evidence for children is less clear. The same genetic risk factors may underlie difficulties in reading and mathematics and cause the high co-morbidity between RD and mathematical difficulties. Implications for interventions are discussed.