Smallholder farming in sub-Saharan Africa keeps many rural households trapped in a cycle of poor productivity and low incomes. Two options to reach a decent income include intensification of production and expansion of farm areas per household. In this study, we explore what is a “viable farm size,” i.e., the farm area that is required to attain a “living income,” which sustains a nutritious diet, housing, education and health care. We used survey data from three contrasting sites in the East African highlands—Nyando (Kenya), Rakai (Uganda), and Lushoto (Tanzania) to explore viable farm sizes in six scenarios. Starting from the baseline cropping system, we built scenarios by incrementally including intensified and re-configured cropping systems, income from livestock and off-farm sources. In the most conservative scenario (baseline cropping patterns and yields, minus basic input costs), viable farm areas were 3.6, 2.4, and 2.1 ha, for Nyando, Rakai, and Lushoto, respectively—whereas current median farm areas were just 0.8, 1.8, and 0.8 ha. Given the skewed distribution of current farm areas, only few of the households in the study sites (0, 27, and 4% for Nyando, Rakai, and Lushoto, respectively) were able to attain a living income. Raising baseline yields to 50% of the water-limited yields strongly reduced the land area needed to achieve a viable farm size, and thereby enabled 92% of the households in Rakai and 70% of the households in Lushoto to attain a living income on their existing farm areas. By contrast, intensification of crop production alone was insufficient in Nyando, although including income from livestock enabled the majority of households (73%) to attain a living income with current farm areas. These scenarios show that increasing farm area and/or intensifying production is required for smallholder farmers to attain a living income from farming. Obviously such changes would require considerable capital and labor investment, as well as land reform and alternative off-farm employment options for those who exit farming.