In order to increase the recycling of phosphogypsum waste, this study explored the feasibility of using phosphogypsum to replace some of the lime and aggregate in the lime-fly ash-crushed stone mixture which is a widely used road base material in China. For this purpose, compaction, compressive strength, composition structures, wetting-drying cycle tests, and shrinkage tests were carried out on the lime-fly ash-phosphogypsum-crushed stone composite to investigate its performance. The results indicate that lime-fly ash-crushed stone modified with phosphogypsum has the required strength of the road base material and favourable performances in environment (wetting-drying cycle) stability. The image processing analysis and shrinkage tests demonstrated that phosphogypsum can significantly improve the compactness and shrinkage performance of lime-fly ash-crushed stone mixture. A suitable content of phosphogypsum and a reasonable content of fine aggregate are conducive to improving the roadway engineering properties (i.e., decreasing shrinkage cracks and increasing compressive strength) of lime-fly ash-phosphogypsum-crushed stone composites.