The Mahu Sag in the Junggar Basin, China, is a hydrocarbon-rich sag. Abundant hydrocarbon source rocks were developed in the late Paleozoic there across the Carboniferous–Permian boundary. However, studies of the source rocks have focused mainly on the lower Permian Fengcheng Formation. Here we compare the Fengcheng Formation with other Carboniferous rocks and those of the lower Permian Jiamuhe and middle Permian Lower-Wuerhe formations. Based on organic petrological and geochemical data, the organic matter precursors, sedimentary facies, and resource potential of these source rocks were investigated. The bio-precursors of Carboniferous, Jiamuhe, and Lower-Wuerhe Formations were dominantly estuarine higher plants, Nematothallus -higher plants, and spores, respectively. The bio-precursors of Fengcheng Formation were mainly bacteria and algae, and the organic matter is abundant, with a high hydrocarbon-generating capacity and significant shale oil potential. In contrast, the other three formations contain poor-quality source rocks, although the Lower-Wuerhe Formation has a higher organic matter abundance than the Jiamuhe Formation. The Carboniferous organic matter contains mainly type II kerogen, whereas the Jiamuhe and Lower-Wuerhe formations contain type III kerogen. The thermal maturities determined from Tmax values is larger than those indicated by biomarkers. The biomarkers show that the three studied formations contain a mixture of terrestrial higher plants and bacteria–algae, with the contribution of green algae being higher than that of bacteria in most samples. However, the ratio of algae to bacteria is lower than that of the Fengcheng Formation. The Fengcheng Formation was deposited in a strongly reducing, high-salinity, and water-stratified sedimentary environment. The other three formations were deposited in an oxidizing–reducing, low-salinity, and water-unstratified environment. They are characterized by the predominance of mudstone over carbonate rocks and the descending distribution type of tricyclic terpenes. Our results provide a basis for research on upper Paleozoic source rocks in western China, and useful information for oil and gas exploration in the Mahu Sag.