Metals and metalloids accumulate in soil, which not only leads to soil degradation and crop yield reduction but also poses hazards to human health. Commonly, source apportionment methods generate an overall relationship between sources and elements and, thus, lack the ability to capture important geographical variations of pollution sources. The present work uses a dataset collected by intensive sampling (1848 topsoil samples containing the metals Cd, Hg, Cr, Pb, and a metalloid of As) in the Shanghai study area and proposes a synthetic approach to source apportionment in the condition of spatial heterogeneity (non-stationarity) through the integration of absolute principal component scores with geographically weighted regression (APCA-GWR). The results showed that three main sources were detected by the APCA, i.e., natural sources, such as alluvial soil materials; agricultural activities, especially the overuse of phosphate fertilizer; and atmospheric deposition pollution from industry coal combustion and transportation activities. APCA-GWR provided more accurate and site-specific pollution source information than the mainstream APCA-MLR, which was verified by higher R2, lower AIC values, and non-spatial autocorrelation of residuals. According to APCA-GWR, natural sources were responsible for As and Cr accumulation in the northern mainland and Pb accumulation in the southern and northern mainland. Atmospheric deposition was the main source of Hg in the entire study area and Pb in the eastern mainland and Chongming Island. Agricultural activities, especially the overuse of phosphate fertilizer, were the main source of Cd across the study area and of As and Cr in the southern regions of the mainland and the middle of Chongming Island. In summary, this study highlights the use of a synthetic APCA-GWR model to efficiently handle source apportionment issues with spatial heterogeneity, which can provide more accurate and specific pollution source information and better references for pollution prevention and human health protection.