Pig manure (PM) is often highly enriched in heavy metals, such as Cu and Zn, due to the wide use of feed additives. To study the potential risks of heavy metal accumulation in the soil and rice grains by the application of PM and other organic manure, a four-year field experiment was conducted in the suburb of Shanghai, southeast China. The contents of Cu, Zn, Pb, and Cd in the soils and rice plants by the treatments of PM and fungal culturing residues (FCR) show a trend of annual increase. Those in the soils and rice by the PM treatment are raised even more significantly. Cu and Zn contents in the soil and rice roots by the PM are significantly higher than those by the non-fertilizer control (CK) during the four years, and Pb and Cd also significantly higher than CK in the latter two years. Heavy metals taken up by the rice plants are mostly retained in the roots. Cu and Zn contents in the rice plants are in the decreasing order of roots > grains > stems > leaves, and Pb and Cd in the order of roots > stems > leaves > grains. Cu, Zn, Pb, and Cd contents in the soils by the PM treatment increase by 73%, 32%, 106%, and 127% on annual average, and those in the brown rice by 104%, 98%, 275%, and 199%, respectively. The contents of Cu, Zn, Pb, and Cd in the brown rice of the treatments are significantly correlated with those in the soils and rice roots (p < 0.05), suggesting the heavy metals accumulated in the rice grains come from the application of PM and FCR. Though the contents of heavy metals in the brown rice during the four experimental years are still within the safe levels, the risks of their accumulative increments, especially by long-term application of PM, can never be neglected.