The concentrations of polycyclic aromatic hydrocarbons (PAHs) in the leachate from the gangue and 20 groundwater samples, which were collected from the 12th Coal Mine around gangue piles in Henan Province, China, were determined by SPE-GC-MS. The characteristics of PAHs pollutants in groundwater were investigated, and compared with the concentrations of PAHs in the leachate from different weathered gangues to discuss the pollution effects of PAHs from coal gangue on groundwater. The results showed that total concentrations of the 16 EPA preferentially controlled PAHs ranged from 146.9 ng/L to 1220.6 ng/L.The components of PAHs such as chrysene, benzo[a]anthracene, benzo[b + k]fluoranthene, indeno[1,2,3-c,d]–pyrene, and dibenz[a,h]anthracene were fairly high. The 2–4 rings PAHs such as naphthalene, phenanthrene, fluorene and chrysene were dominant in groundwater, which was similar to those of the leachate from the different weathered gangues. Therefore, it should be paid much more attention on the transport of lower ring numbered PAHs leached by rains from the coal mines after landfilling and dumping. Based on the spatial distribution of PAHs and the high concentrations of PAHs with 2–4 rings in groundwater and leaching samples, there might be other pollution sources of PAHs except for penetration from coal gangue into groundwater in the Pingdingshan coal mine area.