Climate change and human activities are the major factors affecting runoff and sediment load. We analyzed the inter-annual variation trend of the average rainfall, air temperature, runoff and sediment load in the Xihe River Basin from 1969–2015. Pettitt’s test and the Soil and Water Assessment Tool (SWAT) model were used to detect sudden change in hydro-meteorological variables and simulate the basin hydrological cycle, respectively. According to the simulation results, we explored spatial distribution of soil erosion in the watershed by utilizing ArcGIS10.0, analyzed the average erosion modulus by different type of land use, and quantified the contributions of climate change and human activities to runoff and sediment load in changes. The results showed that: (1) From 1969–2015, both rainfall and air temperature increased, and air temperature increased significantly (p < 0.01) at 0.326 °C/10 a (annual). Runoff and sediment load decreased, and sediment load decreased significantly (p < 0.01) at 1.63 × 105 t/10 a. In 1988, air temperature experienced a sudden increase and sediment load decreased. (2) For runoff, R2 and Nash and Sutcliffe efficiency coefficient (Ens) were 0.92 and 0.91 during the calibration period and 0.90 and 0.87 during the validation period, for sediment load, R2 and Ens were 0.60 and 0.55 during the calibration period and 0.70 and 0.69 during the validation period, meeting the model’s applicability requirements. (3) Soil erosion was worse in the upper basin than other regions, and highest in cultivated land. Climate change exacerbates runoff and sediment load with overall contribution to the total change of −26.54% and −8.8%, respectively. Human activities decreased runoff and sediment load with overall contribution to the total change of 126.54% and 108.8% respectively. Runoff and sediment load change in the Xihe River Basin are largely caused by human activities.