Graph based learning has been an active research topic in machine learning community as well as many application areas including image annotation recently. In order to exploit the correlation between keywords and images, we proposed a novel image annotation method via graph based learning and semantic fusion to estimate the probability of keywords being the caption of an image, and present a new framework to solve the problem. The experiments over Corel images have shown that this approach outperforms other methods and is effective for image annotation.