Field survey is a labour-intensive way to objectively evaluate the grade of building damage triggered by earthquakes. In this paper, we present a decision-tree-based approach to classify the type of building damage by using multiple-source remote sensing from both pre- and postearthquakes. Specifically, the boundary of buildings is delineated from preearthquake multiple-source satellite images using an unsupervised learning method. Then, building damage is classified into four types using decision tree method from postearthquake UAV images, that is, basically intact buildings, slightly damaged buildings, partially collapsed buildings, and completely collapsed buildings. Furthermore, the slightly damaged buildings are determined by the detected roof-holes using joint color and height features. Two experimental areas from Wenchuan and Ya’an earthquakes are used to verify the proposed method.