Video summary is very important for users to grasp a whole video’s content quickly for efficient browsing and editing. In this chapter, we propose a novel video summarization approach based on redundancy removing and content ranking. Firstly, by video parsing and cast indexing, the approach constructs a story board to let user know about the main scenes and the main actors in the video. Then it removes redundant frames to generate a “story-constraint summary” by key frame clustering and repetitive segment detection. To shorten the video summary length to a target length, “time-constraint summary” is constructed by important factor based content ranking. Extensive experiments are carried out on TV series, movies, and cartoons. Good results demonstrate the effectiveness of the proposed method.