Rate control has significant impact on the video coding. However, the conventional rate control algorithm in the latest video coding standard H.264 without take into account the human visual system and result in video quality fluctuation. To address this problem, we propose a novel rate control algorithm for H.264/AVC considering frame activity. First, a method is proposed to detect the scene change in frames of video sequences. Second, bit allocation to frame is related to the frame activity which is calculated on the local motion activity and scene change in frames. And in the MB level, the number of bits allocated to each MB is determined by its local motion activity. Experimental results show that the proposed rate control algorithm can increase average PSNR by 0.9137 comparisons with the JVT-G012, and reduces the bit-rate error by up to 0.06%.