Due to the advancement of multimedia and its requirement of communication over the network, video compression has received much attention among the researchers. One of the popular video codings is scalable video coding, referred to as H.264/AVC standard. The major drawback in the H.264 is that it performs the exhaustive search over the interlayer prediction to gain the best rate-distortion performance. To reduce the computation overhead due to exhaustive search on mode prediction process, this paper presents a new technique for inter prediction mode selection based on the fuzzy holoentropy. This proposed scheme utilizes the pixel values and probabilistic distribution of pixel symbols to decide the mode. The adaptive mode selection is introduced here by analyzing the pixel values of the current block to be coded with those of a motion compensated reference block using fuzzy holoentropy. The adaptively selected mode decision can reduce the computation time without affecting the visual quality of frames. Experimentation of the proposed scheme is evaluated by utilizing five videos, and from the analysis, it is evident that proposed scheme has overall high performance with values of 41.367 dB and 0.992 for PSNR and SSIM respectively.
