In this chapter, we describe a histogram with soft decision using the Hue, Saturation, Intensity Value (HSV) color space for effective detection of video shot boundaries. In the histogram, we choose the relative importance of hue and intensity depending on the saturation of each pixel. In traditional histograms, each pixel contributes to only one component of the histogram. However, we suggest a soft-decision approach in which each pixel contributes to two components of the histogram. We have done a detailed study of the various frame-to-frame distance measures using the proposed histogram and an Red, Green, Blue (RGB) histogram for video shot detection. The results show that the new histogram has a better shot-detection performance for each of the distance measures. A Web-based application has been developed for video retrieval, which is freely accessible to interested users.