MMIR

This chapter introduces an advanced content-based image retrieval (CBIR) system, MMIR, where Markov model mediator (MMM) and multiple instance learning (MIL) techniques are integrated seamlessly and act coherently as a hierarchical learning engine to boost both the retrieval accuracy and efficiency. It is well-understood that the major bottleneck of CBIR systems is the large semantic gap between the low-level image features and the high-level semantic concepts. In addition, the perception subjectivity problem also challenges a CBIR system. To address these issues and challenges, the proposed MMIR system utilizes the MMM mechanism to direct the focus on the image level analysis together with the MIL technique (with the neural network technique as its core) to real-time capture and learn the object-level semantic concepts with some help of the user feedbacks. In addition, from a long-term learning perspective, the user feedback logs are explored by MMM to speed up the learning process and to increase the retrieval accuracy for a query. The comparative studies on a large set of real-world images demonstrate the promising performance of our proposed MMIR system.

Making Image Retrieval and Classification More Accurate Using Time Series and Learned Constraints

After the generation of multimedia data turning digital, an explosion of interest in their data storage, retrieval, and processing, has drastically increased in the database and data mining community. This includes videos, images, and handwriting, where we now have higher expectations in exploiting these data at hand. We argue however, that much of this work’s narrow focus on efficiency and scalability has come at the cost of usability and effectiveness. Typical manipulations are in some forms of video/image processing, which require fairly large amounts for storage and are computationally intensive. In this work, we will demonstrate how these multimedia data can be reduced to a more compact form, that is, time series representation, while preserving the features of interest, and can then be efficiently exploited in Content-Based Image Retrieval. We also introduce a general framework that learns a distance measure with arbitrary constraints on the warping path of the Dynamic Time Warping calculation. We demonstrate utilities of our approach on both classification and query retrieval tasks for time series and other types of multimedia data including images, video frames, and handwriting archives. In addition, we show that incorporating this framework into the relevance feedback system, a query refinement can be used to further improve the precision/recall by a wide margin.

Content Based Image Retrieval Using Active-Nets

Content-based image retrieval has been an active research area in past years. Many different solutions have been proposed to improve performance of retrieval, but the large part of these works have focused on sub-parts of the retrieval problem, providing targeted solutions only for individual aspects (i.e., feature extraction, similarity measures, indexing, etc). In this chapter, we first shortly review some of the main practiced solutions for content-based image retrieval evidencing some of the main issues. Then, we propose an original approach for the extraction of relevant image objects and their matching for retrieval applications, and present a complete image retrieval system which uses this approach (including similarity measures and image indexing). In particular, image objects are represented by a two-dimensional deformable structure, referred to as “active net.” Active net is capable of adapting to relevant image regions according to chromatic and edge information. Extension of the active nets has been defined, which permits the nets to break themselves, thus increasing their capability to adapt to objects with complex topological structure. The resulting representation allows a joint description of color, shape, and structural information of extracted objects. A similarity measure between active nets has also been defined and used to combine the retrieval with an efficient indexing structure. The proposed system has been experimented on two large and publicly available objects databases, namely, the ETH-80 and the ALOI.

Content-Based Retrieval for Mammograms

As distributed mammogram databases at hospitals and breast screening centers are connected together through PACS, a mammogram retrieval system is needed to help medical professionals locate the mammograms they want to aid in medical diagnosis. This chapter presents a complete content-based mammogram retrieval system, seeking images that are pathologically similar to a given example. In the mammogram retrieval system, the pathological characteristics that have been defined in Breast Imaging Reporting and Data System (BI-RADSTM) are used as criteria to measure the similarity of the mammograms. A detailed description of those mammographic features is provided in this chapter. Since the user’s subjective perception should be taken into account in the image retrieval task, a relevance feedback function is also developed to learn individual users’ knowledge to improve the system performance.

Event Detection, Query, and Retrieval for Video Surveillance

Video surveillance automation is used in two key modes: watching for known threats in real-time and searching for events of interest after the fact. Typically, real-time alerting is a localized function, for example, an airport security center receives and reacts to a “perimeter breach alert,” while investigations often tend to encompass a large number of geographically distributed cameras like the London bombing, or Washington sniper incidents. Enabling effective event detection, query and retrieval of surveillance video for preemption, and investigation, involves indexing the video along multiple dimensions. This chapter presents a framework for event detection and surveillance search that includes: video parsing, indexing, query and retrieval mechanisms. It explores video parsing techniques that automatically extract index data from video indexing, which stores data in relational tables; retrieval which uses SQL queries to retrieve events of interest and the software architecture that integrates these technologies.

Content Analysis from User's Relevance Feedback for Content-Based Image Retrieval

An image is a symbolic representation; people interpret an image and associate semantics with it based on their subjective perceptions, which involves the user’s knowledge, cultural background, personal feelings and so on. Content-based image retrieval (CBIR) systems must be able to interact with users and discover the current user’s information needs. An interactive search paradigm that has been developed for image retrieval is machine learning with a user-in-the-loop, guided by relevance feedback, which refers to the notion of relevance of the individual image based on the current user’s subjective judgment. Relevance feedback serves as an information carrier to convey the user’s information needs / preferences to the retrieval system. This chapter not only provides the fundamentals of CBIR systems and relevance feedback for understanding and incorporating relevance feedback into CBIR systems, but also discusses several approaches to analyzing and learning relevance feedback.

A Semantics Sensitive Framework of Organization and Retrieval for Multimedia Databases

Semantics-based retrieval is a trend of the Content-Based Multimedia Retrieval (CBMR). Typically, in multimedia databases, there exist two kinds of clues for query: perceptive features and semantic classes. In this chapter, we proposed a novel framework for multimedia database organization and retrieval, integrating the perceptive features and semantic classes. Thereunto, a semantics supervised cluster-based index organization approach (briefly as SSCI) was developed: the entire data set is divided hierarchically into many clusters until the objects within a cluster are not only close in the perceptive feature space, but also within the same semantic class; then an index entry is built for each cluster. Especially, the perceptive feature vectors in a cluster are organized adjacently in disk. Furthermore, the SSCI supports a relevance feedback approach: users sign the positive and negative examples regarded a cluster as unit rather than a single object. Our experiments show that the proposed framework can improve the retrieval speed and precision of the CBMR systems significantly.

Personalized Content-Based Image Retrieval

In content-based image retrieval (CBIR), a set of low-level features are extracted from an image to represent its visual content. Retrieval is performed by image example where a query image is given as input by the user and an appropriate similarity measure is used to find the best matches in the corresponding feature space. This approach suffers from the fact that there is a large discrepancy between the low-level visual features that one can extract from an image and the semantic interpretation of the image’s content that a particular user may have in a given situation. That is, users seek semantic similarity, but we can only provide similarity based on low-level visual features extracted from the raw pixel data, a situation known as the semantic gap. The selection of an appropriate similarity measure is thus an important problem. Since visual content can be represented by different attributes, the combination and importance of each set of features varies according to the user’s semantic intent. Thus, the retrieval strategy should be adaptive so that it can accommodate the preferences of different users. Relevance feedback (RF) learning has been proposed as a technique aimed at reducing the semantic gap. It works by gathering semantic information from user interaction. Based on the user’s feedback on the retrieval results, the retrieval scheme is adjusted. By providing an image similarity measure under human perception, RF learning can be seen as a form of supervised learning that finds relations between high-level semantic interpretations and low-level visual properties. That is, the feedback obtained within a single query session is used to personalize the retrieval strategy and thus enhance retrieval performance. In this chapter we present an overview of CBIR and related work on RF learning. We also present our own previous work on a RF learning-based probabilistic region relevance learning algorithm for automatically estimating the importance of each region in an image based on the user’s semantic intent.

A Machine Learning-Based Model for Content-Based Image Retrieval

A multimedia index makes it possible to group data according to similarity criteria. Traditional index structures are based on trees and use the k-Nearest Neighbors (k-NN) approach to retrieve databases. Due to some disadvantages of such an approach, the use of neighborhood graphs was proposed. This approach is interesting, but it has some disadvantages, mainly in its complexity. This chapter presents a step in a long process of analyzing, structuring, and retrieving multimedia databases. Indeed, we propose an effective method for locally updating neighborhood graphs, which constitute our multimedia index. Then, we exploit this structure in order to make the retrieval process easy and effective, using queries in an image form in one hand. In another hand, we use the indexing structure to annotate images in order to describe their semantics. The proposed approach is based on an intelligent manner for locating points in a multidimensional space. Promising results are obtained after experimentations on various databases. Future issues of the proposed approach are very relevant in this domain.

Content-Based Image Retrieval

Content-Based Image Retrieval (CBIR) aims to search images that are perceptually similar to the querybased on visual content of the images without the help of annotations. The current CBIR systems use global features (e.g., color, texture, and shape) as image descriptors, or usefeatures extracted from segmented regions (called region-based descriptors). In the former case, descriptors are not discriminative enough at the object level and are sensitive to object occlusion or background clutter, thus fail to give satisfactory result. In the latter case, the features are sensitive to the image segmentation, which is a difficult task in its own right. In addition, the region-based descriptors are still not invariant to varying imaging conditions. In this chapter, we look at the CBIR from the object detection/recognition point of view and introduce the local feature-based image representation methods recently developed in object detection/recognition area. These local descriptors are highly distinctive and robust to imaging condition change. In addition to image representation, we also introduce the other two key issues of CBIR: similarity measurement for image descriptor comparison and the index structure for similarity search.