Issues and Challenges in Web Crawling for Information Extraction

Computational biology and bio inspired techniques are part of a larger revolution that is increasing the processing, storage and retrieving of data in major way. This larger revolution is being driven by the generation and use of information in all forms and in enormous quantities and requires the development of intelligent systems for gathering, storing and accessing information. This chapter describes the concepts, design and implementation of a distributed web crawler that runs on a network of workstations and has been used for web information extraction. The crawler needs to scale (at least) several hundred pages per second, is resilient against system crashes and other events, and is capable to adapted various crawling applications. Further this chapter, focusses on various ways in which appropriate biological and bio inspired tools can be used to implement, automatically locate, understand, and extract online data independent of the source and also to make it available for Semantic web agents like a web crawler.

Introduction

Picture archiving and communications system (PACS) is a filmless and computerized method of communicating and storing medical image data such as computed radiographic, digital radiographic, computed tomographic, ultrasound, fluoroscopic, magnetic resonance and other special X-ray images. A PACS consists of image and data acquisition, storage, display stations integrated with various digital networks. A PACS handling images from various medical imaging modalities is called a full PACS. Small-scale systems that handle images from a single modality (usually connected to a single acquisition device) are sometimes called mini-PACS. A hospital-wide PACS is a PACS which entirely replaces conventional x-ray film by displaying digital images on a network of workstations throughout the hospital. This kind of hospital is called a “Filmless Hospital” (Strickland, 2000). In healthcare environment, the practicing of radiology without X-ray film is called “Filmless Radiology”.