SIMHAR - Smart Distributed Web Crawler for the Hidden Web Using SIM+Hash and Redis Server

A huge number of Hidden Web databases exists over the WWW forming a massive source of high quality information. Retrieval of this information for enriching the repository of the search engine is the prime target of a Hidden web crawler. Besides this, the crawler should perform this task at an affordable cost and resource utilization. This paper proposes a Random ranking mechanism whereby the queries to be raised by the hidden web crawler have been ranked. By ranking the queries according to the proposed mechanism, the Hidden Web crawler is able to make an optimal choice among the candidate queries and efficiently retrieve the Hidden web databases. The Hidden Web crawler proposed here also possesses an extensible and scalable framework to improve the efficiency of crawling. The proposed approach has also been compared with other methods of Hidden Web crawling existing in the literature.

Download Full-text

Design of a Least Cost (LC) Vertical Search Engine Based on Domain Specific Hidden Web Crawler

The Dark Web ◽

10.4018/978-1-5225-3163-0.ch014 ◽

2018 ◽

pp. 319-333

Author(s):

Sudhakar Ranjan ◽

Komal Kumar Bhatia

Keyword(s):

Search Engine ◽

Human Life ◽

Experimental Result ◽

Web Crawler ◽

Domain Specific ◽

Hidden Web ◽

Vertical Search ◽

Communication Time ◽

Vertical Search Engine ◽

Domain Term

Now days with the advent of internet technologies and ecommerce the need for smart search engine for human life is rising. The traditional search engines are not intelligent as well as smart and thus lead to the rise in searching costs. In this paper, architecture of a vertical search engine based on the domain specific hidden web crawler is proposed. To make a least cost vertical search engine improvement in the following techniques like: searching, indexing, ranking, transaction and query interface are suggested. The domain term analyzer filters the useless information to the maximum extent and finally provides the users with high precision information. Through the experimental result it is shown that the system works on accelerating the access, computation, storage, communication time, increased efficiency and work professionally.

Download Full-text

Optimal Query Generation for Hidden Web Extraction Through Response Analysis

The Dark Web ◽

10.4018/978-1-5225-3163-0.ch005 ◽

2018 ◽

pp. 65-83

Author(s):

Sonali Gupta ◽

Komal Kumar Bhatia

Keyword(s):

Optimal Choice ◽

Response Analysis ◽

Web Crawler ◽

Web Databases ◽

Web Extraction ◽

Hidden Web ◽

Query Generation ◽

High Quality Information ◽

Ranking Mechanism ◽

Prime Target

A huge number of Hidden Web databases exists over the WWW forming a massive source of high quality information. Retrieval of this information for enriching the repository of the search engine is the prime target of a Hidden web crawler. Besides this, the crawler should perform this task at an affordable cost and resource utilization. This paper proposes a Random ranking mechanism whereby the queries to be raised by the hidden web crawler have been ranked. By ranking the queries according to the proposed mechanism, the Hidden Web crawler is able to make an optimal choice among the candidate queries and efficiently retrieve the Hidden web databases. The Hidden Web crawler proposed here also possesses an extensible and scalable framework to improve the efficiency of crawling. The proposed approach has also been compared with other methods of Hidden Web crawling existing in the literature.

Download Full-text

A Novel Design of Hidden Web Crawler Using Reinforcement Learning Based Agents

Lecture Notes in Computer Science - Advanced Parallel Processing Technologies ◽

10.1007/978-3-540-76837-1_47 ◽

2007 ◽

pp. 433-440 ◽

Cited By ~ 1

Author(s):

J. Akilandeswari ◽

N. P. Gopalan

Keyword(s):

Reinforcement Learning ◽

Web Crawler ◽

Hidden Web ◽

Novel Design

Download Full-text

Design of a Least Cost (LC) Vertical Search Engine based on Domain Specific Hidden Web Crawler

International Journal of Information Retrieval Research ◽

10.4018/ijirr.2017040102 ◽

2017 ◽

Vol 7 (2) ◽

pp. 19-33

Author(s):

Sudhakar Ranjan ◽

Komal Kumar Bhatia

Keyword(s):

Search Engine ◽

Human Life ◽

Experimental Result ◽

Web Crawler ◽

Domain Specific ◽

Hidden Web ◽

Vertical Search ◽

Communication Time ◽

Vertical Search Engine ◽

Domain Term

Now days with the advent of internet technologies and ecommerce the need for smart search engine for human life is rising. The traditional search engines are not intelligent as well as smart and thus lead to the rise in searching costs. In this paper, architecture of a vertical search engine based on the domain specific hidden web crawler is proposed. To make a least cost vertical search engine improvement in the following techniques like: searching, indexing, ranking, transaction and query interface are suggested. The domain term analyzer filters the useless information to the maximum extent and finally provides the users with high precision information. Through the experimental result it is shown that the system works on accelerating the access, computation, storage, communication time, increased efficiency and work professionally.

Download Full-text