Optimization of Automatic Navigation to Hidden Web Pages by Ranking-Based Browser Preloading

2006 ◽  
pp. 40-49
Author(s):  
Justo Hidalgo ◽  
José Losada ◽  
Manuel Álvarez ◽  
Alberto Pan
Keyword(s):  
Web Pages ◽  
Automatic Navigation ◽  
Hidden Web

scholarly journals Automated Web Application ...

10.29007/vs62 ◽  
2018 ◽  
Author(s):  
Priyank Bhojak ◽  
Vatsal Shah ◽  
Kanu Patel ◽  
Deven Gol
Keyword(s):  
Dynamic Analysis ◽  
Open Source ◽  
Web Application ◽  
Web Applications ◽  
Black Box ◽  
Web Pages ◽  
Software Bugs ◽  
Hidden Web ◽  
A New Technique ◽  
Input Validation

The rate of web application threats is growing more and more now in days. The most of software bugs are result from inappropriate input validation. It should lead to attack of confidential information, breaking of knowledge integrity. We develop a scanner for detecting SQ injection and XSS type software-bugs which is based on hidden web crawl and make open source scanner with the aim of hidden web crawling which may be require authentication. In this research paper we presents a new technique to find vulnerability which include advantages of black-box analysis of different web pages. And at the end we shows evaluation table which mention comparison of our scanner with two other web scanner tool. So finally this paper additionally shows how easy it is to scan web application bugs with dynamic analysis and retrieve hidden web pages from web applications.

scholarly journals A scale for crawler effectiveness on the client-side hidden web

2012 ◽  
Vol 9 (2) ◽  
pp. 561-583 ◽  
Author(s):  
Víctor Prieto ◽  
Manuel Álvarez ◽  
Rafael López-García ◽  
Fidel Cacheda
Keyword(s):  
Open Source ◽  
Search Engines ◽  
Web Site ◽  
Point Of View ◽  
Web Pages ◽  
Common Features ◽  
Hidden Web ◽  
Client Side ◽  
The Web

The main goal of this study is to present a scale that classifies crawling systems according to their effectiveness in traversing the ?clientside? Hidden Web. First, we perform a thorough analysis of the different client-side technologies and the main features of the web pages in order to determine the basic steps of the aforementioned scale. Then, we define the scale by grouping basic scenarios in terms of several common features, and we propose some methods to evaluate the effectiveness of the crawlers according to the levels of the scale. Finally, we present a testing web site and we show the results of applying the aforementioned methods to the results obtained by some open-source and commercial crawlers that tried to traverse the pages. Only a few crawlers achieve good results in treating client-side technologies. Regarding standalone crawlers, we highlight the open-source crawlers Heritrix and Nutch and the commercial crawler WebCopierPro, which is able to process very complex scenarios. With regard to the crawlers of the main search engines, only Google processes most of the scenarios we have proposed, while Yahoo! and Bing just deal with the basic ones. There are not many studies that assess the capacity of the crawlers to deal with client-side technologies. Also, these studies consider fewer technologies, fewer crawlers and fewer combinations. Furthermore, to the best of our knowledge, our article provides the first scale for classifying crawlers from the point of view of the most important client-side technologies.

scholarly journals IHWC: intelligent hidden web crawler for harvesting data in urban domains

2021 ◽  
Author(s):  
Sawroop Kaur ◽  
Aman Singh ◽  
G. Geetha ◽  
Xiaochun Cheng
Keyword(s):  
Large Scale ◽  
Smart Cities ◽  
Quality Data ◽  
Web Pages ◽  
Web Crawler ◽  
Harvest Rate ◽  
Web Interfaces ◽  
Hidden Web ◽  
Special Cases ◽  
Daunting Task

AbstractDue to the massive size of the hidden web, searching, retrieving and mining rich and high-quality data can be a daunting task. Moreover, with the presence of forms, data cannot be accessed easily. Forms are dynamic, heterogeneous and spread over trillions of web pages. Significant efforts have addressed the problem of tapping into the hidden web to integrate and mine rich data. Effective techniques, as well as application in special cases, are required to be explored to achieve an effective harvest rate. One such special area is atmospheric science, where hidden web crawling is least implemented, and crawler is required to crawl through the huge web to narrow down the search to specific data. In this study, an intelligent hidden web crawler for harvesting data in urban domains (IHWC) is implemented to address the relative problems such as classification of domains, prevention of exhaustive searching, and prioritizing the URLs. The crawler also performs well in curating pollution-related data. The crawler targets the relevant web pages and discards the irrelevant by implementing rejection rules. To achieve more accurate results for a focused crawl, ICHW crawls the websites on priority for a given topic. The crawler has fulfilled the dual objective of developing an effective hidden web crawler that can focus on diverse domains and to check its integration in searching pollution data in smart cities. One of the objectives of smart cities is to reduce pollution. Resultant crawled data can be used for finding the reason for pollution. The crawler can help the user to search the level of pollution in a specific area. The harvest rate of the crawler is compared with pioneer existing work. With an increase in the size of a dataset, the presented crawler can add significant value to emission accuracy. Our results are demonstrating the accuracy and harvest rate of the proposed framework, and it efficiently collect hidden web interfaces from large-scale sites and achieve higher rates than other crawlers.

scholarly journals Design of a Parallel and Scalable Crawler for the Hidden Web

2022 ◽  
Vol 12 (1) ◽  
pp. 0-0
Keyword(s):  
Experimental Results ◽  
Web Pages ◽  
Web Crawler ◽  
Huge Amount ◽  
Web Databases ◽  
Specific Approach ◽  
Amount Of Information ◽  
Domain Specific ◽  
Hidden Web ◽  
Enormous Amount

The WWW contains huge amount of information from different areas. This information may be present virtually in the form of web pages, media, articles (research journals / magazine), blogs etc. A major portion of the information is present in web databases that can be retrieved by raising queries at the interface offered by the specific database and is thus called the Hidden Web. An important issue is to efficiently retrieve and provide access to this enormous amount of information through crawling. In this paper, we present the architecture of a parallel crawler for the Hidden Web that avoids download overlaps by following a domain-specific approach. The experimental results further show that the proposed parallel Hidden web crawler (PSHWC), not only effectively but also efficiently extracts and download the contents in the Hidden web databases

Automatic generation of agents for collecting hidden Web pages for data extraction

2004 ◽  
Vol 49 (2) ◽  
pp. 177-196 ◽  
Author(s):  
Juliano Palmieri Lage ◽  
Altigran S. da Silva ◽  
Paulo B. Golgher ◽  
Alberto H.F. Laender
Keyword(s):  
Data Extraction ◽  
Automatic Generation ◽  
Web Pages ◽  
Hidden Web

Appropriate Targets for Search Advertising as Part of Online Gatekeeping for Suicide Prevention

Crisis ◽  
2018 ◽  
Vol 39 (3) ◽  
pp. 197-204 ◽  
Author(s):  
Hajime Sueki ◽  
Jiro Ito
Keyword(s):  
Suicide Prevention ◽  
Daily Life ◽  
Web Pages ◽  
Service Users ◽  
Online Questionnaire ◽  
Consultation Service ◽  
Self Disclosure ◽  
Search Advertising ◽  
Consultation Services ◽  
Age Range

Abstract. Background: Gatekeeper training is an effective suicide prevention strategy. However, the appropriate targets of online gatekeeping have not yet been clarified. Aim: We examined the association between the outcomes of online gatekeeping using the Internet and the characteristics of consultation service users. Method: An advertisement to encourage the use of e-mail-based psychological consultation services among viewers was placed on web pages that showed the results of searches using suicide-related keywords. All e-mails received between October 2014 and December 2015 were replied to as part of gatekeeping, and the obtained data (responses to an online questionnaire and the content of the received e-mails) were analyzed. Results: A total of 154 consultation service users were analyzed, 35.7% of whom were male. The median age range was 20–29 years. Online gatekeeping was significantly more likely to be successful when such users faced financial/daily life or workplace problems, or revealed their names (including online names). By contrast, the activity was more likely to be unsuccessful when it was impossible to assess the problems faced by consultation service users. Conclusion: It may be possible to increase the success rate of online gatekeeping by targeting individuals facing financial/daily life or workplace problems with marked tendencies for self-disclosure.

Sign in / Sign up

Export Citation Format

Share Document