Search engines use crawlers to traverse the Web in order to download web
pages and build their indexes. Maintaining these indexes up-to-date is an
essential task to ensure the quality of search results. However, changes in
web pages are unpredictable. Identifying the moment when a web page changes
as soon as possible and with minimal computational cost is a major challenge.
In this article we present the Web Change Detection system that, in a best
case scenario, is capable to detect, almost in real time, when a web page
changes. In a worst case scenario, it will require, on average, 12 minutes to
detect a change on a low PageRank web site and about one minute on a web site
with high PageRank. Meanwhile, current search engines require more than a
day, on average, to detect a modification in a web page (in both cases).
Exploiting the Web for Semantic Change Detection