Paper:  16.  Crawling towards Eternity: Building an Archive of the World Wide Web 


This paper describes an effort to archive the World Wide Web.  The techniques used to archive the web are very similar to techniques used by search
engines.  That is they archiving needs to start with a seed list, visit web-sites, establish if they have been seen before, and if they are new,
harvest urls from new/changed web pages, establish if the urls are new, and crawl again.  The difference that this paper points out is that in
archiving, indexing is not the most important task.  The emphasis is on versioning pages visited.   

Some ideas introduced in this paper include the use of site based crawling, and the use of bitmap filters to establish if urls have been crawled.
Site based crawling is where one process is dedicated to crawling an entire site.  The process may be responsible for multiple sites, but no other
process will be responsible for the these sites.  This scheme has logistical advantages when it comes to trying to establish if a site has been seen
before since no other process needs to be referenced to establish this.  Bitmap filters are chunks of memory of which bits are set to one if a url has
been visited.  Bits are set on based on multiple hash values of a bitmap and turning on the appropriate bits.

In practice the paper reported that the archiver had been running for about a year between 1996 and 997.  In that time it collected about 2 terabytes
of data.  aA weakness of the paper may be that it does not adequately address the problem of a data set that grows exponentially.