THOR: Deep Web Data Extraction

Project Summary     Motivation     People     Publications    

Project Summary

THOR is a prototype Deep Web data extraction engine. In our first implementation, we have focused on extracting the query-relevant information portions of dynamically-generated web pages, portions we term QA-Pagelets. THOR relies on a two-phase extraction algorithm that first clusters pages according to their structural-similarity and then filters component subtrees according to structural and content similarity.

Motivation

The unabated growth of the Web has resulted in a situation in which more information is available to more people than ever in human history. Along with this unprecedented growth has come the inevitable problem of information overload. To counteract this information overload, users typically rely on search engines (like Google and AllTheWeb) or on manually-created categorization hierarchies (like Yahoo! and the Open Directory Project). Though excellent for accessing Web pages on the so-called "crawlable" web, these approaches overlook a much more massive and high-quality resource: the Deep Web.

The Deep Web (or Hidden Web) comprises all information that resides in autonomous databases behind portals and information providers' web front-ends. Web pages in the Deep Web are dynamically-generated in response to a query through a web site's search form and often contain rich content. A recent study has estimated the size of the Deep Web to be more than 500 billion pages, whereas the size of the "crawlable" web is only 1% of the Deep Web (i.e., less than 5 billion pages). Even those web sites with some static links that are "crawlable" by a search engine often have much more information available only through a query interface. Unlocking this vast deep web content presents a major research challenge.

In analogy to search engines over the "crawlable" web, we argue that one way to unlock the Deep Web is to employ a fully automated approach to extracting, indexing, and searching the query-related information-rich regions from dynamic web pages. Our initial efforts have focused on the first of these: extracting data from the Deep Web.

Extracting the interesting information from a Deep Web site requires many things: including scalable and robust methods for analyzing dynamic web pages of a given web site, discovering and locating the query-related information-rich content regions, and extracting itemized objects within each region. By full automation, we mean that the extraction algorithms should be designed independently of the presentation features or specific content of the web pages, such as the specific ways in which the query-related information is laid out or the specific locations where the navigational links and advertisement information are placed in the web pages.

People

Publications

  • J. Caverlee, L. Liu, and D. Buttler. Probe, Cluster, and Discover: Focused Extraction of QA-Pagelets from the Deep Web, accepted for publication in Proceedings of the 20th IEEE International Conference on Data Engineering (ICDE), 2004. (14% acceptance rate)
  • J. Caverlee, L. Liu, and D. Buttler. THOR: A Cluster-Based System for Extracting QA-Pagelets from the Deep Web, submitted for publication.

This research is partially supported by NSF CNS, NSF CCR, NSF ITR, DoE SciDAC, DARPA, CERCS Research Grant, IBM Faculty Award, IBM SUR grant, HP Equipment Grant, and LLNL LDRD.

Any opinions, findings, and conclusions or recommendations expressed in the project material are those of the authors and do not necessarily reflect the views of the sponsors.