Table of Contents Previous Chapter

Resource Discovery

Edward A. Fox, V. S. Subrahmanian & John M. Carroll

For the foreseeable future, working on the problem of information retrieval in large collections will be of value. There are continual improvements in this field, and allied areas (e.g., natural language processing) also continue to yield new tools and techniques that can be tested and integrated with other approaches. Connecting with hypertext, browsing, and visualization are all important parts of the solution. Success here is certainly of value in solving similar problems for the WWW.

Digital libraries are large, and usually are made up of a number of heterogeneous collections. They will typically include at least a moderate amount of multimedia information, which presents new challenges relative to only searching text. They make it more important to manage variety in item size. In addition to indexing, cataloging is required, usually to handle large items or item collections. Other problems come to roost, like eliminating duplicates or relating multiple versions.

Since traditional libraries are often located and planned to serve a range of user needs, such as university education and research, digital libraries will certainly be required to support important user tasks. Part of this means facilitating search, navigation and browsing in and between multiple spaces, such as document space, concept space, keyword space, and others relating to strings, features, or linguistic constructs. Whether one deals with geographic information, video archives, or data from outer space, many of the same general approaches will apply, especially if integrated with domain specific processing.

Adapting these general approaches to particular types of collections, however, warrants empirical validation in a variety of representative situations. It is also necessary to develop a theory and practice of specification of complex collections and their processing, whereby: specialized methods can be applied to each multimedia data type; new approaches can be developed to handle the complex interrelationships of those data items into common structures (e.g., multimedia streams and performances, composite documents, multimodal interaction schema); and distributed retrieval operations can effectively operate upon such large and complex heterogeneous collections (as found in digital libraries).

Digital libraries are possible now in part due to advances in electronic publishing (interpreted in the broad sense). People create most documents with computers, and also capture a large percentage of multimedia objects into digital forms. With proper standards, much of what is needed by way of indexing, cataloging and metadata can be automatically recorded in usable forms. Such standards and practices can be applied to the WWW at large, especially if urgently needed research develops tools and techniques that allow domain-specialized standards to be used in general purpose environments (like the Web) without loosing their expressive power or requiring labor-intensive coding or translations.

With so much automatic processing, the trend to disintermediation will continue. Thus, an important challenge in developing digital libraries is that of preserving as much as possible of the knowledge and skills expert intermediaries demonstrate. Library and information science have much to contribute, and knowledge acquisition in this regard is of particular value. In addition, models, architectures, prototypes and tests regarding agents must be undertaken to see if such knowledge can be properly applied. Hence:

Another way to think about the future WWW collection of NSF-related information is as a digital library for science and engineering research and education. As such, it deserves serious study, and requires careful coordination and attention. There are enumerable important resources that will be included, and which will support user tasks related to education and research. Program managers have particular information needs(1): managing their own information and also accessing information about the scientific community to find reviewers, panelists, and related activities of other agencies, as well as to contact investigators and task force attendees. The resource discovery needs of investigators operate in reverse, since they must learn about NSF programs, program managers, previous awards, funded proposals, project reports, and a myriad of related documents. They may find it helpful to locate and communicate with investigators funded by NSF, or others who have attended NSF-sponsored workshops. Clearly, it is possible to describe many of the needs of program managers or investigators. Much harder is the task of determining the resource discovery needs of the rest of the nation in terms of NSF-related information. Hence:

With this analysis complete, NSF will have a roadmap to build, through efforts of those it employs or funds, a significant digital library. As this evolves, it will relate to larger and larger portions of the WWW. Thus, it is important for NSF's operations that it encourages the improvement of resource discovery methods on the WWW. It is also of benefit for other uses of the WWW for NSF to provide support. In particular, there is now little funded research dealing with the many scientific problems tied to resource discovery on the WWW. Hence:

• Methods of widely distributed search, and subsequent fusion of results;
  
• Problems of parallel search on the scale of the Web and adapting gracefully to the many types of failures that can occur on such a large and complex networked system;
  
• Techniques to find good starting points or to identify particularly promising collections where more focused searching can begin;
  
• Approaches to computer-assisted browsing, that helps users quickly narrow the scope of their investigation;
  
• Architectures for organizing information and processing elements, such as those using agents, or those working with separate layers (e.g., gatherers, brokers, harvesters, centroids);
  
• Models for searching in such contexts that are marked by extremes of heterogeneity, wide variety of partial structures, and diverse genre;
  
• Integrating the link database of the WWW with results of node analysis and indexing for improved search, navigation, visualization and browsing, allowing for wide variations of node in-degree and out-degree, and inconsistencies, as well as both typed and untyped links;
  
• Constructing limited and isolated distributed test environments, for benchmark comparison purposes, "protecting" the rest of the WWW from high-risk experimental studies, and to help with validation of the abovementioned research (especially parallel search and simulation studies); and
  
• Developing educational and training materials regarding use of advanced information retrieval and resource discovery methods and tools.
  

Fox E., ed., (1993) Sourcebook on Digital Libraries: Report for the National Science Foundation. TR-93-35, VPI&SU Computer Science Dept., Blacksburg, VA. Available by anonymous FTP from directory /pub/DigitalLibrary on fox.cs.vt.edu.

Footnotes

(1)

See position statement by J. Hestenes in Appendix D of the on-line version of this report.

Table of Contents Next Chapter