ABSTRACT

Native Data Representation: An Efficient Wire Format for High Performance Computing

GIT-CC-01-18

Greg Eisenhauer, Fabián E. Bustamante and Karsten Schwan

College of Computing
Georgia Institute of Technology
Atlanta, GA 30332, USA
{fabianb, eisen, schwan, pmw}@cc.gatech.edu

Abstract

Flexible and high-performance data exchange is becoming increasingly important. This trend is due in part to the growing interest among high-performance researchers in tool- and component-based approaches to software development. In trying to reap the well-known benefits of these approaches, the question of what communications infrastructure should be used to link the various application components arises. Traditional HPC-style communication libraries such as MPI offer good performance, but are not intended for loosely-coupled systems. Object- and metadata-based approaches like XML offer the needed plug-and-play flexibility, but with significantly lower performance. We observe that the flexibility and baseline performance of data exchange systems are strongly determined by their `wire formats', or by how they represent data for transmission in the heterogeneous environments. Upon examining the performance implications of using a number of different wire formats, we propose an alternative approach to flexible high-performance data exchange, Native Data Representation, and evaluate its current implementation in the Portable Binary I/O library.

For the full paper PDF PS

Fabián E. Bustamante
Last modified: Sat Aug 4 17:00:38 EDT 2001