Portable Binary Input/Output

Current version: 3.3

What is PBIO?

Storing and transmitting data in binary form is often desirable both to conserve I/O bandwidth and to reduce storage and processing requirements. However, transmission of binary data between machines in heterogeneous environments is problematic. When binary formats are used for long-term data storage, similar problems are encountered with data portability.

PBIO is a general approach to dealing with binary data in storage and transmission. Essentially, PBIO is a data meta-representation. Users register the structure of the data that they wish to transmit/store or receive/read and PBIO transparently masks the differences. In particular, PBIO handles differences in the sizes, locations and even basic types of the fields in the records to be exchanged. Field matching is by field names (provided by the application). Records are C-style structures consisting of fields which can be any of the usual atomic data types or NULL-terminated strings. A previously registered structure may also serve as the basic type of a field, allowing the creation of complex nested record types. PBIO also supports fields which are statically sized one or two dimensional arrays or dynamically sized one dimensional arrays of any of the above datatypes. Record meta-information is transmitted once, when record formats are registered. Thereafter, transmission occurs in the writer's native format and the PBIO library on the receiver transparently handles discrepancies between the writer's format and the format required by the reader. In the case of transfers between homogeneous machines, the only additional overhead imposed by PBIO is the transmission of a 4-byte format ID. Between heterogeneous machines, the extent of the overhead depends upon the degree to which the record formats and atomic type representations differ between the two machines.

PBIO is described in the following publications:

  • Efficient Wire Formats for High Performance Computing, Fabian Bustamente, Greg Eisenhauer, Karsten Schwan and Patrick Widener, SC'2000. postscript, PDF
  • Fast Heterogenous Binary Data Interchange for Event-based Monitoring, Beth Plale, Greg Eisenhauer, Lynn K. Daley, Patrick Widener and Karsten Schwan, Proceedings of the International Conference on Parallel and Distributed Computing Systems (PDCS2000), August 8-10 2000. postscript, PDF
  • Fast Heterogenous Binary Data Interchange, Greg Eisenhauer and Lynn K. Daley, Proceedings of the 9th Heterogeneous Computing Workshop (HCW 2000), pp 90-101. postscript, PDF
  • Morphable Messaging: Efficient Support for Evolution in Distributed Applications, Sandip Agarwala, Greg Eisenhauer and Karsten Schwan, submitted to Challenges of Large Applications in Distributed Environments (CLADE), June 2004. PostScript [220K] | PDF Format [129K]
  • What platforms does PBIO support?

    PBIO has been compiled and tested amongst heterogenous combinations of the following platforms:

    PBIO is distributed with a configure script and should adapt itself to most rational Unix-based platforms.

    Where can I get source and documentation?

    PBIO is available under the new BSD license and can be downloaded in gzip or compress formats directly. A paper describing PBIO and containing basic usage information is available (pdf). An earlier version of this paper appeared as College of Computing Tech Report GIT-CC-94-45, which has the following bibtex entry:

      author =       "Greg Eisenhauer",
      title =        "Portable Self-Describing Binary Data Streams",
      institution =  "College of Computing, Georgia Institute of Technology",
      year =         "1994",
      number =       "GIT-CC-94-45",
      note =         "{\it (anon. ftp from ftp.cc.gatech.edu)}",

    Older versions of PBIO source are available below:

    Known limitations

    Recent PBIO news.

    Debugging PBIO programs.

    Some general questions about debugging PBIO programs are answered in another document.

    This page is maintained by PBIO author Greg Eisenhauer
    Last Modified Sep 27, 2010.