libSynk

What's New

About

libSynk is a compact, portable library for adding fast communication and synchronization to distributed applications. It runs on several platforms, including Windows, Mac and Unix/Linux.  It includes automatic routing software that seamlessly & transparently routes messages across Shared Memory, Myrinet, LAN (TCP) and WAN (TCP).

libSynk's simplified API makes it a convenient alternative to other standards-based messaging packages.  Its synchronization scalability has been demonstrated on configurations of over fifteen hundred processors.

Platforms

libSynk runs on many platforms, from PDAs to Supercomputers. The following is a partial list:

Scale	Tested Size	Platforms
Handhelds/PDAs	4	Windows CE (Pocket PC), Familiar Linux (iPAQ)
Desktops	Dozens	Windows XP/2K/NT/98/95 (Intel), Mac OS X, Linux
Clusters	up to 136 CPUs	Linux (Pentiums & IA32/IA64), Sun Solaris, SGI Irix
SMPs	up to 512 CPUs	Linux, Sun Solaris, SGI Irix, HP UX
Supercomputers	up to 1536 CPUs	HP UX, SGI Irix, HP Tru64, AIX 64

Back to top

Quick Start

For those impatient, like me:

Back to top

Features

Software Architecture

The following picture depicts libSynk's software modules and dependencies.

• TM = Time Management
• TM RED = Reductions-based TM
• TM NULL = Null message-based TM
• FM = "Fast" Messaging (terminology is a relic of earlier software)
• FM MYR = Myrinet-based module of FM
• FM TCP = TCP-based (LAN/WAN) module of FM
• FM SHM = Shared memory-based module of FM
• FM MPI = MPI-based module of FM
• RM = Asynchronous reductions-based Reductions Management
• RM BAR = RM-based scalable barrier.

Some of the modules provide two sets of interfaces: one interface is intended for applications that use the module, while the other interface is intended for extensibility of its implementation.

For example, FM provides a simple message-passing interface for applications, and another internal interface for adding new communication modules.  The Shared Memory, Myrinet and TCP modules are incorporated into this framework as extensions based on FM's internal interface.

Similarly, TM exports a simple interface for time-synchronized applications needing to compute lower bounds on timestamps (LBTS) of events.  Internally, it supports an interface to incorporate new LBTS algorithms.  The reductions-based algorithm and null message-based algorithm are two modules included with libSynk that conform to TM's internal interface.

Synergetic Time Management

Time synchronization is a well-studied problem in Parallel and Distributed Simulation literature.  Unfortunately, no single synchronization is universally optimal in practice.  Run-time performance of any synchronization depends on many factors, such as topology, look-ahead and timestamp distribution.  Recognizing this heterogeneity, libSynk supports a synergetic approach to synchronization.  Multiple synchronization algorithms are executed simultaneously.  Each algorithm is included as a TM sub-module, conforming to TM's internal extension interface.  libSynk includes a module with a reductions-based algorithm, and another module incorporating a variant of Null Message algorithm.  Further, the TM framework is designed to include an expanded set of synchronization algorithms, to be added over time.

Synchronization based on global reductions is useful to scale to large number of processors.  This approach is largely resilient to application characteristics such as event time distributions and topology of event exchanges, and thus provides uniformly stable performance across a range of applications.  This algorithm is included as a sub-module of libSynk.  We have found this algorithm to perform very satisfactorily over a large set of network simulation experiments.  However, for applications with specialized communication characteristics, such as localized event exchange, more optimized synchronization approaches are possible.

In order to deliver optimized performance for such specialized applications, a variant of the well-known Null Message algorithm (Chandy-Misra-Bryant) is included as another sub-module.  The Null Message algorithm is suited for localized communication applications, performing well during periods when the application is engaged in well-behaved local interactions.  The reductions-based algorithm acts as a shock-absorbing layer, kicking in at times when the Null Message algorithm assumptions are temporarily violated.  When look-ahead reduces to zero, Null message can live-lock (or deadlock); reductions easily continue to deliver advances until look-ahead increases.  When time increments are significantly greater than look-ahead, null messages suffer from "time creep"; reductions deliver the requisite "time leaps".  Large fan-out can degenerate communication to broadcast; reductions run faster, obviating null messages.  Directed Acyclic Graph (DAG) topology can needlessly constrain time advances; reductions kick in to deliver faster advances.

Thus, our approach of combined simultaneous execution of multiple synchronization algorithms can help achieve optimized performance in an application-independent manner.

Composite Timestamps

Timestamps are, by default, defined to be simple double variables (TM_Time in tm.h).  However, it is now possible to redefine timestamps to be any arbitrary C struct (by redefining TM_Time in tm.h inside #if COMPOSITETIME).  For applications demanding the efficiency of using simple doubles as timestamps, that option is still available (which, in fact, is the default).  However, those applications that need additional functionality can define the preprocessor constant COMPOSITETIME to 1, and place the specialized definitions of TM_Time and associated macros in tm.h.  One such example is included in tm.h, in which a "tie breaker" field is defined in addition to the traditional double timestamp, and TM_Time is defined as a struct.

MPI Support

MPI is now supported for communication (starting with release 03Feb05).  To use MPI, edit libsynk/Makefile to set -DMPI_AVAILABLE=1.  Also set CC to your MPI installation's C compiler wrapper (e.g., mpicc).  To run libSynk processes with MPI, simply use your MPI installation's parallel job spawning tool (e.g., mpirun) to spawn the processes.  Specify SPMD style of parallel execution (which is most likely the default mode in your installation, in which case you don't need to worry about choosing the mode).  As of 03Feb05 release, MPI communication cannot be mixed with other modes (such as TCP, Shared Memory and/or GM).  This might change in the future, to accommodate richer composition of communication methods.  There is indeed support already built into libSynk code (fm.c) to hook multiple MPI installations together via TCP, etc, but it is not exercised for now, for simplicity.  When MPI is used, libSynk will assume existence of only a single MPI installation, and FM_nodeid has one-to-one mapping with that process' MPI_COMM_WORLD rank.  Due to this one-to-one mapping with MPI node configuration, it is not necessary to set the NODEINFO environment variable when using MPI.

Handhelds

libSynk has been ported to compile and run on handheld platforms, with almost no modifications.  The communication code has been adapted to the Windows CE platform, and its memory footprint configured to fit in the constrained memory space typical for handhelds.

Specifically, libSynk has been tested on Compaq iPAQ 3850 PDAs running Pocket PC 2002 (ARM processor), and also on on emulated x86 processors running Pocket PC 2002.  libSynk compiles and runs unmodified using embedded software tools on these platforms.

Back to top

Run-time Performance

Run-time performance of libSynk has been benchmarked on a range of platforms.  The largest studied configuration included 1536 processors of the LeMieux supercomputer at the Pittsburgh Supercomputing Center.

The following chart shows the time to perform asynchronous reductions with increasing number of processors.  Note that the y-axis is shown in logarithmic scale.

The following chart shows the time to perform a time-management operation (TimeAdvanceRequest) in a simple RTI benchmark.

Back to top

Download

• Please accept the terms and disclaimer before downloading and using the software.
  Downloading implies acceptance.
• Download the most recent version from the following:
  • libsynk-current.zip [Windows]
  • libsynk-current.tar.Z [Unix, Linux, MacOSX]
  • Or browse the ftp directory for earlier versions.
  
  
• On Windows, unzip it to C:\ (resulting in a C:\libsynk directory).
  On Unix/Linux, untar it to your home directory (resulting in a $HOME/libsynk directory).
• The rest of the instructions here assume libSynk has been extracted to home or C:\. If you extract to a different directory, substitute the path in the instructions accordingly.

Back to top

Install

On Windows (assuming MS Visual Studio/C++ or equivalent):

cd C:\libsynk nmake	This creates synk.lib
cd C:\libsynk\fdkcompat nmake	This creates brti.lib

On Unix/Linux/MacOSX:

Optional: Edit ~/libsynk/Makefile Edit ~/libsynk/fdkcompat/Makefile	Customize per locale/platform. E.g., compiler settings (LDLIBS vs. LDFLAGS, etc.).
cd ~/libsynk make	This creates libsynk.a
cd ~/libsynk/fdkcompat make	This creates libbrti.a

If Myrinet/GM is available, libSynk can use it.  Define the preprocessor macro -DGM_AVAILABLE=1 in CFLAGS, and add the GM library (-lgm on Unix, gm.dll[?] on Windows) to the list of linked libraries.

Back to top

Run

All libSynk applications use the following distributed execution framework: The user chooses N machines named m₀ m₁ ... m_N-1.  A libSynk application process with FM ID i is mapped to machine m_i, where m_i is the hostname of machine on which the process i is run, i=[0..N-1].

NODEINFO Environment Variable

All libSynk applications require the user to specify an environment variable named NODEINFO for each process.  Based on the terminology defined above, the value of NODEINFO at machine i should be of the form N:m₀,m₁,...,m_N-1:i.

Example:
Suppose we pick two machines named gaia (gaia.cc.gatech.edu) and marquesas (marquesas.cc.gatech.edu).  We will use shortened machine names (gaia, marquesas) for simplicity here.  Use fully qualified domain names if your DNS settings require so.

• For the process on gaia.cc.gatech.edu, the NODEINFO should be set as:

  setenv NODEINFO "2:gaia,marquesas:0".

• And, for the process on marquesas.cc.gatech.edu, it should be set as:

  setenv NODEINFO "2:gaia,marquesas:1".

Note the colon followed by 0 for the process on gaia, to say it is federate 0, and colon one for marquesas to say it is the federate 1.

fmtest

A messaging test program is included as libsynk/fmtest.c.  On each machine m_i, run it as fmtest with the appropriate NODEINFO setting for each instance.

Example:
Based on the preceding two-machine example:

• On gaia.cc.gatech.edu, run

  setenv NODEINFO "2:gaia,marquesas:0"
  ./fmtest

• And, on marquesas.cc.gatech.edu, run

  setenv NODEINFO "2:gaia,marquesas:1"
  ./fmtest

rmtest

A program to test asynchronous distributed reductions is included in libsynk/rmtest.c.  Execution is similar to fmtest above, except replace fmtest with rmtest in the commands.

ping

A ping test program is included as libsynk/fdkcompat/brtiping.c.

Example:
Based on the preceding two-machine example:

• On gaia.cc.gatech.edu, run

  setenv NODEINFO "2:gaia,marquesas:0"
  ./brtiping

• And, on marquesas.cc.gatech.edu, run

  setenv NODEINFO "2:gaia,marquesas:1"
  ./brtiping

Optimize

For best performance, you can specify the network topology to libSynk using a network description file.  A sample file is included as libsynk/net.txt.  At runtime, libSynk looks for a file named net.txt in the current working directory of the process.  The contents of the file need to be exactly the same for all the processes (either by using a single specification file shared via a shared file system, or by manually making copies of a single specification file on to each machine).  libSynk runs an "all pairs shortest path" algorithm to find the best routes among the processors.

Illustration of mixture of different communication links among the processors.  The process labeled m_i routes messages into and out of its uni/multi processor machine, using the path with least latency between m_i and the message destination.  Currently, the link types supported are: Shared Memory (SHM), Myrinet/GM (MYR), Local Area Network (LAN) and Wide Area Network (WAN).  Any mixture of these links can be specified and instantiated, depending on your particular network configuration.  Note that only the leader/router process uses multiple links to route messages; other processes on the same machine only have shared memory links to their local leader process.

Shared Memory Multiprocessors

You can run more than one process per machine.  In other words, m₀..m_N-1 need not be distinct from each other -- the same machine name can appear more than once.  This is useful on machines that have more than one processor.  In such shared-memory (symmetric) multi-processor machines, libSynk automatically uses shared memory communication among processes on the same machine.  All off-machine communication is routed via a "machine leader" process, which is the process with the smallest ID assigned to the same machine.

Example:
If marquesas had two-processors, we could run two copies of ping on marquesas, and one on gaia.

• On gaia.cc.gatech.edu, run

  setenv NODEINFO "3:gaia,marquesas,marquesas:0"
  brtiping

• And, on marquesas.cc.gatech.edu, run

  setenv NODEINFO "3:gaia,marquesas,marquesas:1"
  brtiping

• In another window of marquesas.cc.gatech.edu, run

  setenv NODEINFO "3:gaia,marquesas,marquesas:2"
  brtiping

Note that messages between the two processes 1 and 2 (on marquesas) are routed via shared memory.  Messages from process 2 (marquesas) to process 0 (gaia) first go via shared memory to process 1 (marquesas leader), and then via TCP (or Myrinet if available) to process 0 (gaia).

Also, note that libSynk implicitly detects shared memory links and instantiates them.  Hence, shared memory links do not appear in the network configuration file (see Optimize above).

Firewalls

libSynk can communicate across firewalls using the port-forwarding features of secure shells.  I will soon add details on how to set up the port numbers for your application.  Meanwhile, contact me if you need further information.

Back to top

Library APIs

libSynk FM, TM and RM application programming interface is described in this document (extracted from the FDK user guide).

Back to top

Compatibility

libSynk is significantly simpler to install, execute and modify.  It is also portable across Unix and Windows platforms.

Existing FDK applications can use libSynk with tiny modifications.

• BRTI applications can be run almost unmodified on both Windows and Unix/Linux platforms.
• HLA RTI (formerly called the FDK DRTI) applications can also be run.  Currently, this RTI runs only on Linux.  Porting to Solaris/Irix/Unix and Windows is in progress.  HLA RTI is not included in the public libSynk release due to license/export restrictions.  It can, however, be released on a per-user basis.  Please send email to obtain.

Porting existing BRTI/DRTI applications:

• Remove all previous #included FDK/BRTI headers in the application (e.g., remove #include "tm-kit.h", etc.).  Instead, only include "brti.h".
• To compile, add -Ilibsynk/fdkcompat and -Ilibsynk to include paths.
• Add libsynk/fdkcompat/libbrti.a and libsynk/libsynk.a (in that order) to linked libraries.
• Similar modifications apply to DRTI applications.

Back to top

Runtime Configuration via Environment Variables

Environment variables that can be specified by the user to change various runtime parameter settings:

Variable	Module	Used in	Format	Notes
FM_NUMNODES	FM	fm.c	Integer	[1..MAXPE].  No default value.  Used to specify the total number of processes used in the application.  libSynk applications must set this variable.  However, RTI applications should use the NODEINFO variable instead.
FM_NODEID	FM	fm.c	Integer	[0..$FM_NUMNODES-1].  No default value.  Used to specify the index of this process.  libSynk applications must set this variable.  However, RTI applications should use the NODEINFO variable instead.
FM_NODENAME_i	FM	fm.c	String	Used to specify the hostname of process whose FM_NODEID is i.  libSynk applications must set this variable for all 0<=i<$FM_NUMNODES.  However, RTI applications should use the NODEINFO variable instead.
FM_NETFILE	FM	fm.c	String	Default "./net.txt".  Used to specify the name of the network configuration file.
FM_DEBUG	FM	fm.c	Integer	Set debug level of FM to given value.  Higher values generate more output.
FMGM_DEBUG	FM GM	fmgm.c	Integer	Set debug level of FM GM to given value.  Higher values generate more output.
FMSHM_DEBUG	FM SHM	fmshm.c	Integer	Set debug level of FM SHM to given value.  Higher values generate more output.
FMSHM_KEY	FM SHM	fmshm.c	Integer	Used to specify/override the key of shared memory segment created on a machine on which more than one process is mapped.  Default based on UID of the user on that machine.
FMSHM_BUFSPERPEQ	FM SHM	fmshm.c	Integer
FMSHM_TOTMEMORY	FM SHM	fmshm.c	Integer
FMSHM_BUFSPERPE	FM SHM	fmshm.c	Integer
FMTCP_DEBUG	FM TCP	fmtcp.c	Integer	Set debug level of FM TCP to given value.  Higher values generate more output.
HOME	FM TCP	fmtcp.c	String
FMTCP_SESSIONNAME	FM TCP	fmtcp.c	String
FMTCP_MASTERPORT	FM TCP	fmtcp.c	Integer
FMTCP_NOPORTFILE	FM TCP	fmtcp.c	0 or 1
FMTEST_DEBUG	FM TEST	fmtest.c	Integer	Set debug level of FMTEST to given value.  Higher values generate more output.
FMTEST_MAXTRANS	FM TEST	fmtest.c	Integer
RM_DEBUG	RM	rm.c	Integer	Set debug level of RM to given value.  Higher values generate more output.
RM_SCHEDULE	RM	rm.c	String
RM_NODEGROUPS	RM	rm.c	String
RMTEST_DEBUG	RM TEST	rmtest.c	Integer	Set debug level of RMTEST to given value.  Higher values generate more output.
RMTEST_MAXTRANS	RM TEST	rmtest.c	Integer
TM_DEBUG	TM	tm.c	Integer	Set debug level of TM to given value.  Higher values generate more output.
TM_LPROB	TM	tm.c	Double
TM_MSGDELAY	TM	tm.c	Double
TM_USEUDP	TM	tm.c	0 or 1
TM_UDPALWAYS	TM	tm.c	0 or 1
BRTI_HEAPSIZE	BRTI	fdkcompat/brti.c	Integer
BRTI_HEAPINCR	BRTI	fdkcompat/brti.c	Integer
RTIC_OPTNER	RTI CORE	fdkcompat/rticore.c	0 or 1
RTIC_OPTMINLBTS	RTI CORE	fdkcompat/rticore.c	0 or 1
RTIC_DEBUG	RTI CORE	fdkcompat/rticore.c	Integer	Set debug level of RTI CORE to given value.  Higher values generate more output.
RTICHM_DEBUG	RTI CORE	fdkcompat/rticore.c	Integer	Set debug level of RTI CORE's handle manager to given value.  Higher values generate more output.
NODEINFO	RTI CORE	fdkcompat/rticore.c	String

 

Back to top

Frequently Asked Questions

Questions/answers are added over time with feedback.

1. Q: When I run brtiping, execution fails saying "NODEINFO not set...".
   A: You need to set the environment variable named NODEINFO for each process.  See Run above on how to format the value for the NODEINFO environment variable.

   ---

2. Q: On Solaris, my application build failed at link time -- socket, hostname, etc. not found by linker.
   A: Add -lnsl -lsocket to compiler command line.

   ---

3. Q: On windows, nmake is not available on command line.
   A: Install MS Visual Studio.  Enable command line access during installation, or run vcvars32.bat (see MS Visual C++ help).

   ---

4. Q: Where is the HLA RTI? I can't find it in the libSynk release.
   A: The HLA RTI (formerly called the DRTI) that works with libSynk is not included in the public release, due to export/license restrictions from our former sponsors.  It is however available for release separately upon request, on an individual basis.  See Compatibility above.

   ---

5. Q: I am having trouble getting libSynk applications to run over separate machine that DO NOT share a common file system. Basically, the TCP port file is not updated correctly.
   A: Let's assume two machines named A & B, and you are starting federate/process 0 on machine A, federate 1 on machine B. When using TCP, the federate 0 searches for a vacant port number and binds a listening socket to it. The rest of the processes synchronize by first contacting the federate 0, and so all federates need to know the port number of federate 0. This is achieved, by default, using a shared file in home directory.
   If A & B don't share home directories, there are two methods to make it work. In the following, csh or tcsh shell is assumed. For ksh/sh/bash, replace "setenv X val" with "export X=val".

   • ---

     METHOD 1: By default, federate 0 creates a file whose content is a single integer representing its listening port number. Once the federate 0 comes up and creates this file, it is sufficient to copy this file to all other machines, and then start the rest of the processes.
     • Start on the first machine:
       % (setenv NODEINFO "2:A,B:0"; ./brtiping) &
       Note the colon 0 to specify that this federate 0. Ampersand puts it in background.
     • This will create a file named "fmtcp-portfile" in your HOME directory. Copy that file from machine A to your HOME directory on machine B.
     • Then start the process on machine B:
       % (setenv NODEINFO "2:A,B:1"; ./brtiping) &
       Note the colon 1 to specify that this is federate 1.
     They should now see each other and synchronize. For any subsequent runs, the port number should remain the same, and so the file content will remain the same. Hence, you can skip the file copying for any subsequent runs. Note that, using this method, you will need to start federate 0 before any other federate is started.

   • ---

     METHOD 2: You can force federate 0 to use a specific port number, rather than choosing a vacant port number dynamically. This way, there is no need for conveying a dynamic port number via portfile. The statically assigned port number can be specified to all the federates using a simple environment variable FMTCP_MASTERPORT.
     • Start on first machine:
       % (setenv NODEINFO "2:A,B:0"; setenv FMTCP_MASTERPORT 12345; ./brtiping) &
       
     • Start on the second machine:
       % (setenv NODEINFO "2:A,B:1"; setenv FMTCP_MASTERPORT 12345; ./brtiping) &
       
     This will force federate 0 (master) to bind to the specified port. The federates should now see each other and synchronize. This method is useful to avoid file copying. Also, using this method, the federates can be started in any order (it is NOT necessary to start the federate 0 first). There is a potential drawback, however. If the specified port on the machine of federate 0 is already in use by another process, it will fail to bind, and it will exit abnormally. However, this is usually easy to avoid by choosing a sufficiently random port number, and I have seen it to seldom fail.

   ---

Back to top

Credits

• libSynk is written by Kalyan Perumalla.
• The fdkcompat backward compatibility layer is based on portions of FDK written by (in alpha order): Steve Ferenci, Richard Fujimoto, Thom McLean, Kalyan Perumalla, Ivan Tacic and others.

Back to top

Usage Terms and Disclaimer

libSynk and associated software is copyright © 2003 by the authors and the Georgia Tech Research Corporation, Atlanta, Georgia. All Rights Reserved.  libSynk and associated software can be used only for non-commercial purposes (research and education). Commercial use and/or redistribution requires licensing agreements with the author(s) and the Georgia Tech Research Corporation.  Download and use implies agreement to the terms and conditions.

Permission to use, copy and modify this software and its documentation for any research purpose and without fee is hereby granted, provided that the above copyright notice appears in all copies and that both the copyright notice and this permission notice appear in supporting documentation. The use or inclusion of this software or its documentation in any commercial product or distribution of this software to any other party without specific, written prior permission is prohibited.

THE SOFTWARE IS PROVIDED AS IS AND GEORGIA TECH RESEARCH CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL GEORGIA TECH RESEARCH CORPORATION OR THE AUTHORS BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTUOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Back to top