Overview

Background 

Federated simulations in which different simulators interoperate with each other in executing a single simulation are increasingly becoming important in many areas, such as battle field simulation and distributed multi-user games. The significance of interoperability of different simulators is highlighted by the High Level Architecture (HLA) initiative of the US Department of Defense (see the special issue of the journal SIMULATION, Volume 71, Number 6, December 1998, by SCS). The FDK software is designed to facilitate in building efficient run-time infrastructures (RTIs) that are necessary in such federated simulations. The FDK is currently being used by Georgia Tech researchers in various federated simulation projects, such as in building an efficient and extensible RTI for the HLA. The FDK software is being released to university and industry researchers as part of the University Outreach program spronsored by the Defense Modeling and Simulation Organization (DMSO) . This is aimed at promoting research in designing and building new, faster and better RTIs. 

Obtaining the Software

To obtain a copy of the FDK software, send email to fdk@cc.gatech.edu. When the software is extracted, it expands to two directories: FEDSIM and JANE. The FEDSIM directory contains the RTIKIT and RTI software, and the JANE directory contains the graphical interface software.

Installation

1. Installing FEDSIM: RTIKIT library, RTIs and RTI example applications.
2. Installing JANE: Client/Server-based graphical interface to executing RTI applications.

Installing FEDSIM

FEDSIM software consists of the RTI Kit and two RTI implementations. The RTI Kit is designed to work with a variety of communication architectures based on the Fast Messages (FM) interface. Currently, they include: Myrinet, TCP, and Shared-memory. The Myrinet FM interface requires the Myrinet switching hardware. The TCP FM interface is based on Unix BSD stream socket interface. A shared-memory FM interface is implemented using shared-memory communication on shared-memory multiprocessors.

FM codes:
Currently, the FM interfaces have been tested as follows: The Myrinet version has been tested on 8 Sun UltraSPARC nodes connected by a single Myrinet switch. The TCP version has been tested on a configuration of Sun UltraSPARC nodes, and on a configuration of SGI Indy/O2 nodes, and on an SGI Origin 2000, and on a configuration of four dual-processor Intel i86pc nodes. The Shared-memory version has been tested on an SGI Origin 2000 which has 16 processors.

Architecture codes:
The entire FEDSIM software is designed so that a single copy of the source code directory can be used for multiple architectures simultaneously. Currently, SGI uniprocessors, SGI multiprocessors, Sun Sparcs and Intel i86pcs are supported, with architecture codes of SGI, SGIMP, SPARC, INTELSOL and LINUX respectively for SGI 32-bit processors running Irix 6.x, SGI 64-bit multiprocessors running Irix 6.x, Sun UltraSPARCS running Solaris 2.x, Intel PCs running Solaris 2.x, and Intel PCs running Linux 2.0.x. We shall use the word ARCH to denote the architecture code of the node on which FEDSIM is being installed. Thus, henceforth, ARCH must be interpreted as SPARC if Sun Sparcs are being used, SGIMP if SGI multiprocessor is being used, and so on.

After extracting the source code for the software, do the following:

Choose FM type: The default FMTYPE is TCP. If you need to change it, you need to choose from one of the supported communication interfaces. This can be done by setting the variable FMTYPE to TCP if TCP is desired, SHM if shared memory is desired, and MYR if Myrinet FM is desired.

This can be done using any one of the following two ways:

1. Specify FMTYPE when you execute make. For example, the following can be executed to build the FDK using Myrinet FM:
2. (cd FDK/FEDSIM; make FMTYPE=MYR)
3. Set the environment variable FMTYPE. For example, using the C-shell (csh), the following can be executed to build the FDK using TCP FM:
4. (setenv FMTYPE TCP; cd FDK/FEDSIM; make)
   The advantage to using this method is that it avoids the need to set the FMTYPE variable every time make is invoked. The FMTYPE variable can be set once in your shell profile file (for example, .cshrc).

Compile sources: Change directory to FEDSIM, and type make to initiate the software build. This will build both the RTIKIT as well as the RTIs and their example applications. The architecture type is automatically sensed (i.e., the ARCH code gets assigned automatically), and libraries and executables are stored in LIB/$ARCH and OBJ/$ARCH/$FMTYPE directories respectively under each software component.
Upon installation, the following will be accessible: 

Multiple architectures and FM types: If you need to use more than one architecture and/or more than one FM implementation, you can make a separate installation for each type of architecture and FM, or you can share the same installation by compiling the same source with different FMTYPE. It is not necessary to make separate installation for different node architectures or different FM types -- the same source copy can be shared by multiple architectures as well as multiple FM types.

For example, to compile the same source of the FDK with both TCP and SHM FM (on, say, an SGIMP machine), the following can be executed:
(cd FDK/FEDSIM; make FMTYPE=TCP; make FMTYPE=SHM)
The same source, can also be compiled on another architecture (on, say, a SPARC machine with TCP FM), by executing the following:
(cd FDK/FEDSIM; make FMTYPE=TCP)
Note that the architecture is not specified on the command line since it is automatically sensed by the makefiles.

Installing JANE

The JANE system consists of server and client components.

• Server: Pre-built server executables are included in the release, for different architectures, under FDK/JANE/SERVER.
• Client: A Java-based client is included in FEDSIM/JANE/CLIENT. It is in the form of pre-built Java class library jane-classes.jar in the directory JANECLIENT. Two scripts are also included in that directory for running the client: a C-shell script janeclient.csh for UNIX and a batch file janeclient.bat for Windows.
  1. The client needs Java runtime environment as well as Java Foundation Classes (JFC/Swing). It requires either JDK 1.1 with JFC/Swing 1.1 (or higher); alternatively, it also runs with JDK 1.2 (in which JFC/Swing are already included). Both JDK and JFC/Swing may be obtained from the Java/Sun site.
  2. Edit the scripts janeclient.csh and/or janeclient.bat to set the JDKHOME, SWINGHOME and JANEHOME appropriately. The variable SWINGHOME is not important if you are using JDK1.2. JANEHOME must point to the directory containing jane-classes.jar. It is desirable for JANEHOME to be an absolute path, so that the scripts can be invoked by anyone from any directory.

Building Simulations

1. Makefiles: Copying and customizing Makefiles.
2. Compilation: Compiling the application.

Makefiles

Create your application directory, called, say, MYAPP. It can be created at any location, and it need not be created under the FDK installation directory. Place all your source files in that directory. You have a choice of using the supplied Makefiles, or creating your own Makefiles.

1. Using the supplied Makefiles.
2. Using your own Makefiles.

1. Copy the files Makefile and makefile.aimk from the FEDSIM/RTIS/DRTI/APP/PING directory into your MYAPP directory (or from FEDSIM/RTIS/BRTI/APP/PING if you are using BRTI).
2. Edit the Makefile to correctly set the variable FEDSIMINSTALLDIR to point to your installed FEDSIM directory. It should point to the directory that contains the RTIKIT and RTIS directories.
3. Edit the makefile.aimk to suit to your application source dependencies. Note that it is set up to generate all the intermediate object files as well as the final application executable in the subdirectory OBJ/$ARCH/$FMTYPE under your application directory, where ARCH is your architecture code, and FMTYPE is your FM code.

Alternatively, you can use the makefile.aimk methodology that can help to automate and ease the build process. This method is illustrated in the PING application of DRTI (FEDSIM/RTIS/DRTI/APP/PING). The advantages of this approach is that the same copy of the source code can be used with little modifications to compile on different node architectures and using different FM types. The object files and executables can in fact co-exist independently. In addition, the required configuration information such as include paths and libraries are automatically configured for the chosen architectures and FM types.

1. If you desire to use the makefile.aimk methodology, do the following:
1. If you already have a Makefile, then simply rename it to makefile.aimk. Otherwise, copy the makefile.aimk from the directory FEDSIM/RTIS/DRTI/APP/PING into your application directory.
2. Copy the Makefile from FEDSIM/RTIS/DRTI/APP/PING to your application directory.
3. In the makefile.aimk, you can assume that the following variables are automatically assigned the correct values, and hence these variables can be used in your make rules. These variables take on the correct set of values based on the architecture and FM types chosen in the RTIKIT/DRTI.
• ARCH is set to the correct architecture code.
• FMTYPE is set to the correct FM type code.
• DRTI_CC is set to the C++ compiler that was used to compile the DRTI.
• DRTI_DEFINES is set to the C/C++ compiler macro definitions that are required to use the DRTI and the RTIKIT.
• DRTI_INCLS is set to the include directories that must be searched for DRTI and RTIKIT header files.
• DRTI_CFLAGS is set to the set of compiler flags that were used to compile the DRTI and the RTIKIT.
• DRTI_LIBS is set to the correct set of libraries that must be linked to a DRTI application.
4. Customize your makefile.aimk to suit your needs, making use of the automatically pre-defined convenience variables. Again, you can use the DRTI's PING application makefiles as illustration.
5. When you type "make" to build your application, first the Makefile is invoked, which in turn invokes the aimk script included with the RTIKIT (FEDSIM/MISC/aimk). This script collects the appropriate configuration specifications that are specified under the CONF directories of the RTIKIT and DRTI modules, and invokes the makefile.aimk of your application. Thus, by the time the rules of your makefile.aimk get invoked, the correct set of values get assigned to the variables described above.


2. If you do not want to use makefile.aimk, you can use the following steps to create your own build procedures.
1. Set the include path of your compiler to find the RTIKIT header files in FEDSIM/RTIKIT/INCLUDE, and the DRTI header files in FEDSIM/RTIS/DRTI/SRC.
2. Link your application to the RTIKIT library FEDSIM/RTIKIT/LIB/libRTIKIT-$ARCH-$FMYPE.a and to the DRTI library FEDSIM/RTIS/LIB/libDRTI-$ARCH-$FMYPE.a
3. Define compiler flags and C/C++ macro definitions as required for your particular choice of ARCH and FMTYPE pair. For this, you can observe the command-lines executed for the DRTI's PING application by (re)compiling that application, and mimic those options (you can execute "cd FEDSIM/RTIS/DRTI/APP/PING; make clean; make" to recompile). For example, on ARCH=SPARC and FMTYPE=TCP, additional libraries -lnsl and -lsocket are required.

Compilation

Type make in your application directory to compile your application. If you need to specify an FMTYPE other then the default TCP FM, you can do so by invoking make FMTYPE=MYR or make FMTYPE=SHM, for Myrinet and shared-memory respectively. Alternatively, you can specify FMTYPE as an environment variable once, and invoke make without the FMTYPE specification on its command line.

If you are using the supplied Makefiles, the final executable shall be placed in OBJ/$ARCH/$FMTYPE directory.

When using the Myrinet FM, you may need additional files, such as fmconfig and lcp32 (from the FEDSIM/RTIS/BRTI/APP/PING directory) in your MYAPP directory.

Executing Simulations

The federations can be executed in the following ways:

1. Manual execution by physically logging into and executing the federate on each node in the federation.
2. Automated execution, from a single regular text window/console.
3. Execution using the Jane graphical interface.

Executing Federates Manually

Federates can be started individually on each node by the user by physically logging into each node and initiating them. This is the ground-zero way of starting the federation. More complex and automated methods can be constructed which are based on this simple mechanism (say, by building your own scripts to automate this process).

The following are the steps in executing federations manually.

1. Log into each node in the federation, and go to the directory in which the federate must be executed, and run the federate after setting up the following information.
2. Each federate requires an environment variable called NODEINFO whose value describes the federation configuration. The value should be of the form npe:nodename,...,nodename:index where npe is the total number of federates in the federation, nodename,...,nodename is the list of nodes on which the federates are executing, and index is the index of this federate into this node list (0..npe-1). Note that the order of the node list must be specified to be exactly the same for all federates. Note also that the node corresponding to the index must be the same as the node on which the indexth federate is executed.
3. Each federate requires an environment variable called SESSIONNAME whose value is unique across multiple simulation runs. A simple way of assigning the session name to make it unique across runs is to use the process ID of one of the federates (say, the first federate).

   In the current TCP FM implementation, the federates exchange certain information (port numbers) using a file named after the session name in the home directory of the user. If the session name is not unique across executions, old information may be exchanged in later sessions, leading to undesirable behavior.

4. To avoid cluttering up user's shell profiles with environment variables, the environment variables required by the federates can be set on the same command line as the federate. For example, using the shells csh or tcsh, the tm_ping example can be executed on the node hawaii as:

   hawaii% (setenv NODEINFO "3:maui,hawaii,lanai:1"; setenv SESSIONNAME 12345; OBJ/SPARC/TCP/tm_ping)
   

   One way of automatically generating a unique session name is to use the process ID of the sub-shell of the first federate. This can be done, for example, using the $$ variable of the C-shell.

   Note: In the current version of the TCP FM, the federates create and use a file named after the unique session name (in the home directory) to exchange information such as TCP port numbers. This file is automatically removed by the first federate after the federation is initialized. In case the first federate fails before RTI is fully intialized successfully, then the file may be left behind in the home directory, which must be removed manually by the user. If that file is left undeleted, then federates in later executions may receive the old information resulting in undesirable behavior.

Automated Federation Execution

The FDK includes a spawner executable, FEDSIM/RTIKIT/BIN/spawner-$ARCH, that helps in automatically spawning the federation from a single location.

Select the node on which the spawner is to be executed. It should be possible to rsh from this node to other nodes in the federation, since the spawner uses rsh to spawn the federation on the other nodes. (You can use a .rhosts file in your home directory to facilitate this.) Execute the spawner on this node. The spawner that is thus initiated first is called the master spawner. The master spawner actually spawns copies of itself on other nodes in the federation. These remote copies are called the slave spawners. Each spawner is responsible for setting up the environment for its local federate based on the federation information. After setting up the environment, each spawner executes the local federate on its node. That is, the master spawner executes the local federate on the master's node, and each slave executes the local federate on its node. The master spawner waits until all the federates exit, whereas each slave spawner exec's its local federate (and hence ceases to exist once the federate starts running).

The (master) spawner takes several arguments that specify the federation execution information.

Example: See the sample script FEDSIM/RTIS/DRTI/APP/PING/run for an illustration of how to use the spawner.

Depending on the FM type selected to build your application, arguments are either necessary or optional. The format of the arguments to the master spawner is as follows:

• -npe: must be followed by the number of nodes used in the simulation. This includes the node on which the master spawner is executing. This option must be specified before the -nodes option.
• -endtime: must be followed by the total simulated time until which the federates must execute. This can be a (positive) floating point number.
• -nodes: must be followed by the names of all the nodes used in the federation.
• • The number of names must be exactly equal to the number of nodes specified above.
  • The name of the first node must be the same as the name of the node on which the master spawner is initiated.
  • When TCP FM is used, there is currently a limitation: It is assumed that all the nodes in the federation share the same home directory on a network file system, such as using NFS.
  • 
    XXX Reminder to author: remove this note when this constraint is solved by other means.
  • Each node name can be optionally followed by node-specific arguments, as described next.
  • • -spawner slave-spawner:
    • 
      to specify the absolute path name of the spawner executable file on this node. This allows different slave spawners to be specified on different nodes.
    • -cd working directory:
    • 
      to specify the absolute path name of the directory in which the simulation must execute. This allows different federates to execute in diferent directories.
    • -exec executable
    • 
      to specify the absolute path of the federate executable on this node. Different nodes can use different federate executables. The federate executable must be specified at least for the first node. If this option is not specified for any other node, then the same executable name as for the first federate is used for that node also.
    • -startargs arguments -endargs
    • 
      to specify that arguments must be supplied to the federate as its command-line arguments. Not specifying this option is the same as specifying zero arguments to the federate.
  • The node arguments must appear in the above specified order. However, omissions of any/all of the node options are allowed.
  • If the federate is built using the shared-memory FM interface, the -nodes specification is optional (all the node names will default to the same name as the server node name). For federates using TCP or Myrinet FM, the -nodes specification is mandatory.
  • If the federates are built using TCP or Shared-memory FM, then more than one federate can be run on the same node. Hence, the same node can appear more than once in the node name list. For federates using Myrinet FM, however, at most one federate can be run on any given node because Myrinet software requires exclusive access to the Myrinet switch on each node.
• -setenv: must be followed by a name=value assignment. Note that there should be no space in the "name=value" assignment specification. This will set the environment variable name to value in every federate.
• 
  This is useful to specify the environment variables such as FMLCPFILE for federates using Myrinet FM. This option can be used more than once, to specify several environment variable definitions.

Federate Execution Using Jane

The following are the steps in executing federations using the Jane system.

1. Start the JANE Server.
2. Start the JANE Client.
3. Run your Simulation.
4. Collaborate with Multiple Clients.

Start the JANE Server. First the JANE server must be started on any one of the nodes which will be used in the simulation/federation. This node should be such that processes can be spawned from this node using rsh to the other nodes used in the federation. Once started, the server can be left running, and need not be restarted for each client.

1. Select the node on which the server is to be started. It should be possible to rsh from this node to other nodes in the federation. The first federate is spawned on this node, and all other federates are spawned via rsh on the other nodes.
2. Execute the server on the previously selected node as follows:

3. JANE/SERVER/janeserver-$ARCH |∓mp;mp; tee /tmp/yourname.log ∓mp;mp;
   

   where, ARCH is the architecture code of the node, and yourname is replaced by your name (to avoid clash with other users). The output from the server will be captured in yourname.log, so that you can refer to it when necessary.
4. The server prints out the port number on which it will accept connections from clients. This is the port number that will need to be specified to the client when connecting to the server. The server attempts to make the port number equal to (or close to) the user ID of your account on that node. If you ever forget the port number, it can be found in the first few lines of the log file into which you are capturing the server output.
5. When you want to kill the server, use the kill command on the process ID of the server.

Start the JANE Client. The JANE client can be started on any node on which JAVA is installed, and from which the server can be reached across a network.

1. On UNIX nodes, run the JANE/CLIENT/janeclient.csh C-shell script. On Windows nodes, run the JANE\CLIENT\janeclient.bat batch file. In case you run into problems executing the scripts, you may want to briefly view the contents of the script files to understand the problem.
2. The client displays a connection dialog. Specify the node on which the server has been started, along with its port number. Specify your name, which will be used to identify you during collaborative executions. This will connect you to the server.

Run your Simulation.

1. Spawning the Federation. Once the client is successfully connected to the server, you can start your simulation using the client's graphical interface.
   In the Simulation-Start dialog, specify the federation information. On-line help is available in that dialog for information on how to specify the federation execution information. The information is essentially a graphical interface to that of the spawner as described in the Automated Federation Execution.
   
   
2. Federation Input/Output. Once the simulation is successfully started, the output from the federates is displayed by the client in separate tabbed text windows. Standard input can be entered to each individual federate by selecting its window, and entering the input and selecting the Send button. Thus, each federate can invoke input functions, such as getchar(), to receive individual user input.
   It is important, however, to not invoke input functions in the federate before the RTI is initialized..

Collaborate with Multiple Clients The JANE client/server system permits multiple clients to collaborate in controlling a federation.

1. One client connects to the server using its original port number. Then that user enables collaboration by using the corresponding menu option in the client window. The server responds with a new collaboration port number to which other clients can now connect. This collaboration port number is different from the original port number to which the first client connected. When other clients connect to this collaboration port (instead of to the original port), then all the collaborating clients receive identical displays of the simulation activity. The clients can communicate with each other using the chat area in the client window.
2. 
   Note: There is no master-slave distinction among the collaborating clients. All the active clients have identical capabilities.
3. If a client joins for collaboration after a simulation has been started, then that new client may not get the most accurate display of the activity. It is best to let all the clients join for collaboration before starting simulations. However, it is permissible for any client to exit from collaboration at any time, even during the execution of a simulation.
4. Once the collaboration is enabled, and all the interested clients have joined, further joins can be disabled by closing the collaboration port via the corresponding menu option in any of the joined clients.

Frequently Asked Questions