NAME
cthread_monitoring_intro - introduction to monitoring using
Falcon
LIBRARY
C Threads Library (libcthreads.a)
SYNOPSIS
#include <cthread.h>
DESCRIPTION
Instrumenting a target application and its run-time system
is the first step toward application steering. Falcon is a
set of tools integrated with cthreads that collectively sup-
ports instrumentation and on-line monitoring of cthreads-
based parallel and distributed applications. This man page
describes how to use the specification language and instru-
mentation tool, two major components of Falcon. It does not
discuss the high level view language; for information on
this see [Ph.D. Thesis, W. Gu, 1995]. For information on
the workings of Falcon see [GIT-CC-94-21]. In Falcon
software instrumentation points are called sensors. A sen-
sor generates timestamped event records that may be used
immediately for on-line monitoring or stored for postmortem
analysis. Event records are stored in trace buffers from
which they are removed by local monitors.
In order to control monitoring overheads, sensors can be
controlled dynamically in two ways. First, sensors may be
turned off if events captured by those sensors are not
needed. Second, sampling and tracing rates can be adjusted
depending on monitoring load and tolerance of inaccuracies
in monitored information. As an example of the latter, a
tracing sensor that monitors a frequently accessed mutex
lock can produce an event record once every five mutex lock
accesses, thereby improving monitoring perturbation at the
cost of reducing trace accuracy.
Writing a Sensor Specification
Falcon's sensor specification language is used to define
application-specific sensors for capturing program and per-
formance behavior. The file in which the sensor specifica-
tion resides must be named with a .s extension, e.g.,
my_app.s. A sample sensor specification is shown below:
sensor work_load {
attributes {
int process_num;
int double work_load;
}
};
The specification describes the structure of the application
data to be contained in the trace record generated by this
sensor. The declaration when compiled by the sensor com-
piler generates something like the following sensor subrou-
tine (though less readable):
int
user_sensor_work_load(int process_num, double work_load)
{
if (sensor_switch_flag(SENSOR_NUMBER_WORK_LOAD) == ON)
{
sensor_type_work_load data;
data.type = SENSOR_NUMBER_WORK_LOAD;
data.perturbation = 0;
data.timestamp = cthread_timestamp();
data.thread = cthread_self();
data.process_num = process_num;
data.work_load = work_load;
while (write_buffer(get_buffer(cthread_self()),
&data,
sizeof(sensor_type_work_load)) ==
FAILED) {
data.perturbation = cthread_timestamp() -
data.timestamp;
}
}
}
Note that there are four implicit fields for any event
record describing the event's sensor type, timestamp, thread
id, and perturbation. The body of this subroutine generates
entries for an event data structure, then writes that struc-
ture into a trace buffer. A local monitor later retrieves
this structure from the buffer. Each sensor's code body is
also surrounded by an if statement, so it can be enabled
or disabled during program execution.
Compiling a Sensor Specification
To compile a sensor specification named my_app.s type:
sensor_spec/sensor my_app.s
The sensor compiler generates a number of files and places
them in your current working directory:
my_app.eventformat.c
Contains event format definitions needed by PBIO.
my_app.sensorcode.c
Actual sensors converted to C functions. When the
application is executing, these functions generate
events that are forwarded to the hubserver by the local
monitor threads.
my_app.sensorextern.h
Contains extern declarations for the sensor functions
defined in my_app.sensorcode.c. You must modify your
code to include this .h file.
my_app.sensors.h
Assigns unique identifier to, and contains data struc-
ture definitions for, each user specified sensor.
Automatically included in both my_app.eventformat.c and
my_app.sensorcode.c.
my_app.sf
Intermediate representation of sensors (not currently
used).
Instrumenting Your Application
Once the sensors have been generated, instrument your
application by inserting calls to the sensors (C functions)
defined in my_app.sensorcode.c.
Compiling and Linking Your Application
Compile my_app.eventformat.c and my_app.sensorcode.c using
a standard C compiler. Link these object files with your
application program. Note: when linking, ensure
my_app.eventformat.o precedes the ithreads library,
libcthreads.a, otherwise the specified sensors will not be
recognized by the Falcon's run-time monitoring system and no
events will be generated.
Start the Application
The example below reflects a typical on-line monitoring
scenario. The application connects to a hubserver via an
internet domain socket. The monitoring client, the one
receiving the event stream, connects also to the hubserver.
Below are three windows, depicting the startup of the appli-
cation my_app, a single hubserver, and the monitoring
client. The hubserver must be started first followed by the
client with the application beginning last.
WINDOW 1: look at usage options, then start the datahub
forge% hubserver -u
usage: hubserver { -isock <hostname> <port number> } {
-usock <socket name> }
{ -wfile <file name> } { -rfile <file name> } {-
debug}
forge% hubserver
Data Hub at INET socket connection <forge.cc.gatech.edu,
1057>
Data Hub at UNIX socket connection </tmp/datahub_8dc>
WINDOW 2: start the client, connecting it to the hubserver
on machine "forge" via an internet domain socket.
santanni% monitoring_client -isock_in forge 1057
WINDOW 3: start the sample application. The options given
are ithread options, available by default to ithread appli-
cations. To see a full list of cthread options, pull up the
chaos man page "cthread_parse_args". The first option con-
nects the application to the hubserver via an internet
domain socket. The second option specifies default monitor-
ing so events are generated not only for user specified sen-
sors but for the instrumented ithreads as well.
santanni% my_app -cthread_monitor_socket forge 1057
-cthread_default_monitoring
Terminating Execution
The hubserver can be killed by typing Ctrl-C. Be aware
that if the hubserver is killed before the application com-
pletes, the hubserver will be unable to capture the remain-
ing event stream. Applications still connected to an absent
hubserver can detect a broken socket and should quit grace-
fully.
LIMITS AND DEFAULTS
SEE ALSO
cthread_intro(3), cthread_parse_args(3)x
AUTHOR
Falcon was written and maintained by many people.
This man page written by Beth Schroeder.