January 28, 2002

TOPIC: Mesh Routing, MRC, Alewife, ...
 

Reading:

• Charles M. Flaig, VLSI Mesh Routing Systems, Technical Report 5241:TR:87, Computer Science Department, California Institute of Technology.
• John Kubiatowicz and Anant Agarwal, Anatomy of a Message in the Alewife Multiprocessor, Proc. of the 7th ACM Intl. Conference on Supercomputing (ICS), July 1993.
• Transmission Control Protocol, RFC793, September 1981.

The first paper addresses mesh routing chips (MRCs) which are organized as shown in the following diagram:

TRC (Torus Routing Chip)/Mosaic: J-Machine, M-Machine.
MRC (Mesh Routing Chip): Caltech Machines, Intel "Delta", Alewife.

Each chip has 10 ports:

Issues dealt with in this class are:

• (A) signaling conventions,
• (B) packet format, and
• (C) flow control/reliability/deadlocks.

(A) SIGNALING CONVENTIONS:

no clock, only ACK, min. number of transactions: 2-phase asynchronous protocol.
 

(B) PACKET FORMAT:

            Packet format:

X and Y indicate the number of hops in the x- and y-direction in the network, followed by the data and the tail.
Example: X=+2 and Y=+1. The packet is forwarded twice in the x-direction from W(est) to E(ast) (+2) and then once in the y-direction from S(outh) to N(orth) (+1):

The routing mechanism is implemented in two levels, first packets are forwarded in their x-direction, then in their y-direction:

"Wormhole routing" allows for sending out long messages at a router before all of the message has been received.
 

(C) FLOW CONTROL/RELIABILITY/DEADLOCKS:

    (i)   Flow Control: ACKs in hardware.
    (ii)  Reliability: assumed :-)
    (iii) Deadlocks...

     DEADLOCKS:

Compare to OS: deadlocks are possible if 'circular waiting' occurs. Solution: ordering of locks (ordering of the 'resources'). Here: Routers forward packet in x direction until x=0, then in y-direction until y=0 ===> ordering achieved by first forwarding in X and then in Y. Disadvantage: no other routing decisions possible, e.g., if a path is congested, router can not decide to go around (less freedom).

Possible deadlocks:

• network deadlocks (solved, see above)
• protocol deadlocks
• application deadlocks

(i) Protocol deadlocks: e.g.: RPC: request message followed by reply message, e.g.:

    req_handler {                    while (i) {   /* outer loop */

        send reply;                      poll (i);

    };                                   do request;

                                     };

e.g.: try to send before we receive next message and don't get any reply...
    ---->   more buffer
    ----> req_handler {
                        while (! ready to send)
                            buffer message;
                        reply;
                    };

that is, implement your own buffering.

(ii) Duplicate network: 1 network for request, 1 network for replies (shared-memory machines)
DASH-> 2 sets of networks
multiplexing wires, one wire says which network ("virtual lanes")

(iii) Drop messages (TCP/IP)