Transport Multiplexing Protocol (TMux)

Status of this Memo

This document specifies an Internet standards track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the "Internet Official Protocol Standards" (STD 1) for the standardization state and status of this protocol. Distribution of this memo is unlimited.

Abstract

One of the problems with the use of terminal servers is the large number of small packets they can generate. Frequently, most of these packets are destined for only one or two hosts. TMux is a protocol which allows multiple short transport segments, independent of application type, to be combined between a server and host pair.

Acknowledgments

This specification is the result of the merger of two documents: the original TMux proposal which was the result of several discussions and related initiatives through IETF working groups; and IEN 90 [1] originally proposed by Danny Cohen and Jon Postel in May 1979.

Applicability Statement

The TMux protocol is intended to optimize the transmission of large numbers of small data packets that are generated in situations where many interactive Telnet and Rlogin sessions are connected to a few hosts on the network. In these situations, TMux can improve both network and host performance. TMux is not intended for multiplexing long streams composed of large blocks of data that are typically transmitted by such applications as FTP.

The TMux protocol may be applicable to other situations where small packets are generated, but this was not considered in the design.

The use of the TMux protocol in any other situation may require some modification.

1. Introduction

When network designers consider which protocols generate the most load, they naturally tend to consider protocols which transfer large blocks of data (e.g., FTP, NFS). What is often not considered is the load generated by Telnet and Rlogin because of the assumption that users type slowly and the packets are very small. This is a grave underestimation of the load on networks and hosts which have many Telnet and Rlogin ports on multiple terminal servers.

The problem stems from the fact that the work a host must do to process a 1-octet packet is very nearly as much as the work it must do to process a 1500-octet packet. That is, it is the overhead of processing a packet which consumes a host's resources, not the processing of the data.

In particular, communication load is not measured only in bits per seconds but also in packets per seconds, and in many situation the latter is the true performance limit, not the former. The proposed multiplexing is aimed at alleviating this situation.

If one assumes that most users connected to a terminal server will be connecting to only a few hosts, then it should be obvious that the network and host load could be greatly reduced if traffic from multiple users, destined for the same host, could be sent in the same packet.

TMux is designed to improve network utilization and reduce the interrupt load on hosts which conduct multiple sessions involving many short packets. It does this by multiplexing transport traffic onto a single IP datagram [2], thereby resulting in fewer, larger packets. TMux is highly constrained in its method of accomplishing this task, seeking simplicity rather than sophistication.

2. Protocol Design

IP hosts may engage in the use of TMux transparently, and may even switch back and forth between use of TMux and carriage of transport segments in the usual, independent IP datagrams.

TMux operates by placing a set of transport segments into the same IP datagram. Each segment is preceded by a TMux mini-header which specifies the segment length and the actual segment transport protocol. The receiving host demultiplexes the individual transport segments and presents them to the transport layer as if they had been received in the usual IP/transport packaging. The transport layer is, therefore, unaware of the special encapsulation which was used.

Hence, a TMux message appears as:

     | IP hdr | TM hdr | Tport segment | TM hdr | Tport segment| ...|

Where:

   TM hdr         is a TMux mini-header and specifies the following
   
                  Tport segment.
   
   Tport segment  refers to the entire transport segment, including
                  transport headers.

The TMux Protocol is defined to allow the combining of transmission units of different higher level protocols in one transmission unit of a lower level protocol. Only segments with the same Internet Protocol (IP) header, (with the possible exception of the protocol and check- sum fields) may be combined. For example, the segment (H1, B1) and the segment (H2, B2), where Hi and Bi are the headers and the bodies of the segment, respectively, may be combined (multiplexed) only if H=H1=H2. The combined TMux message is either (H, B1, B2) or (H, B2, B1).

The receiver of this combined message should treat it as if the two original segments, (H,B1), and (H,B2), arrived separately. It is recommended, though not a requirement, that the segments in the TMux message should be processed in the same order that they are in the TMux message.

The multiplexing is achieved by combining the individual segments, (H,B1) through (H,Bn), into a single message. This single message has an IP header which is equal to H, but having in the PROTOCOL field the value 18 which is the protocol number of the TMux protocol. This IP header is followed by all the segments, B1 through Bn. Each segment, Bi, is preceded by a 4 octet TMux mini header. This contains the number of the protocol to which this segment is addressed. It also contains the total length of this segment, including this mini header. Since this mini header is not otherwise protected by a check- sum, it also includes a checksum field which just covers this mini header.

2.1. IP Protocol field value

TMux is indicated in an IP datagram by the Protocol (ID) value of 18 (22 octal), see [3].

2.2. Header Format

Each 4 octet TMux mini-header has the following general format:

                     +-------------------------------+
                     |         Length high           |
                     +-------------------------------+
                     |          Length low           |
                     +-------------------------------+
                     |         Protocol ID           |
                     +-------------------------------+
                     |          Checksum             |
                     +-------------------------------+
                     |      Transport segment        |
                     |       ...                     |
                     |       ...                     |

The LENGTH field specifies the octet count for this mini header and the following transport segment, from 0-65535 octets. Hence, the length field has a minimum value of 4. For segments that are larger than the maximum allowed for TMux (see section 5.1), individual IP datagrams should be sent.

The Protocol ID field contains the value that would normally have been placed in the IP header Protocol field.

The 'Checksum' field is the XOR of the first 3 octets.

To ensure that TCP, UDP and other segments keep their 32 bit alignment, where the segments being multiplexed are not a multiple of 32 bits long, extra octets will be added to re-align the end of the segment, and hence the next segment. These octets will be ignored on input. This padding will not affect the LENGTH field, it will still contain the real length of the segment.

2.3. Sending Data

Host endpoints may choose to use TMux at any time and in either (or both) directions. They also may switch back and forth between use of TMux packaging and the usual individual IP datagrams for individual transport associations. The only barrier to the use of TMux is for the sender to know whether TMux is supported by the receiver. This is important, since early use of TMux is likely to be limited.

The easiest way to detect TMUX support is to only send TMux messages to hosts from which a valid TMux message has already been received. This then leaves the problem of one host starting the TMux connection. This is most easily accomplished by the host sending an IP datagram with no data (i.e., with the IP total length field of 20), but with an IP Protocol field value of 18 for TMux. This is referred to as a TMux ENQ (enquiry) message. The host receiving this message then knows that the originator supports TMux, and can start to send TMux messages. This will in turn cause the originator of the ENQ message to start to use TMux. If for any reason the receiver does not intend to send TMux messages to the originator, but is prepared to accept them, then it can reply with another ENQ message.

If an ENQ message does not get a response, then it is reasonable to resend the ENQ a while later in case the original ENQ message was lost. If this again is lost, the ENQ may be repeated as often as needed, but the time between requests should increase exponentially up to a limit of about 1 hour. Suitable times between ENQs would be 15 seconds, 30 seconds, 60 seconds, 120 seconds etc.

Note that this checking process does not need to impede any of the transport (user) data, which may be sent as convenient, albeit in its less-efficient IP datagram form.

The only problem with this scheme is that a host which supports TMux may stop supporting it, as might happen when the host is re-booted. Other hosts need to learn of this change. The solution to this is to maintain a Time To Live (TTL) value for hosts from which TMux messages have been received. This TTL is a timed TTL, rather than a count as used in the IP TTL field, and this time stamp is updated every time a TMux message is received. This can then be used to expire the information held by TMux on the host after a suitable time, e.g., 1 minute.

This TTL time stamp is used as follows. When TMux is passed a segment to be sent to a host, a check is made to see if the time to live has expired. If the TTL has not expired, the segment is sent in a TMux message as normal. If the TTL has expired, the host is marked as being unable to TMux, but the segment is STILL sent as a TMux message (i.e., with the normal delay to allow other segments to be multiplexed). If the host is really unable to TMux anymore (a rare occurrence) then this segment will be timed out and retried by the transport provider i.e., TCP. Because the host was marked as not able to TMux, the retry will be sent as a normal IP datagram. If the remote host is still able to TMux then it should send back TMux traffic (even if it has been rebooted), typically a TCP window update, and the local host will mark it as able to TMux again. This way of operating removes any performance problem caused by continually dropping out of TMuxing and having to send probe messages. If the IP datagram to be sent is from UDP, then the remote host may not send anything in reply. So for UDP this scheme will not be any better than just stopping sending TMux messages to the host, but it is also no worse.

3. Protocol Behavior

3.1. Transport Flow Control

TMux operates as an extension to the IP datagram protocol. Hence, it has no impact on most flow control mechanisms, since they operate at the transport layer -- above TMux.

3.2. Connection Management

The concept of a connection pertains to certain transport protocols, but not to IP or to TMux. Hence, when connection management is required by a transport protocol using TMux, it occurs in the same fashion as it does for IP. In fact, the transport protocol is not to be aware that TMux is being used.

3.3 Multiplexed Message Construction

When a transport provider (e.g., TCP or UDP) sends a segment, TMux first removes the IP header (if present) and adds a TMux mini-header and the segment to the Multiplexed Message under construction for the host specified by the destination address of the segment.

When the first message to be transmitted is placed into the Multiplexed Message under construction, a timer is started. When the timer expires, the Multiplexed Message under construction is transmitted. This ensures that all segments available for sending before the timer expires are sent in a single Multiplexed Message. If, during construction of the Multiplexed Message, the buffer holding the message fills, the Multiplexed Message is transmitted immediately.

The delay time should be user configurable; a reasonable time is 20 to 30 milliseconds. The time period should be large enough to give a reasonable probability of sending multiple segments but not so large that the echo response time becomes a problem. This suggests that the upper limit for the timer is probably 1/10th second. As the cost of using timeouts on many systems is quite large, it is recommended that a single timer be used and that all TMux messages under construction are sent when the timer expires.

Additionally, configuration options may limit the number of included data segments or the maximum size of the Multiplexed Message before it is transmitted. It is also suggested that larger segments (e.g., those over 700 octets) should be sent as standard IP datagrams, and not multiplexed. This is to ensure that the delay caused by the TMux timer does not put a delay on those segments for which it is inadvisable. The size of the largest segments to be multiplexed should (if possible) be configurable.

4. Protocol Example

This example shows a TMux message consisting of three multiplexed segments:

A TCP segment consisting of a 20 octet TCP header, 5 octets of data and 3 octets of padding. Thus the length field is

             Mini header + TCP header + data
           =     4       +     20     +  5
           =     29

The padding is NOT included in the length.

A TCP segment consisting of a 20 octet TCP header, 4 octets of data. This segment does not require padding.

A UDP segment consisting of a 4 octet UDP header, 41 octets of data and 3 octets of padding.

                     +-------------------------------+
                     |         Length = 29           |
                     |         (2 octets)            |
                     +-------------------------------+
                     |     Protocol ID = 6 (TCP)     |
                     +-------------------------------+
                     |          Checksum             |
                     +-------------------------------+
                     |         TCP Header            |
                     |        (20 octets)            |
                     +-------------------------------+
                     |          TCP data             |
                     |         (5 octets)            |
                     +-------------------------------+
                     |          Padding              |
                     |         (3 octets)            |
                     +-------------------------------+
                     |         Length = 28           |
                     |         (2 octets)            |
                     +-------------------------------+
                     |     Protocol ID = 6 (TCP)     |
                     +-------------------------------+
                     |          Checksum             |
                     +-------------------------------+
                     |         TCP Header            |
                     |        (20 octets)            |
                     +-------------------------------+
                     |          TCP data             |
                     |         (4 octets)            |
                     +-------------------------------+
                     |         Length = 49           |
                     |         (2 octets)            |
                     +-------------------------------+
                     |    Protocol ID = 17 (UDP)     |
                     +-------------------------------+
                     |          Checksum             |
                     +-------------------------------+
                     |         UDP Header            |
                     |         (4 octets)            |
                     +-------------------------------+
                     |          UDP data             |
                     |         (41 octets)           |
                     +-------------------------------+
                     |          Padding              |
                     |         (3 octets)            |
                     +-------------------------------+

5. Implementation Suggestion

5.1 Maximum TMux Message Size

In section 3.3, a note is made about sending messages immediately if the limit on TMux message size is reached. On systems where Path MTU Discovery (as per RFC 1191 [4]) has been implemented this should be used to discover the maximum message size that can be transmitted, and this should be used as the maximum TMux message size.

5.2 Deciding Which Segments to Multiplex

It is the responsibility of the sender to decide which segments should be TMux'd and which should not. For example, segments sent by FTP should not normally be multiplexed. In many situations, it may be sensible to restrict the sessions that can be multiplexed to just those involved in interactive traffic (Telnet and Rlogin) by examining the source and destination TCP port numbers. However, if a segment that would not normally be multiplexed is to be sent and a TMux message is already under construction, then the extra segment can be added to the TMux message under construction, and this complete message should be sent immediately, rather than waiting for the timer to expire.

6. Implementation notes

The following notes are the result of experience gained during the testing of early implementations of TMux. Whilst they do not form part of the actual standard, they should be followed if possible to ensure compatibility with other implementations.

Because the TMux mini-header does not contain a TOS field, only segments with the same IP TOS field should be contained in a single TMux message. As most systems do not use the TOS feature, this is not a major restriction. Where the TOS field is used, it may be desirable to hold several messages under construction for a host, one for each TOS value.

Segments containing IP options should not be multiplexed.

Only unicast addresses should be considered for multiplexing.

Segments addressed to the loopback address (127.0.0.1) are not candidates for multiplexing.

Only segments with a source or destination port that is for an interactive session (i.e., Telnet and Rlogin) should be considered for multiplexing using TMux.

If an error is discovered in a checksum of a TMux header, the rest of the message, starting there, is ignored. If an unknown PROTOCOL field is discovered in any TMux header, this segment, and only this one, is ignored.

If the TMux implementation is continually sending TMux messages containing exactly one segment (because is there is little traffic to multiplex), then TMux may be turned off. This implies that TMux may be switched off when there is no congestion.

To prevent intermediate nodes from fragmenting and reconstructing TMux frames, implementations may want to set the "do not fragment" flag in the IP datagram of TMux messages.

If host B receives a TMux ENQ message from host A, but does not have any data for host A, then it may also send back an ENQ message. However, host A may send another ENQ message in response to this, so causing B to respond and so on. Thus if this facility is used, code must be included to prevent this looping behavior happening. Sending an ENQ in response to an ENQ is not recommended, except in special circumstances.

It is recommended that the following aspects of the TMux protocol be user configurable:

The maximum size of a segment that can be multiplexed by TMux.

The delay between the first segment being placed into the message under construction and the message being sent.

7. Security Considerations

Because TMux is effectively an extension to IP, it does not have any more impact on site security than does IP. Security should be dealt with by upper layer protocols.

Because some routers filter packets on the TCP port numbers, any segments sent using TMux will not be subject to this filtering as it will obscure the TCP port number However, larger segments for the same TCP connection will still be sent as IP datagrams, and so will be subject to filtering, thus giving rise to a potential problem. For this reason, any routers that do not support TMux, but which do support this type of filtering should not allow TMux messages through (in either direction). This will cause both hosts to think the other does not support TMux, so all segments will be sent as IP datagrams, thus eliminating this problem.

A better solution to this problem, is for routers to understand the TMux protocol, and to inspect each of the multiplexed segments and remove those segments that fail the filtering.

8. References

   [1] Cohen, D., and Postel, J., "Multiplexing Protocol", IEN 90,
   
       USC/Information Sciences Institute,, May 1979.
   
   [2] Postel, J., "Internet Protocol", STD 5, RFC 791, USC/Information
       Sciences Institute, September 1981.
   
   [3] Reynolds, J. and J. Postel, "Assigned Numbers", STD 2, RFC 1340,
       USC/Information Sciences Institute, March 1990.
   
   [4] Mogul, J., and S. Deering, "Path MTU discovery", RFC 1191, DECWRL
       and Stanford University, November 1990.

9. Authors' Addresses

Peter Cameron
Xylogics International, Ltd.
Featherstone Rd.
Wolverton Mill
Milton Keynes
MK12 5RD
United Kingdom

       Phone: +44  908 222112
       Fax:   +44  908 222115
       EMail: cameron@xylint.co.uk
       
       David Crocker
       Silicon Graphics, Inc.
       2011 N. Shoreline Blvd.
       P.O. Box 7311
       Mountain View, CA 94039-7311
       USA
       
       Phone: +1 415 390 1804
       Fax:   +1 415 962 8404
       EMail: dcrocker@sgi.com

Danny Cohen
Myricom
325 N. Santa Anita Ave.
Arcadia, CA 91006
USA

       Phone: +1 818 821 5555
       EMail: Cohen@myricom.com
       
       Jon Postel
       USC/Information Sciences Institute
       4676 Admiralty Way
       Marina del Rey, CA  90292-6695
       USA
       
       Phone: +1 310 822 1511
       Fax:   +1 310 823 6714
       EMail: Postel@ISI.EDU

10. Discussion List

There is a discussion list for this protocol, which for historical reasons is called:

           cmp-id@xylint.co.uk

Requests to join the list should be sent to:

           cmp-id-request@xylint.co.uk