RFC: 817

MODULARITY AND EFFICIENCY IN PROTOCOL IMPLEMENTATION

                             David D. Clark
                  MIT Laboratory for Computer Science
               Computer Systems and Communications Group
                               July, 1982

1. Introduction

     Many  protocol implementers have made the unpleasant discovery that

their packages do not run quite as fast as they had hoped. The blame

for this widely observed problem has been attributed to a variety of

causes, ranging from details in the design of the protocol to the

underlying structure of the host operating system. This RFC will

discuss some of the commonly encountered reasons why protocol

implementations seem to run slowly.

     Experience  suggests  that  one  of  the  most important factors in

determining the performance of an implementation is the manner in which

that implementation is modularized and integrated into the host

operating system. For this reason, it is useful to discuss the question

of how an implementation is structured at the same time that we consider

how it will perform. In fact, this RFC will argue that modularity is

one of the chief villains in attempting to obtain good performance, so

that the designer is faced with a delicate and inevitable tradeoff

between good structure and good performance. Further, the single factor

which most strongly determines how well this conflict can be resolved is

2. Efficiency Considerations

     There  are  many aspects to efficiency.  One aspect is sending data

at minimum transmission cost, which is a critical aspect of common

carrier communications, if not in local area network communications.

Another aspect is sending data at a high rate, which may not be possible

at all if the net is very slow, but which may be the one central design

constraint when taking advantage of a local net with high raw bandwidth.

The final consideration is doing the above with minimum expenditure of

computer resources. This last may be necessary to achieve high speed,

but in the case of the slow net may be important only in that the

resources used up, for example cpu cycles, are costly or otherwise

needed. It is worth pointing out that these different goals often

conflict; for example it is often possible to trade off efficient use of

the computer against efficient use of the network. Thus, there may be

no such thing as a successful general purpose protocol implementation.

The simplest measure of performance is throughput, measured in bits

per second. It is worth doing a few simple computations in order to get

a feeling for the magnitude of the problems involved. Assume that data

is being sent from one machine to another in packets of 576 bytes, the

maximum generally acceptable internet packet size. Allowing for header

overhead, this packet size permits 4288 bits in each packet. If a

useful throughput of 10,000 bits per second is desired, then a data

bearing packet must leave the sending host about every 430 milliseconds,

a little over two per second. This is clearly not difficult to achieve.

the packet must leave the host every 43 milliseconds, and to achieve one

megabit per second, which is not at all unreasonable on a high-speed

local net, the packets must be spaced no more than 4.3 milliseconds.

These latter numbers are a slightly more alarming goal for which to

set one's sights. Many operating systems take a substantial fraction of

a millisecond just to service an interrupt. If the protocol has been

structured as a process, it is necessary to go through a process

scheduling before the protocol code can even begin to run. If any piece

of a protocol package or its data must be fetched from disk, real time

delays of between 30 to 100 milliseconds can be expected. If the

protocol must compete for cpu resources with other processes of the

system, it may be necessary to wait a scheduling quantum before the

protocol can run. Many systems have a scheduling quantum of 100

milliseconds or more. Considering these sorts of numbers, it becomes

immediately clear that the protocol must be fitted into the operating

system in a thorough and effective manner if any like reasonable

throughput is to be achieved.

     There is one obvious conclusion immediately suggested by even  this

simple analysis. Except in very special circumstances, when many

packets are being processed at once, the cost of processing a packet is

dominated by factors, such as cpu scheduling, which are independent of

the packet size. This suggests two general rules which any

implementation ought to obey. First, send data in large packets.

Obviously, if processing time per packet is a constant, then throughput

unneeded packet. Unneeded packets use up just as many resources as a

packet full of data, but perform no useful function. RFC 813, "Window

and Acknowledgement Strategy in TCP", discusses one aspect of reducing

the number of packets sent per useful data byte. This document will

mention other attacks on the same problem.

     The  above  analysis  suggests that there are two main parts to the

problem of achieving good protocol performance. The first has to do

with how the protocol implementation is integrated into the host

operating system. The second has to do with how the protocol package

itself is organized internally. This document will consider each of

these topics in turn.

3. The Protocol vs. the Operating System

There are normally three reasonable ways in which to add a protocol

to an operating system. The protocol can be in a process that is

provided by the operating system, or it can be part of the kernel of the

operating system itself, or it can be put in a separate communications

processor or front end machine. This decision is strongly influenced by

details of hardware architecture and operating system design; each of

these three approaches has its own advantages and disadvantages.

     The  "process"  is the abstraction which most operating systems use

to provide the execution environment for user programs. A very simple

path for implementing a protocol is to obtain a process from the

operating system and implement the protocol to run in it.

modifications to the kernel are not required, the job can be done by

someone who is not an expert in the kernel structure. Since it is often

impossible to find somebody who is experienced both in the structure of

the operating system and the structure of the protocol, this path, from

a management point of view, is often extremely appealing. Unfortunately,

putting a protocol in a process has a number of disadvantages, related

to both structure and performance. First, as was discussed above,

process scheduling can be a significant source of real-time delay.

There is not only the actual cost of going through the scheduler, but

the problem that the operating system may not have the right sort of

priority tools to bring the process into execution quickly whenever

there is work to be done.

     Structurally, the difficulty with putting a protocol in  a  process

is that the protocol may be providing services, for example support of

data streams, which are normally obtained by going to special kernel

entry points. Depending on the generality of the operating system, it

may be impossible to take a program which is accustomed to reading

through a kernel entry point, and redirect it so it is reading the data

from a process. The most extreme example of this problem occurs when

implementing server telnet. In almost all systems, the device handler

for the locally attached teletypes is located inside the kernel, and

programs read and write from their teletype by making kernel calls. If

server telnet is implemented in a process, it is then necessary to take

the data streams provided by server telnet and somehow get them back

down inside the kernel so that they mimic the interface provided by

modification is necessary to achieve this structure, which somewhat

defeats the benefit of having removed the protocol from the kernel in

the first place.

Clearly, then, there are advantages to putting the protocol package

in the kernel. Structurally, it is reasonable to view the network as a

device, and device drivers are traditionally contained in the kernel.

Presumably, the problems associated with process scheduling can be

sidesteped, at least to a certain extent, by placing the code inside the

kernel. And it is obviously easier to make the server telnet channels

mimic the local teletype channels if they are both realized in the same

level in the kernel.

     However, implementation of protocols in the kernel has its own  set

of pitfalls. First, network protocols have a characteristic which is

shared by almost no other device: they require rather complex actions

to be performed as a result of a timeout. The problem with this

requirement is that the kernel often has no facility by which a program

can be brought into execution as a result of the timer event. What is

really needed, of course, is a special sort of process inside the

kernel. Most systems lack this mechanism. Failing that, the only

execution mechanism available is to run at interrupt time.

     There are substantial drawbacks to implementing a protocol  to  run

at interrupt time. First, the actions performed may be somewhat complex

and time consuming, compared to the maximum amount of time that the

operating system is prepared to spend servicing an interrupt. Problems

bad when running as a result of a clock interrupt, which can imply that

the clock interrupt is masked. Second, the environment provided by an

interrupt handler is usually extremely primitive compared to the

environment of a process. There are usually a variety of system

facilities which are unavailable while running in an interrupt handler.

The most important of these is the ability to suspend execution pending

the arrival of some event or message. It is a cardinal rule of almost

every known operating system that one must not invoke the scheduler

while running in an interrupt handler. Thus, the programmer who is

forced to implement all or part of his protocol package as an interrupt

handler must be the best sort of expert in the operating system

involved, and must be prepared for development sessions filled with

obscure bugs which crash not just the protocol package but the entire

operating system.

     A final problem with processing  at  interrupt  time  is  that  the

system scheduler has no control over the percentage of system time used

by the protocol handler. If a large number of packets arrive, from a

foreign host that is either malfunctioning or fast, all of the time may

be spent in the interrupt handler, effectively killing the system.

There are other problems associated with putting protocols into an

operating system kernel. The simplest problem often encountered is that

the kernel address space is simply too small to hold the piece of code

in question. This is a rather artificial sort of problem, but it is a

severe problem none the less in many machines. It is an appallingly

for every byte of new feature put in one must find some other byte of

old feature to throw out. It is hopeless to expect an effective and

general implementation under this kind of constraint. Another problem

is that the protocol package, once it is thoroughly entwined in the

operating system, may need to be redone every time the operating system

changes. If the protocol and the operating system are not maintained by

the same group, this makes maintenance of the protocol package a

perpetual headache.

     The  third  option  for  protocol  implementation  is  to  take the

protocol package and move it outside the machine entirely, on to a

separate processor dedicated to this kind of task. Such a machine is

often described as a communications processor or a front-end processor.

There are several advantages to this approach. First, the operating

system on the communications processor can be tailored for precisely

this kind of task. This makes the job of implementation much easier.

Second, one does not need to redo the task for every machine to which

the protocol is to be added. It may be possible to reuse the same

front-end machine on different host computers. Since the task need not

be done as many times, one might hope that more attention could be paid

to doing it right. Given a careful implementation in an environment

which is optimized for this kind of task, the resulting package should

turn out to be very efficient. Unfortunately, there are also problems

with this approach. There is, of course, a financial problem associated

with buying an additional computer. In many cases, this is not a

problem at all since the cost is negligible compared to what the

fundamentally, the communications processor approach does not completely

sidestep any of the problems raised above. The reason is that the

communications processor, since it is a separate machine, must be

attached to the mainframe by some mechanism. Whatever that mechanism,

code is required in the mainframe to deal with it. It can be argued

that the program to deal with the communications processor is simpler

than the program to implement the entire protocol package. Even if that

is so, the communications processor interface package is still a

protocol in nature, with all of the same structural problems. Thus, all

of the issues raised above must still be faced. In addition to those

problems, there are some other, more subtle problems associated with an

outboard implementation of a protocol. We will return to these problems

later.

     There  is  a  way  of  attaching  a  communications  processor to a

mainframe host which sidesteps all of the mainframe implementation

problems, which is to use some preexisting interface on the host machine

as the port by which a communications processor is attached. This

strategy is often used as a last stage of desperation when the software

on the host computer is so intractable that it cannot be changed in any

way. Unfortunately, it is almost inevitably the case that all of the

available interfaces are totally unsuitable for this purpose, so the

result is unsatisfactory at best. The most common way in which this

form of attachment occurs is when a network connection is being used to

mimic local teletypes. In this case, the front-end processor can be

attached to the mainframe by simply providing a number of wires out of

plugged into teletype ports on the mainframe computer. (Because of the

appearance of the physical configuration which results from this

arrangement, Michael Padlipsky has described this as the "milking

machine" approach to computer networking.) This strategy solves the

immediate problem of providing remote access to a host, but it is

extremely inflexible. The channels being provided to the host are

restricted by the host software to one purpose only, remote login. It

is impossible to use them for any other purpose, such as file transfer

or sending mail, so the host is integrated into the network environment

in an extremely limited and inflexible manner. If this is the best that

can be done, then it should be tolerated. Otherwise, implementors

should be strongly encouraged to take a more flexible approach.

4. Protocol Layering

     The  previous  discussion suggested that there was a decision to be

made as to where a protocol ought to be implemented. In fact, the

decision is much more complicated than that, for the goal is not to

implement a single protocol, but to implement a whole family of protocol

layers, starting with a device driver or local network driver at the

bottom, then IP and TCP, and eventually reaching the application

specific protocol, such as Telnet, FTP and SMTP on the top. Clearly,

the bottommost of these layers is somewhere within the kernel, since the

physical device driver for the net is almost inevitably located there.

Equally clearly, the top layers of this package, which provide the user

his ability to perform the remote login function or to send mail, are

whether the protocol family shall be inside or outside the kernel, but

how it shall be sliced in two between that part inside and that part

outside.

     Since  protocols  come  nicely layered, an obvious proposal is that

one of the layer interfaces should be the point at which the inside and

outside components are sliced apart. Most systems have been implemented

in this way, and many have been made to work quite effectively. One

obvious place to slice is at the upper interface of TCP. Since TCP

provides a bidirectional byte stream, which is somewhat similar to the

I/O facility provided by most operating systems, it is possible to make

the interface to TCP almost mimic the interface to other existing

devices. Except in the matter of opening a connection, and dealing with

peculiar failures, the software using TCP need not know that it is a

network connection, rather than a local I/O stream that is providing the

communications function. This approach does put TCP inside the kernel,

which raises all the problems addressed above. It also raises the

problem that the interface to the IP layer can, if the programmer is not

careful, become excessively buried inside the kernel. It must be

remembered that things other than TCP are expected to run on top of IP.

The IP interface must be made accessible, even if TCP sits on top of it

inside the kernel.

     Another obvious place to slice is above Telnet.  The  advantage  of

slicing above Telnet is that it solves the problem of having remote

login channels emulate local teletype channels. The disadvantage of

been included there is getting remarkably large. In some early

implementations, the size of the network package, when one includes

protocols at the level of Telnet, rivals the size of the rest of the

supervisor. This leads to vague feelings that all is not right.

     Any attempt to slice through a lower layer  boundary,  for  example

between internet and TCP, reveals one fundamental problem. The TCP

layer, as well as the IP layer, performs a demultiplexing function on

incoming datagrams. Until the TCP header has been examined, it is not

possible to know for which user the packet is ultimately destined.

Therefore, if TCP, as a whole, is moved outside the kernel, it is

necessary to create one separate process called the TCP process, which

performs the TCP multiplexing function, and probably all of the rest of

TCP processing as well. This means that incoming data destined for a

user process involves not just a scheduling of the user process, but

scheduling the TCP process first.

     This suggests an  alternative  structuring  strategy  which  slices

through the protocols, not along an established layer boundary, but

along a functional boundary having to do with demultiplexing. In this

approach, certain parts of IP and certain parts of TCP are placed in the

kernel. The amount of code placed there is sufficient so that when an

incoming datagram arrives, it is possible to know for which process that

datagram is ultimately destined. The datagram is then routed directly

to the final process, where additional IP and TCP processing is

performed on it. This removes from the kernel any requirement for timer

user. This structure has the additional advantage of reducing the

amount of code required in the kernel, so that it is suitable for

systems where kernel space is at a premium. The RFC 814, titled "Names,

Addresses, Ports, and Routes," discusses this rather orthogonal slicing

strategy in more detail.

     A related discussion of protocol layering and multiplexing  can  be

found in Cohen and Postel [1].

5. Breaking Down the Barriers

     In  fact, the implementor should be sensitive to the possibility of

even more peculiar slicing strategies in dividing up the various

protocol layers between the kernel and the one or more user processes.

The result of the strategy proposed above was that part of TCP should

execute in the process of the user. In other words, instead of having

one TCP process for the system, there is one TCP process per connection.

Given this architecture, it is not longer necessary to imagine that all

of the TCPs are identical. One TCP could be optimized for high

throughput applications, such as file transfer. Another TCP could be

optimized for small low delay applications such as Telnet. In fact, it

would be possible to produce a TCP which was somewhat integrated with

the Telnet or FTP on top of it. Such an integration is extremely

important, for it can lead to a kind of efficiency which more

traditional structures are incapable of producing. Earlier, this paper

pointed out that one of the important rules to achieving efficiency was

to send the minimum number of packets for a given amount of data. The

goal, because independent layers have independent ideas about when

packets should be sent, and unless these layers can somehow be brought

into cooperation, additional packets will flow. The best example of

this is the operation of server telnet in a character at a time remote

echo mode on top of TCP. When a packet containing a character arrives

at a server host, each layer has a different response to that packet.

TCP has an obligation to acknowledge the packet. Either server telnet

or the application layer above has an obligation to echo the character

received in the packet. If the character is a Telnet control sequence,

then Telnet has additional actions which it must perform in response to

the packet. The result of this, in most implementations, is that

several packets are sent back in response to the one arriving packet.

Combining all of these return messages into one packet is important for

several reasons. First, of course, it reduces the number of packets

being sent over the net, which directly reduces the charges incurred for

many common carrier tariff structures. Second, it reduces the number of

scheduling actions which will occur inside both hosts, which, as was

discussed above, is extremely important in improving throughput.

The way to achieve this goal of packet sharing is to break down the

barrier between the layers of the protocols, in a very restrained and

careful manner, so that a limited amount of information can leak across

the barrier to enable one layer to optimize its behavior with respect to

the desires of the layers above and below it. For example, it would

represent an improvement if TCP, when it received a packet, could ask

the layer above whether or not it would be worth pausing for a few

upper layer would have any outgoing data to send. Dallying before

sending the acknowledgement produces precisely the right sort of

optimization if the client of TCP is server Telnet. However, dallying

before sending an acknowledgement is absolutely unacceptable if TCP is

being used for file transfer, for in file transfer there is almost never

data flowing in the reverse direction, and the delay in sending the

acknowledgement probably translates directly into a delay in obtaining

the next packets. Thus, TCP must know a little about the layers above

it to adjust its performance as needed.

     It would be possible to imagine a general  purpose  TCP  which  was

equipped with all sorts of special mechanisms by which it would query

the layer above and modify its behavior accordingly. In the structures

suggested above, in which there is not one but several TCPs, the TCP can

simply be modified so that it produces the correct behavior as a matter

of course. This structure has the disadvantage that there will be

several implementations of TCP existing on a single machine, which can

mean more maintenance headaches if a problem is found where TCP needs to

be changed. However, it is probably the case that each of the TCPs will

be substantially simpler than the general purpose TCP which would

otherwise have been built. There are some experimental projects

currently under way which suggest that this approach may make designing

of a TCP, or almost any other layer, substantially easier, so that the

total effort involved in bringing up a complete package is actually less

if this approach is followed. This approach is by no means generally

     The  general conclusion to be drawn from this sort of consideration

is that a layer boundary has both a benefit and a penalty. A visible

layer boundary, with a well specified interface, provides a form of

isolation between two layers which allows one to be changed with the

confidence that the other one will not stop working as a result.

However, a firm layer boundary almost inevitably leads to inefficient

operation. This can easily be seen by analogy with other aspects of

operating systems. Consider, for example, file systems. A typical

operating system provides a file system, which is a highly abstracted

representation of a disk. The interface is highly formalized, and

presumed to be highly stable. This makes it very easy for naive users

to have access to disks without having to write a great deal of

software. The existence of a file system is clearly beneficial. On the

other hand, it is clear that the restricted interface to a file system

almost inevitably leads to inefficiency. If the interface is organized

as a sequential read and write of bytes, then there will be people who

wish to do high throughput transfers who cannot achieve their goal. If

the interface is a virtual memory interface, then other users will

regret the necessity of building a byte stream interface on top of the

memory mapped file. The most objectionable inefficiency results when a

highly sophisticated package, such as a data base management package,

must be built on top of an existing operating system. Almost

inevitably, the implementors of the database system attempt to reject

the file system and obtain direct access to the disks. They have

sacrificed modularity for efficiency.

The concept of a protocol is still unknown and frightening to most naive

programmers. The idea that they might have to implement a protocol, or

even part of a protocol, as part of some application package, is a

dreadful thought. And thus there is great pressure to hide the function

of the net behind a very hard barrier. On the other hand, the kind of

inefficiency which results from this is a particularly undesirable sort

of inefficiency, for it shows up, among other things, in increasing the

cost of the communications resource used up to achieve the application

goal. In cases where one must pay for one's communications costs, they

usually turn out to be the dominant cost within the system. Thus, doing

an excessively good job of packaging up the protocols in an inflexible

manner has a direct impact on increasing the cost of the critical

resource within the system. This is a dilemma which will probably only

be solved when programmers become somewhat less alarmed about protocols,

so that they are willing to weave a certain amount of protocol structure

into their application program, much as application programs today weave

parts of database management systems into the structure of their

application program.

     An extreme example of putting the protocol package  behind  a  firm

layer boundary occurs when the protocol package is relegated to a front-

end processor. In this case the interface to the protocol is some other

protocol. It is difficult to imagine how to build close cooperation

between layers when they are that far separated. Realistically, one of

the prices which must be associated with an implementation so physically

modularized is that the performance will suffer as a result. Of course,

the mainframe architecture, with interprocessor co-ordination signals,

shared memory, and similar features. Such a physical modularity might

work very well, but there is little documented experience with this

closely coupled architecture for protocol support.

6. Efficiency of Protocol Processing

     To this point, this document has considered how a protocol  package

should be broken into modules, and how those modules should be

distributed between free standing machines, the operating system kernel,

and one or more user processes. It is now time to consider the other

half of the efficiency question, which is what can be done to speed the

execution of those programs that actually implement the protocols. We

will make some specific observations about TCP and IP, and then conclude

with a few generalities.

     IP  is a simple protocol, especially with respect to the processing

of normal packets, so it should be easy to get it to perform

efficiently. The only area of any complexity related to actual packet

processing has to do with fragmentation and reassembly. The reader is

referred to RFC 815, titled "IP Datagram Reassembly Algorithms", for

specific consideration of this point.

     Most costs in the IP layer come from table look  up  functions,  as

opposed to packet processing functions. An outgoing packet requires two

translation functions to be performed. The internet address must be

translated to a target gateway, and a gateway address must be translated

network). It is easy to build a simple implementation of these table

look up functions that in fact performs very poorly. The programmer

should keep in mind that there may be as many as a thousand network

numbers in a typical configuration. Linear searching of a thousand

entry table on every packet is extremely unsuitable. In fact, it may be

worth asking TCP to cache a hint for each connection, which can be

handed down to IP each time a packet is sent, to try to avoid the

overhead of a table look up.

     TCP   is   a   more   complex  protocol,  and  presents  many  more

opportunities for getting things wrong. There is one area which is

generally accepted as causing noticeable and substantial overhead as

part of TCP processing. This is computation of the checksum. It would

be nice if this cost could be avoided somehow, but the idea of an end-

to-end checksum is absolutely central to the functioning of TCP. No

host implementor should think of omitting the validation of a checksum

on incoming data.

Various clever tricks have been used to try to minimize the cost of

computing the checksum. If it is possible to add additional microcoded

instructions to the machine, a checksum instruction is the most obvious

candidate. Since computing the checksum involves picking up every byte

of the segment and examining it, it is possible to combine the operation

of computing the checksum with the operation of copying the segment from

one location to another. Since a number of data copies are probably

already required as part of the processing structure, this kind of

the modularity of the program. Finally, computation of the checksum

seems to be one place where careful attention to the details of the

algorithm used can make a drastic difference in the throughput of the

program. The Multics system provides one of the best case studies of

this, since Multics is about as poorly organized to perform this

function as any machine implementing TCP. Multics is a 36-bit word

machine, with four 9-bit bytes per word. The eight-bit bytes of a TCP

segment are laid down packed in memory, ignoring word boundaries. This

means that when it is necessary to pick up the data as a set of 16-bit

units for the purpose of adding them to compute checksums, horrible

masking and shifting is required for each 16-bit value. An early

version of a program using this strategy required 6 milliseconds to

checksum a 576-byte segment. Obviously, at this point, checksum

computation was becoming the central bottleneck to throughput. A more

careful recoding of this algorithm reduced the checksum processing time

to less than one millisecond. The strategy used was extremely dirty.

It involved adding up carefully selected words of the area in which the

data lay, knowing that for those particular words, the 16-bit values

were properly aligned inside the words. Only after the addition had

been done were the various sums shifted, and finally added to produce

the eventual checksum. This kind of highly specialized programming is

probably not acceptable if used everywhere within an operating system.

It is clearly appropriate for one highly localized function which can be

clearly identified as an extreme performance bottleneck.

Another area of TCP processing which may cause performance problems

occur in each incoming packet. One paper, by Bunch and Day [2], asserts

that the overhead of packet header processing is actually an important

limiting factor in throughput computation. Not all measurement

experiments have tended to support this result. To whatever extent it

is true, however, there is an obvious strategy which the implementor

ought to use in designing his program. He should build his program to

optimize the expected case. It is easy, especially when first designing

a program, to pay equal attention to all of the possible outcomes of

every test. In practice, however, few of these will ever happen. A TCP

should be built on the assumption that the next packet to arrive will

have absolutely nothing special about it, and will be the next one

expected in the sequence space. One or two tests are sufficient to

determine that the expected set of control flags are on. (The ACK flag

should be on; the Push flag may or may not be on. No other flags should

be on.) One test is sufficient to determine that the sequence number of

the incoming packet is one greater than the last sequence number

received. In almost every case, that will be the actual result. Again,

using the Multics system as an example, failure to optimize the case of

receiving the expected sequence number had a detectable effect on the

performance of the system. The particular problem arose when a number

of packets arrived at once. TCP attempted to process all of these

packets before awaking the user. As a result, by the time the last

packet arrived, there was a threaded list of packets which had several

items on it. When a new packet arrived, the list was searched to find

the location into which the packet should be inserted. Obviously, the

number, because one is expecting to receive a packet which comes after

those already received. By mistake, the list was searched from front to

back, starting with the packets with the lowest sequence number. The

amount of time spent searching this list backwards was easily detectable

in the metering measurements.

Other data structures can be organized to optimize the action which

is normally taken on them. For example, the retransmission queue is

very seldom actually used for retransmission, so it should not be

organized to optimize that action. In fact, it should be organized to

optimized the discarding of things from it when the acknowledgement

arrives. In many cases, the easiest way to do this is not to save the

packet at all, but to reconstruct it only if it needs to be

retransmitted, starting from the data as it was originally buffered by

the user.

     There is another generality, at least as  important  as  optimizing

the common case, which is to avoid copying data any more times than

necessary. One more result from the Multics TCP may prove enlightening

here. Multics takes between two and three milliseconds within the TCP

layer to process an incoming packet, depending on its size. For a 576-

byte packet, the three milliseconds is used up approximately as follows.

One millisecond is used computing the checksum. Six hundred

microseconds is spent copying the data. (The data is copied twice, at

.3 milliseconds a copy.) One of those copy operations could correctly

be included as part of the checksum cost, since it is done to get the

However, the copy also performs another necessary transfer at the same

time. Header processing and packet resequencing takes .7 milliseconds.

The rest of the time is used in miscellaneous processing, such as

removing packets from the retransmission queue which are acknowledged by

this packet. Data copying is the second most expensive single operation

after data checksuming. Some implementations, often because of an

excessively layered modularity, end up copying the data around a great

deal. Other implementations end up copying the data because there is no

shared memory between processes, and the data must be moved from process

to process via a kernel operation. Unless the amount of this activity

is kept strictly under control, it will quickly become the major

performance bottleneck.

7. Conclusions

     This document has addressed two aspects  of  obtaining  performance

from a protocol implementation, the way in which the protocol is layered

and integrated into the operating system, and the way in which the

detailed handling of the packet is optimized. It would be nice if one

or the other of these costs would completely dominate, so that all of

one's attention could be concentrated there. Regrettably, this is not

so. Depending on the particular sort of traffic one is getting, for

example, whether Telnet one-byte packets or file transfer maximum size

packets at maximum speed, one can expect to see one or the other cost

being the major bottleneck to throughput. Most implementors who have

studied their programs in an attempt to find out where the time was

equally to all parts of their program. With the possible exception of

checksum processing, very few people have ever found that their

performance problems were due to a single, horrible bottleneck which

they could fix by a single stroke of inventive programming. Rather, the

performance was something which was improved by painstaking tuning of

the entire program.

     Most  discussions  of protocols begin by introducing the concept of

layering, which tends to suggest that layering is a fundamentally

wonderful idea which should be a part of every consideration of

protocols. In fact, layering is a mixed blessing. Clearly, a layer

interface is necessary whenever more than one client of a particular

layer is to be allowed to use that same layer. But an interface,

precisely because it is fixed, inevitably leads to a lack of complete

understanding as to what one layer wishes to obtain from another. This

has to lead to inefficiency. Furthermore, layering is a potential snare

in that one is tempted to think that a layer boundary, which was an

artifact of the specification procedure, is in fact the proper boundary

to use in modularizing the implementation. Again, in certain cases, an

architected layer must correspond to an implemented layer, precisely so

that several clients can have access to that layer in a reasonably

straightforward manner. In other cases, cunning rearrangement of the

implemented module boundaries to match with various functions, such as

the demultiplexing of incoming packets, or the sending of asynchronous

outgoing packets, can lead to unexpected performance improvements

compared to more traditional implementation strategies. Finally, good

program. Since performance is influenced, not just by the fine detail,

but by the gross structure, it is sometimes the case that in order to

obtain a substantial performance improvement, it is necessary to

completely redo the program from the bottom up. This is a great

disappointment to programmers, especially those doing a protocol

implementation for the first time. Programmers who are somewhat

inexperienced and unfamiliar with protocols are sufficiently concerned

with getting their program logically correct that they do not have the

capacity to think at the same time about the performance of the

structure they are building. Only after they have achieved a logically

correct program do they discover that they have done so in a way which

has precluded real performance. Clearly, it is more difficult to design

a program thinking from the start about both logical correctness and

performance. With time, as implementors as a group learn more about the

appropriate structures to use for building protocols, it will be

possible to proceed with an implementation project having more

confidence that the structure is rational, that the program will work,

and that the program will work well. Those of us now implementing

protocols have the privilege of being on the forefront of this learning

process. It should be no surprise that our programs sometimes suffer

Citations

     [1]  Cohen  and  Postel,  "On  Protocol  Multiplexing",  Sixth Data

Communications Symposium, ACM/IEEE, November 1979.

     [2] Bunch and Day, "Control Structure Overhead in TCP", Trends  and

Applications: Computer Networking, NBS Symposium, May 1980.