A Really

Simple Mail Transfer Protocol

(DRAFT of 28, 2006)

Geoff Collyer

geoff@plan9.bell-labs.com

Bell Laboratories

Murray Hill, New Jersey 07974

ABSTRACT

A new mail transport protocol is proposed for use over a reliable byte-stream transport. This protocol is faster, simpler and more streaming than SMTP, and deals better with binary and Unicode data.

1. Introduction

A new mail transport protocol is proposed for use over a reliable byte-stream transport such as TCP or ATM AAL5, called rsmtp. Rsmtp has these properties:

∙ takes fewer protocol round trips .ix round~trips .ix stop-and-wait .ix turn (pairs of ‘stop-and-waits’ or ‘turnarounds’) per message transferred.

∙ incurs no per-byte massaging on advanced operating systems, only new-line massaging in text on others.

∙ presents a clear, 8-bit channel .ix clear channel .ix 8-bit channel (except for text messages or textual parts of multipart MIME messages on legacy operating systems, .ix legacy operating~systems .ix newline conventions which must perform new-line translations), so no MIME Content-Transfer-Encodings are needed or permitted. .ix MIME .ix Content-Transfer-Encodings

∙ imposes no line-length limits.

∙ .ix SMTP chit-chat omits chit-chat of SMTP25, which often contains only a few bits of information per line.

∙ .ix duplicate suppression permits and encourages suppression of duplicate messages.

∙ permits and requires .ix loop detection loop detection (detecting a system connecting to itself).

∙ permits and encourages .ix short-circuit delivery short-circuit delivery by the sender .ix message store if sender and receiver share a message store.

∙ provides no analogues of SMTP’s vrfy, .ix SMTP [vrfy] .ix SMTP [expn] expn, etc.

∙ subsumes NNTP (the Internet Network News Transfer Protocol)17. .ix NNTP Network News Transfer Protocol

To elaborate, bytes are transmitted raw and received without further required per-byte processing, rather than, as in SMTP, having high bits stripped, .ix SMTP high-bit whacking .ix SMTP hidden dots .ix SMTP CRLF stuffing .ix SMTP CRLF whacking dots and (on advanced operating systems) carriage-returns stuffed (and unstuffed), and case of alphabetics folded (except that domain names must be treated as case-insensitive, alas, in rsmtp). An octet, or byte, contains 8 bits and rsmtp uses them all. .ix 8-bit byte .ix 8-bit octet CRLFs are not required anywhere and are discouraged everywhere; a new line is a newline character in text. Implementations on operating systems that use other new-line conventions must convert between their native representations and the use of a single newline character in text messages or textual parts of multipart MIME messages. The lack of per-byte hand-crafting on advanced operating systems (and minimal fiddling on others) permits higher performance, greater simplicity and less confusion, particularly during transmission of messages (as opposed to envelopes), where neither party need be aware of any structuring into lines (since there may not be any) and can simply copy bytes en masse. Similarly, rsmtp provides no analogues of SMTP’s vrfy, .ix SMTP [vrfy] .ix SMTP [expn] expn, etc.; systems requiring these functions can continue to use SMTP on TCP port 25. Rsmtp is purely for transport, not inquiries, and these commands tend to be slow, unnecessary and invasions of privacy.

NNTP was never needed. .ix NNTP unneeded Network file systems provide superior news reading. .ix network file systems For sending news, creative use of SMTP to a well-known port other .ix SMTP than TCP 25, with the message-id supplied in the MAIL FROM line, and a receiver that refused articles whose message-ids were listed in its netnews history file would have done the job; there was sufficient mechanism within the SMTP protocol. As an added enhancement, one could have required the sender to put the Newsgroups: list in the envelope RCPT as a local-part. Rsmtp makes an even better replacement for NNTP.

.ix complexity of protocol specifications

rsmtp is quite simple on the wire. Using specification size in pages as a coarse measure of complexity, Figure 1 compares rsmtp with earlier mail transport protocols. RFC page counts are from the InterNIC’s copies; the others are pages of troff output. This paper cites RFC 821 on occasion, so the comparison isn’t completely fair, but gives an approximate idea of relative complexity. Admittedly, rsmtp encourages asynchrony in I/O .ix asynchrony and performs the equivalent of a two-phase commit to ensure that messages .ix two-phase commit really are delivered and only once, and so is harder to program than SMTP, at least on systems without convenient address-space sharing among processes.

Figure 1: Specification sizes

l’6i’

2. Protocol

2.1. Grammar

.ix {rsmtp} grammar .ix {yacc}

Figure 2 is the BNF grammar of the complete conversation to transmit a single message via rsmtp, in yacc’s16 notation.

greeting: ’r’ ’s’ ’m’ ’t’ ’p’ ’ ’ role ’ ’ DOMAIN ’ ’ DOMAIN ’\n’;

role:   ’t’ | ’r’;

envelope: ’i’ ’\n’ | ’i’ ADDRESS ’ ’ numb ’ ’ notify ’\n’ ’m’ NON_NL ’\n’

   optext ’s’ ADDRESS ’\n’ rcpts ’\n’;

notify: ’n’ | ’z’ | ’b’ | ’c’;

optext: | LETTER NON_NL ’\n’ optext;

rcpts:  ’t’ ADDRESS ’\n’ | ’t’ ADDRESS ’\n’ rcpts;

envstatus:  optextrepl rcpstats sendstatus ’\n’;

optextrepl: | optext ’\n’;

rcpstats:   rcpstat ’\n’ | rcpstat ’\n’ rcpstats;

rcpstat:    msgstat | rcpreject NON_NL;

rcpreject:  ’n’ | ’f’;

sendstatus: ’Y’ | ’R’ numb ’ ’ NON_NL | sendreject NON_NL;

sendreject: ’N’ | ’F’;

message:    fragment | fragment message ; 

fragment:   numb ’\n’ ANYBYTES ’.’ ; 

msgstatus:  msgstat ’\n’;

msgstat:    ’y’ | ’r’ numb ’ ’ NON_NL;

numb:   digit | digit numb;

digit:  ’0’ | ’1’ | ’2’ | ’3’ | ’4’ | ’5’ | ’6’ | ’7’ | ’8’ | ’9’;

Figure 2: Grammar of an rsmtp conversation

l’6i’

Literal characters are enclosed in quotes; C language escapes are honoured. Alternatives are separated by vertical bars. ANYBYTES is any sequence of zero or more bytes. NON_NL is any sequence of zero or more non-newline characters. DOMAIN is an Internet domain, ADDRESS is normally an Internet mail address (a mailbox in RFC 821 terms); both are currently defined by RFC 821 and amended by later RFCs18,28,9,10,19,20,22,15, notably the hosts requirements RFCs6,7. See the section Address Resolution for other addressing possibilities. LETTER is any Unicode8,1 letter, encoded as UTF-830,31, the obvious choice; .ix Unicode .ix utf-8 .ix utf all characters with values above 8016 are assumed to be letters for this purpose.

This grammar is simple enough that yacc is not needed to parse it, in contrast to the two yacc grammars, one for SMTP and one for RFC 822, used by at least one extant mail system.

2.2. Protocol Overview .ix {rsmtp} overview

Upon connection, the receiver sends a greeting; the transmitter replies with greeting and an envelope. .ix {rsmtp} envelope If the envelope indicates that the transmitter is done for this session, the call is over and both sides shut down; otherwise the receiver replies with an envstatus. If the sendstatus is Y, the transmitter sends a message and the receiver replies with a msgstatus. The process then starts over from the transmitter sending an envelope (without a greeting).

Figure 3 shows the flow of an rsmtp conversation as it would be if the protocol were completely synchronous (‘half-duplex’). One trip through the main loop transfers one message. .ix asynchrony .ix duplicity The preferred implementation will be asynchronous (‘full-duplex’), with both parties potentially having data in flight simultaneously. Henceforth, we shall abbreviate ‘the transmitter’ to ‘T’ and ‘the receiver’ to ‘R’. .ix abbreviations T might be referred to by others as the client and R as the server. ‘Current message’ is hereinafter abbreviated to ‘CM’ and ‘previous message within this session, if any, if sent’ to ‘PM’.

Legend: solid lines indicate control; dashed lines indicate data. Italics indicate grammar production rules from Figure 2.

Figure 3: Flow of control, data & actions in an rsmtp conversation

l’6i’

.ix {rsmtp} flow

If either party detects loss of synchronisation with the other, it drops the connection. .ix two-phase commit .ix synchronisation loss In order to implement a two-phase commit, the usual graceful end to an rsmtp conversation is for T to drop the connection after sending its greeting.

.ix addresses

Envelope addresses are per RFC 821, as modified by later RFCs. Messages are per RFC 822, minus the CRs, .ix CR excluded .ix MIME and the many MIME RFCs, notably the defining ones11,12,23,13,14. The bodies of text messages or textual parts of multipart MIME messages should be in the Unicode character set, encoded as UTF-8, as recommended by the IAB29, and must contain no carriage returns. The rest of the conversation (the envelope) is in UTF-8. .ix utf-8

2.2.1. Round Trips

.ix round~trips

A protocol round trip is the full cycle of T sending data, stopping and waiting for R to reply, R sending back a reply and stopping and waiting for T. (Each individual ‘stop-and-wait’ is also called a ‘line turnaround’.) Let m be the number of messages T tries to send in a single connection, s be the number actually sent (s≦m) and r be the total number of recipients in all those messages. Rsmtp takes fewer protocol round trips: m+1, at most, per connection in rsmtp rather than 2m+s+r+2½ in SMTP. For example, 2 for rsmtp versus 5½+r in the simple case of a successful session transferring only a single message. This leads to fewer packets and less delay. In fact, due to the asynchronous nature of rsmtp, it is possible that each party will find the data it needs to read next already waiting for it in protocol stack buffers because the other party sent it in advance of need.

ESMTP with the .ix ESMTP PIPELINING extension15 negotiated gets the round trips down for purely ASCII messages to 3½ in the singleton case and 2m+2½ in the multiple-message case, though the numbers go up to 2m+c+2½, where c is the number of ‘chunks’ sent on that connection, if the CHUNKING and BINARYMIME extensions28 are negotiated to send binary data (8-bit MIME messages) without high-bit stripping, similar to rsmtp. (CHUNKING is a prerequisite for BINARYMIME.) c is roughly the number of message bytes sent on a connection divided by the chunk size, which is selected by the ESMTP transmitter.

A benefit to needing at most one protocol round trip per message, as opposed to the greater number for SMTP, is that latencies, in the hosts or the network(s) involved, are amplified by round trips, but rsmtp minimises them. The lockstep or ‘stop-and-wait’ nature of most Internet transfer protocols .ix lockstep .ix stop-and-wait contributes to this amplification. Existing major mail relay sites that use SMTP to exchange mail among themselves already find that the delay due to SMTP’s many round trips is a bottleneck in draining their outgoing SMTP queues.

2.2.2. Comparison with SMTP

Rsmtp .ix comparision is superior to SMTP in several respects; the serious comparison is with ESMTP. If CHUNKING, BINARYMIME and PIPELINING are all enabled, .ix ESMTP ESMTP approaches the efficiencies of rsmtp, but rsmtp still takes fewer round trips using a simpler protocol. And with all these hacks (extensions) piled on, ESMTP still has many of the usual SMTP irritations: .ix pointless SMTP CRLFs .ix cabalistic SMTP reply codes .ix SMTP reply codes frippery .ix micro-managed, complex SMTP syntax the pointless CRLFs in the conversation, the cabalistic reply codes, the frippery of the text following non-error reply codes and the multi-line responses, the complex and micro-managed syntax, the sloppy specification of message responsibility hand-off, and no specified means of loop detection. The number of bags nailed on the side of SMTP is really .ix bags nailed on side of SMTP quite astonishing and the sight quite grotesque. Conversion to rsmtp provides a means of, over time, replacing it outright.

In addition, ESMTP must negotiate which extensions it may use on the wire, so decisions about Content-Transfer-Encodings are optimally done in the ESMTP transmitter. .ix unneeded Content-Transfer-Encodings in {rsmtp} .ix ESMTP If negotiations fail and encodings are needed, ESMTP will transmit appreciably more bytes than sending the unencoded message via rsmtp. Given an unencoded MIME message, rsmtp guarantees a clear channel and no need for encoding.

2.2.3. Hand-off of Message Delivery Responsibility

.ix delivery responsibility

Both parties have reliable knowledge of the completeness of delivery. T initially is responsible for delivery of a message. There exists a window in the greeting exchange and envelope transmission in which a dropped connection may cause disagreement about responsibility for delivering any prior message (both parties may briefly claim it). To eliminate the possibility of duplicate messages being sent due to this window, R should retain a history in permanent storage of messages in transit and refuse duplicates by message-id. More history could optionally be retained, say, to prevent duplicates as a result of a transmitting system restarting after a crash.

2.3. The Protocol in Detail

.ix {rsmtp} detail

In the non-message part of the conversation, lines end with a single newline character (128) and ‘linear whitespace’ is permitted only where specified and consists of a single space character (408) in each instance.

2.3.1. Connection & Greetings

.ix {rsmtp} port .ix {rsmtp} detail greetings

Upon connection to R’s rsmtp port (2390 is assigned by the IANA for TCP), R sends a single line greeting containing these strings, separated by a single space each: an rsmtp identification string (‘rsmtp’), a role identifier (‘r’), R’s preferred (or ‘canonical’) fully-qualified domain name (FQDN), and R’s message store name .ix message store (the FQDN of the message store, or set of mailboxes, that R answers for and stores received messages into). The message store name might be as fine-grained as the hostname of a single machine with its own unshared set of mailboxes or as broad as a departmental domain which contains a single message store shared via a network filesystem. If a domain, excluding its subdomains, contains multiple message stores, unless one of them is considered the canonical message store for the domain, then hosts within that domain, again excluding its subdomains, should use fully-qualified hostnames as the message store names for rsmtp. Three common configurations come to mind:

∙ a single Unix� machine with a local store;

∙ a cluster of machines with NFS mounts of the message store (and a working NFS lock daemon!);

∙ a gateway (possibly with no store).

Observe that these are orthogonal; one could have a gateway with a local or imported store that forwards mail destined for other stores.

T responds with a similar greeting line, containing T’s domains and the role identifier of ‘t’. If R’s preferred FQDN is exactly T’s preferred FQDN, T has connected to itself somehow, which is an error. .ix loop detection Left undetected, a mail system could forward mail through itself endlessly in this situation. Similarly, .ix short-circuit delivery .ix message store if R’s message store name is exactly T’s message store name, they share a message store and thus R can deliver the message directly into its local message store and avoid sending it to T, thereby conserving local network bandwidth (a ‘short-circuit local delivery’). This is an optimisation, so degenerating to using hostnames as message store names will only hurt performance, not correctness.

2.3.2. Hand-off of Responsibility & Possible Shutdown

.ix delivery responsibility .ix {rsmtp} shutdown

When T sends a message, it knows the disposition of the PM, if any. Once R reads the message T just sent, R knows that T knows the disposition of any PM, so if R accepted and successfully received the PM, then T, once it has sent its CM, is no longer responsible for delivery of the PM (if any) to the recipients named with it. R accepted (shared) responsibility when it sent back message-reception status; once R has read the message T just sent (the CM), R knows that it has sole responsibility for the PM, if any.

T can send an end-of-session envelope (a single line containing just ‘i’), read PM’s reception status, send a greeting, flush its output buffering, and drop the connection here and R will consider this a graceful shutdown.

2.3.3. Envelope Exchange

If T continues, it sends an envelope, which starts with a line identifying the next message and containing these strings, separated by a single space each: ‘i’ and a globally-unique string (a message-id, though not necessarily an RFC 821/2 Message-ID; syntactically an ADDRESS), the exact byte count of the number of bytes of the message about to be sent, and an indicator of a request for delivery notification. .ix delivery notification The delivery notification cases and their indicators are:

n today’s ‘normal’ case: .ix bounce .ix vacation messages only negative delivery reports (‘bounces’) and automatically-generated responses (‘vacation messages’) are to be returned to the sender.

z the ‘mailing list’ case: no delivery reports are to be returned.

b the ‘automated mailing list’ case: only negative delivery reports are to be returned to the sender, presumably for automatic processing such as deleting users whose mail consistently bounces.

c the ‘control freak manager’ case: .ix control freak manager all delivery reports, positive, negative, indifferent, and automatically-generated are to be returned to the sender. .ix stoned control freaks (Yes, he or she will be stoned by the resulting barrage, and perhaps that’s enough to control overuse.)

It is R’s responsibility to generate these reports (positive ones upon local delivery and negative ones upon any permanent failure) and pass the obligation on to the next system if the message is forwarded. If the obligation can’t be passed on with the message (e.g., the message is to be forwarded via SMTP), R must generate a negative delivery report noting that it cannot be responsible for any further delivery reports for this message. Note that these delivery reports are distinct from .ix read receipts read receipts generated by mail-reading software. .ix privacy Read receipts are obviously a privacy issue and although control freaks like them (despite the resultant stoning), end users ought to have control over generation of them.

R is expected to compare the size in bytes transmitted in the envelope with its free file system space and refuse a message if there is too little file system space to have a reasonable chance of receiving the message. (An ‘end-of-session’ envelope is a single line containing just ‘i’.) In normal (non-end-of-session) envelopes, the next line sent contains an ‘m’ and the message’s MIME Content-Type; ‘text/plain; charset=UTF-8’ .ix utf-8 .ix MIME .ix Content-Type is encouraged for ordinary text. .ix protocol extension It then sends zero or more lines of protocol extension (optext), the contents of which are not yet defined, except that each line must begin with a letter not used as an initial letter in envelope exchanges by either party in this specification (for uniqueness).

T then sends a line consisting of ‘s’ and the sender address; if this address is empty, no bounces, other failure notifications, nor vacation messages are to be returned. T then sends the list of recipients, one per line (rcpts). Each line consists of ‘t’ and a recipient address. The list of recipients, and the envelope itself, is terminated by sending a single empty line.

2.3.4. Line Turnaround During Envelope Exchange

R then sends the envelope status (envstatus), .ix protocol extension which starts with zero or more lines of protocol extension replies (optextrepl), the contents of which are not yet defined, but each line of which begins with the same letter as the corresponding protocol extension request line. These responses appear in the same order as the corresponding requests.

R reads and waits for the envelope T just sent. R then sends status for each recipient (rcpstats), one per line, in the same order as T sent the list of recipients. The first character of each rcpstat is one of

y for a valid recipient,

n followed by a reason for a duplicate recipient or other no-op recipient that will be ignored by R,

f followed by a reason that this user will never be acceptable (e.g., no such user), or

r followed by a suggested retry delay in seconds, a space, and a reason for the temporary failure (e.g., mail forwarding file is mangled).

R then sends a go/no-go indicator for the whole message (sendstatus) on its own line:

Y for ‘go ahead and send’,

N followed by a reason; for ‘don’t send; a no-op’ (e.g., this message is a duplicate),

F followed by a reason; for ‘don’t send; there’s something wrong with this message’ (e.g., no valid recipients),

R followed by a suggested retry delay in seconds, a space, and a reason for a temporary failure (e.g., full disk).

T reads R’s greeting the first time through only. T always reads PM’s reception status (msgstat) and CM’s envstatus.

Note that both the envelope’s list of recipients and envstatus’s list of recipient statuses may be quite long; this is one place where asynchrony can help avoid stopping and waiting. T may send all of envelope and then read all of envstatus, but this will produce a stop-and-wait lock-step. Likewise, R may read the entire envelope then send all of envstatus. The preferred implementation is that both parties send and receive whenever possible. If there is protocol stack buffering (writing data won’t block), each should attempt to send any queued data for the other while also attempting to read and process any data that arrives. Validating addresses can take a little while for R, so T is likely to get ahead of R during envelope transmission. In the preferred implementation, with R incrementally processing recipient addresses, T should read the final recipient’s status from R shortly after sending the final recipient address to R, and T should be able to start sending message (assuming sendstatus was Y) and the envelope of the next message, if any, immediately.

2.3.5. Optional Message Transmission

If and only if R sends back a message status of ‘Y’ (go ahead), the rest of this section (Optional Message Transmission) is performed. T sends the entire message, RFC 822 headers and all, but minus carriage returns in text: just the raw data, unmodified (on advanced systems; on other systems, stripped of carriage returns in text), unstripped, undotted, but in framed fragments. There must be at least one fragment. .ix framed fragments Each fragment consists of a decimal byte count of the number of data bytes in the fragment, a newline, that many data bytes and the synchronisation pattern (‘.’, .ix synchronisation pattern a single dot, with no newline following). The final fragment of a message is zero-length, thus consisting of the bytes 0\n. (in C language string notation). This message should be of the size (in bytes) and MIME Content-Type promised in the envelope. The size may be only an estimate if the message is being generated on the fly and its size is not known in advance. .ix MIME .ix Content-Type It must end with a newline if the body is textual.

R reads the message, then sends back final message reception status (msgstatus) as a line:

y if successful, or

r and a suggested retry delay in seconds, a space and a reason (e.g., disk filled up during reception).

When R sends the status, it knows the disposition of the CM (received successfully or unsuccessfully). If the message was successfully received, even if the connection drops before the next greeting exchange can complete, R should take responsibility for and process the message. When T reads the status, it also will know the disposition.

2.3.6. Start Over

T’s CM becomes its PM. R discards any memory of the PM as ‘in transit’. The whole process may now start over for a new message by going back to ‘Hand-off of Responsibility & Possible Shutdown’.

3. ‘Off-the-Wire’ Considerations

Some details of the protocol are not visible on the wire. Many of these are semantics of parts of the protocol, discussed elsewhere in this paper.

3.1. Address Resolution

We expect that, at least initially, within the Internet, addresses used in rsmtp will typically be resolved as for SMTP: .ix address resolution .ix DNS A records .ix DNS MX records DNS lookup of MX records for the destination domain, followed by DNS lookup of A records for the domains in the MX records (or the destination domain itself if it has no MX records), followed by connecting to each of the resulting IP addresses and attempting delivery until successful connection and delivery (or rejection).

However, some sites may wish to resolve addresses differently, and cooperating sites may use other forms of address or resolution. Rsmtp is agnostic about addressing. One possibility is to connect to the TCP MUX21 port (TCP port 1), or some similar service, and ask the ‘mail redirector’ service where mail should be delivered.

4. Implementation

.ix implementation

All output should be buffered and flushed at line turnarounds (notably after steps T2, T6, R1 and R4) so that, after the initial ‘rsmtp r’ line at the start of a session, each message transferred involves a minimum number of packets and round trips: one half round trip consisting of a message, if any, and the next message’s envelope, and a second half trip consisting of message status and the next message’s envelope’s status.

Rsmtp has been implemented in Limbo on the Inferno operating system .ix {upas} .ix Unix .ix Limbo .ix Plan 9 .ix C and will soon be implemented in C as part of the upas26 mail system on Unix and Plan 924.

Both parties can be sending data at the same time, so to avoid the possibility of deadlock .ix deadlock avoidance caused by exhausting the buffering in the sending and receiving protocol stacks, one may need to use Unix’s select(2) or multiple processes. .ix asynchrony .ix multiple~processes In our experiments, we never were able to consume more than about 120KB (using TCP between Plan 9 systems), which is a large (but finite) buffer for our purposes. In particular, R can deadlock reading recipient addresses and writing statuses of same in one process if T is also single-threaded.

Step T2 may be overlapped with steps T3 and T4.

To encourage the spread of rsmtp, it may be worthwhile for mail systems to attempt a connection by default .ix {rsmtp} opportunistic to a remote machine’s rsmtp port for a few seconds and fall back to using SMTP if there is no quick connection, unless it is known a priori that that machine does not listen for rsmtp.

We believe that spam may be cured by other means ultimately (see Problems Solved Elsewhere), but in the interim, .ix spam we offer these anti-spam suggestions:

∙ check the sender’s domain and IP address against lists of known spammers, for example, Paul Vixie’s ‘black hole’ list at http://www.vix.com/spam and http://maps.vix.com/rbl;

∙ accept mail only for your own domain(s) and those whose MXs are known to point to your machines, to defeat use of your machines as relays for spammers;

∙ periodically summarise mail volume by sender or sending domain to find spammers;

∙ consider (large) quotas on mailboxes. Beware though that these can backfire: don’t do as Microsoft� Exchange� (up to and including at least version 5.5) does and send mail every few minutes to someone telling them that their mailbox is over quota.

5. Sample Conversations

.ix example conversations

Figure 4 is a sample SMTP conversation. Responses are preceded by ‘< ’. All lines end with CRLF pairs. ESMTP substitutes ehlo for helo and spews more 250 responses. This took 8½ protocol round trips; more recipients would take more round trips.

< 220 uunet.ca SMTP blathering, at your service<CR>

  helo bell-labs.com<CR>

< 250 delighted to make your acquaintance, bell-labs.com<CR>

  mail from:<geoff@bell-labs.com><CR>

< 250 did i ever tell you about the time i hitch-hiked to Kiev?<CR>

  rcpt to:<tg@uunet.ca><CR>

< 250 no? well it was a dark and stormy night, back in 19-ought-2,<CR>

  rcpt to:<smd@uunet.ca><CR>

< 250 when i was struck with the desire to get as far from Cheesequake<CR>

  rcpt to:<honey@uunet.ca><CR>

< 550 No such user here, better luck elsewhere.<CR>

  data<CR>

< 354 get on with it!<CR>

  RFC 822 message: 7-bit data (modulo ‘binary’ ESMTP) with hidden dots & CRLFs

  .<CR>

< 250 as is humanly possible.  Flagging down a passing steamship,<CR>

  quit<CR>              one could instead start over at ‘mail from:’ here

< 221 so long, farewell, goodbye<CR>

Figure 4: Sample SMTP conversation

l’6i’

Figure 5 is a sample rsmtp conversation. This took 1½ round trips and more recipients would not increase the number of round trips.

< rsmtp r ghost.uunet.ca uunet.ca

  transmitter does not wait here

  rsmtp t speed.x.bell-labs.com x.bell-labs.com

  i967@x.bell-labs.com 154 n

  mtext/plain; charset=UTF-8

  sgeoff@bell-labs.com

  ttg@uunet.ca

  tsmd@uunet.ca

  thoney@uunet.ca

< y

< y

< fno such user

< Y

  154

  message sent here as raw bytes

  . no newline here

  0

  . no newline here

< y

  i

  rsmtp t speed.x.bell-labs.com x.bell-labs.com

Figure 5: Sample rsmtp conversation

l’6i’

6. Problems Solved Elsewhere

.ix problems~solved~elsewhere .ix privacy .ix authentication

Others are at work solving privacy and possibly authentication at lower layers (e.g., IPsec3,4,5,27, SSL, Inferno’s authenticated or encrypted mounts), .ix SSL .ix Inferno where they can be solved once rather than in every protocol. End-to-end authentication of users is probably best done by PGP2 or something similar; it’s unclear how much the mail transport can help here.

.ix spam

Spam (unsolicited commercial mail) is under attack technologically and in the courts. At the time of writing, one major spammer has been driven out of business by the courts. Technological approaches to the problem may be implemented in an rsmtp receiver, but support for anti-spam measures in the protocol appears to be unnecessary.

SMTP has a .ix SMTP [turn] turn command to reverse the roles of transmitter and receiver. It reduces security somewhat. The DNS forward map for .ix DNS a domain will usually be correct, of necessity, but the reverse map may be missing or wrong, due to error or malice. The forward map is used to place a call, yet the caller’s own reverse map may be wrong or missing, so upon line turnaround, the (now) transmitter can’t have much confidence that the (now) receiver is who it claims to be. The initial motivation for turn was, at least in part, the perceived high cost of establishing connections (e.g., via low-speed modem); the cost doesn’t seem so great now with higher bandwidth networks available and the tuned TCP implementations in use. Also, with SMTP implementations generally placing calls to deliver mail on demand, turn isn’t likely to find any work to do. Many SMTP implementations omit turn and we omit it from our protocol.

SMTP includes vrfy and expn .ix SMTP [vrfy] .ix SMTP [expn] commands to, respectively, verify and expand mailbox names (‘local-parts’). Part of the motivation for them may have been the idea (as in X.400) that people would try to guess each other’s userids, which isn’t done in practice. These commands are arguably security and privacy violations, do not implement mail transport, have often not been implemented, and assume a certain model of the mail system that in turn encourages security exposure. .ix security .ix privacy Some network mail receivers accept all plausible mail and leave it up to the local mail routing software to expand aliases, decide if user names are valid, etc. We omit any form of these SMTP commands.

7. Future Directions

.ix future~directions

Most of the Internet transfer protocols, including this one, seem ripe for replacement by a good remote file system protocol. .ix remote file system protocols .ix transfer protocols .ix message store For mail, mounting the remote message store, with appropriate access rights, and depositing the message(s) to be transferred directly into the remote message store seems the right approach. Mounting a remote message store on /msg and reading /msg/netnew might create a new directory, say /msg/123 containing at least a file named ctl. Writing an acceptable envelope into /msg/123/ctl might make /msg/123/data appear or become writable by T, ready for T to drop the message in.

There are still some problems to be solved with remote file systems: how to authenticate and perhaps map userids to make remote file systems work, at least minimally, between completely disparate user communities seem the most glaring. Latency is also an issue for some RPC-based network file systems when operating over great distances.

8. Security Considerations

See .ix security Problems Solved Elsewhere.

9. Acknowledgements

.ix acknowledgements

Discussions with Mark Moraes, .ix Mark Moraes Michael Scott Baldwin, .ix Michael Scott Baldwin and Paul Lustgarten .ix Paul Lustgarten helped flesh out details of rsmtp. In particular, Michael explained the use of two-phase commit. Paul also read drafts of this document and provided some insider history. Mark provided insight, particularly with respect to spam and latency, gleaned from operating a large mail relay. Greg Kochanski suggested how to shave off a single read (pause) .ix Greg Kochanski and that led to a rethinking that produced the current protocol of at most one round trip per message. Norman Wilson suggested the use of framed fragments to permit .ix Norman Wilson on-the-fly fabrication of messages whose size is not known a priori.

10. References

As always when dealing with the Internet, .ix references .ix rfcs .ix out-of-date rfcs all RFCs cited may have been updated or replaced by later ones.

1. ISO/IEC DIS 10646-1:1993: Information technology - Universal Multiple-Octet Coded Character Set (UCS) — Part 1: Architecture and Basic Multilingual Plane, ISO (1993).

2. D. Atkins, W. Stallings, and P. Zimmermann, RFC 1991: PGP Message Exchange Formats, ISI, Marina Del Ray, CA (August 1996).

3. R. Atkinson, RFC 1825: Security Architecture for the Internet Protocol, ISI, Marina Del Ray, CA (August 1995).

4. R. Atkinson, RFC 1826: IP Authentication Header, ISI, Marina Del Ray, CA (August 1995).

5. R. Atkinson, RFC 1827: IP Encapsulating Security Payload (ESP), ISI, Marina Del Ray, CA (August 1995).

6. R. Braden (ed), RFC 1123: Requirements for Internet Hosts -- Application and Support, ISI, Marina Del Ray, CA (October 1, 1989).

7. R. Braden (ed), RFC 1122: Requirements for Internet Hosts -- Communication Layers, ISI, Marina Del Ray, CA (October 1, 1989).

8. The Unicode Consortium, The Unicode Standard, Version 2.0, Addison-Wesley, New York (1996).

9. D. Crocker, N. Freed, and A. Cargille, RFC 1845: SMTP Service Extension for Checkpoint/Restart, ISI, Marina Del Ray, CA (September 1995).

10. A. Durand and F. Dupont, RFC 1846: SMTP 521 Reply Code, ISI, Marina Del Ray, CA (September 1995).

11. N. Freed and N. Borenstein, RFC 2045: Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, ISI, Marina Del Ray, CA (November 1996).

12. N. Freed and N. Borenstein, RFC 2046: Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types, ISI, Marina Del Ray, CA (November 1996).

13. N. Freed, J. Klensin, and J. Postel, RFC 2048: Multipurpose Internet Mail Extensions (MIME) Part Four: Registration Procedures, ISI, Marina Del Ray, CA (November 1996).

14. N. Freed and N. Borenstein, RFC 2049: Multipurpose Internet Mail Extensions (MIME) Part Five: Conformance Criteria and Examples, ISI, Marina Del Ray, CA (November 1996).

15. N. Freed, RFC 2197: SMTP Service Extension for Command Pipelining, ISI, Marina Del Ray, CA (September 1997).

16. S. C. Johnson, ‘‘Yacc — Yet Another Compiler-Compiler,’’ Comp. Sci. Tech. Rep. No. 32, Bell Laboratories, Murray Hill, New Jersey (July 1975).

17. Brian Kantor and Phil Lapsley, RFC 977: Network News Transfer Protocol, ISI, Marina Del Ray, CA (February 1986).

18. J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker, RFC 1652: SMTP Service Extension for 8bit-MIMEtransport, ISI, Marina Del Ray, CA (July 1994).

19. J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker, RFC 1869: SMTP Service Extensions, ISI, Marina Del Ray, CA (November 1995).

20. J. Klensin, N. Freed, and K. Moore, RFC 1870: SMTP Service Extension for Message Size Declaration, ISI, Marina Del Ray, CA (November 1995).

21. M. Lottor, RFC 1078: TCP Port Service Multiplexer (TCPMUX), ISI, Marina Del Ray, CA (November 1, 1988).

22. K. Moore, RFC 1891: SMTP Service Extension for Delivery Status Notifications, ISI, Marina Del Ray, CA (January 1996).

23. K. Moore, RFC 2047: MIME (Multipurpose Internet Mail Extensions) Part Three: Message Header Extensions for Non-ASCII Text, ISI, Marina Del Ray, CA (November 1996).

24. Rob Pike, Dave Presotto, Ken Thompson, and Howard Trickey, ‘‘Plan 9 from Bell Labs,’’ Proc. of the Summer 1990 UKUUG Conf., London, pp. 1-9} (July, 1990).

25. Jonathan B. Postel, RFC 821: Simple Mail Transfer Protocol, ISI, Marina Del Ray, CA (August 1982).

26. David L. Presotto, ‘‘Upas - a simpler approach to network mail,’’ pp. 533-538 in USENIX Conference Proceedings, USENIX, Portland, OR (Summer 1985).

27. various} , ftp://ietf.org/internet-drafts/*sec*} .

28. G. Vaudreuil, RFC 1830: SMTP Service Extensions for Transmission of Large and Binary MIME Messages, ISI, Marina Del Ray, CA (August 1995).

29. C. Weider, C. Preston, K. Simonsen, H. Alvestrand, R. Atkinson, M. Crispin, and P. Svanberg, RFC 2130: The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996, ISI, Marina Del Ray, CA (April 1997).

30. X/Open} , File System Safe UCS Transformation Format (FSS-UTF), X/Open Preliminary Specification} , ISO designation is ISO/IEC JTC1/SC2/WG2 N 1036, dated 1994-08-01. } 1993} .

31. F. Yergeau, RFC 2279: UTF-8, a transformation format of ISO 10646, ISI, Marina Del Ray, CA (January 1998).