idnits 2.17.00 (12 Aug 2021) /tmp/idnits38107/draft-leiba-imap-implement-guide-08.txt: ** The Abstract section seems to be numbered Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Cannot find the required boilerplate sections (Copyright, IPR, etc.) in this document. Expected boilerplate is as follows today (2022-05-20) according to https://trustee.ietf.org/license-info : IETF Trust Legal Provisions of 28-dec-2009, Section 6.a: This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 2: Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 3: This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** Missing expiration date. The document expiration date should appear on the first and last page. ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity. ** The document seems to lack a 1id_guidelines paragraph about the list of current Internet-Drafts. ** The document seems to lack a 1id_guidelines paragraph about the list of Shadow Directories. ** The document is more than 15 pages and seems to lack a Table of Contents. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an Introduction section. ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** The document seems to lack separate sections for Informative/Normative References. All references will be assumed normative when checking for downward references. ** There is 1 instance of too long lines in the document, the longest one being 19 characters in excess of 72. ** The abstract seems to contain references ([RFC-2060]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 1999) is 8375 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'UIDVALIDITY 824708485' is mentioned on line 274, but not defined == Missing Reference: 'UNSEEN 9921' is mentioned on line 275, but not defined == Missing Reference: 'UNSEEN' is mentioned on line 280, but not defined == Missing Reference: 'TRYCREATE' is mentioned on line 473, but not defined == Missing Reference: 'READ-ONLY' is mentioned on line 475, but not defined == Missing Reference: 'UIDVALIDITY 12345' is mentioned on line 618, but not defined -- Looks like a reference, but probably isn't: '1' on line 674 == Unused Reference: 'UTF-7' is defined on line 986, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2060 (Obsoleted by RFC 3501) ** Downref: Normative reference to an Informational RFC: RFC 2180 ** Obsolete normative reference: RFC 2044 (ref. 'UTF-8') (Obsoleted by RFC 2279) ** Downref: Normative reference to an Informational RFC: RFC 2152 (ref. 'UTF-7') -- Possible downref: Non-RFC (?) normative reference: ref. 'NAMESPACE' Summary: 17 errors (**), 0 flaws (~~), 8 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group B. Leiba 3 Internet Draft IBM T.J. Watson Research Center 4 Document: draft-leiba-imap-implement-guide-08.txt December 1998 5 Expires June 1999 7 IMAP4 Implementation Recommendations 9 Status of this Document 11 This document provides information for the Internet community. This 12 document does not specify an Internet standard of any kind. 13 Distribution of this document is unlimited. 15 This document is an Internet Draft. Internet Drafts are working 16 documents of the Internet Engineering Task Force (IETF), its Areas, 17 and its Working Groups. Note that other groups may also distribute 18 working documents as Internet Drafts. 20 Internet Drafts are draft documents valid for a maximum of six 21 months. Internet Drafts may be updated, replaced, or obsoleted by 22 other documents at any time. It is not appropriate to use Internet 23 Drafts as reference material or to cite them other than as a "working 24 draft" or "work in progress". 26 To view the entire list of current Internet-Drafts, please check the 27 "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow 28 Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern 29 Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific 30 Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). 32 A revised version of this draft document will be submitted to the RFC 33 editor. Discussion and suggestions for improvement are requested. 34 This document will expire by the end of June 1999. 36 1. Abstract 38 The IMAP4 specification [RFC-2060] describes a rich protocol for use 39 in building clients and servers for storage, retrieval, and 40 manipulation of electronic mail. Because the protocol is so rich and 41 has so many implementation choices, there are often trade-offs that 42 must be made and issues that must be considered when designing such 43 clients and servers. This document attempts to outline these issues 44 and to make recommendations in order to make the end products as 45 interoperable as possible. 47 Internet DRAFT Implementation Recommendations December 1998 49 2. Conventions used in this document 51 In examples, "C:" indicates lines sent by a client that is connected 52 to a server. "S:" indicates lines sent by the server to the client. 54 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 55 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 56 document are to be interpreted as described in [RFC-2119]. 58 3. Interoperability Issues and Recommendations 60 3.1. Accessibility 62 This section describes the issues related to access to servers and 63 server resources. Concerns here include data sharing and maintenance 64 of client/server connections. 66 3.1.1. Multiple Accesses of the Same Mailbox 68 One strong point of IMAP4 is that, unlike POP3, it allows for 69 multiple simultaneous access to a single mailbox. A user can, thus, 70 read mail from a client at home while the client in the office is 71 still connected; or the help desk staff can all work out of the same 72 inbox, all seeing the same pool of questions. An important point 73 about this capability, though is that NO SERVER IS GUARANTEED TO 74 SUPPORT THIS. If you are selecting an IMAP server and this facility 75 is important to you, be sure that the server you choose to install, 76 in the configuration you choose to use, supports it. 78 If you are designing a client, you MUST NOT assume that you can 79 access the same mailbox more than once at a time. That means 80 1. you MUST handle gracefully the failure of a SELECT command if 81 the server refuses the second SELECT, 82 2. you MUST handle reasonably the severing of your connection (see 83 "Severed Connections", below) if the server chooses to allow the 84 second SELECT by forcing the first off, 85 3. you MUST avoid making multiple connections to the same mailbox 86 in your own client (for load balancing or other such reasons), 87 and 88 4. you MUST avoid using the STATUS command on a mailbox that you 89 have selected (with some server implementations the STATUS 90 command has the same problems with multiple access as do the 91 SELECT and EXAMINE commands). 93 A further note about STATUS: The STATUS command is sometimes used to 94 check a non-selected mailbox for new mail. This mechanism MUST NOT 95 be used to check for new mail in the selected mailbox; section 5.2 of 97 Internet DRAFT Implementation Recommendations December 1998 99 [RFC-2060] specifically forbids this in its last paragraph. Further, 100 since STATUS takes a mailbox name it is an independent operation, not 101 operating on the selected mailbox. Because of this, the information 102 it returns is not necessarily in synchronization with the selected 103 mailbox state. 105 3.1.2. Severed Connections 107 The client/server connection may be severed for one of three reasons: 108 the client severs the connection, the server severs the connection, 109 or the connection is severed by outside forces beyond the control of 110 the client and the server (a telephone line drops, for example). 111 Clients and servers must both deal with these situations. 113 When the client wants to sever a connection, it's usually because it 114 has finished the work it needed to do on that connection. The client 115 SHOULD send a LOGOUT command, wait for the tagged response, and then 116 close the socket. But note that, while this is what's intended in 117 the protocol design, there isn't universal agreement here. Some 118 contend that sending the LOGOUT and waiting for the two responses 119 (untagged BYE and tagged OK) is wasteful and unnecessary, and that 120 the client can simply close the socket. The server should interpret 121 the closed socket as a log out by the client. The counterargument is 122 that it's useful from the standpoint of cleanup, problem 123 determination, and the like, to have an explicit client log out, 124 because otherwise there is no way for the server to tell the 125 difference between "closed socket because of log out" and "closed 126 socket because communication was disrupted". If there is a 127 client/server interaction problem, a client which routinely 128 terminates a session by breaking the connection without a LOGOUT will 129 make it much more difficult to determine the problem. 131 Because of this disagreement, server designers must be aware that 132 some clients might close the socket without sending a LOGOUT. In any 133 case, whether or not a LOGOUT was sent, the server SHOULD NOT 134 implicitly expunge any messages from the selected mailbox. If a 135 client wants the server to do so, it MUST send a CLOSE or EXPUNGE 136 command explicitly. 138 When the server wants to sever a connection it's usually due to an 139 inactivity timeout or is because a situation has arisen that has 140 changed the state of the mail store in a way that the server can not 141 communicate to the client. The server SHOULD send an untagged BYE 142 response to the client and then close the socket. Sending an 143 untagged BYE response before severing allows the server to send a 144 human-readable explanation of the problem to the client, which the 145 client may then log, display to the user, or both (see section 7.1.5 146 of [RFC-2060]). 148 Internet DRAFT Implementation Recommendations December 1998 150 Regarding inactivity timeouts, there is some controversy. Unlike 151 POP, for which the design is for a client to connect, retrieve mail, 152 and log out, IMAPs design encourages long-lived (and mostly 153 inactive) client/server sessions. As the number of users grows, this 154 can use up a lot of server resources, especially with clients that 155 are designed to maintain sessions for mailboxes that the user has 156 finished accessing. To alleviate this, a server MAY implement an 157 inactivity timeout, unilaterally closing a session (after first 158 sending an untagged BYE, as noted above). Some server operators have 159 reported dramatic improvements in server performance after doing 160 this. As specified in [RFC-2060], if such a timeout is done it MUST 161 NOT be until at least 30 minutes of inactivity. The reason for this 162 specification is to prevent clients from sending commands (such as 163 NOOP) to the server at frequent intervals simply to avert a too-early 164 timeout. If the client knows that the server may not time out the 165 session for at least 30 minutes, then the client need not poll at 166 intervals more frequent than, say, 25 minutes. 168 3.2. Scaling 170 IMAP4 has many features that allow for scalability, as mail stores 171 become larger and more numerous. Large numbers of users, mailboxes, 172 and messages, and very large messages require thought to handle 173 efficiently. This document will not address the administrative 174 issues involved in large numbers of users, but we will look at the 175 other items. 177 3.2.1. Flood Control 179 There are three situations when a client can make a request that will 180 result in a very large response - too large for the client reasonably 181 to deal with: there are a great many mailboxes available, there are a 182 great many messages in the selected mailbox, or there is a very large 183 message part. The danger here is that the end user will be stuck 184 waiting while the server sends (and the client processes) an enormous 185 response. In all of these cases there are things a client can do to 186 reduce that danger. 188 There is also the case where a client can flood a server, by sending 189 an arbitratily long command. We'll discuss that issue, too, in this 190 section. 192 3.2.1.1. Listing Mailboxes 194 Some servers present Usenet newsgroups to IMAP users. Newsgroups, 195 and other such hierarchical mailbox structures, can be very numerous 197 Internet DRAFT Implementation Recommendations December 1998 199 but may have only a few entries at the top level of hierarchy. Also, 200 some servers are built against mail stores that can, unbeknownst to 201 the server, have circular hierarchies - that is, it's possible for 202 "a/b/c/d" to resolve to the same file structure as "a", which would 203 then mean that "a/b/c/d/b" is the same as "a/b", and the hierarchy 204 will never end. The LIST response in this case will be unlimited. 206 Clients that will have trouble with this are those that use 207 C: 001 LIST "" * 208 to determine the mailbox list. Because of this, clients SHOULD NOT 209 use an unqualified "*" that way in the LIST command. A safer 210 approach is to list each level of hierarchy individually, allowing 211 the user to traverse the tree one limb at a time, thus: 213 C: 001 LIST "" % 214 S: * LIST () "/" Banana 215 S: * LIST ...etc... 216 S: 001 OK done 217 and then 218 C: 002 LIST "" Banana/% 219 S: * LIST () "/" Banana/Apple 220 S: * LIST ...etc... 221 S: 002 OK done 223 Using this technique the client's user interface can give the user 224 full flexibility without choking on the voluminous reply to "LIST *". 225 Of course, it is still possible that the reply to 226 C: 005 LIST "" alt.fan.celebrity.% 227 may be thousands of entries long, and there is, unfortunately, 228 nothing the client can do to protect itself from that. This has not 229 yet been a notable problem. 231 Servers that may export circular hierarchies (any server that 232 directly presents a UNIX file system, for instance) SHOULD limit the 233 hierarchy depth to prevent unlimited LIST responses. A suggested 234 depth limit is 20 hierarchy levels. 236 3.2.1.2. Fetching the List of Messages 238 When a client selects a mailbox, it is given a count, in the untagged 239 EXISTS response, of the messages in the mailbox. This number can be 240 very large. In such a case it might be unwise to use 241 C: 004 FETCH 1:* ALL 242 to populate the user's view of the mailbox. One good method to avoid 243 problems with this is to batch the requests, thus: 245 Internet DRAFT Implementation Recommendations December 1998 247 C: 004 FETCH 1:50 ALL 248 S: * 1 FETCH ...etc... 249 S: 004 OK done 250 C: 005 FETCH 51:100 ALL 251 S: * 51 FETCH ...etc... 252 S: 005 OK done 253 C: 006 FETCH 101:150 ALL 254 ...etc... 256 Using this method, another command, such as "FETCH 6 BODY[1]" can be 257 inserted as necessary, and the client will not have its access to the 258 server blocked by a storm of FETCH replies. (Such a method could be 259 reversed to fetch the LAST 50 messages first, then the 50 prior to 260 that, and so on.) 262 As a smart extension of this, a well designed client, prepared for 263 very large mailboxes, will not automatically fetch data for all 264 messages AT ALL. Rather, the client will populate the user's view 265 only as the user sees it, possibly pre-fetching selected information, 266 and only fetching other information as the user scrolls to it. For 267 example, to select only those messages beginning with the first 268 unseen one: 270 C: 003 SELECT INBOX 271 S: * 10000 EXISTS 272 S: * 80 RECENT 273 S: * FLAGS (\Answered \Flagged \Deleted \Draft \Seen) 274 S: * OK [UIDVALIDITY 824708485] UID validity status 275 S: * OK [UNSEEN 9921] First unseen message 276 S: 003 OK [READ-WRITE] SELECT completed 277 C: 004 FETCH 9921:* ALL 278 ... etc... 280 If the server does not return an OK [UNSEEN] response, the client may 281 use SEARCH UNSEEN to obtain that value. 283 This mechanism is good as a default presentation method, but only 284 works well if the default message order is acceptable. A client may 285 want to present various sort orders to the user (by subject, by date 286 sent, by sender, and so on) and in that case (lacking a SORT 287 extension on the server side) the client WILL have to retrieve all 288 message descriptors. A client that provides this service SHOULD NOT 289 do it by default and SHOULD inform the user of the costs of choosing 290 this option for large mailboxes. 292 3.2.1.3. Fetching a Large Body Part 294 The issue here is similar to the one for a list of messages. In the 296 Internet DRAFT Implementation Recommendations December 1998 298 BODYSTRUCTURE response the client knows the size, in bytes, of the 299 body part it plans to fetch. Suppose this is a 70 MB video clip. 300 The client can use partial fetches to retrieve the body part in 301 pieces, avoiding the problem of an uninterruptible 70 MB literal 302 coming back from the server: 304 C: 022 FETCH 3 BODY[1]<0.20000> 305 S: * 3 FETCH (FLAGS(\Seen) BODY[1]<0> {20000} 306 S: ...data...) 307 S: 022 OK done 308 C: 023 FETCH 3 BODY[1]<20001.20000> 309 S: * 3 FETCH (BODY[1]<20001> {20000} 310 S: ...data...) 311 S: 023 OK done 312 C: 024 FETCH 3 BODY[1]<40001.20000> 313 ...etc... 315 3.2.1.4. BODYSTRUCTURE vs. Entire Messages 317 Because FETCH BODYSTRUCTURE is necessary in order to determine the 318 number of body parts, and, thus, whether a message has "attachments", 319 clients often use FETCH FULL as their normal method of populating the 320 user's view of a mailbox. The benefit is that the client can display 321 a paperclip icon or some such indication along with the normal 322 message summary. However, this comes at a significant cost with some 323 server configurations. The parsing needed to generate the FETCH 324 BODYSTRUCTURE response may be time-consuming compared with that 325 needed for FETCH ENVELOPE. The client developer should consider this 326 issue when deciding whether the ability to add a paperclip icon is 327 worth the tradeoff in performance, especially with large mailboxes. 329 Some clients, rather than using FETCH BODYSTRUCTURE, use FETCH BODY[] 330 (or the equivalent FETCH RFC822) to retrieve the entire message. 331 They then do the MIME parsing in the client. This may give the 332 client slightly more flexibility in some areas (access, for instance, 333 to header fields that aren't returned in the BODYSTRUCTURE and 334 ENVELOPE responses), but it can cause severe performance problems by 335 forcing the transfer of all body parts when the user might only want 336 to see some of them - a user logged on by modem and reading a small 337 text message with a large ZIP file attached may prefer to read the 338 text only and save the ZIP file for later. Therefore, a client 339 SHOULD NOT normally retrieve entire messages and SHOULD retrieve 340 message body parts selectively. 342 3.2.1.5. Long Command Lines 344 A client can wind up building a very long command line in an effort 346 Internet DRAFT Implementation Recommendations December 1998 348 to try to be efficient about requesting information from a server. 349 This can typically happen when a client builds a message set from 350 selected messages and doesn't recognise that contiguous blocks of 351 messages may be group in a range. Suppose a user selects all 10,000 352 messages in a large mailbox and then unselects message 287. The 353 client could build that message set as "1:286,288:10000", but a 354 client that doesn't handle that might try to enumerate each message 355 individually and build "1,2,3,4, [and so on] ,9999,10000". Adding 356 that to the fetch command results in a command line that's almost 357 49,000 octets long, and, clearly, one can construct a command line 358 that's even longer. 360 A client SHOULD limit the length of the command lines it generates to 361 approximately 1000 octets (including all quoted strings but not 362 including literals). If the client is unable to group things into 363 ranges so that the command line is within that length, it SHOULD 364 split the request into multiple commands. The client SHOULD use 365 literals instead of long quoted strings, in order to keep the command 366 length down. 368 For its part, a server SHOULD allow for a command line of at least 369 8000 octets. This provides plenty of leeway for accepting reasonable 370 length commands from clients. The server SHOULD send a BAD response 371 to a command that does not end within the server's maximum accepted 372 command length. 374 3.2.2. Subscriptions 376 The client isn't the only entity that can get flooded: the end user, 377 too, may need some flood control. The IMAP4 protocol provides such 378 control in the form of subscriptions. Most servers support the 379 SUBSCRIBE, UNSUBSCRIBE, and LSUB commands, and many users choose to 380 narrow down a large list of available mailboxes by subscribing to the 381 ones that they usually want to see. Clients, with this in mind, 382 SHOULD give the user a way to see only subscribed mailboxes. A 383 client that never uses the LSUB command takes a significant usability 384 feature away from the user. Of course, the client would not want to 385 hide the LIST command completely; the user needs to have a way to 386 choose between LIST and LSUB. The usual way to do this is to provide 387 a setting like "show which mailboxes?: [] all [] subscribed only". 389 3.2.3. Searching 391 IMAP SEARCH commands can become particularly troublesome (that is, 392 slow) on mailboxes containing a large number of messages. So let's 393 put a few things in perspective in that regard. 395 Internet DRAFT Implementation Recommendations December 1998 397 The flag searches SHOULD be fast. The flag searches (ALL, [UN]SEEN, 398 [UN]ANSWERED, [UN]DELETED, [UN]DRAFT, [UN]FLAGGED, NEW, OLD, RECENT) 399 are known to be used by clients for the client's own use (for 400 instance, some clients use "SEARCH UNSEEN" to find unseen mail and 401 "SEARCH DELETED" to warn the user before expunging messages). 403 Other searches, particularly the text searches (HEADER, TEXT, BODY) 404 are initiated by the user, rather than by the client itself, and 405 somewhat slower performance can be tolerated, since the user is aware 406 that the search is being done (and is probably aware that it might be 407 time-consuming). A smart server might use dynamic indexing to speed 408 commonly used text searches. 410 The client MAY allow other commands to be sent to the server while a 411 SEARCH is in progress, but at the time of this writing there is 412 little or no server support for parallel processing of multiple 413 commands in the same session (and see "Multiple Accesses of the Same 414 Mailbox" above for a description of the dangers of trying to work 415 around this by doing your SEARCH in another session). 417 Another word about text searches: some servers, built on database 418 back-ends with indexed search capabilities, may return search results 419 that do not match the IMAP spec's "case-insensitive substring" 420 requirements. While these servers are in violation of the protocol, 421 there is little harm in the violation as long as the search results 422 are used only in response to a user's request. Still, developers of 423 such servers should be aware that they ARE violating the protocol, 424 should think carefully about that behaviour, and MUST be certain that 425 their servers respond accurately to the flag searches for the reasons 426 outlined above. 428 In addition, servers SHOULD support CHARSET UTF-8 [UTF-8] in 429 searches. 431 3.3 Avoiding Invalid Requests 433 IMAP4 provides ways for a server to tell a client in advance what is 434 and isn't permitted in some circumstances. Clients SHOULD use these 435 features to avoid sending requests that a well designed client would 436 know to be invalid. This section explains this in more detail. 438 3.3.1. The CAPABILITY Command 440 All IMAP4 clients SHOULD use the CAPABILITY command to determine what 441 version of IMAP and what optional features a server supports. The 442 client SHOULD NOT send IMAP4rev1 commands and arguments to a server 443 that does not advertize IMAP4rev1 in its CAPABILITY response. 445 Internet DRAFT Implementation Recommendations December 1998 447 Similarly, the client SHOULD NOT send IMAP4 commands that no longer 448 exist in IMAP4rev1 to a server that does not advertize IMAP4 in its 449 CAPABILITY response. An IMAP4rev1 server is NOT required to support 450 obsolete IMAP4 or IMAP2bis commands (though some do; do not let this 451 fact lull you into thinking that it's valid to send such commands to 452 an IMAP4rev1 server). 454 A client SHOULD NOT send commands to probe for the existance of 455 certain extensions. All standard and standards-track extensions 456 include CAPABILITY tokens indicating their presense. All private and 457 experimental extensions SHOULD do the same, and clients that take 458 advantage of them SHOULD use the CAPABILITY response to determine 459 whether they may be used or not. 461 3.3.2. Don't Do What the Server Says You Can't 463 In many cases, the server, in response to a command, will tell the 464 client something about what can and can't be done with a particular 465 mailbox. The client SHOULD pay attention to this information and 466 SHOULD NOT try to do things that it's been told it can't do. 467 Examples: 468 * Do not try to SELECT a mailbox that has the \Noselect flag set. 469 * Do not try to CREATE a sub-mailbox in a mailbox that has the 470 \Noinferiors flag set. 471 * Do not respond to a failing COPY or APPEND command by trying to 472 CREATE the target mailbox if the server does not respond with a 473 [TRYCREATE] response code. 474 * Do not try to expunge a mailbox that has been selected with the 475 [READ-ONLY] response code. 477 3.4. Miscellaneous Protocol Considerations 479 We describe here a number of important protocol-related issues, the 480 misunderstanding of which has caused significant interoperability 481 problems in IMAP4 implementations. One general item is that every 482 implementer should be certain to take note of and to understand 483 section 2.2.2 and the preamble to section 7 of the IMAP4rev1 spec 484 [RFC-2060]. 486 3.4.1. Well Formed Protocol 488 We cannot stress enough the importance of adhering strictly to the 489 protocol grammar. The specification of the protocol is quite rigid; 490 do not assume that you can insert blank space for "readability" if 491 none is called for. Keep in mind that there are parsers out there 492 that will crash if there are protocol errors. There are clients that 494 Internet DRAFT Implementation Recommendations December 1998 496 will report every parser burp to the user. And in any case, 497 information that cannot be parsed is information that is lost. Be 498 careful in your protocol generation. And see "A Word About Testing", 499 below. 501 In particular, note that the string in the INTERNALDATE response is 502 NOT an RFC-822 date string - that is, it is not in the same format as 503 the first string in the ENVELOPE response. Since most clients will, 504 in fact, accept an RFC-822 date string in the INTERNALDATE response, 505 it's easy to miss this in your interoperability testing. But it will 506 cause a problem with some client, so be sure to generate the correct 507 string for this field. 509 3.4.2. Special Characters 511 Certain characters, currently the double-quote and the backslash, may 512 not be sent as-is inside a quoted string. These characters MUST be 513 preceded by the escape character if they are in a quoted string, or 514 else the string MUST be sent as a literal. Both clients and servers 515 MUST handle this, both on output (they MUST send these characters 516 properly) and on input (they MUST be able to receive escaped 517 characters in quoted strings). Example: 519 C: 001 LIST "" % 520 S: * LIST () "" INBOX 521 S: * LIST () "\\" TEST 522 S: * LIST () "\\" {12} 523 S: "My" mailbox 524 S: 001 OK done 525 C: 002 LIST "" "\"My\" mailbox\\%" 526 S: * LIST () "\\" {17} 527 S: "My" mailbox\Junk 528 S: 002 OK done 530 Note that in the example the server sent the hierarchy delimiter as 531 an escaped character in the quoted string and sent the mailbox name 532 containing imbedded double-quotes as a literal. The client used only 533 quoted strings, escaping both the backslash and the double-quote 534 characters. 536 The CR and LF characters may be sent ONLY in literals; they are not 537 allowed, even if escaped, inside quoted strings. 539 And while we're talking about special characters: the IMAP spec, in 540 the section titled "Mailbox International Naming Convention", 541 describes how to encode mailbox names in modified UTF-7 [UTF-7 and 542 RFC-2060]. Implementations MUST adhere to this in order to be 543 interoperable in the international market, and servers SHOULD 545 Internet DRAFT Implementation Recommendations December 1998 547 validate mailbox names sent by client and reject names that do not 548 conform. 550 3.4.3. UIDs and UIDVALIDITY 552 Servers that support existing back-end mail stores often have no good 553 place to save UIDs for messages. Often the existing mail store will 554 not have the concept of UIDs in the sense that IMAP has: strictly 555 increasing, never re-issued, 32-bit integers. Some servers solve 556 this by storing the UIDs in a place that's accessible to end users, 557 allowing for the possibility that the users will delete them. Others 558 solve it by re-assigning UIDs every time a mailbox is selected. 560 The server SHOULD maintain UIDs permanently for all messages if it 561 can. If that's not possible, the server MUST change the UIDVALIDITY 562 value for the mailbox whenever any of the UIDs may have become 563 invalid. Clients MUST recognize that the UIDVALIDITY has changed and 564 MUST respond to that condition by throwing away any information that 565 they have saved about UIDs in that mailbox. There have been many 566 problems in this area when clients have failed to do this; in the 567 worst case it will result in loss of mail when a client deletes the 568 wrong piece of mail by using a stale UID. 570 It seems to be a common misunderstanding that "the UIDVALIDITY and 571 the UID, taken together, form a 64-bit identifier that uniquely 572 identifies a message on a server". This is absolutely NOT TRUE. 573 There is no assurance that the UIDVALIDITY values of two mailboxes be 574 different, so the UIDVALIDITY in no way identifies a mailbox. The 575 ONLY purpose of UIDVALIDITY is, as its name indicates, to give the 576 client a way to check the validity of the UIDs it has cached. While 577 it is a valid implementation choice to put these values together to 578 make a 64-bit identifier for the message, the important concept here 579 is that UIDs are not unique between mailboxes; they are only unique 580 WITHIN a given mailbox. 582 Some server implementations have attempted to make UIDs unique across 583 the entire server. This is inadvisable, in that it limits the life 584 of UIDs unnecessarily. The UID is a 32-bit number and will run out 585 in reasonably finite time if it's global across the server. If you 586 assign UIDs sequentially in one mailbox, you will not have to start 587 re-using them until you have had, at one time or another, 2**32 588 different messages in that mailbox. In the global case, you will 589 have to reuse them once you have had, at one time or another, 2**32 590 different messages in the entire mail store. Suppose your server has 591 around 8000 users registered (2**13). That gives an average of 2**19 592 UIDs per user. Suppose each user gets 32 messages (2**5) per day. 593 That gives you 2**14 days (16000+ days = about 45 years) before you 594 run out. That may seem like enough, but multiply the usage just a 596 Internet DRAFT Implementation Recommendations December 1998 598 little (a lot of spam, a lot of mailing list subscriptions, more 599 users) and you limit yourself too much. 601 What's worse is that if you have to wrap the UIDs, and, thus, you 602 have to change UIDVALIDITY and invalidate the UIDs in the mailbox, 603 you have to do it for EVERY mailbox in the system, since they all 604 share the same UID pool. If you assign UIDs per mailbox and you have 605 a problem, you only have to kill the UIDs for that one mailbox. 607 Under extreme circumstances (and this is extreme, indeed), the server 608 may have to invalidate UIDs while a mailbox is in use by a client - 609 that is, the UIDs that the client knows about in its active mailbox 610 are no longer valid. In that case, the server MUST immediately 611 change the UIDVALIDITY and MUST communicate this to the client. The 612 server MAY do this by sending an unsolicited UIDVALIDITY message, in 613 the same form as in response to the SELECT command. Clients MUST be 614 prepared to handle such a message and the possibly coincident failure 615 of the command in process. For example: 617 C: 032 UID STORE 382 +Flags.silent \Deleted 618 S: * OK [UIDVALIDITY 12345] New UIDVALIDITY value! 619 S: 032 NO UID command rejected because UIDVALIDITY changed! 620 C: ...invalidates local information and re-fetches... 621 C: 033 FETCH 1:* UID 622 ...etc... 624 At the time of the writing of this document, the only server known to 625 do this does so only under the following condition: the client 626 selects INBOX, but there is not yet a physical INBOX file created. 627 Nonetheless, the SELECT succeeds, exporting an empty INBOX with a 628 temporary UIDVALIDITY of 1. While the INBOX remains selected, mail 629 is delivered to the user, which creates the real INBOX file and 630 assigns a permanent UIDVALIDITY (that is likely not to be 1). The 631 server reports the change of UIDVALIDITY, but as there were no 632 messages before, so no UIDs have actually changed, all the client 633 must do is accept the change in UIDVALIDITY. 635 Alternatively, a server may force the client to re-select the 636 mailbox, at which time it will obtain a new UIDVALIDITY value. To do 637 this, the server closes this client session (see "Severed 638 Connections" above) and the client then reconnects and gets back in 639 synch. Clients MUST be prepared for either of these behaviours. 641 We do not know of, nor do we anticipate the future existance of, a 642 server that changes UIDVALIDITY while there are existing messages, 643 but clients MUST be prepared to handle this eventuality. 645 Internet DRAFT Implementation Recommendations December 1998 647 3.4.4. FETCH Responses 649 When a client asks for certain information in a FETCH command, the 650 server MAY return the requested information in any order, not 651 necessarily in the order that it was requested. Further, the server 652 MAY return the information in separate FETCH responses and MAY also 653 return information that was not explicitly requested (to reflect to 654 the client changes in the state of the subject message). Some 655 examples: 657 C: 001 FETCH 1 UID FLAGS INTERNALDATE 658 S: * 5 FETCH (FLAGS (\Deleted)) 659 S: * 1 FETCH (FLAGS (\Seen) INTERNALDATE "..." UID 345) 660 S: 001 OK done 661 (In this case, the responses are in a different order. Also, the 662 server returned a flag update for message 5, which wasn't part of the 663 client's request.) 665 C: 002 FETCH 2 UID FLAGS INTERNALDATE 666 S: * 2 FETCH (INTERNALDATE "...") 667 S: * 2 FETCH (UID 399) 668 S: * 2 FETCH (FLAGS ()) 669 S: 002 OK done 670 (In this case, the responses are in a different order and were 671 returned in separate responses.) 673 C: 003 FETCH 2 BODY[1] 674 S: * 2 FETCH (FLAGS (\Seen) BODY[1] {14} 675 S: Hello world! 676 S: ) 677 S: 003 OK done 678 (In this case, the FLAGS response was added by the server, since 679 fetching the body part caused the server to set the \Seen flag.) 681 Because of this characteristic a client MUST be ready to receive any 682 FETCH response at any time and should use that information to update 683 its local information about the message to which the FETCH response 684 refers. A client MUST NOT assume that any FETCH responses will come 685 in any particular order, or even that any will come at all. If after 686 receiving the tagged response for a FETCH command the client finds 687 that it did not get all of the information requested, the client 688 SHOULD send a NOOP command to the server to ensure that the server 689 has an opportunity to send any pending EXPUNGE responses to the 690 client (see [RFC-2180]). 692 3.4.5. RFC822.SIZE 694 Some back-end mail stores keep the mail in a canonical form, rather 696 Internet DRAFT Implementation Recommendations December 1998 698 than retaining the original MIME format of the messages. This means 699 that the server must reassemble the message to produce a MIME stream 700 when a client does a fetch such as RFC822 or BODY[], requesting the 701 entire message. It also may mean that the server has no convenient 702 way to know the RFC822.SIZE of the message. Often, such a server 703 will actually have to build the MIME stream to compute the size, only 704 to throw the stream away and report the size to the client. 706 When this is the case, some servers have chosen to estimate the size, 707 rather than to compute it precisely. Such an estimate allows the 708 client to display an approximate size to the user and to use the 709 estimate in flood control considerations (q.v.), but requires that 710 the client not use the size for things such as allocation of buffers, 711 because those buffers might then be too small to hold the actual MIME 712 stream. Instead, a client SHOULD use the size that's returned in the 713 literal when you fetch the data. 715 The protocol requires that the RFC822.SIZE value returned by the 716 server be EXACT. Estimating the size is a protocol violation, and 717 server designers must be aware that, despite the performance savings 718 they might realize in using an estimate, this practice will cause 719 some clients to fail in various ways. If possible, the server SHOULD 720 compute the RFC822.SIZE for a particular message once, and then save 721 it for later retrieval. If that's not possible, the server MUST 722 compute the value exactly every time. Incorrect estimates do cause 723 severe interoperability problems with some clients. 725 3.4.6. Expunged Messages 727 If the server allows multiple connections to the same mailbox, it is 728 often possible for messages to be expunged in one client unbeknownst 729 to another client. Since the server is not allowed to tell the 730 client about these expunged messages in response to a FETCH command, 731 the server may have to deal with the issue of how to return 732 information about an expunged message. There was extensive 733 discussion about this issue, and the results of that discussion are 734 summarized in [RFC-2180]. See that reference for a detailed 735 explanation and for recommendations. 737 3.4.7. The Namespace Issue 739 Namespaces are a very muddy area in IMAP4 implementation right now 740 (see [NAMESPACE] for a proposal to clear the water a bit). Until the 741 issue is resolved, the important thing for client developers to 742 understand is that some servers provide access through IMAP to more 743 than just the user's personal mailboxes, and, in fact, the user's 744 personal mailboxes may be "hidden" somewhere in the user's default 746 Internet DRAFT Implementation Recommendations December 1998 748 hierarchy. The client, therefore, SHOULD provide a setting wherein 749 the user can specify a prefix to be used when accessing mailboxes. 750 If the user's mailboxes are all in "~/mail/", for instance, then the 751 user can put that string in the prefix. The client would then put 752 the prefix in front of any name pattern in the LIST and LSUB 753 commands: 754 C: 001 LIST "" ~/mail/% 755 (See also "Reference Names in the LIST Command" below.) 757 3.4.8. Creating Special-Use Mailboxes 759 It may seem at first that this is part of the namespace issue; it is 760 not, and is only indirectly related to it. A number of clients like 761 to create special-use mailboxes with particular names. Most 762 commonly, clients with a "trash folder" model of message deletion 763 want to create a mailbox with the name "Trash" or "Deleted". Some 764 clients want to create a "Drafts" mailbox, an "Outbox" mailbox, or a 765 "Sent Mail" mailbox. And so on. There are two major 766 interoperability problems with this practice: 767 1. different clients may use different names for mailboxes with 768 similar functions (such as "Trash" and "Deleted"), or may manage the 769 same mailboxes in different ways, causing problems if a user switches 770 between clients and 771 2. there is no guarantee that the server will allow the creation of 772 the desired mailbox. 774 The client developer is, therefore, well advised to consider 775 carefully the creation of any special-use mailboxes on the server, 776 and, further, the client MUST NOT require such mailbox creation - 777 that is, if you do decide to do this, you MUST handle gracefully the 778 failure of the CREATE command and behave reasonably when your 779 special-use mailboxes do not exist and can not be created. 781 In addition, the client developer SHOULD provide a convenient way for 782 the user to select the names for any special-use mailboxes, allowing 783 the user to make these names the same in all clients used and to put 784 them where the user wants them. 786 3.4.9. Reference Names in the LIST Command 788 Many implementers of both clients and servers are confused by the 789 "reference name" on the LIST command. The reference name is intended 790 to be used in much the way a "cd" (change directory) command is used 791 on Unix, PC DOS, Windows, and OS/2 systems. That is, the mailbox 792 name is interpreted in much the same way as a file of that name would 793 be found if one had done a "cd" command into the directory specified 794 by the reference name. For example, in Unix we have the following: 796 Internet DRAFT Implementation Recommendations December 1998 798 > cd /u/jones/junk 799 > vi banana [file is "/u/jones/junk/banana"] 800 > vi stuff/banana [file is "/u/jones/junk/stuff/banana"] 801 > vi /etc/hosts [file is "/etc/hosts"] 803 The interoperability problems with this, in practice, are several. 804 First, while some IMAP servers are built on Unix or PC file systems, 805 many others are not, and the file system semantics do not make sense 806 in those configurations. Second, while some IMAP servers expose the 807 underlying file system to the clients, others allow access only to 808 the user's personal mailboxes, or to some other limited set of files, 809 making such file-system-like semantics less meaningful. Third, 810 because the IMAP spec leaves the interpretation of the reference name 811 as "implementation-dependent", the various server implementations 812 handle it in vastly differing ways, and fourth, many implementers 813 simply do not understand it and misuse it, do not use it, or ignore 814 it as a result. 816 The following statement gets somewhat into the religious issues that 817 we've tried to avoid scrupulously here: because of the confusion 818 around the reference name, its use by a client is a dangerous thing, 819 prone to result in interoperability problems. There are servers that 820 interpret it as originally intended; there are servers that ignore it 821 completely; there are servers that simply prepend it to the mailbox 822 name (with or without inserting a hierarchy delimiter in between). 823 Because a client can't know which of these four behaviours to expect, 824 a client SHOULD NOT use a reference name itself, expecting a 825 particular server behavior. However, a client SHOULD permit a USER, 826 by configuration, to use a reference name. 828 There is in no way universal agreement about the use or non-use of 829 the reference name. The last words here are, "Be aware." 831 3.4.12. Mailbox Hierarchy Delimiters 833 The server's selection of what to use as a mailbox hierarchy 834 delimiter is a difficult one, involving several issues: What 835 characters do users expect to see? What characters can they enter 836 for a hierarchy delimiter if it is desired (or required) that the 837 user enter it? What character can be used for the hierarchy 838 delimiter, noting that the chosen character can not otherwise be used 839 in the mailbox name? 841 Because some interfaces show users the hierarchy delimiters or allow 842 users to enter qualified mailbox names containing them, server 843 implementations SHOULD use delimiter characters that users generally 844 expect to see as name separators. The most common characters used 846 Internet DRAFT Implementation Recommendations December 1998 848 for this are "/" (as in Unix file names), "\" (as in OS/2 and Windows 849 file names), and "." (as in news groups). There is little to choose 850 among these apart from what users may expect or what is dictated by 851 the underlying file system, if any. One consideration about using 852 "\" is that it's also a special character in the IMAP protocol. 853 While the use of other hierarchy delimiter characters is permissible, 854 A DESIGNER IS WELL ADVISED TO STAY WITH ONE FROM THIS SET unless the 855 server is intended for special purposes only. Implementers might be 856 thinking about using characters such as "-", "_", ";", "&", "#", "@", 857 and "!", but they should be aware of the surprise to the user as well 858 as of the effect on URLs and other external specifications (since 859 some of these characters have special meanings there). Also, a 860 server that uses "\" (and clients of such a server) MUST remember to 861 escape that character in quoted strings or to send literals instead. 862 Literals are recommended over escaped characters in quoted strings in 863 order to maintain compatibility with older IMAP versions that did not 864 allow escaped characters in quoted strings (but check the grammar to 865 see where literals are allowed): 866 C: 001 LIST "" {13} 867 S: + send literal 868 C: this\%\%\%\h* 869 S: * LIST () "\\" {27} 870 S: this\is\a\mailbox\hierarchy 871 S: 001 OK LIST complete 873 In any case, a server SHOULD NOT use normal alpha-numeric characters 874 (such as "X" or "0") as delimiters; a user would be very surprised to 875 find that "EXPENDITURES" actually represented a two-level hierarchy. 876 And a server SHOULD NOT use characters that are non-printable or 877 difficult or impossible to enter on a standard US keyboard. Control 878 characters, box-drawing characters, and characters from non-US 879 alphabets fit into this category. Their use presents 880 interoperability problems that are best avoided. 882 The UTF-7 encoding of mailbox names also raises questions about what 883 to do with the hierarchy delimiters in encoded names: do we encode 884 each hierarchy level and separate them with delimiters, or do we 885 encode the fully qualified name, delimiters and all? The answer for 886 IMAP is the former: encode each hierarchy level separately, and 887 insert delimiters between. This makes it particularly important not 888 to use as a hierarchy delimiter a character that might cause 889 confusion with IMAP's modified UTF-7 [UTF-7 and RFC-2060] encoding. 891 To repeat: a server SHOULD use "/", "\", or "." as its hierarchy 892 delimiter. The use of any other character is likely to cause 893 problems and is STRONGLY DISCOURAGED. 895 Internet DRAFT Implementation Recommendations December 1998 897 3.4.11. ALERT Response Codes 899 The protocol spec is very clear on the matter of what to do with 900 ALERT response codes, and yet there are many clients that violate it 901 so it needs to be said anyway: "The human-readable text contains a 902 special alert that MUST be presented to the user in a fashion that 903 calls the user's attention to the message." That should be clear 904 enough, but I'll repeat it here: Clients MUST present ALERT text 905 clearly to the user. 907 3.4.12. Deleting Mailboxes 909 The protocol does not guarantee that a client may delete a mailbox 910 that is not empty, though on some servers it is permissible and is, 911 in fact, much faster than the alternative or deleting all the 912 messages from the client. If the client chooses to try to take 913 advantage of this possibility it MUST be prepared to use the other 914 method in the even that the more convenient one fails. Further, a 915 client SHOULD NOT try to delete the mailbox that it has selected, but 916 should first close that mailbox; some servers do not permit the 917 deletion of the selected mailbox. 919 That said, a server SHOULD permit the deletion of a non-empty 920 mailbox; there's little reason to pass this work on to the client. 921 Moreover, forbidding this prevents the deletion of a mailbox that for 922 some reason can not be opened or expunged, leading to possible 923 denial-of-service problems. 925 Example: 926 [User tells the client to delete mailbox BANANA, which is 927 currently selected...] 928 C: 008 CLOSE 929 S: 008 OK done 930 C: 009 DELETE BANANA 931 S: 009 NO Delete failed; mailbox is not empty. 932 C: 010 SELECT BANANA 933 S: * ... untagged SELECT responses 934 S: 010 OK done 935 C: 011 STORE 1:* +FLAGS.SILENT \DELETED 936 S: 011 OK done 937 C: 012 CLOSE 938 S: 012 OK done 939 C: 013 DELETE BANANA 940 S: 013 OK done 942 Internet DRAFT Implementation Recommendations December 1998 944 3.5. A Word About Testing 946 Since the whole point of IMAP is interoperability, and since 947 interoperability can not be tested in a vacuum, the final 948 recommendation of this treatise is, "Test against EVERYTHING." Test 949 your client against every server you can get an account on. Test 950 your server with every client you can get your hands on. Many 951 clients make limited test versions available on the Web for the 952 downloading. Many server owners will give serious client developers 953 guest accounts for testing. Contact them and ask. NEVER assume that 954 because your client works with one or two servers, or because your 955 server does fine with one or two clients, you will interoperate well 956 in general. 958 In particular, in addition to everything else, be sure to test 959 against the reference implementations: the PINE client, the 960 University of Washington server, and the Cyrus server. 962 See the following URLs on the web for more information here: 963 IMAP Products and Sources: http://www.imap.org/products.html 964 IMC MailConnect: http://www.imc.org/imc-mailconnect 966 4. Security Considerations 968 This document describes behaviour of clients and servers that use the 969 IMAP4 protocol, and as such, has the same security considerations as 970 described in [RFC-2060]. 972 5. References 974 [RFC-2060]; Crispin, M.; "Internet Message Access Protocol - Version 975 4rev1"; RFC 2060; University of Washington; December 1996. 977 [RFC-2119]; Bradner, S.; "Key words for use in RFCs to Indicate 978 Requirement Levels"; RFC 2119; Harvard University; March 1997. 980 [RFC-2180]; Gahrns, M.; "IMAP4 Multi-Accessed Mailbox Practice"; RFC 981 2180; Microsoft; July 1997. 983 [UTF-8]; Yergeau, F.; " UTF-8, a transformation format of Unicode and 984 ISO 10646"; RFC 2044; Alis Technilogies; October 1996. 986 [UTF-7]; Goldsmith, D. & Davis, M.; "UTF-7, a Mail-Safe 987 Transformation Format of Unicode"; RFC 2152; Apple Computer, Inc. & 988 Taligent, Inc.; May 1997. 990 [NAMESPACE]; Gahrns, M. & Newman, C.; "IMAP4 Namespace"; draft 992 Internet DRAFT Implementation Recommendations December 1998 994 document ; Microsoft & Innosoft; 995 June 1997. 997 6. Author's Address 999 Barry Leiba 1000 IBM T.J. Watson Research Center 1001 30 Saw Mill River Road 1002 Hawthorne, NY 10532 1004 Phone: 1-914-784-7941 1005 Email: leiba@watson.ibm.com