idnits 2.17.00 (12 Aug 2021) /tmp/idnits60690/draft-melnikov-imap-disc-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3667, Section 5.1 on line 13. -- Found old boilerplate from RFC 3978, Section 5.5 on line 1477. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 1450. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 1457. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 1463. ** The document seems to lack an RFC 3978 Section 5.1 IPR Disclosure Acknowledgement -- however, there's a paragraph with a matching beginning. Boilerplate error? ** This document has an original RFC 3978 Section 5.4 Copyright Line, instead of the newer IETF Trust Copyright according to RFC 4748. ** This document has an original RFC 3978 Section 5.5 Disclaimer, instead of the newer disclaimer which includes the IETF Trust according to RFC 4748. ** The document uses RFC 3667 boilerplate or RFC 3978-like boilerplate instead of verbatim RFC 3978 boilerplate. After 6 May 2005, submission of drafts without verbatim RFC 3978 boilerplate is not accepted. The following non-3978 patterns matched text found in the document. That text should be removed or replaced: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- ** The document seems to lack a 1id_guidelines paragraph about Internet-Drafts being working documents. ** The document seems to lack a 1id_guidelines paragraph about 6 months document validity. ** The document is more than 15 pages and seems to lack a Table of Contents. == No 'Intended status' indicated for this document; assuming Proposed Standard == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 1524 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack an IANA Considerations section. (See Section 2.2 of https://www.ietf.org/id-info/checklist for how to handle the case when there are no actions for IANA.) ** There are 325 instances of too long lines in the document, the longest one being 23 characters in excess of 72. ** The abstract seems to contain references ([IMAP4]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. == There are 7 instances of lines with non-RFC6890-compliant IPv4 addresses in the document. If these are example addresses, they should be changed. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (October 2004) is 6420 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'HEADER' is mentioned on line 897, but not defined -- Looks like a reference, but probably isn't: '3' on line 906 -- Looks like a reference, but probably isn't: '4' on line 906 -- Looks like a reference, but probably isn't: '5' on line 906 -- Looks like a reference, but probably isn't: '6' on line 906 -- Looks like a reference, but probably isn't: '7' on line 906 -- Looks like a reference, but probably isn't: '8' on line 906 -- Looks like a reference, but probably isn't: '9' on line 907 -- Looks like a reference, but probably isn't: '10' on line 907 -- Looks like a reference, but probably isn't: '11' on line 907 -- Looks like a reference, but probably isn't: '13' on line 907 -- Looks like a reference, but probably isn't: '14' on line 907 -- Looks like a reference, but probably isn't: '15' on line 907 -- Looks like a reference, but probably isn't: '16' on line 907 -- Looks like a reference, but probably isn't: '21' on line 908 == Missing Reference: 'MIME' is mentioned on line 984, but not defined == Missing Reference: 'UNSEEN 12' is mentioned on line 1345, but not defined == Missing Reference: 'UIDVALIDITY 3857529045' is mentioned on line 1346, but not defined == Missing Reference: 'HIGHESTMODSEQ 20010715194045007' is mentioned on line 1349, but not defined == Missing Reference: 'IMAP' is mentioned on line 1370, but not defined ** Obsolete normative reference: RFC 3501 (ref. 'IMAP4') (Obsoleted by RFC 9051) ** Obsolete normative reference: RFC 2359 (ref. 'UIDPLUS') (Obsoleted by RFC 4315) -- No information found for draft-melnikov-imap-condstore-XX - is the name correct? -- Possible downref: Normative reference to a draft: ref. 'CONDSTORE' ** Downref: Normative reference to an Informational RFC: RFC 2683 -- No information found for draft-ietf-imapext-acl-XX - is the name correct? Summary: 14 errors (**), 0 flaws (~~), 10 warnings (==), 24 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 IMAPEXT Working Group A. Melnikov 2 Internet Draft: IMAP4 Disconnected Access Editor 3 Document: draft-melnikov-imap-disc-06.txt October 2004 4 Expires: April 2005 6 Synchronization operations for disconnected IMAP4 clients 8 Status of this Memo 10 By submitting this Internet-Draft, I certify that any applicable 11 patent or other IPR claims of which I am aware have been disclosed, or 12 will be disclosed, and any of which I become aware will be disclosed, 13 in accordance with RFC 3668. 15 Internet Drafts are working documents of the Internet Engineering 16 Task Force (IETF), its Areas, and its Working Groups. Note that 17 other groups may also distribute working documents as Internet 18 Drafts. Internet Drafts are draft documents valid for a maximum of 19 six months. Internet Drafts may be updated, replaced, or obsoleted 20 by other documents at any time. It is not appropriate to use 21 Internet Drafts as reference material or to cite them other than as 22 ``work in progress''. 24 The list of current Internet-Drafts can be accessed at 25 http://www.ietf.org/ietf/1id-abstracts.txt 27 The list of Internet-Draft Shadow Directories can be accessed at 28 http://www.ietf.org/shadow.html. 30 This is a draft document based on the expired draft written by 31 the IETF IMAP Working Group. A revised version of this draft document 32 will be submitted to the RFC editor as an Informational (or BCP) RFC for 33 the Internet Community. Discussion and suggestions for improvement are 34 requested, and should be sent to imap@CAC.Washington.EDU. 36 This memo is for informational use and does not constitute a 37 standard. Distribution of this memo is unlimited. 39 Abstract 41 This document attempts to address some of the issues involved in building 42 a disconnected IMAP4 [IMAP4] client. In particular, it deals with the 43 issues of what might be called the "driver" portion of the synchronization 44 tool: the portion of the code responsible for issuing the correct set 45 of IMAP4 commands to synchronize the disconnected client in the way 46 that is most likely to make the human who uses the disconnected 47 client happy. 49 This note describes different strategies that can be used by disconnected 50 clients as well as shows how to use IMAP protocol in order to minimize the 51 time of synchronization process. 53 This note also lists IMAP extensions that a server should implement in 54 order to provide better synchronization facilities to disconnected clients. 56 1. Introduction 58 Several recommendations presented in this document are generally 59 applicable to all types of IMAP clients. However this document tries 60 to concentrate on disconnected mail clients [IMAP-MODEL]. It also suggests 61 some IMAP extensions* that should be implemented by IMAP servers in order 62 to make the life of disconnected clients easier. In particular, the [UIDPLUS] 63 extension was specifically designed to streamline certain disconnected 64 operations, like expunging, uploading and copying messages 65 (see Sections 4.2.1, 4.2.2.1 and 4.2.4). 67 Readers of this document are also strongly advised to read RFC 2683 68 [RFC 2683]. 70 * - note, that the functionality provided by the base IMAP protocol 71 [IMAP4] is sufficient to perform basic synchronization. 73 1.1. Conventions Used in this Document 75 In examples, "C:" and "S:" indicate lines sent by the client and 76 server respectively. 78 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 79 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 80 document are to be interpreted as described in RFC 2119 [KEYWORDS]. 82 Let's call an IMAP command idempotent, if the result of executing the 83 command twice sequentially is the same as the result of executing the 84 command just once. 86 Editorial comments/questions or missing paragraphs are marked in the 87 text with << and >>. 89 2. Design Principles 91 All mailbox state or content information stored on the disconnected 92 client should be viewed strictly as a cache of the state of the 93 server. The "master" state remains on the server, just as it would 94 with an interactive IMAP4 client. The one exception to this rule is 95 that information about the state of the disconnected client's cache 96 (the state includes flag changes while offline and scheduled message 97 uploads) remains on the disconnected client: that is, the IMAP4 server 98 is not responsible for remembering the state of the disconnected IMAP4 99 client. 101 We assume that a disconnected client is a client that, for whatever 102 reason, wants to minimize the length of time that it is "on the 103 phone" to the IMAP4 server. Often this will be because the client is 104 using a dialup connection, possibly with very low bandwidth, but 105 sometimes it might just be that the human is in a hurry to catch an 106 airplane, or some other event beyond our control. Whatever the 107 reason, we assume that we must make efficient use of the network 108 connection, both in the usual sense (not generating spurious traffic) 109 and in the sense that we would prefer not to have the connection 110 sitting idle while the client and/or the server is performing 111 strictly local computation or I/O. Another, perhaps simpler way of 112 stating this is that we assume that network connections are 113 "expensive". 115 Practical experience with disconnected mail systems has shown that 116 there is no single synchronization strategy that is appropriate 117 for all cases. Different humans have different preferences, 118 and the same human's preference will vary depending both on 119 external circumstance (how much of a hurry the human is in today) 120 and on the value that the human places on the messages being 121 transferred. The point here is that there is no way that 122 the synchronization program can guess exactly what the human 123 wants to do, so the human will have to provide some guidance. 125 Taken together, the preceding two principles lead to the conclusion 126 that the synchronization program must make its decisions based on 127 some kind of guidance provided by the human by selecting the appropriate 128 options in UI or through some sort of configuration file, but almost 129 certainly should not pause for I/O with the human during the middle 130 of the synchronization process. The human will almost certainly have 131 several different configurations for the synchronization program, for 132 different circumstances. 134 Since a disconnected client has no way of knowing what changes might 135 have occurred to the mailbox while it was disconnected, message 136 numbers are not useful to a disconnected client. All disconnected 137 client operations should be performed using UIDs, so that the client 138 can be sure that it and the server are talking about the same 139 messages during the synchronization process. 141 3. Overall picture of synchronization 143 The basic strategy for synchronization is outlined below. 144 Note that the real strategy may vary from one application to another 145 or may depend on a synchronization mode. 147 a) Process any "actions" that were pending on the client that 148 were not associated with any mailbox (in particular sending 149 messages composed offline with SMTP. This is not part of IMAP 150 synchronization, but it is mentioned here for completeness); 152 b) Fetch the current list of "interesting" mailboxes (The disconnected 153 client should allow the user to skip this step completely); 155 c) "Client-to-server synchronization" - for each IMAP "action" that 156 were pending on the client: 158 1) If the action implies opening a new mailbox (any operation 159 that operates on messages) - open the mailbox. Check its UID 160 validity value (see section 4.1 for more details) returned in 161 the UIDVALIDITY response code. If the UIDVALIDITY value returned 162 by the server differs, the client MUST empty the local cache of 163 the mailbox and remove any pending "actions" which refer to UIDs 164 in that mailbox (and consider them failed). Note, this doesn't 165 affect actions performed on client generated fake UIDs (see 166 section 5). 168 2) Perform the action. If the action is to delete a mailbox (DELETE), 169 make sure that the mailbox is closed first (see also Section 170 3.4.12 of [RFC 2683]). 172 d) "Server-to-client synchronization" - for each mailbox that requires 173 synchronization, do the following: 175 1) Check the mailbox UIDVALIDITY (see section 4.1 for more details). 176 with SELECT/EXAMINE/STATUS. 177 If UIDVALIDITY value returned by the server differs, 178 the client MUST 180 * empty the local cache of that mailbox; 181 * remove any pending "actions" which refer to UIDs in 182 that mailbox and consider them failed; 183 * skip step 2-II; 185 2) Fetch the current "descriptors"; 187 I) Discover new messages. 189 II) Discover changes to old messages. 191 3) Fetch the bodies of any "interesting" messages that the client 192 doesn't already have. 194 e) Close all open mailboxes not required for further operations 195 (if staying online) or disconnect all open connections (if going 196 offline). 198 Terms used: 200 "Actions" are queued requests that were made by the human to the 201 client's MUA software while the client was disconnected. 203 Let define "descriptors" as a set of IMAP4 FETCH data items. 204 Conceptually, a message's descriptor is that set of 205 information that allows the synchronization program to decide what 206 protocol actions are necessary to bring the local cache to the 207 desired state for this message; since this decision is really up 208 to the human, this information probably includes a at least a few 209 header fields intended for human consumption. Exactly what will 210 constitute a descriptor depends on the client implementation. At 211 a minimum, the descriptor contains the message's UID and FLAGS. 212 Other likely candidates are the RFC822.SIZE, RFC822.HEADER, 213 BODYSTRUCTURE or ENVELOPE data items. 215 Comments: 217 1). The list of actions should be ordered. E.g., if the human deletes 218 message A1 in mailbox A, then expunges mailbox A, then deletes 219 message A2 in mailbox A, the human will expect that message A1 is 220 gone and that message A2 is still present but is now deleted. 222 By processing all the actions before proceeding with 223 synchronization, we avoid having to compensate for the local MUA's 224 changes to the server's state. That is, once we have processed 225 all the pending actions, the steps that the client must take to 226 synchronize itself will be the same no matter where the changes to 227 the server's state originated. 229 2). Steps a) and b) can be performed in parallel. Alternatively step a) 230 can be performed after d). 232 3). On step b) the set of "interesting" mailboxes pretty much has to be 233 determined by the human. What mailboxes belong to this set may 234 vary between different IMAP4 sessions with the same server, 235 client, and human. An interesting mailbox can be a mailbox 236 returned by LSUB command. The special mailbox "INBOX" SHOULD always 237 be considered "interesting". 239 4). On step d-2-II) the client also finds out about 240 changes to the flags of messages that the client already has in 241 its local cache, as well as finding out about messages in the 242 local cache that no longer exist on the server (i.e., messages that 243 have been expunged). 245 5). "Interesting" messages are those messages that the synchronization 246 program thinks the human wants to have cached locally, based on 247 the configuration and the data retrieved in step (b). 249 6). A disconnected IMAP client is a special case of an IMAP client, 250 so it MUST be able to handle any "unexpected" unsolicited responses, 251 like EXISTS and EXPUNGE, at any time. 252 The disconnected client MAY ignore EXPUNGE response during 253 "client-to-server" synchronization phase (step c)). 255 The rest of this discussion will focus primarily on the synchronization 256 issues for a single mailbox. 258 4. Mailbox synchronization steps and strategies 260 4.1. Checking UID Validity 262 The "UID validity" of a mailbox is a number returned in an 263 UIDVALIDITY response code in an OK untagged response at mailbox 264 selection time. The UID validity value changes between sessions when 265 UIDs fail to persist between sessions. 267 Whenever the client selects a mailbox, the client must compare the 268 returned UID validity value with the value stored in the local cache. 269 If the UID validity values differ, the UIDs in the client's cache are 270 no longer valid. The client MUST then empty the local cache of 271 that mailbox and remove any pending "actions" which refer to UIDs in 272 that mailbox. The client MAY also issue a warning to the human. 273 The client MUST NOT cancel any scheduled uploads (i.e. APPENDs) for 274 the mailbox. 276 Note that UIDVALIDITY is not only returned on a mailbox selection. 277 The COPYUID and APPENDUID response codes defined in the [UIDPLUS] extension 278 (see also 4.2.2) and the UIDVALIDITY STATUS response data item also contain 279 a UIDVALIDITY value for some other mailbox. The client SHOULD behave as 280 described in the previous paragraph (but it should act on the other mailbox's 281 cache), no matter how it obtained the UIDVALIDITY value. 283 4.2. Synchronizing local changes with the server 285 4.2.1. Uploading messages to the mailbox 287 Two of the most common examples of operations resulting in message 288 uploads are: 290 1) Saving a draft message 291 2) Copying a message between remote mailboxes on two different IMAP servers 292 or a local mailbox and a remote mailbox. 294 Message upload is performed with the APPEND command. A message scheduled to be 295 uploaded has no UID associated with it, as all UIDs are assigned by the 296 server. The APPEND command will effectively associate a UID with the uploaded 297 message that can be stored in the local cache for future reference. 298 However [IMAP4] doesn't describe a simple mechanism to discover the message UID 299 by just performing the APPEND command. In order to discover the UID the client can 300 do one of the following: 302 1) Remove the uploaded message from cache. After that use the mechanism described 303 in 4.3 to fetch the information about the uploaded message as if it had been uploaded 304 by some other client. 306 2) Try to fetch header information as described in 4.2.2 in order to find a message 307 that corresponds to the uploaded message. One strategy for doing this is described 308 in 4.2.2. 310 Case 1) describes a not particularly smart client. 312 C: A003 APPEND Drafts (\Seen $MDNSent) {310} 313 S: + Ready for literal data 314 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 315 C: From: Fred Foobar 316 C: Subject: afternoon meeting 317 C: To: mooch@owatagu.siam.edu 318 C: Message-Id: 319 C: MIME-Version: 1.0 320 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 321 C: 322 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 323 C: 324 S: A003 OK APPEND Completed 326 Fortunately there is a simpler way to discover the message UID in the presence 327 of the [UIDPLUS] extension: 329 C: A003 APPEND Drafts (\Seen $MDNSent) {310} 330 S: + Ready for literal data 331 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 332 C: From: Fred Foobar 333 C: Subject: afternoon meeting 334 C: To: mooch@owatagu.siam.edu 335 C: Message-Id: 336 C: MIME-Version: 1.0 337 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 338 C: 339 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 340 C: 341 S: A003 OK [APPENDUID 1022843275 77712] APPEND completed 343 The UID of the appended message is the second parameter of APPENDUID 344 response code. 346 4.2.2. Optimizing "move" and "copy" operations 348 Practical experience with IMAP, and other mailbox access 349 protocols that support multiple mailboxes suggests that moving a 350 message from one mailbox to another is an extremely common operation. 352 4.2.2.1. Moving a message between two mailboxes on the same server 354 In IMAP4 a "move" operation between two mailboxes on the same server 355 is really a combination of a COPY operation and a STORE +FLAGS (\Deleted) 356 operation. This makes good protocol sense for IMAP, but it leaves 357 a simple-minded disconnected client in the silly position of deleting 358 and possibly expunging its cached copy of a message, then fetching 359 an identical copy via the network. 361 However, the presence of the UIDPLUS extension in the server can help: 362 C: A001 UID COPY 567,414 "Interesting Messages" 363 S: A001 OK [COPYUID 1022843275 414,567 5:6] Completed 364 This tells the client that the message with UID 414 in the current mailbox 365 was successfully copied to the mailbox "Interesting Messages" and was given 366 the UID 5, and that the message with UID 567 was given the UID 6. 368 In the absence of UIDPLUS extension support in the server the following 369 trick can be used. By including the Message-ID: header and the INTERNALDATE 370 data item as part of the descriptor, the client can check the descriptor 371 of a "new" message against messages that are already in its cache, and 372 avoid fetching the extra copy. Of course, it's possible that the 373 cost of checking to see if the message is already in the local cache 374 may exceed the cost of just fetching it, so this technique should not 375 be used blindly. If the MUA implements a "move" command, it makes 376 special provisions to use this technique when it knows that a 377 copy/delete sequence is the result of a "move" command. 379 Note, that servers are not required (although they are strongly encouraged 380 with "SHOULD language") to preserve INTERNALDATE when copying messages. 382 Also note, since it's theoretically possible for this algorithm to find 383 the wrong message (given sufficiently malignant Message-ID headers), 384 implementors should provide a way to disable this optimization, both 385 permanently and on a message-by-message basis. 387 Example: Copying a message in the absence of UIDPLUS extension. 389 At some point in time the client has fetch the source message 390 and some information was cached: 392 C: C021 UID FETCH (BODY.PEEK[] INTERNALDATE FLAGS) 393 ... 394 S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600" 395 FLAGS (\Draft $MDNSent) BODY[] {1036} 396 S: ... 397 S: Message-Id: <20040903110856.22a127cd@chardonnay> 398 S: ... 399 S: ...message body... 400 S: ) 401 ... 402 S: C021 OK fetch completed 404 Later on the client decides to copy the message: 406 C: C035 UID COPY 123 "Interesting Messages" 407 S: C035 OK Completed 409 As the server hasn't provided the COPYUID response code, the client 410 tries the optimization described above: 412 C: C036 SELECT "Interesting Messages" 413 ... 414 C: C037 UID SEARCH ON 31-May-2002 HEADER 415 "Message-Id" "20040903110856.22a127cd@chardonnay" 416 S: SEARCH 12368 417 S: C037 OK completed 419 Note, that if the server has returned multiple UIDs in the SEARCH 420 response the client MUST NOT use any of the returned UID. 422 4.2.2.2. Moving a message from a remote mailbox to a local 424 Moving a message from a remote mailbox to a local is done with FETCH 425 (that includes FLAGS and INTERNALDATE) followed by 426 UID STORE +FLAGS.SILENT (\Deleted): 428 C: A003 UID FETCH 123 (BODY.PEEK[] INTERNALDATE FLAGS) 429 S: * 27 FETCH (UID 123 INTERNALDATE "31-May-2002 05:26:59 -0600" 430 FLAGS (\Seen $MDNSent) BODY[] 431 S: ...message body... 432 S: ) 433 S: A003 OK UID FETCH completed 434 C: A004 UID STORE +FLAGS.SILENT (\Deleted) 435 S: A004 STORE completed 437 Note, that there is no reason to fetch the message during synchronization 438 if it's already in the client's cache. Also, the client SHOULD preserve 439 delivery date in the local cache. 441 4.2.2.3. Moving a message from a local mailbox to a remote 443 Moving a message from a local mailbox to a remote is done with APPEND: 445 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310} 446 S: + Ready for literal data 447 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 448 C: From: Fred Foobar 449 C: Subject: afternoon meeting 450 C: To: mooch@owatagu.siam.edu 451 C: Message-Id: 452 C: MIME-Version: 1.0 453 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 454 C: 455 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 456 C: 457 S: A003 OK [APPENDUID 1022843275 77712] completed 459 The client SHOULD specify delivery date from the local cache in the APPEND. 461 If the [LITERAL+] extension is available, the client can save a round trip*: 463 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310+} 464 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 465 C: From: Fred Foobar 466 C: Subject: afternoon meeting 467 C: To: mooch@owatagu.siam.edu 468 C: Message-Id: 469 C: MIME-Version: 1.0 470 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 471 C: 472 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 473 C: 474 S: A003 OK [APPENDUID 1022843275 77712] completed 476 * - Note that there is a risk that the server will reject the message due to 477 its size. If this happens, the client will waste bandwidth transferring 478 the whole message. If the client wouldn't have used the LITERAL+, this 479 could have been avoided: 481 C: A003 APPEND Drafts (\Seen $MDNSent) "31-May-2004 05:26:59 -0600" 482 {16777215} 483 S: A003 NO Sorry, message is too big 485 4.2.2.4. Moving a message between two mailboxes on different servers 487 Moving a message between two mailbox on two different servers is a 488 combination of the operations described in 4.2.2.2 followed by the 489 operations described in 4.2.2.3. 491 4.2.2.5. Uploading multiple messages to a remote mailbox with MULTIAPPEND 493 When there is a need to upload multiple messages to a remote mailbox 494 (e.g. as per 4.2.2.3), the presence of certain IMAP extensions may 495 significantly improve performance. One of them is [MULTIAPPEND]. 497 For some mail stores opening a mailbox for appending might be expensive. 498 [MULTIAPPEND] tells the server to open mailbox once (instead of opening 499 and closing it "n" times per "n" messages to be uploaded) and keep it 500 open while a group of messages is being uploaded to the server. 502 Also, if the server supports both [MULTIAPPEND] and [LITERAL+] extensions, 503 the entire upload is accomplished in a single command/response round trip. 505 Note: Client implementors should be aware, that [MULTIAPPEND] performs 506 append of multiple messages atomically. This means, for example, 507 if there is not enough space to save "n"-th message (or the message 508 has invalid structure and is rejected by the server) after successful 509 upload of "n-1" messages, the whole upload operation fails and no 510 message will be saved in the mailbox. Although, this behavior might 511 be desirable in certain situations, it might not be what you want. 512 Otherwise, the client should use the regular APPEND command (Section 513 4.2.2.3), possibly utilizing the [LITERAL+] extension. 514 See also section 5.1 for discussions about error recovery. 516 Note: MULTIAPPEND can be used together with the UIDPLUS extension in a way 517 similar to what was described in section 4.2.1. [MULTIAPPEND] 518 extends the syntax of the APPENDUID response code to allow for multiple 519 message UIDs in the second parameter. 521 Example: 522 An example below demonstrates the use of MULTIAPPEND together with 523 UIDPLUS (synchronization points where the client waits for confirmations 524 from the server are marked with "<--->"): 526 C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310} 527 <---> 528 S: + Ready for literal data 529 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 530 C: From: Fred Foobar 531 C: Subject: afternoon meeting 532 C: To: mooch@owatagu.siam.edu 533 C: Message-Id: 534 C: MIME-Version: 1.0 535 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 536 C: 537 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 538 C: (\Seen) " 1-Jun-2002 22:43:04 -0800" {286} 539 <---> 540 S: + Ready for literal data 541 C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST) 542 C: From: Joe Mooch 543 C: Subject: Re: afternoon meeting 544 C: To: foobar@blurdybloop.com 545 C: Message-Id: 546 C: MIME-Version: 1.0 547 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 548 C: 549 C: 3:30 is fine with me. 550 C: 551 S: A003 OK [APPENDUID 1022843275 77712,77713] completed 553 The upload takes 3 round trips. 555 Example: 556 The example above was modified for the case when the server supports 557 MULTIAPPEND, LITERAL+ and UIDPLUS. The upload takes only 1 round trip. 559 C: A003 APPEND Jan-2002 (\Seen $MDNSent) "31-May-2002 05:26:59 -0600" {310+} 560 C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) 561 C: From: Fred Foobar 562 C: Subject: afternoon meeting 563 C: To: mooch@owatagu.siam.edu 564 C: Message-Id: 565 C: MIME-Version: 1.0 566 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 567 C: 568 C: Hello Joe, do you think we can meet at 3:30 tomorrow? 569 C: (\Seen) " 1-Jun-2002 22:43:04 -0800" {286+} 570 C: Date: Mon, 7 Feb 1994 22:43:04 -0800 (PST) 571 C: From: Joe Mooch 572 C: Subject: Re: afternoon meeting 573 C: To: foobar@blurdybloop.com 574 C: Message-Id: 575 C: MIME-Version: 1.0 576 C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII 577 C: 578 C: 3:30 is fine with me. 579 C: 580 S: A003 OK [APPENDUID 1022843275 77712,77713] completed 582 4.2.3. Replaying local flag changes 584 The disconnected client uses STORE command to synchronize local flag state 585 with the server. The disconnected client SHOULD use +FLAGS.SILENT or -FLAGS.SILENT 586 in order to set or unset flags modified by the user while offline. The FLAGS 587 form MUST NOT be used, as there is a risk that this will overwrite flags 588 on the server that has been changed by some other client. 590 Example: 591 For the message with UID 15, the disconnected client stores the following 592 flags \Seen and $Highest. The flags were modified on the server by some other 593 client: \Seen, \Answered and $Highest. 594 While offline the user requested to remove $Highest flags and to add \Deleted. 595 The flag synchronization sequence for the message should look like: 597 C: A001 UID STORE 15 +FLAGS.SILENT (\Deleted) 598 S: A001 STORE completed 599 C: A002 UID STORE 15 -FLAGS.SILENT ($Highest) 600 S: A002 STORE completed 602 If the disconnected client is able to store an additional binary state 603 information (or a piece of information that can take a value from a predefined 604 set of values) in the local cache of an IMAP mailbox or in a local mailbox 605 (e.g. message priority), and if the server supports storing of arbitrary 606 keywords, the client MUST use keywords to store this state on the server. 608 Example: 609 Imagine a speculative mail client that can mark a message as one of work-related 610 ($Work), personal ($Personal) or spam ($Spam). In order to mark a message as 611 personal the client issues: 613 C: A001 UID STORE 15 +FLAGS.SILENT ($Personal) 614 S: A001 STORE completed 615 C: A002 UID STORE 15 -FLAGS.SILENT ($Work $Spam) 616 S: A002 STORE completed 618 In order to mark the message as neither work, nor personal, not spam, the client 619 issues: 621 C: A003 UID STORE 15 -FLAGS.SILENT ($Personal $Work $Spam) 622 S: A003 STORE completed 624 4.2.4. Processing mailbox compression (EXPUNGE) requests 626 A naive disconnected client implementation that supports compressing a mailbox 627 while offline may decide to issue an EXPUNGE command to the server in order 628 to expunge messages marked \Deleted. The problem with this command during 629 synchronization is that it permanently erases all messages with the \Deleted flag set, 630 i.e. even those messages that were marked as \Deleted on the server while the user 631 was offline. Doing this might result in an unpleasant surprise for the user. 633 Fortunately the [UIDPLUS] extension can help in this case as well. The extension 634 introduces UID EXPUNGE command, that, unlike EXPUNGE, takes a UID set parameter, 635 that lists UIDs of all messages that can be expunged. When processing this command 636 the server erases only messages with \Deleted flag listed in the UID list. Thus, 637 messages not listed in the UID set will not be expunged even if they have the \Deleted 638 flag set. 640 Example: While offline 3 messages with UIDs 7, 27 and 65 were marked \Deleted 641 when the user requested to compress the open mailbox. Another client marked 642 a message \Deleted on the server (UID 34). During synchronization the 643 disconnected client issues: 645 C: A001 UID EXPUNGE 7,27,65 646 S: * ... EXPUNGE 647 S: * ... EXPUNGE 648 S: * ... EXPUNGE 649 S: A001 UID EXPUNGE completed 651 If another client issues UID SEARCH DELETED command (to find all messages with 652 \Deleted flag) before and after the UID EXPUNGE it will get: 654 Before: 655 C: B001 UID SEARCH DELETED 656 S: * SEARCH 65 34 27 7 657 S: B001 UID SEARCH completed 659 After: 660 C: B002 UID SEARCH DELETED 661 S: * SEARCH 34 662 S: B002 UID SEARCH completed 664 In the absence of the [UIDPLUS] extension the following sequence of command can be 665 used as an approximation. Note: It's possible for another client to mark additional 666 messages as deleted while this sequence is being performed. In this case, these 667 additional messages will be expunged as well. 669 1). Find all messages marked \Deleted on the server: 671 C: A001 UID SEARCH DELETED 672 S: * SEARCH 65 34 27 7 673 S: A001 UID SEARCH completed 675 2). Find all messages that must not be erased (for the previous example 676 the list will consist of the message with UID 34) 678 3). Temporary remove \Deleted flag on all messages found in step 2) 680 C: A002 UID STORE 34 -FLAGS.SILENT (\Deleted) 681 S: A002 UID STORE completed 683 4). Expunge the mailbox 685 C: A003 EXPUNGE 686 S: * 20 EXPUNGE 687 S: * 7 EXPUNGE 688 S: * 1 EXPUNGE 689 S: A003 EXPUNGE completed 691 Here message with UID 7 has message number 1; with UID 27 - message 692 number 7 and with UID 65 - message number 20. 694 5). Restore \Deleted flag on all messages found when performing step 2) 696 C: A004 UID STORE 34 +FLAGS.SILENT (\Deleted) 697 S: A004 UID STORE completed 699 4.2.5. Closing a mailbox 701 When the disconnected client has to close a mailbox, it should not use 702 CLOSE command, because CLOSE does a silent EXPUNGE (section 4.2.4 explains 703 why EXPUNGE should not be used by a disconnected client). It is safe to use 704 CLOSE only if the mailbox was opened with EXAMINE. 706 If the mailbox was opened with SELECT, the client can use one of the 707 following commands to implicitly close the mailbox and prevent the silent 708 expunge: 710 1). UNSELECT - This is a command described in [UNSELECT] that works as 711 CLOSE, but doesn't cause the silent EXPUNGE. This command is 712 supported by the server if it reports UNSELECT in its CAPABILITY list. 713 2). SELECT - SELECT causes implicit CLOSE without EXPUNGE. 714 3). If the client intends to issue LOGOUT after closing the mailbox, it may 715 just issue LOGOUT, because LOGOUT causes implicit CLOSE without EXPUNGE 716 as well. 717 4). SELECT - if the client knows a mailbox that doesn't 718 exist or can't be selected, it MAY SELECT it. 720 If the client opened the mailbox with SELECT and just wants to avoid 721 implicit EXPUNGE without closing the mailbox, it may also use the following: 723 5). EXAMINE - reselect the same mailbox in read-only mode. 725 4.3. Details of "Normal" synchronization of a single mailbox 727 The most common form of synchronization is where the human trusts the 728 integrity of the client's copy of the state of a particular mailbox, 729 and simply wants to bring the client's cache up to date so that it 730 accurately reflects the mailbox's current state on the server. 732 4.3.1. Discovering new messages and changes to old messages 734 Let represent the highest UID that the client knows about 735 in this mailbox. Since UIDs are allocated in strictly ascending 736 order, this is simply the UID of the last message in the mailbox that 737 the client knows about. Let represent 's UID 738 plus one. Let represent a list consisting of all the 739 FETCH data item items that the implementation considers to be part of 740 the descriptor; at a minimum this is just the FLAGS data item, but 741 it usually also includes BODYSTRUCTURE and RFC822.SIZE. At this step 742 SHOULD NOT include RFC822. 744 With no further information, the client can issue the following 745 two commands: 746 tag1 UID FETCH :* 747 tag2 UID FETCH 1: FLAGS 749 The first command will request some information about "new" messages 750 (i.e. messages received by the server since the last synchronization). 751 It will also allow the client to build a message number to UID map 752 (only for new messages). The second command allows the client to 753 1) update cached flags for old messages; 754 2) find out which old messages got expunged; 755 3) build a mapping between message numbers and UIDs (for old messages). 757 The order here is significant. We want the server to start returning 758 the list of new message descriptors as fast as it can, so that the 759 client can start issuing more FETCH commands, so we start out by 760 asking for the descriptors of all the messages we know the client 761 cannot possibly have cached yet. The second command fetches the 762 information we need to determine what changes may have occurred to 763 messages that the client already has cached. Note, that the former 764 command should only be issued if the UIDNEXT value cached by the client 765 differs from the one returned by the server. Once the client has 766 issued these two commands, there's nothing more the client can do 767 with this mailbox until the responses to the first command start 768 arriving. A clever synchronization program might use this time to 769 fetch its local cache state from disk, or start the process of 770 synchronizing another mailbox. 772 Example of the first FETCH: 773 C: A011 UID fetch 131:* (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE) 775 Note #1: The first FETCH may result in huge volume of data sent by 776 the server. A smart disconnected client should use message ranges 777 (see also section 3.2.1.2 of [RFC 2683]), so that the user is able to 778 execute a different operation between fetching information for 779 a group of new messages. 781 Example: Knowing the new UIDNEXT returned by the server on SELECT or 782 EXAMINE (), the client can split the UID range 783 : 784 into groups, e.g. 100 messages. After that the client can issue: 786 C: A011 UID fetch : 787 (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE) 788 ... 789 C: A012 UID fetch : 790 (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE) 791 ... 792 ... 793 C: A0FF UID fetch : 794 (FLAGS BODYSTRUCTURE INTERNALDATE RFC822.SIZE) 796 Note, that without issuing a SEARCH command it is not possible to 797 determine how many messages will fall into a subrange, as UIDs are 798 not necessarily contiguous. 800 Note #2: The client SHOULD ignore any unsolicited EXPUNGE responses 801 received during the first FETCH command. EXPUNGE responses contain 802 message numbers which are useless to a client that doesn't have 803 the message-number-to-UID translation table. 805 The second FETCH command will result in zero or more untagged fetch 806 responses. Each response will have a corresponding UID FETCH data item. 807 All messages that didn't have a matching untagged FETCH response 808 MUST be removed from the local cache. 810 For example, if the had a value 15000 and the local cache 811 contained 3 messages with the UIDs 12, 777 and 14999 respectively, than 812 after receiving the following responses from the server: 814 S: * 1 FETCH (UID 12 FLAGS (\Seen)) 815 S: * 2 FETCH (UID 777 FLAGS (\Answered \Deleted)) 817 the client must remove the message with UID 14999 from its local cache. 819 Note #3: If the client is not interested in flag changes (i.e. the client 820 only wants to know which old messages are still on the server), the second 821 FETCH command can be substituted with: 822 tag2 UID SEARCH UID 1: 824 This command will generate less traffic. However an implementor should be 825 aware that in order to build the mapping table from message numbers to UIDs 826 the output of the SEARCH command MUST be sorted first, because there is 827 no requirement for a server to return UIDs in SEARCH response in any 828 particular order. 830 4.3.2. Searching for "interesting" messages. 832 This step is either performed entirely on the client (from the information received 833 in step 4.3.1), entirely on the server or some combination of both. 834 The decision on what is an "interesting" message is up to the client software 835 and the human. One easy criterion that should probably be implemented in any 836 client is whether the message is "too big" for automatic retrieval, where "too big" 837 is a parameter defined in the client's configuration. 839 Another commonly used criterion is the age of a message. For example, the client 840 may choose to download only messages received in the last week (in this case, 841 would be today's date minus 7 days): 843 tag3 UID SEARCH UID SINCE 845 Keep in mind that a date search disregards time and timezone. 846 The client can avoid doing this search if it specified INTERNALDATE in 847 on step 4.3.1. If the client did, it can perform the local search on its message cache. 849 At this step the client also decides what kind of information about a particular 850 message to fetch from the server. In particular, even for a message that is considered 851 to be "too big" the client MAY choose to fetch some part(s) of it. For example, 852 if the message is a multipart/mixed containing a text part and a MPEG attachment, 853 there is no reason for the client not to fetch the text part. The decision of which 854 part should or should not be fetched can be based on the information received in 855 the BODYSTRUCTURE FETCH response data item (i.e. if BODYSTRUCTURE was included in 856 on step 4.3.1). 858 4.3.3. Populating cache with "interesting" messages. 860 Once the client has found out which messages are "interesting", it 861 can start issuing appropriate FETCH commands for "interesting" messages or 862 parts thereof. 864 It is important to note that fetching a message into the disconnected 865 client's local cache does NOT imply that the human has (or even will) 866 read the message. Thus, the synchronization program for a 867 disconnected client should always be careful to use the .PEEK 868 variants of the FETCH data items that implicitly set the \Seen flag. 870 Once the last descriptor has arrived and the last FETCH command has 871 been issued, the client simply needs to process the incoming fetch 872 items, using them to update the local message cache. 874 In order to avoid deadlock problems, the client must give processing 875 of received messages priority over issuing new FETCH commands during 876 this synchronization process. This may necessitate temporary local 877 queuing of FETCH requests that cannot be issued without causing a 878 deadlock. In order to achieve the best use of the "expensive" network 879 connection, the client will almost certainly need to pay careful 880 attention to any flow-control information that it can obtain from the 881 underlying transport connection (usually a TCP connection). 883 Note: The requirement stated in the previous paragraph might result in 884 an unpleasant user experience, if followed blindly. For example, the 885 user might be unwilling to wait for the client to finish synchronization 886 before starting to process the user's requests. A smart disconnected client 887 should allow the user to perform requested operations in between IMAP 888 commands which are part of the synchronization process. See also the 889 Note #1 in section 4.3.1. 891 Example: After fetching a message BODYSTRUCTURE the client discovers 892 a complex MIME message. Than it decides to fetch MIME headers 893 of the nested MIME messages and some body parts. 895 C: A011 UID fetch 11 (BODYSTRUCTURE) 896 S: ... 897 C: A012 UID fetch 11 (BODY[HEADER] BODY[1.MIME] BODY[1.1.MIME] 898 BODY[1.2.MIME] BODY[2.MIME] BODY[3.MIME] BODY[4.MIME] BODY[5.MIME] 899 BODY[6.MIME] BODY[7.MIME] BODY[8.MIME] BODY[9.MIME] BODY[10.MIME] 900 BODY[11.MIME] BODY[12.MIME] BODY[13.MIME] BODY[14.MIME] BODY[15.MIME] 901 BODY[16.MIME] BODY[17.MIME] BODY[18.MIME] BODY[19.MIME] BODY[20.MIME] 902 BODY[21.MIME]) 903 S: ... 904 C: A013 UID fetch 11 (BODY[1.1] BODY[1.2]) 905 S: ... 906 C: A014 UID fetch 11 (BODY[3] BODY[4] BODY[5] BODY[6] BODY[7] BODY[8] 907 BODY[9] BODY[10] BODY[11] BODY[13] BODY[14] BODY[15] BODY[16] 908 BODY[21]) 909 S: ... 911 4.3.4. User initiated synchronization 913 After the client has finished the main synchronization process as described in 914 4.3.1-4.3.3, the user may optionally request additional synchronization steps 915 while the client is still online. This is not any different from the process 916 described in 4.3.2 and 4.3.3. 918 Typical examples are: 919 1) fetch all messages selected in UI. 920 2) fetch all messages marked as \Flagged on the server. 922 4.4. Special case: descriptor-only synchronization 924 For some mailboxes, fetching the descriptors might be the entire 925 synchronization step. Practical experience with IMAP has shown that 926 a certain class of mailboxes (e.g., "archival" mailboxes) are used 927 primarily for long-term storage of important messages that the human 928 wants to have instantly available on demand but does not want 929 cluttering up the disconnected client's cache at any other time. 930 Messages in this kind of mailbox would be fetched exclusively by 931 explicit actions queued by the local MUA. Thus, the only 932 synchronization desirable on this kind of mailbox is fetching enough 933 descriptor information for the user to be able to identify messages 934 for subsequent download. 936 Special mailboxes that receive messages from a high volume, low 937 priority mailing list might also be in this category, at least when 938 the human is in a hurry. 940 4.5. Special case: fast new-only synchronization 942 In some cases the human might be in such a hurry that s/he doesn't 943 care about changes to old messages, just about new messages. In this 944 case, the client can skip the UID FETCH command that obtains the 945 flags and UIDs for old messages (1:). 947 4.6. Special case: blind FETCH 949 In some cases the human may know (for whatever reason) that s/he 950 always wants to fetch any new messages in a particular mailbox, 951 unconditionally. In this case, the client can just fetch the 952 messages themselves, rather than just the descriptors, by using a 953 command like: 954 tag1 UID FETCH :* (FLAGS BODY.PEEK[]) 956 Note, that this example ignores the fact that the messages can 957 be arbitrary long. The disconnected client MUST always check 958 for message size before downloading, unless explicitly told otherwise. 959 A well behaved client should use instead something like the following: 961 1) Issue "tag1 UID FETCH :* (FLAGS RFC822.SIZE)" 962 2) From the message sizes returned in step 1 construct UID set 963 964 3) Issue "tag2 UID FETCH (BODY.PEEK[])" 966 or 968 1) Issue "tag1 UID FETCH :* (FLAGS)" 969 2) Construct UID set from the responses of 1) 970 3) Issue "tag2 SEARCH UID SMALLER " 971 Construct UID set from the result of 972 the SEARCH command. 973 4) Issue "tag3 UID FETCH (BODY.PEEK[])" 975 or 977 1) Issue "tag1 UID FETCH :* (FLAGS BODY.PEEK[]<0.>)", 978 where should be replaced with the maximal message size 979 the client is willing to download. 980 Note: In response to such a command, the server will only return 981 partial data if the message is longer than . It will return 982 the full message data for any message whose size is smaller than or 983 equal to . In the former case, the client will not be able 984 to extract the full [MIME] structure of the message from the truncated 985 data, so the client should include BODYSTRUCTURE in the UID FETCH 986 command as well. 988 5. Implementation considerations 990 Below are listed some common implementation pitfalls that should be 991 considered when implementing a disconnected client. 993 1) Implementing fake UIDs on the client. 995 A message scheduled to be uploaded has no UID, as UIDs are selected by 996 the server. The client may implement fake UIDs internally in order to 997 reference not yet uploaded messages in further operations. For example, 998 a message could be scheduled to be uploaded, but subsequently marked as deleted or copied 999 to another mailbox). Here the client MUST NOT under any circumstances 1000 sent these fake UIDs to the server. Also, client implementors should 1001 be reminded that according to [IMAP4] an UID is a 32bit unsigned integer 1002 excluding 0. So, both 4294967295 and 2147483648 are valid UIDs and 0 and -1 1003 are both invalid. Some disconnected mail clients have been known to send 1004 negative numbers (e.g. "-1") as message UIDs to servers during synchronization. 1006 Example 1: The user starts composing a new message, edits it, saves it, 1007 continues to edit and saves it again. 1009 A disconnected client may record in its replay log (log of operations 1010 to be replayed on the server during synchronization) the sequence of 1011 operations as shown below. For the purpose of this example we assume 1012 that all draft messages are stored in the mailbox called Drafts on an 1013 IMAP server. We will also use the following conventions: 1014 UID of the intermediate version of the draft when it was saved 1015 for the first time. This is a fake UID generated on the client. 1016 UID of the final version of the draft. This is another fake UID 1017 generated on the client. 1019 1). APPEND Drafts (\Seen $MDNSent \Drafts) {} 1020 ...first version of the message follows... 1021 2). APPEND Drafts (\Seen $MDNSent \Drafts) {} 1022 ...final version of the message follows... 1023 3). STORE +FLAGS (\Deleted) 1025 Step 1 corresponds to the first attempt to save the draft message, 1026 step 2 corresponds to the second attempt to save the draft message 1027 and the step 3 deletes the first version of the draft message saved 1028 in step 1. 1030 A naive disconnected client may send the command in step 3 without 1031 replacing the fake client generated with the value returned 1032 by the server in step 1. A server will probably reject this command, 1033 which will make the client believe that the synchronization sequence 1034 has failed. 1036 2) Section 5.1 talks about common implementation errors related to error 1037 recovery during playback. 1039 3) Don't assume that the disconnected client is the only client used by 1040 the user. 1042 <> 1043 Example 2: Some clients may use the \Deleted flag as an indicator that 1044 the message should not appear in the user's view. Usage of the \Deleted 1045 flag for this purpose is not safe, as other clients (e.g. online 1046 clients) might EXPUNGE the mailbox at any time. 1048 4) Beware of data dependencies between synchronization operations. 1050 It might be very tempting for a client writer to perform some 1051 optimizations on the playback log. Such optimizations might include 1052 removing redundant operations (for example, see the optimization #2 1053 in section 5.3), or their reordering. 1055 It is not always safe to reorder or remove redundant operations during 1056 synchronization, because some operations may have dependencies. So if 1057 in doubt, don't do this. The following example demonstrates this: 1059 Example 3: The user copies a message out of a mailbox and then deletes 1060 the mailbox. 1062 C: A001 SELECT Old-Mail 1063 S: ... 1064 C: A002 UID COPY 111 ToDo 1065 S: A002 OK [COPYUID 1022843345 111 94] Copy completed 1066 ... 1067 C: A015 CLOSE 1068 S: A015 OK Completed 1069 C: A016 DELETE Old-Mail 1070 S: A016 OK Mailbox deletion completed successfully 1072 If the client performs DELETE (tag A016) first and COPY (tag A002) 1073 second, than the COPY fails. Also, the message that the user so 1074 carefully copied into another mailbox, has been lost. 1076 5.1. Error recovery during playback 1078 Error recovery during synchronization is one of the trickiest parts 1079 to get right. Below, we will discuss certain error conditions 1080 and suggest possible choices to handle them: 1082 1). Lost connection to the server. 1084 The client MUST remember the current position in playback (replay) log and 1085 replay it starting from the interrupted operation (the last command 1086 issued by the client, but not acknowledged by the server) next time it 1087 successfully connects to the same server. If the connection was lost while 1088 executing a non-idempotent IMAP command (see the definition in Section 1), when 1089 reconnected the client MUST make sure that the interrupted command was 1090 indeed not executed. If it wasn't executed, the client must restart playback 1091 from the interrupted command, otherwise from the following command. 1093 When reconnected, care must be taken in order to properly reapply logical 1094 operations that are represented by multiple IMAP commands, e.g. UID EXPUNGE 1095 emulation when UID EXPUNGE is not supported by the server (see section 4.2.4). 1097 Once the client detects that the connection to the server was lost, 1098 it MUST stop replaying its log. There are existing disconnected clients 1099 that, to the great annoyance of users, pop up an error dialog 1100 for each and every playback operation that fails. 1102 2). Copying/appending messages to a mailbox that doesn't exist. 1103 (The server advertises this condition by sending the TRYCREATE response 1104 code in the tagged NO response to the APPEND or COPY command.) 1106 The user should be advised about the situation and be given 1107 one of the following choices: 1108 a). Try to recreate a mailbox; 1109 b). Copy/upload messages to another mailbox; 1110 c). Skip copy/upload. 1111 d). Abort replay. 1113 3). Copying messages from, rename or get/change ACLs [ACL] on 1114 a mailbox that doesn't exist: 1116 a). Skip operation 1117 b). Abort replay 1119 4). Deleting mailboxes or deleting/expunging messages that no longer exist. 1121 This is actually is not an error and should be ignored by the client. 1123 5). Performing operations on messages that no longer exist. 1125 a). Skip operation 1126 b). Abort replay 1128 In the case of changing flags on an expunged message the client should 1129 silently ignore the error. 1131 Note 1: Several synchronization operations map to multiple IMAP commands 1132 (for example "move" described in 4.2.2). The client must guarantee 1133 atomicity of each such multistep operation. For example, 1134 when performing a "move" between two mailboxes on the same server, 1135 if the server is unable to copy messages, the client MUST NOT attempt to 1136 set the \Deleted flag on the messages being copied, let alone expunge 1137 them. However, the client MAY consider that move operation succeeded 1138 even if the server was unable to set the \Deleted flag on copied messages. 1140 Note 2: Many synchronization operations have data dependencies. 1141 A failed operation must cause all dependent operations to fail as 1142 well. The client should check that and MUST NOT try to perform 1143 all dependent operations blindly (unless the user corrected the original 1144 problem). For example, a message may be scheduled to be appended to 1145 a mailbox on the server and later on the appended message may be copied 1146 to another mailbox. If the APPEND operation fails, the client must not 1147 attempt to COPY the failed message later on. (See also Section 5, example 3). 1149 5.2. Quality of implementation issues. 1151 Below listed some quality of implementation issues for disconnected clients. 1152 They will help to write a disconnected client that works correctly, performs 1153 synchronization as quickly as possible (and thus can make the user 1154 happier as well as save her some money) and minimizes the server load: 1156 1) Don't lose information. 1158 No matter how smart your client is in other areas, if it loses information 1159 users will get very upset. 1161 2) Don't do work unless explicitly asked. Be flexible. Ask all questions 1162 BEFORE starting synchronization, if possible. 1164 3) Minimize traffic 1166 The client MUST NOT issue a command if the client already received 1167 the required information from the server. 1169 The client MUST make use of UIDPLUS extension if it is supported 1170 by the server. 1172 See also optimization #1 in Section 5.3. 1174 4) Minimize number of round trips. 1176 Round trips kill performance, especially on links with high latency. 1177 Sections 4.2.2.5 and 5.2 give some advices how to minimize number of 1178 round trips. 1180 See also optimization #1 in Section 5.3. 1182 5.3. Optimizations 1184 Some useful optimizations are described in this section. A disconnected 1185 client that supports the recommendations listed below will give the user 1186 a more pleasant experience. 1188 1) The initial OK or PREAUTH responses may contain the CAPABILITY response code 1189 as described in section 7.1 of [IMAP4]. This response code gives 1190 the same information as returned by the CAPABILITY command(*). 1191 A disconnected client that pays attention to this response code can 1192 avoid sending CAPABILITY command and will save a round trip. 1194 (*) - Note: Some servers report in the CAPABILITY response code 1195 extensions that are only relevant in unauthenticated state or 1196 in all states. Such servers usually send another CAPABILITY 1197 response code upon successful authentication using LOGIN or 1198 AUTHENTICATE command (that negotiates no security layer, see 1199 section 6.2.2 of [IMAP4]). The CAPABILITY response code 1200 sent upon successful LOGIN/AUTHENTICATE might be different 1201 from the CAPABILITY response code in the initial OK response, 1202 as extensions only relevant for unauthenticated state will not 1203 be advertised and some additional extensions available only 1204 in authenticated and/or selected state will be. 1206 Example 1: 1208 S: * OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=DIGEST-MD5 AUTH=SRP] 1209 imap.example.com ready 1210 C: 2 authenticate DIGEST-MD5 1211 ... 1212 S: 2 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN SORT 1213 THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User authenticated 1214 (no layer) 1216 2) An advanced disconnected client may choose to optimize its replay log. 1217 For example, there might be some operations which are redundant (the list 1218 is not complete): 1219 a) an EXPUNGE followed by another EXPUNGE or CLOSE; 1220 b) changing flags (other than the \Deleted flag) on a message that 1221 gets immediately expunged; 1222 c) opening and closing the same mailbox. 1224 When optimizing, be careful about data dependencies between commands. 1225 For example, if the client is wishing to optimize (see case b) above) 1227 tag1 UID STORE +FLAGS (\Deleted) 1228 ... 1229 tag2 UID STORE +FLAGS (\Flagged) 1230 ... 1231 tag3 UID COPY "Backup" 1232 ... 1233 tag4 UID EXPUNGE 1235 it can't remove the second UID STORE command, because the message is being 1236 copied before it gets expunged. 1238 In general, it might be a good idea to keep mailboxes open during 1239 synchronization (see case c) above), if possible. This can be more easily 1240 achieved in conjunction with optimization #3 described below. 1242 3) Perform some synchronization steps in parallel, if possible. 1244 Several synchronization steps don't depend on each other and thus can 1245 be performed in parallel. Because the server machine is usually more 1246 powerful than the client machine and can perform some operations in 1247 parallel, this may speed up the total time of synchronization. 1249 In order to achieve such parallelization the client will have to open 1250 more than one connection to the same server. Client writers should not 1251 forget about non-trivial cost associated with establishing a TCP connection 1252 and performing an authentication. The disconnected client MUST NOT use 1253 one connection per mailbox. In most cases it is sufficient to have 1254 two connections. The disconnected client SHOULD avoid selecting the same 1255 mailbox in more than one connection, see section 3.1.1 of the [RFC 2683] 1256 for more details. 1258 Any mailbox synchronization MUST start with checking of the UIDVALIDITY 1259 as described in section 4.1 of this document. The client MAY use STATUS 1260 command to check UID Validity of a non selected mailbox. This is preferable 1261 to opening many connections to the same server to perform synchronization 1262 of multiple mailboxes simultaneously. As described in section 5.3.10 of 1263 [IMAP4], this SHOULD NOT be used on the selected mailbox. 1265 6. IMAP extensions that may help 1267 The following extensions can save traffic and/or number of round trips: 1269 1) The use of [UIDPLUS] is discussed in sections 4.1, 4.2.1, 4.2.2.1 and 4.2.4. 1271 2) The use of the MULTIAPPEND and LITERAL+ extensions for uploading messages 1272 is discussed in 4.2.2.5. 1274 3) Use the CONDSTORE extension (see section 6.1) for quick flag resynchronization. 1276 6.1. CONDSTORE extension 1278 An advance disconnected mail client should use the [CONDSTORE] extension 1279 when it is supported by the server. The client must cache the value from 1280 HIGHESTMODSEQ OK response code received on mailbox opening and update 1281 it whenever the server sends MODSEQ FETCH data items. 1283 If the client receives NOMODSEQ OK untagged response instead of 1284 HIGHESTMODSEQ, it MUST remove the last known HIGHESTMODSEQ value from its 1285 cache and follow more general instructions in section 3. 1287 When the client opens the mailbox for synchronization it first compares 1288 UIDVALIDITY as described in step d)1) in section 3. If the cached 1289 UIDVALIDITY value matches the one returned by the server, the client 1290 MUST compare the cached value of HIGHESTMODSEQ with the one returned 1291 by the server. If the cached HIGHESTMODSEQ value also matches the 1292 one returned by the server, then the client MUST NOT fetch flags for 1293 cached messages, as they hasn't changed. If the value on the server 1294 is higher than the cached one, the client MAY use 1295 "SEARCH MODSEQ " to find all messages with flags 1296 changed since the last time the client was online and had the mailbox 1297 opened. Alternatively the client MAY use 1298 "FETCH 1:* (FLAGS) (CHANGEDSINCE )". The latter operation 1299 combines searching for changed messages and fetching new information. 1301 In all cases the client still needs to fetch information about new 1302 messages (if requested by the user), as well as discover which messages have been 1303 expunged. 1305 Step d) ("Server-to-client synchronization") in section 4 in the presence 1306 of the CONDSTORE extension is amended as follows: 1308 d) "Server-to-client synchronization" - for each mailbox that requires 1309 synchronization, do the following: 1311 1a) Check the mailbox UIDVALIDITY (see section 4.1 for more details). 1312 with SELECT/EXAMINE/STATUS. 1313 If the UIDVALIDITY value returned by the server differs, 1314 the client MUST 1316 * empty the local cache of that mailbox; 1317 * "forget" the cached HIGHESTMODSEQ value for the mailbox; 1318 * remove any pending "actions" which refer to UIDs in 1319 that mailbox. Note, this doesn't affect actions performed on 1320 client generated fake UIDs (see section 5); 1321 * skip steps 1b and 2-II; 1323 1b) Check the mailbox HIGHESTMODSEQ. If the cached value is the same 1324 as the one returned by the server, skip fetching message flags 1325 on step 2-II, i.e. the client only has to find out which messages 1326 got expunged. 1328 2) Fetch the current "descriptors"; 1330 I) Discover new messages. 1332 II) Discover changes to old messages using 1333 "FETCH 1:* (FLAGS) (CHANGEDSINCE )" or 1334 "SEARCH MODSEQ ". 1336 3) Fetch the bodies of any "interesting" messages that the client 1337 doesn't already have. 1339 Example (the UIDVALIDITY value is the same, but the HIGHESTMODSEQ value 1340 has changed on the server while the client was offline): 1342 C: A142 SELECT INBOX 1343 S: * 172 EXISTS 1344 S: * 1 RECENT 1345 S: * OK [UNSEEN 12] Message 12 is first unseen 1346 S: * OK [UIDVALIDITY 3857529045] UIDs valid 1347 S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) 1348 S: * OK [PERMANENTFLAGS (\Deleted \Seen \*)] Limited 1349 S: * OK [HIGHESTMODSEQ 20010715194045007] 1350 S: A142 OK [READ-WRITE] SELECT completed 1352 after that either: 1353 C: A143 UID FETCH 1:* (FLAGS) (CHANGEDSINCE 20010715194032001) 1354 S: * 2 FETCH (UID 6 MODSEQ (20010715205008000) FLAGS (\Deleted)) 1355 S: * 5 FETCH (UID 9 MODSEQ (20010715195517000) FLAGS ($NoJunk 1356 $AutoJunk $MDNSent)) 1357 ... 1358 S: A143 OK FETCH completed 1360 or: 1362 C: A143 SEARCH MODSEQ 20010715194032001 1363 S: * SEARCH 2 5 6 7 11 12 18 19 20 23 (MODSEQ 20010917162500) 1364 S: A143 OK Search complete 1366 7. Security Considerations 1368 It is believed that this document does not raise any new security concerns 1369 that are not already present in the base [IMAP] protocol, and these issues 1370 are discussed in [IMAP]. Additional security considerations may be found 1371 in different extensions mentioned in this document, in particular in 1372 [UIDPLUS], [LITERAL+], [CONDSTORE], [MULTIAPPEND] and [UNSELECT]. 1374 Implementors are also reminded about the importance of thorough testing. 1376 8. References 1378 8.1. Normative References 1380 [KEYWORDS] Bradner, "Key words for use in RFCs to Indicate 1381 Requirement Levels", RFC 2119, Harvard University, March 1997. 1383 [IMAP4] Crispin, M., "Internet Message Access Protocol - Version 1384 4rev1", RFC 3501, University of Washington, March 2003. 1386 [UIDPLUS] Myers, J., "IMAP4 UIDPLUS extension", RFC 2359, June 1988. 1388 [LITERAL+] Myers, J. "IMAP4 non-synchronizing literals", RFC 2088, 1389 January 1997. 1391 [CONDSTORE] Melnikov, A., Hole, S., "IMAP Extension for Conditional 1392 STORE operation", Work in progress, draft-melnikov-imap-condstore-XX.txt, 1393 Isode Limited, ACI WorldWide/MessagingDirect. 1395 [MULTIAPPEND] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - 1396 MULTIAPPEND EXTENSION", RFC 3502, University of Washington, 1397 March 2003. 1399 [UNSELECT] Melnikov, A., "Internet Message Access Protocol (IMAP) 1400 UNSELECT command", RFC 3691, Isode Limited, February 2004. 1402 [RFC 2683] Leiba, B., "IMAP4 Implementation Recommendations", RFC 2683, 1403 September 1999. 1405 8.2. Informative References 1407 [ACL] Myers, J., "IMAP4 ACL Extension", RFC 2086, January 1997. 1408 and 1409 Melnikov, A., "IMAP4 ACL Extension", draft-ietf-imapext-acl-XX.txt, 1410 Work in Progress. 1412 [IMAP-MODEL] Crispin, M. "Distributed Electronic Mail Models in 1413 IMAP4", RFC 1733, University of Washington, December 1994. 1415 9. Acknowledgment 1417 This document is based on the draft-ietf-imap-disc-01.txt written 1418 by Rob Austein in November 1994. 1420 The editor appreciate comments posted by Mark Crispin to the IMAP mailing 1421 list and the comments/corrections/ideas received from Grant Baillie, 1422 Cyrus Daboo, John G. Myers, Chris Newman and Timo Sirainen. 1424 The editor would also like to thank the developers of Netscape Messenger 1425 and Mozilla mail clients for providing examples of disconnected mail clients 1426 that served as a base for many recommendations in this document. 1428 10. Editor's Address 1430 Alexey Melnikov 1431 mailto: alexey.melnikov@isode.com 1433 Isode Limited 1434 5 Castle Business Village, 1435 36 Station Road, 1436 Hampton, Middlesex, 1437 United Kingdom, TW12 2BX 1439 Phone: +44 77 53759732 1441 11. Intellectual Property 1443 The IETF takes no position regarding the validity or scope of any 1444 Intellectual Property Rights or other rights that might be claimed to 1445 pertain to the implementation or use of the technology described in 1446 this document or the extent to which any license under such rights 1447 might or might not be available; nor does it represent that it has 1448 made any independent effort to identify any such rights. Information 1449 on the procedures with respect to rights in RFC documents can be 1450 found in BCP 78 and BCP 79. 1452 Copies of IPR disclosures made to the IETF Secretariat and any 1453 assurances of licenses to be made available, or the result of an 1454 attempt made to obtain a general license or permission for the use of 1455 such proprietary rights by implementers or users of this 1456 specification can be obtained from the IETF on-line IPR repository at 1457 http://www.ietf.org/ipr. 1459 The IETF invites any interested party to bring to its attention any 1460 copyrights, patents or patent applications, or other proprietary 1461 rights that may cover technology that may be required to implement 1462 this standard. Please address the information to the IETF at 1463 ietf-ipr@ietf.org. 1465 12. Full Copyright Statement 1467 Copyright (C) The Internet Society (2004). This document is subject 1468 to the rights, licenses and restrictions contained in BCP 78, and 1469 except as set forth therein, the authors retain all their rights. 1471 This document and the information contained herein are provided on an 1472 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 1473 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET 1474 ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, 1475 INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE 1476 INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 1477 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1479 Acknowledgement 1481 Funding for the RFC Editor function is currently provided by 1482 the Internet Society.