idnits 2.17.00 (12 Aug 2021) /tmp/idnits26677/draft-yergeau-rfc2279bis-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == There are 4 instances of lines with non-ascii characters in the document. == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 1 instance of lines with non-RFC2606-compliant FQDNs in the document. -- The abstract seems to indicate that this document obsoletes RFC2279, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (October 9, 2002) is 7163 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Missing reference section? 'UNICODE' on line 507 looks like a reference -- Missing reference section? 'US-ASCII' on line 517 looks like a reference -- Missing reference section? 'RFC2119' on line 498 looks like a reference -- Missing reference section? 'CESU-8' on line 481 looks like a reference -- Missing reference section? 'RFC2234' on line 501 looks like a reference -- Missing reference section? '1' on line 521 looks like a reference -- Missing reference section? 'RFC2978' on line 504 looks like a reference -- Missing reference section? 'RFC2045' on line 494 looks like a reference Summary: 1 error (**), 0 flaws (~~), 4 warnings (==), 11 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group F. Yergeau 3 Internet-Draft Alis Technologies 4 Expires: April 9, 2003 October 9, 2002 6 UTF-8, a transformation format of ISO 10646 7 draft-yergeau-rfc2279bis-02 9 Status of this Memo 11 This document is an Internet-Draft and is in full conformance with 12 all provisions of Section 10 of RFC2026. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that 16 other groups may also distribute working documents as Internet- 17 Drafts. 19 Internet-Drafts are draft documents valid for a maximum of six months 20 and may be updated, replaced, or obsoleted by other documents at any 21 time. It is inappropriate to use Internet-Drafts as reference 22 material or to cite them other than as "work in progress." 24 The list of current Internet-Drafts can be accessed at http:// 25 www.ietf.org/ietf/1id-abstracts.txt. 27 The list of Internet-Draft Shadow Directories can be accessed at 28 http://www.ietf.org/shadow.html. 30 This Internet-Draft will expire on April 9, 2003. 32 Copyright Notice 34 Copyright (C) The Internet Society (2002). All Rights Reserved. 36 Abstract 38 <1> 39 ISO/IEC 10646-1 defines a large character set called the Universal 40 Character Set (UCS) which encompasses most of the world's writing 41 systems. The originally proposed encodings of the UCS, however, were 42 not compatible with many current applications and protocols, and this 43 has led to the development of UTF-8, the object of this memo. UTF-8 44 has the characteristic of preserving the full US-ASCII range, 45 providing compatibility with file systems, parsers and other software 46 that rely on US-ASCII values but are transparent to other values. 47 This memo updates and replaces RFC 2279. 48 <2> 49 Discussion of this draft should take place on the ietf- 50 charsets@iana.org mailing list. 52 Table of Contents 54 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 55 2. Notational conventions . . . . . . . . . . . . . . . . . . . . 5 56 3. UTF-8 definition . . . . . . . . . . . . . . . . . . . . . . . 6 57 4. Syntax of UTF-8 Byte Sequences . . . . . . . . . . . . . . . . 8 58 5. Versions of the standards . . . . . . . . . . . . . . . . . . 9 59 6. Byte order mark (BOM) . . . . . . . . . . . . . . . . . . . . 10 60 7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 61 8. MIME registration . . . . . . . . . . . . . . . . . . . . . . 13 62 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 63 10. Security Considerations . . . . . . . . . . . . . . . . . . . 15 64 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . 16 65 Author's Address . . . . . . . . . . . . . . . . . . . . . . . 17 66 A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18 67 B. Changes from RFC 2279 . . . . . . . . . . . . . . . . . . . . 19 68 Full Copyright Statement . . . . . . . . . . . . . . . . . . . 20 70 1. Introduction 71 <3> 72 ISO/IEC 10646 [ISO.10646-1] defines a large character set called the 73 Universal Character Set (UCS), which encompasses most of the world's 74 writing systems. The same set of characters is defined by the 75 Unicode standard [UNICODE], which further defines additional 76 character properties and other application details of great interest 77 to implementors. Up to the present time, changes in Unicode and 78 amendments and additions to ISO/IEC 10646 have tracked each other, so 79 that the character repertoires and code point assignments have 80 remained in sync. The relevant standardization committees have 81 committed to maintain this very useful synchronism. 82 <4> 83 ISO/IEC 10646 and Unicode define several encoding forms of their 84 common repertoire: UTF-8, UCS-2, UTF-16, UCS-4 and UTF-32. In an 85 encoding form, each character is represented as one or more encoding 86 units. All standard UCS encoding forms except UTF-8 have an encoding 87 unit larger than one octet, making them hard to use in many current 88 applications and protocols that assume 8 or even 7 bit characters. 89 <5> 90 UTF-8, the object of this memo, has a one-octet encoding unit. It 91 uses all bits of an octet, but has the quality of preserving the full 92 US-ASCII [US-ASCII] range: US-ASCII characters are encoded in one 93 octet having the normal US-ASCII value, and any octet with such a 94 value can only stand for an US-ASCII character, and nothing else. 95 <6> 96 UTF-8 encodes UCS characters as a varying number of octets, where the 97 number of octets, and the value of each, depend on the integer value 98 assigned to the character in ISO/IEC 10646 (the character number, 99 a.k.a. code point or Unicode scalar value). This encoding form has 100 the following characteristics (all values are in hexadecimal): 101 <7> 102 o Character numbers from U+0000 to U+007F (US-ASCII repertoire) 103 correspond to octets 00 to 7F (7 bit US-ASCII values). A direct 104 consequence is that a plain ASCII string is also a valid UTF-8 105 string. 106 <8> 107 o US-ASCII octet values do not appear otherwise in a UTF-8 encoded 108 character stream. This provides compatibility with file systems 109 or other software (e.g. the printf() function in C libraries) 110 that parse based on US-ASCII values but are transparent to other 111 values. 112 <9> 113 o Round-trip conversion is easy between UTF-8 and other encoding 114 forms. 115 <10> 116 o The first octet of a multi-octet sequence indicates the number of 117 octets in the sequence. 119 <11> 120 o The octet values C0, C1, FE and FF never appear. If the range of 121 character numbers is restricted to U+0000..U+10FFFF (the UTF-16 122 accessible range), then the octet values F5..FD also never appear. 123 <12> 124 o Character boundaries are easily found from anywhere in an octet 125 stream. 126 <13> 127 o The lexicographic sorting order of UTF-8 strings is the same as if 128 ordered by character numbers. Of course this is of limited 129 interest since a sort order based on character numbers is not 130 culturally valid. 131 <14> 132 o The Boyer-Moore fast search algorithm can be used with UTF-8 data. 133 <15> 134 o UTF-8 strings can be fairly reliably recognized as such by a 135 simple algorithm, i.e. the probability that a string of 136 characters in any other encoding appears as valid UTF-8 is low, 137 diminishing with increasing string length. 139 <16> 140 UTF-8 was originally a project of the X/Open Joint 141 Internationalization Group XOJIG with the objective to specify a File 142 System Safe UCS Transformation Format [FSS_UTF] that is compatible 143 with UNIX systems, supporting multilingual text in a single encoding. 144 The original authors were Gary Miller, Greger Leijonhufvud and John 145 Entenmann. Later, Ken Thompson and Rob Pike did significant work for 146 the formal definition of UTF-8. 148 2. Notational conventions 149 <17> 150 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 151 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 152 document are to be interpreted as described in [RFC2119]. 153 <18> 154 UCS characters are designated by the U+HHHH notation, where HHHH is a 155 string of from 4 to 6 hexadecimal digits representing the character 156 number in ISO/IEC 10646. 158 3. UTF-8 definition 159 <19> 160 UTF-8 is defined by Annex D of ISO/IEC 10646-1 [ISO.10646-1]. 161 Descriptions and formulae can also be found in the Unicode Standard 162 [UNICODE] and in [FSS_UTF]. 163 <20> 164 In UTF-8, characters are encoded using sequences of 1 to 6 octets. 165 If the range of character numbers is restricted to U+0000..U+10FFFF 166 (the UTF-16 accessible range), then only sequences of one to four 167 octets will occur. The only octet of a "sequence" of one has the 168 higher-order bit set to 0, the remaining 7 bits being used to encode 169 the character number. In a sequence of n octets, n>1, the initial 170 octet has the n higher-order bits set to 1, followed by a bit set to 171 0. The remaining bit(s) of that octet contain bits from the number 172 of the character to be encoded. The following octet(s) all have the 173 higher-order bit set to 1 and the following bit set to 0, leaving 6 174 bits in each to contain bits from the character to be encoded. 175 <21> 176 The table below summarizes the format of these different octet types. 177 The letter x indicates bits available for encoding bits of the 178 character number. 180 Char. number range | UTF-8 octet sequence 181 (hexadecimal) | (binary) 182 --------------------+--------------------------------------------- 183 0000 0000-0000 007F | 0xxxxxxx 184 0000 0080-0000 07FF | 110xxxxx 10xxxxxx 185 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx 186 0001 0000-001F FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 187 0020 0000-03FF FFFF | 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 188 0400 0000-7FFF FFFF | 1111110x 10xxxxxx ... 10xxxxxx 189 <22> 190 Encoding a character to UTF-8 proceeds as follows: 191 <23> 192 1. Determine the number of octets required from the character number 193 and the first column of the table above. It is important to note 194 that the rows of the table are mutually exclusive, i.e. there is 195 only one valid way to encode a given character. 196 <24> 197 2. Prepare the high-order bits of the octets as per the second 198 column of the table. 199 <25> 200 3. Fill in the bits marked x from the bits of the character number, 201 expressed in binary. Start by putting the lowest-order bit of 202 the character number in the lowest-order position of the last 203 octet of the sequence, then put the next higher-order bit of the 204 character number in the next higher-order position of that octet, 205 etc. When the x bits of the last octet are filled in, move on to 206 the next to last octet, then to the preceding one, etc. until 207 all x bits are filled in. 209 <26> 210 The definition of UTF-8 prohibits encoding character numbers between 211 U+D800 and U+DFFF, which are reserved for use with the UTF-16 212 encoding form (as surrogate pairs) and do not directly represent 213 characters. When encoding in UTF-8 from UTF-16 data, it is necessary 214 to first decode the UTF-16 data to obtain character numbers, which 215 are then encoded in UTF-8 as described above. This contrasts with 216 CESU-8 [CESU-8], which is a UTF-8-like encoding that is not meant for 217 use on the Internet. CESU-8 operates similarly to UTF-8 but encodes 218 the UTF-16 code values (16-bit quantities) instead of the character 219 number (code point). This leads to different results for character 220 numbers above 0xFFFF; the CESU-8 encoding of those characters is NOT 221 valid UTF-8. 222 <27> 223 Decoding a UTF-8 character proceeds as follows: 224 <28> 225 1. Initialize a binary number with all bits set to 0. Up to 31 bits 226 may be needed (up to 21 if the range of character numbers is 227 known to be restricted to the UTF-16 accessible range). 228 <29> 229 2. Determine which bits encode the character number from the number 230 of octets in the sequence and the second column of the table 231 above (the bits marked x). 232 <30> 233 3. Distribute the bits from the sequence to the binary number, first 234 the lower-order bits from the last octet of the sequence and 235 proceeding to the left until no x bits are left. The binary 236 number is now equal to the character number. 238 <31> 239 Implementations of the decoding algorithm above MUST protect against 240 decoding invalid sequences. For instance, a naive implementation may 241 decode the overlong UTF-8 sequence C0 80 into the character U+0000, 242 or the surrogate pair ED A1 8C ED BE B4 into U+233B4. Decoding 243 invalid sequences may have security consequences or cause other 244 problems. See Security Considerations (Section 10) below. 246 4. Syntax of UTF-8 Byte Sequences 247 <32> 248 A UTF-8 string is a sequence of octets representing a sequence of UCS 249 characters. An octet sequence is valid UTF-8 only if it matches the 250 following syntax, which is derived from the rules for encoding UTF-8 251 and is expressed in the ABNF of [RFC2234]. 253 UTF8-octets = *( UTF8-char ) 254 UTF8-char = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4 / UTF8-5 / UTF8-6 255 UTF8-1 = %x00-7F 256 UTF8-2 = %xC2-DF UTF8-tail 257 UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) / 258 %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail ) 259 UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F7 3( UTF8-tail ) 260 UTF8-5 = %xF8 %x88-BF 3( UTF8-tail ) / %xF9-FB 4( UTF8-tail ) 261 UTF8-6 = %xFC %x84-BF 4( UTF8-tail ) / %xFD 5( UTF8-tail ) 262 UTF8-tail = %x80-BF 264 5. Versions of the standards 265 <33> 266 ISO/IEC 10646 is updated from time to time by publication of 267 amendments and additional parts; similarly, new versions of the 268 Unicode standard are published over time. Each new version obsoletes 269 and replaces the previous one, but implementations, and more 270 significantly data, are not updated instantly. 271 <34> 272 In general, the changes amount to adding new characters, which does 273 not pose particular problems with old data. In 1996, Amendment 5 to 274 the 1993 edition of ISO/IEC 10646 and Unicode 2.0 moved and expanded 275 the Korean Hangul block, thereby making any previous data containing 276 Hangul characters invalid under the new version. Unicode 2.0 has the 277 same difference from Unicode 1.1. The justification for allowing 278 such an incompatible change was that there were no major 279 implementations and no significant amounts of data containing Hangul. 280 The incident has been dubbed the "Korean mess", and the relevant 281 committees have pledged to never, ever again make such an 282 incompatible change (see Unicode Consortium Policies [1]). 283 <35> 284 New versions, and in particular any incompatible changes, have 285 consequences regarding MIME charset labels, to be discussed in MIME 286 registration (Section 8). 288 6. Byte order mark (BOM) 289 <36> 290 The UCS character U+FEFF "ZERO WIDTH NO-BREAK SPACE" is also known 291 informally as "BYTE ORDER MARK" (abbreviated "BOM"). This character 292 can be used as a genuine "ZERO WIDTH NO-BREAK SPACE" within text, but 293 the BOM name hints at a second possible usage of the character: to 294 prepend a U+FEFF character to a stream of UCS characters as a 295 "signature". A receiver of such a serialized stream may then use the 296 initial character as a hint that the stream consists of UCS 297 characters and also to recognize which UCS encoding is involved and, 298 with encodings having a multi-octet encoding unit, as a way to 299 recognize the serialization order of the octets. UTF-8 having a 300 single-octet encoding unit, this last function is useless and the BOM 301 will always appear as the octet sequence EF BB BF. 302 <37> 303 It is important to understand that the character U+FEFF appearing at 304 any position other than the beginning of a stream MUST be interpreted 305 with the semantics for the zero-width non-breaking space, and MUST 306 NOT be interpreted as a signature. When interpreted as a signature, 307 the Unicode standard suggests than an initial U+FEFF character may be 308 stripped before processing the text. Such stripping is necessary in 309 some cases (e.g. when concatenating two strings, because otherwise 310 the resulting string may contain an unintended "ZERO WIDTH NO-BREAK 311 SPACE" at the connection point), but might affect an external process 312 at a different layer (such as a digital signature or a count of the 313 characters) that is relying on the presence of all characters in the 314 stream. It is therefore RECOMMENDED to avoid stripping an initial 315 U+FEFF interpreted as a signature without a good reason, to ignore it 316 instead of stripping it when appropriate (such as for display) and to 317 strip it only when really necessary. 318 <38> 319 U+FEFF in the first position of a stream MAY be interpreted as a 320 zero-width non-breaking space, and is not always a signature. In an 321 attempt at diminishing this uncertainty, Unicode 3.2 adds a new 322 character, U+2060 "WORD JOINER", with exactly the same semantics and 323 usage as U+FEFF except for the signature function, and strongly 324 recommends its exclusive use for expressing word-joining semantics. 325 Eventually, following this recommendation will make it all but 326 certain that any initial U+FEFF is a signature, not an intended "ZERO 327 WIDTH NO-BREAK SPACE". 328 <39> 329 In the meantime, the uncertainty unfortunately remains and may affect 330 Internet protocols. Protocol specifications MAY restrict usage of 331 U+FEFF as a signature in order to reduce or eliminate the potential 332 ill effects of this uncertainty. In the interest of striking a 333 balance between the advantages (reduction of uncertainty) and 334 drawbacks (loss of the signature function) of such restrictions, it 335 is useful to distinguish a few cases: 337 <40> 338 o A protocol SHOULD forbid use of U+FEFF as a signature for those 339 textual protocol elements that the protocol mandates to be always 340 UTF-8, the signature function being totally useless in those 341 cases. 342 <41> 343 o A protocol SHOULD also forbid use of U+FEFF as a signature for 344 those textual protocol elements for which the protocol provides 345 character encoding identification mechanisms, when it is expected 346 that implementations of the protocol will be in a position to 347 always use the mechanisms properly. This will be the case when 348 the protocol elements are maintained tightly under the control of 349 the implementation from the time of their creation to the time of 350 their (properly labelled) transmission. 351 <42> 352 o A protocol SHOULD NOT forbid use of U+FEFF as a signature for 353 those textual protocol elements for which the protocol does not 354 provide character encoding identification mechanisms, when a ban 355 would be unenforceable, or when it is expected that 356 implementations of the protocol will not be in a position to 357 always use the mechanisms properly. The latter two cases are 358 likely to occur with larger protocol elements such as MIME 359 entities, especially when implementations of the protocol will 360 obtain such entities from file systems, from protocols that do not 361 have encoding identification mechanisms for payloads (such as FTP) 362 or from other protocols that do not guarantee proper 363 identification of character encoding (such as HTTP). 365 <43> 366 When a protocol forbids use of U+FEFF as a signature for a certain 367 protocol element, then any initial U+FEFF in that protocol element 368 MUST be interpreted as a "ZERO WIDTH NO-BREAK SPACE". When a 369 protocol does NOT forbid use of U+FEFF as a signature for a certain 370 protocol element, then implementations SHOULD be prepared to handle a 371 signature in that element and react appropriately: using the 372 signature to identify the character encoding as necessary and 373 stripping or ignoring the signature as appropriate. 375 7. Examples 376 <44> 377 The character sequence U+0041 U+2262 U+0391 U+002E "A." is encoded in UTF-8 as follows: 380 --+--------+-----+-- 381 41 E2 89 A2 CE 91 2E 382 --+--------+-----+-- 383 <45> 384 The character sequence U+D55C U+AD6D U+C5B4 (Korean "hangugeo", 385 meaning "the Korean language") is encoded in UTF-8 as follows: 387 --------+--------+-------- 388 ED 95 9C EA B5 AD EC 96 B4 389 --------+--------+-------- 390 <46> 391 The character sequence U+65E5 U+672C U+8A9E (Japanese "nihongo", 392 meaning "the Japanese language") is encoded in UTF-8 as follows: 394 --------+--------+-------- 395 E6 97 A5 E6 9C AC E8 AA 9E 396 --------+--------+-------- 397 <47> 398 The character U+233B4 (a Chinese character meaning 'stump of tree'), 399 prepended with a UTF-8 BOM, is encoded in UTF-8 as follows: 401 --------+----------- 402 EF BB BF F0 A3 8E B4 403 --------+----------- 405 8. MIME registration 406 <48> 407 This memo serves as the basis for registration of the MIME charset 408 parameter for UTF-8, according to [RFC2978]. The charset parameter 409 value is "UTF-8". This string labels media types containing text 410 consisting of characters from the repertoire of ISO/IEC 10646 411 including all amendments at least up to amendment 5 of the 1993 412 edition (Korean block), encoded to a sequence of octets using the 413 encoding scheme outlined above. UTF-8 is suitable for use in MIME 414 content types under the "text" top-level type. 415 <49> 416 It is noteworthy that the label "UTF-8" does not contain a version 417 identification, referring generically to ISO/IEC 10646. This is 418 intentional, the rationale being as follows: 419 <50> 420 A MIME charset label is designed to give just the information needed 421 to interpret a sequence of bytes received on the wire into a sequence 422 of characters, nothing more (see [RFC2045], section 2.2). As long as 423 a character set standard does not change incompatibly, version 424 numbers serve no purpose, because one gains nothing by learning from 425 the tag that newly assigned characters may be received that one 426 doesn't know about. The tag itself doesn't teach anything about the 427 new characters, which are going to be received anyway. 428 <51> 429 Hence, as long as the standards evolve compatibly, the apparent 430 advantage of having labels that identify the versions is only that, 431 apparent. But there is a disadvantage to such version-dependent 432 labels: when an older application receives data accompanied by a 433 newer, unknown label, it may fail to recognize the label and be 434 completely unable to deal with the data, whereas a generic, known 435 label would have triggered mostly correct processing of the data, 436 which may well not contain any new characters. 437 <52> 438 Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible 439 change, in principle contradicting the appropriateness of a version 440 independent MIME charset label as described above. But the 441 compatibility problem can only appear with data containing Korean 442 Hangul characters encoded according to Unicode 1.1 (or equivalently 443 ISO/IEC 10646 before amendment 5), and there is arguably no such data 444 to worry about, this being the very reason the incompatible change 445 was deemed acceptable. 446 <53> 447 In practice, then, a version-independent label is warranted, provided 448 the label is understood to refer to all versions after Amendment 5, 449 and provided no incompatible change actually occurs. Should 450 incompatible changes occur in a later version of ISO/IEC 10646, the 451 MIME charset label defined here will stay aligned with the previous 452 version until and unless the IETF specifically decides otherwise. 454 9. IANA Considerations 455 <54> 456 The entry for UTF-8 in the IANA charset registry should be updated to 457 point to this memo. 459 10. Security Considerations 460 <55> 461 Implementors of UTF-8 need to consider the security aspects of how 462 they handle illegal UTF-8 sequences. It is conceivable that in some 463 circumstances an attacker would be able to exploit an incautious UTF- 464 8 parser by sending it an octet sequence that is not permitted by the 465 UTF-8 syntax. 466 <56> 467 A particularly subtle form of this attack can be carried out against 468 a parser which performs security-critical validity checks against the 469 UTF-8 encoded form of its input, but interprets certain illegal octet 470 sequences as characters. For example, a parser might prohibit the 471 NUL character when encoded as the single-octet sequence 00, but 472 erroneously allow the illegal two-octet sequence C0 80 and interpret 473 it as a NUL character. Another example might be a parser which 474 prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the 475 illegal octet sequence 2F C0 AE 2E 2F. This last exploit has 476 actually been used in a widespread virus attacking Web servers in 477 2001; the security threat is thus very real. 479 Bibliography 481 [CESU-8] Phipps, T., "Compatibility Encoding Scheme for UTF-16: 482 8-Bit (CESU-8)", UTR 26, April 2002, . 485 [FSS_UTF] X/Open Company Ltd., "X/Open CAE Specification C501 -- 486 File System Safe UCS Transformation Format (FSS_UTF)", 487 ISBN 1-85912-082-2, April 1995. 489 [ISO.10646-1] International Organization for Standardization, 490 "Information Technology - Universal Multiple-octet 491 coded Character Set (UCS) - Part 1: Architecture and 492 Basic Multilingual Plane", ISO Standard 10646-1, 2000. 494 [RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet 495 Mail Extensions (MIME) Part One: Format of Internet 496 Message Bodies", RFC 2045, November 1996. 498 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 499 Requirement Levels", BCP 14, RFC 2119, March 1997. 501 [RFC2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax 502 Specifications: ABNF", RFC 2234, November 1997. 504 [RFC2978] Freed, N. and J. Postel, "IANA Charset Registration 505 Procedures", BCP 19, RFC 2978, October 2000. 507 [UNICODE] The Unicode Consortium, "The Unicode Standard -- 508 Version 3.2", defined by The Unicode Standard, 509 Version 3.0 (Reading, MA, Addison-Wesley, 2000. ISBN 510 0-201-61633-5), as amended by the Unicode Standard 511 Annex #27: Unicode 3.1 (see http://www.unicode.org/ 512 reports/tr27) and by the Unicode Standard Annex #28: 513 Unicode 3.2 (see http://www.unicode.org/reports/tr28), 514 March 2002, . 517 [US-ASCII] American National Standards Institute, "Coded 518 Character Set - 7-bit American Standard Code for 519 Information Interchange", ANSI X3.4, 1986. 521 [1] 523 Author's Address 525 FranȺois Yergeau 526 Alis Technologies 527 100, boul. Alexis-Nihon, bureau 600 528 MontrȨal, QC H4M 2P2 529 Canada 531 Phone: +1 514 747 2547 532 Fax: +1 514 747 2561 533 EMail: fyergeau@alis.com 535 Appendix A. Acknowledgements 536 <65> 537 The following have participated in the drafting and discussion of 538 this memo: James E. Agenbroad, Harald Alvestrand, Andries Brouwer, 539 Mark Davis, Martin J. DÈ­rst, Patrick FÈñltstrȵm, Ned Freed, David 540 Goldsmith, Tony Hansen, Edwin F. Hart, Paul Hoffman, David Hopwood, 541 Simon Josefsson, Kent Karlsson, Markus Kuhn, Michael Kung, Alain 542 LaBontȨ, Ira McDonald, Alexey Melnikov, John Gardiner Myers, Dan 543 Oscarsson, Murray Sargent, Markus Scherer, Keld Simonsen, Arnold 544 Winkler, Kenneth Whistler and Misha Wolf. 546 Appendix B. Changes from RFC 2279 547 <66> 548 <67> 549 o Significantly shortened Introduction. No more mention of UTF-1 or 550 UTF-7, of Transformation Formats. 551 <68> 552 o Straightened out terminology. UTF-8 now described in terms of an 553 encoding form of the character number. UCS-2 and UCS-4 almost 554 disappeared. 555 <69> 556 o Note warning against decoding of invalid sequences turned into a 557 normative MUST NOT. 558 <70> 559 o New section about the UTF-8 BOM, with advice for protocols. 560 <71> 561 o Updated a couple of references (10646-1:2000, Unicode 3.2, RFC 562 2978). 563 <72> 564 o Added TOC. 565 <73> 566 o Removed suggested UNICODE-1-1-UTF-8 MIME charset registration. 567 <74> 568 o New "Notational conventions" section about RFC 2119 and U+HHHH 569 notation. 570 <75> 571 o Pointer to Unicode Consortium Policies added in "Versions of the 572 standards" section. 573 <76> 574 o Added a fourth example with a non-BMP character and a BOM. 575 <77> 576 o Added a paragraph about U+2060 WORD JOINER. 577 <78> 578 o Enumerate more byte values impossible in UTF-8, either as a result 579 of forbidding overlong sequences or of restricting to the UTF-16 580 accessible range. 581 <79> 582 o Added "IANA Considerations" section to ask that the UTF-8 entry in 583 the charset registry point to this memo. 584 <80> 585 o Added an ABNF syntax for valid UTF-8 octet sequences 586 <81> 587 o Added some warning language about CESU-8 589 Full Copyright Statement 591 Copyright (C) The Internet Society (2002). All Rights Reserved. 593 This document and translations of it may be copied and furnished to 594 others, and derivative works that comment on or otherwise explain it 595 or assist in its implementation may be prepared, copied, published 596 and distributed, in whole or in part, without restriction of any 597 kind, provided that the above copyright notice and this paragraph are 598 included on all such copies and derivative works. However, this 599 document itself may not be modified in any way, such as by removing 600 the copyright notice or references to the Internet Society or other 601 Internet organizations, except as needed for the purpose of 602 developing Internet standards in which case the procedures for 603 copyrights defined in the Internet Standards process must be 604 followed, or as required to translate it into languages other than 605 English. 607 The limited permissions granted above are perpetual and will not be 608 revoked by the Internet Society or its successors or assigns. 610 This document and the information contained herein is provided on an 611 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 612 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 613 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 614 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 615 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 617 Acknowledgement 619 Funding for the RFC Editor function is currently provided by the 620 Internet Society.