idnits 2.17.00 (12 Aug 2021) /tmp/idnits55058/draft-yergeau-rfc2279bis-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** Looks like you're using RFC 2026 boilerplate. This must be updated to follow RFC 3978/3979, as updated by RFC 4748. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == No 'Intended status' indicated for this document; assuming Proposed Standard Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- -- The abstract seems to indicate that this document obsoletes RFC2279, but the header doesn't have an 'Obsoletes:' line to match this. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the RFC 3978 Section 5.4 Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (June 9, 2003) is 6914 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Possible downref: Non-RFC (?) normative reference: ref. 'UNICODE' -- Obsolete informational reference (is this intentional?): RFC 2234 (Obsoleted by RFC 4234) Summary: 1 error (**), 0 flaws (~~), 2 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group F. Yergeau 3 Internet-Draft Alis Technologies 4 Expires: December 8, 2003 June 9, 2003 6 UTF-8, a transformation format of ISO 10646 7 draft-yergeau-rfc2279bis-05 9 Status of this Memo 11 This document is an Internet-Draft and is in full conformance with 12 all provisions of Section 10 of RFC2026. 14 Internet-Drafts are working documents of the Internet Engineering 15 Task Force (IETF), its areas, and its working groups. Note that other 16 groups may also distribute working documents as Internet-Drafts. 18 Internet-Drafts are draft documents valid for a maximum of six months 19 and may be updated, replaced, or obsoleted by other documents at any 20 time. It is inappropriate to use Internet-Drafts as reference 21 material or to cite them other than as "work in progress." 23 The list of current Internet-Drafts can be accessed at http:// 24 www.ietf.org/ietf/1id-abstracts.txt. 26 The list of Internet-Draft Shadow Directories can be accessed at 27 http://www.ietf.org/shadow.html. 29 This Internet-Draft will expire on December 8, 2003. 31 Copyright Notice 33 Copyright (C) The Internet Society (2003). All Rights Reserved. 35 Abstract 37 ISO/IEC 10646-1 defines a large character set called the Universal 38 Character Set (UCS) which encompasses most of the world's writing 39 systems. The originally proposed encodings of the UCS, however, were 40 not compatible with many current applications and protocols, and this 41 has led to the development of UTF-8, the object of this memo. UTF-8 42 has the characteristic of preserving the full US-ASCII range, 43 providing compatibility with file systems, parsers and other software 44 that rely on US-ASCII values but are transparent to other values. 45 This memo obsoletes and replaces RFC 2279. 47 Table of Contents 49 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 50 2. Notational conventions . . . . . . . . . . . . . . . . . . . . 4 51 3. UTF-8 definition . . . . . . . . . . . . . . . . . . . . . . . 4 52 4. Syntax of UTF-8 Byte Sequences . . . . . . . . . . . . . . . . 6 53 5. Versions of the standards . . . . . . . . . . . . . . . . . . 6 54 6. Byte order mark (BOM) . . . . . . . . . . . . . . . . . . . . 7 55 7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 56 8. MIME registration . . . . . . . . . . . . . . . . . . . . . . 9 57 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 58 10. Security Considerations . . . . . . . . . . . . . . . . . . . 11 59 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11 60 12. Changes from RFC 2279 . . . . . . . . . . . . . . . . . . . . 12 61 Normative references . . . . . . . . . . . . . . . . . . . . . 12 62 Informative references . . . . . . . . . . . . . . . . . . . . 13 63 Author's Address . . . . . . . . . . . . . . . . . . . . . . . 14 64 Intellectual Property and Copyright Statements . . . . . . . . 15 66 1. Introduction 68 ISO/IEC 10646 [ISO.10646] defines a large character set called the 69 Universal Character Set (UCS), which encompasses most of the world's 70 writing systems. The same set of characters is defined by the Unicode 71 standard [UNICODE], which further defines additional character 72 properties and other application details of great interest to 73 implementers. Up to the present time, changes in Unicode and 74 amendments and additions to ISO/IEC 10646 have tracked each other, so 75 that the character repertoires and code point assignments have 76 remained in sync. The relevant standardization committees have 77 committed to maintain this very useful synchronism. 79 ISO/IEC 10646 and Unicode define several encoding forms of their 80 common repertoire: UTF-8, UCS-2, UTF-16, UCS-4 and UTF-32. In an 81 encoding form, each character is represented as one or more encoding 82 units. All standard UCS encoding forms except UTF-8 have an encoding 83 unit larger than one octet, making them hard to use in many current 84 applications and protocols that assume 8 or even 7 bit characters. 86 UTF-8, the object of this memo, has a one-octet encoding unit. It 87 uses all bits of an octet, but has the quality of preserving the full 88 US-ASCII [US-ASCII] range: US-ASCII characters are encoded in one 89 octet having the normal US-ASCII value, and any octet with such a 90 value can only stand for a US-ASCII character, and nothing else. 92 UTF-8 encodes UCS characters as a varying number of octets, where the 93 number of octets, and the value of each, depend on the integer value 94 assigned to the character in ISO/IEC 10646 (the character number, 95 a.k.a. code position, code point or Unicode scalar value). This 96 encoding form has the following characteristics (all values are in 97 hexadecimal): 99 o Character numbers from U+0000 to U+007F (US-ASCII repertoire) 100 correspond to octets 00 to 7F (7 bit US-ASCII values). A direct 101 consequence is that a plain ASCII string is also a valid UTF-8 102 string. 104 o US-ASCII octet values do not appear otherwise in a UTF-8 encoded 105 character stream. This provides compatibility with file systems 106 or other software (e.g. the printf() function in C libraries) that 107 parse based on US-ASCII values but are transparent to other 108 values. 110 o Round-trip conversion is easy between UTF-8 and other encoding 111 forms. 113 o The first octet of a multi-octet sequence indicates the number of 114 octets in the sequence. 116 o The octet values C0, C1, F5 to FF never appear. 118 o Character boundaries are easily found from anywhere in an octet 119 stream. 121 o The byte-value lexicographic sorting order of UTF-8 strings is the 122 same as if ordered by character numbers. Of course this is of 123 limited interest since a sort order based on character numbers is 124 not culturally valid. 126 o The Boyer-Moore fast search algorithm can be used with UTF-8 data. 128 o UTF-8 strings can be fairly reliably recognized as such by a 129 simple algorithm, i.e. the probability that a string of characters 130 in any other encoding appears as valid UTF-8 is low, diminishing 131 with increasing string length. 133 UTF-8 was originally a project of the X/Open Joint 134 Internationalization Group XOJIG with the objective to specify a File 135 System Safe UCS Transformation Format [FSS_UTF] that is compatible 136 with UNIX systems, supporting multilingual text in a single encoding. 137 The original authors were Gary Miller, Greger Leijonhufvud and John 138 Entenmann. Later, Ken Thompson and Rob Pike did significant work for 139 the formal definition of UTF-8. 141 2. Notational conventions 143 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 144 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 145 document are to be interpreted as described in [RFC2119]. 147 UCS characters are designated by the U+HHHH notation, where HHHH is a 148 string of from 4 to 6 hexadecimal digits representing the character 149 number in ISO/IEC 10646. 151 3. UTF-8 definition 153 UTF-8 is defined by the Unicode Standard [UNICODE]. Descriptions and 154 formulae can also be found in Annex D of ISO/IEC 10646-1 [ISO.10646] 156 In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16 157 accessible range) are encoded using sequences of 1 to 4 octets. The 158 only octet of a "sequence" of one has the higher-order bit set to 0, 159 the remaining 7 bits being used to encode the character number. In a 160 sequence of n octets, n>1, the initial octet has the n higher-order 161 bits set to 1, followed by a bit set to 0. The remaining bit(s) of 162 that octet contain bits from the number of the character to be 163 encoded. The following octet(s) all have the higher-order bit set to 164 1 and the following bit set to 0, leaving 6 bits in each to contain 165 bits from the character to be encoded. 167 The table below summarizes the format of these different octet types. 168 The letter x indicates bits available for encoding bits of the 169 character number. 171 Char. number range | UTF-8 octet sequence 172 (hexadecimal) | (binary) 173 --------------------+--------------------------------------------- 174 0000 0000-0000 007F | 0xxxxxxx 175 0000 0080-0000 07FF | 110xxxxx 10xxxxxx 176 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx 177 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 179 Encoding a character to UTF-8 proceeds as follows: 181 1. Determine the number of octets required from the character number 182 and the first column of the table above. It is important to note 183 that the rows of the table are mutually exclusive, i.e. there is 184 only one valid way to encode a given character. 186 2. Prepare the high-order bits of the octets as per the second 187 column of the table. 189 3. Fill in the bits marked x from the bits of the character number, 190 expressed in binary. Start by putting the lowest-order bit of the 191 character number in the lowest-order position of the last octet 192 of the sequence, then put the next higher-order bit of the 193 character number in the next higher-order position of that octet, 194 etc. When the x bits of the last octet are filled in, move on to 195 the next to last octet, then to the preceding one, etc. until all 196 x bits are filled in. 198 The definition of UTF-8 prohibits encoding character numbers between 199 U+D800 and U+DFFF, which are reserved for use with the UTF-16 200 encoding form (as surrogate pairs) and do not directly represent 201 characters. When encoding in UTF-8 from UTF-16 data, it is necessary 202 to first decode the UTF-16 data to obtain character numbers, which 203 are then encoded in UTF-8 as described above. This contrasts with 204 CESU-8 [CESU-8], which is a UTF-8-like encoding that is not meant for 205 use on the Internet. CESU-8 operates similarly to UTF-8 but encodes 206 the UTF-16 code values (16-bit quantities) instead of the character 207 number (code point). This leads to different results for character 208 numbers above 0xFFFF; the CESU-8 encoding of those characters is NOT 209 valid UTF-8. 211 Decoding a UTF-8 character proceeds as follows: 213 1. Initialize a binary number with all bits set to 0. Up to 21 bits 214 may be needed. 216 2. Determine which bits encode the character number from the number 217 of octets in the sequence and the second column of the table 218 above (the bits marked x). 220 3. Distribute the bits from the sequence to the binary number, first 221 the lower-order bits from the last octet of the sequence and 222 proceeding to the left until no x bits are left. The binary 223 number is now equal to the character number. 225 Implementations of the decoding algorithm above MUST protect against 226 decoding invalid sequences. For instance, a naive implementation may 227 decode the overlong UTF-8 sequence C0 80 into the character U+0000, 228 or the surrogate pair ED A1 8C ED BE B4 into U+233B4. Decoding 229 invalid sequences may have security consequences or cause other 230 problems. See Security Considerations (Section 10) below. 232 4. Syntax of UTF-8 Byte Sequences 234 For the convenience of implementors using ABNF, a definition of UTF-8 235 in ABNF syntax is given here. 237 A UTF-8 string is a sequence of octets representing a sequence of UCS 238 characters. An octet sequence is valid UTF-8 only if it matches the 239 following syntax, which is derived from the rules for encoding UTF-8 240 and is expressed in the ABNF of [RFC2234]. 242 UTF8-octets = *( UTF8-char ) 243 UTF8-char = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4 244 UTF8-1 = %x00-7F 245 UTF8-2 = %xC2-DF UTF8-tail 246 UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) / 247 %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail ) 248 UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) / 249 %xF4 %x80-8F 2( UTF8-tail ) 250 UTF8-tail = %x80-BF 252 NOTE -- The authoritative definition of UTF-8 is in [UNICODE]. This 253 grammar is believed to describe the same thing as what Unicode 254 describes, but does not claim to be authoritative. Implementors are 255 urged to rely on the authoritative source, rather than on this ABNF. 257 5. Versions of the standards 258 ISO/IEC 10646 is updated from time to time by publication of 259 amendments and additional parts; similarly, new versions of the 260 Unicode standard are published over time. Each new version obsoletes 261 and replaces the previous one, but implementations, and more 262 significantly data, are not updated instantly. 264 In general, the changes amount to adding new characters, which does 265 not pose particular problems with old data. In 1996, Amendment 5 to 266 the 1993 edition of ISO/IEC 10646 and Unicode 2.0 moved and expanded 267 the Korean Hangul block, thereby making any previous data containing 268 Hangul characters invalid under the new version. Unicode 2.0 has the 269 same difference from Unicode 1.1. The justification for allowing such 270 an incompatible change was that there were no major implementations 271 and no significant amounts of data containing Hangul. The incident 272 has been dubbed the "Korean mess", and the relevant committees have 273 pledged to never, ever again make such an incompatible change (see 274 Unicode Consortium Policies [1]). 276 New versions, and in particular any incompatible changes, have 277 consequences regarding MIME charset labels, to be discussed in MIME 278 registration (Section 8). 280 6. Byte order mark (BOM) 282 The UCS character U+FEFF "ZERO WIDTH NO-BREAK SPACE" is also known 283 informally as "BYTE ORDER MARK" (abbreviated "BOM"). This character 284 can be used as a genuine "ZERO WIDTH NO-BREAK SPACE" within text, but 285 the BOM name hints at a second possible usage of the character: to 286 prepend a U+FEFF character to a stream of UCS characters as a 287 "signature". A receiver of such a serialized stream may then use the 288 initial character as a hint that the stream consists of UCS 289 characters and also to recognize which UCS encoding is involved and, 290 with encodings having a multi-octet encoding unit, as a way to 291 recognize the serialization order of the octets. UTF-8 having a 292 single-octet encoding unit, this last function is useless and the BOM 293 will always appear as the octet sequence EF BB BF. 295 It is important to understand that the character U+FEFF appearing at 296 any position other than the beginning of a stream MUST be interpreted 297 with the semantics for the zero-width non-breaking space, and MUST 298 NOT be interpreted as a signature. When interpreted as a signature, 299 the Unicode standard suggests than an initial U+FEFF character may be 300 stripped before processing the text. Such stripping is necessary in 301 some cases (e.g. when concatenating two strings, because otherwise 302 the resulting string may contain an unintended "ZERO WIDTH NO-BREAK 303 SPACE" at the connection point), but might affect an external process 304 at a different layer (such as a digital signature or a count of the 305 characters) that is relying on the presence of all characters in the 306 stream. It is therefore RECOMMENDED to avoid stripping an initial 307 U+FEFF interpreted as a signature without a good reason, to ignore it 308 instead of stripping it when appropriate (such as for display) and to 309 strip it only when really necessary. 311 U+FEFF in the first position of a stream MAY be interpreted as a 312 zero-width non-breaking space, and is not always a signature. In an 313 attempt at diminishing this uncertainty, Unicode 3.2 adds a new 314 character, U+2060 "WORD JOINER", with exactly the same semantics and 315 usage as U+FEFF except for the signature function, and strongly 316 recommends its exclusive use for expressing word-joining semantics. 317 Eventually, following this recommendation will make it all but 318 certain that any initial U+FEFF is a signature, not an intended "ZERO 319 WIDTH NO-BREAK SPACE". 321 In the meantime, the uncertainty unfortunately remains and may affect 322 Internet protocols. Protocol specifications MAY restrict usage of 323 U+FEFF as a signature in order to reduce or eliminate the potential 324 ill effects of this uncertainty. In the interest of striking a 325 balance between the advantages (reduction of uncertainty) and 326 drawbacks (loss of the signature function) of such restrictions, it 327 is useful to distinguish a few cases: 329 o A protocol SHOULD forbid use of U+FEFF as a signature for those 330 textual protocol elements that the protocol mandates to be always 331 UTF-8, the signature function being totally useless in those 332 cases. 334 o A protocol SHOULD also forbid use of U+FEFF as a signature for 335 those textual protocol elements for which the protocol provides 336 character encoding identification mechanisms, when it is expected 337 that implementations of the protocol will be in a position to 338 always use the mechanisms properly. This will be the case when 339 the protocol elements are maintained tightly under the control of 340 the implementation from the time of their creation to the time of 341 their (properly labeled) transmission. 343 o A protocol SHOULD NOT forbid use of U+FEFF as a signature for 344 those textual protocol elements for which the protocol does not 345 provide character encoding identification mechanisms, when a ban 346 would be unenforceable, or when it is expected that 347 implementations of the protocol will not be in a position to 348 always use the mechanisms properly. The latter two cases are 349 likely to occur with larger protocol elements such as MIME 350 entities, especially when implementations of the protocol will 351 obtain such entities from file systems, from protocols that do not 352 have encoding identification mechanisms for payloads (such as FTP) 353 or from other protocols that do not guarantee proper 354 identification of character encoding (such as HTTP). 356 When a protocol forbids use of U+FEFF as a signature for a certain 357 protocol element, then any initial U+FEFF in that protocol element 358 MUST be interpreted as a "ZERO WIDTH NO-BREAK SPACE". When a protocol 359 does NOT forbid use of U+FEFF as a signature for a certain protocol 360 element, then implementations SHOULD be prepared to handle a 361 signature in that element and react appropriately: using the 362 signature to identify the character encoding as necessary and 363 stripping or ignoring the signature as appropriate. 365 7. Examples 367 The character sequence U+0041 U+2262 U+0391 U+002E "A." is encoded in UTF-8 as follows: 370 --+--------+-----+-- 371 41 E2 89 A2 CE 91 2E 372 --+--------+-----+-- 374 The character sequence U+D55C U+AD6D U+C5B4 (Korean "hangugeo", 375 meaning "the Korean language") is encoded in UTF-8 as follows: 377 --------+--------+-------- 378 ED 95 9C EA B5 AD EC 96 B4 379 --------+--------+-------- 381 The character sequence U+65E5 U+672C U+8A9E (Japanese "nihongo", 382 meaning "the Japanese language") is encoded in UTF-8 as follows: 384 --------+--------+-------- 385 E6 97 A5 E6 9C AC E8 AA 9E 386 --------+--------+-------- 388 The character U+233B4 (a Chinese character meaning 'stump of tree'), 389 prepended with a UTF-8 BOM, is encoded in UTF-8 as follows: 391 --------+----------- 392 EF BB BF F0 A3 8E B4 393 --------+----------- 395 8. MIME registration 397 This memo serves as the basis for registration of the MIME charset 398 parameter for UTF-8, according to [RFC2978]. The charset parameter 399 value is "UTF-8". This string labels media types containing text 400 consisting of characters from the repertoire of ISO/IEC 10646 401 including all amendments at least up to amendment 5 of the 1993 402 edition (Korean block), encoded to a sequence of octets using the 403 encoding scheme outlined above. UTF-8 is suitable for use in MIME 404 content types under the "text" top-level type. 406 It is noteworthy that the label "UTF-8" does not contain a version 407 identification, referring generically to ISO/IEC 10646. This is 408 intentional, the rationale being as follows: 410 A MIME charset label is designed to give just the information needed 411 to interpret a sequence of bytes received on the wire into a sequence 412 of characters, nothing more (see [RFC2045], section 2.2). As long as 413 a character set standard does not change incompatibly, version 414 numbers serve no purpose, because one gains nothing by learning from 415 the tag that newly assigned characters may be received that one 416 doesn't know about. The tag itself doesn't teach anything about the 417 new characters, which are going to be received anyway. 419 Hence, as long as the standards evolve compatibly, the apparent 420 advantage of having labels that identify the versions is only that, 421 apparent. But there is a disadvantage to such version-dependent 422 labels: when an older application receives data accompanied by a 423 newer, unknown label, it may fail to recognize the label and be 424 completely unable to deal with the data, whereas a generic, known 425 label would have triggered mostly correct processing of the data, 426 which may well not contain any new characters. 428 Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible 429 change, in principle contradicting the appropriateness of a version 430 independent MIME charset label as described above. But the 431 compatibility problem can only appear with data containing Korean 432 Hangul characters encoded according to Unicode 1.1 (or equivalently 433 ISO/IEC 10646 before amendment 5), and there is arguably no such data 434 to worry about, this being the very reason the incompatible change 435 was deemed acceptable. 437 In practice, then, a version-independent label is warranted, provided 438 the label is understood to refer to all versions after Amendment 5, 439 and provided no incompatible change actually occurs. Should 440 incompatible changes occur in a later version of ISO/IEC 10646, the 441 MIME charset label defined here will stay aligned with the previous 442 version until and unless the IETF specifically decides otherwise. 444 9. IANA Considerations 446 The entry for UTF-8 in the IANA charset registry should be updated to 447 point to this memo. 449 10. Security Considerations 451 Implementers of UTF-8 need to consider the security aspects of how 452 they handle illegal UTF-8 sequences. It is conceivable that in some 453 circumstances an attacker would be able to exploit an incautious 454 UTF-8 parser by sending it an octet sequence that is not permitted by 455 the UTF-8 syntax. 457 A particularly subtle form of this attack can be carried out against 458 a parser which performs security-critical validity checks against the 459 UTF-8 encoded form of its input, but interprets certain illegal octet 460 sequences as characters. For example, a parser might prohibit the 461 NUL character when encoded as the single-octet sequence 00, but 462 erroneously allow the illegal two-octet sequence C0 80 and interpret 463 it as a NUL character. Another example might be a parser which 464 prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the 465 illegal octet sequence 2F C0 AE 2E 2F. This last exploit has actually 466 been used in a widespread virus attacking Web servers in 2001; the 467 security threat is thus very real. 469 Another security issue occurs when encoding to UTF-8: the ISO/IEC 470 10646 description of UTF-8 allows encoding character numbers up to 471 U+7FFFFFFF, yielding sequences of up to 6 bytes. There is therefore 472 a risk of buffer overflow if the range of character numbers is not 473 explicitly limited to U+10FFFF or if buffer sizing doesn't take into 474 account the possibility of 5- and 6-byte sequences. 476 Security may also be impacted by a characteristic of several 477 character encodings, including UTF-8: the "same thing" (as far as a 478 user can tell) can be represented by several distinct character 479 sequences. For instance, an e with acute accent can be represented by 480 the precomposed U+00E9 E ACUTE character or by the canonically 481 equivalent sequence U+0065 U+0301 (E + COMBINING ACUTE). Even though 482 UTF-8 provides a single byte sequence for each character sequence, 483 the existence of multiple character sequences for "the same thing" 484 may have security consequences whenever string matching, indexing, 485 searching, sorting, regular expression matching and selection are 486 involved. An example would be string matching of an identifier 487 appearing in a credential and in access control list entries. This 488 issue is amenable to solutions based on Unicode Normalization Forms, 489 see [UAX15]. 491 11. Acknowledgements 493 The following have participated in the drafting and discussion of 494 this memo: James E. Agenbroad, Harald Alvestrand, Andries Brouwer, 495 Mark Davis, Martin J. Duerst, Patrick Faltstrom, Ned Freed, David 496 Goldsmith, Tony Hansen, Edwin F. Hart, Paul Hoffman, David Hopwood, 497 Simon Josefsson, Kent Karlsson, Dan Kohn, Markus Kuhn, Michael Kung, 498 Alain LaBonte, Ira McDonald, Alexey Melnikov, MURATA Makoto, John 499 Gardiner Myers, Chris Newman, Dan Oscarsson, Roozbeh Pournader, 500 Murray Sargent, Markus Scherer, Keld Simonsen, Arnold Winkler, 501 Kenneth Whistler and Misha Wolf. 503 12. Changes from RFC 2279 505 o Restricted the range of characters to 0000-10FFFF (the UTF-16 506 accessible range). 508 o Made Unicode the source of the normative definition of UTF-8, 509 keeping ISO/IEC 10646 as the reference for characters. 511 o Straightened out terminology. UTF-8 now described in terms of an 512 encoding form of the character number. UCS-2 and UCS-4 almost 513 disappeared. 515 o Turned the note warning against decoding of invalid sequences into 516 a normative MUST NOT. 518 o Added a new section about the UTF-8 BOM, with advice for 519 protocols. 521 o Removed suggested UNICODE-1-1-UTF-8 MIME charset registration. 523 o Added an ABNF syntax for valid UTF-8 octet sequences 525 o Expanded Security Considerations section, in particular impact of 526 Unicode normalization 528 Normative references 530 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 531 Requirement Levels", BCP 14, RFC 2119, March 1997. 533 [ISO.10646] 534 International Organization for Standardization, 535 "Information Technology - Universal Multiple-octet coded 536 Character Set (UCS)", ISO/IEC Standard 10646, comprised 537 of ISO/IEC 10646-1:2000, "Information technology -- 538 Universal Multiple-Octet Coded Character Set (UCS) -- Part 539 1: Architecture and Basic Multilingual Plane", ISO/IEC 540 10646-2:2001, "Information technology -- Universal 541 Multiple-Octet Coded Character Set (UCS) -- Part 2: 542 Supplementary Planes" and ISO/IEC 10646-1:2000/Amd 1:2002, 543 "Mathematical symbols and other characters". 545 [UNICODE] The Unicode Consortium, "The Unicode Standard -- Version 546 4.0", defined by The Unicode Standard, Version 4.0 547 (Reading, MA, Addison-Wesley, 2003. ISBN 0-321-18578-1), 548 April 2003, . 551 Informative references 553 [CESU-8] Phipps, T., "Unicode Technical Report #26: Compatibility 554 Encoding Scheme for UTF-16: 8-Bit (CESU-8)", UTR 26, April 555 2002, . 557 [FSS_UTF] X/Open Company Ltd., "X/Open CAE Specification C501 -- 558 File System Safe UCS Transformation Format (FSS_UTF)", 559 ISBN 1-85912-082-2, April 1995. 561 [RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet Mail 562 Extensions (MIME) Part One: Format of Internet Message 563 Bodies", RFC 2045, November 1996. 565 [RFC2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax 566 Specifications: ABNF", RFC 2234, November 1997. 568 [RFC2978] Freed, N. and J. Postel, "IANA Charset Registration 569 Procedures", BCP 19, RFC 2978, October 2000. 571 [UAX15] Davis, M. and M. Duerst, "Unicode Standard Annex #15: 572 Unicode Normalization Forms", An integral part of The 573 Unicode Standard, Version 4.0.0, April 2003, . 576 [US-ASCII] 577 American National Standards Institute, "Coded Character 578 Set - 7-bit American Standard Code for Information 579 Interchange", ANSI X3.4, 1986. 581 URIs 583 [1] 585 Author's Address 587 Francois Yergeau 588 Alis Technologies 589 100, boul. Alexis-Nihon, bureau 600 590 Montreal, QC H4M 2P2 591 Canada 593 Phone: +1 514 747 2547 594 Fax: +1 514 747 2561 595 EMail: fyergeau@alis.com 597 Intellectual Property Statement 599 The IETF takes no position regarding the validity or scope of any 600 intellectual property or other rights that might be claimed to 601 pertain to the implementation or use of the technology described in 602 this document or the extent to which any license under such rights 603 might or might not be available; neither does it represent that it 604 has made any effort to identify any such rights. Information on the 605 IETF's procedures with respect to rights in standards-track and 606 standards-related documentation can be found in BCP-11. Copies of 607 claims of rights made available for publication and any assurances of 608 licenses to be made available, or the result of an attempt made to 609 obtain a general license or permission for the use of such 610 proprietary rights by implementors or users of this specification can 611 be obtained from the IETF Secretariat. 613 The IETF invites any interested party to bring to its attention any 614 copyrights, patents or patent applications, or other proprietary 615 rights which may cover technology that may be required to practice 616 this standard. Please address the information to the IETF Executive 617 Director. 619 Full Copyright Statement 621 Copyright (C) The Internet Society (2003). All Rights Reserved. 623 This document and translations of it may be copied and furnished to 624 others, and derivative works that comment on or otherwise explain it 625 or assist in its implementation may be prepared, copied, published 626 and distributed, in whole or in part, without restriction of any 627 kind, provided that the above copyright notice and this paragraph are 628 included on all such copies and derivative works. However, this 629 document itself may not be modified in any way, such as by removing 630 the copyright notice or references to the Internet Society or other 631 Internet organizations, except as needed for the purpose of 632 developing Internet standards in which case the procedures for 633 copyrights defined in the Internet Standards process must be 634 followed, or as required to translate it into languages other than 635 English. 637 The limited permissions granted above are perpetual and will not be 638 revoked by the Internet Society or its successors or assignees. 640 This document and the information contained herein is provided on an 641 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING 642 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING 643 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION 644 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF 645 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 647 Acknowledgement 649 Funding for the RFC Editor function is currently provided by the 650 Internet Society.