idnits 2.17.00 (12 Aug 2021) /tmp/idnits10433/draft-ietf-idnabis-rationale-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** It looks like you're using RFC 3978 boilerplate. You should update this to the boilerplate described in the IETF Trust License Policy document (see https://trustee.ietf.org/license-info), which is required now. -- Found old boilerplate from RFC 3978, Section 5.1 on line 16. -- Found old boilerplate from RFC 3978, Section 5.5, updated by RFC 4748 on line 2059. -- Found old boilerplate from RFC 3979, Section 5, paragraph 1 on line 2070. -- Found old boilerplate from RFC 3979, Section 5, paragraph 2 on line 2077. -- Found old boilerplate from RFC 3979, Section 5, paragraph 3 on line 2083. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The document seems to lack a both a reference to RFC 2119 and the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. RFC 2119 keyword, line 876: '... CONTEXTUAL RULE REQUIRED category and...' RFC 2119 keyword, line 2013: '... MUST, SHOULD, or MAY) and small ...' Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust Copyright Line does not match the current year -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (December 15, 2008) is 4904 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'ASCII' is defined on line 1775, but no explicit reference was found in the text == Unused Reference: 'RFC1034' is defined on line 1852, but no explicit reference was found in the text == Unused Reference: 'RFC1035' is defined on line 1855, but no explicit reference was found in the text == Unused Reference: 'RFC2181' is defined on line 1861, but no explicit reference was found in the text == Unused Reference: 'RFC2673' is defined on line 1867, but no explicit reference was found in the text == Unused Reference: 'Unicode-UTR36' is defined on line 1912, but no explicit reference was found in the text == Outdated reference: draft-ietf-idnabis-defs has been published as RFC 5890 == Outdated reference: draft-ietf-idnabis-tables has been published as RFC 5892 ** Obsolete normative reference: RFC 3490 (Obsoleted by RFC 5890, RFC 5891) -- Obsolete informational reference (is this intentional?): RFC 810 (Obsoleted by RFC 952) -- Obsolete informational reference (is this intentional?): RFC 2673 (Obsoleted by RFC 6891) -- Obsolete informational reference (is this intentional?): RFC 3454 (Obsoleted by RFC 7564) -- Obsolete informational reference (is this intentional?): RFC 3491 (Obsoleted by RFC 5891) Summary: 3 errors (**), 0 flaws (~~), 9 warnings (==), 11 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group J. Klensin 3 Internet-Draft December 15, 2008 4 Intended status: Informational 5 Expires: June 18, 2009 7 Internationalized Domain Names for Applications (IDNA): Background, 8 Explanation, and Rationale 9 draft-ietf-idnabis-rationale-06.txt 11 Status of this Memo 13 By submitting this Internet-Draft, each author represents that any 14 applicable patent or other IPR claims of which he or she is aware 15 have been or will be disclosed, and any of which he or she becomes 16 aware will be disclosed, in accordance with Section 6 of BCP 79. 18 Internet-Drafts are working documents of the Internet Engineering 19 Task Force (IETF), its areas, and its working groups. Note that 20 other groups may also distribute working documents as Internet- 21 Drafts. 23 Internet-Drafts are draft documents valid for a maximum of six months 24 and may be updated, replaced, or obsoleted by other documents at any 25 time. It is inappropriate to use Internet-Drafts as reference 26 material or to cite them other than as "work in progress." 28 The list of current Internet-Drafts can be accessed at 29 http://www.ietf.org/ietf/1id-abstracts.txt. 31 The list of Internet-Draft Shadow Directories can be accessed at 32 http://www.ietf.org/shadow.html. 34 This Internet-Draft will expire on June 18, 2009. 36 Abstract 38 Several years have passed since the original protocol for 39 Internationalized Domain Names (IDNs) was completed and deployed. 40 During that time, a number of issues have arisen, including the need 41 to update the system to deal with newer versions of Unicode. Some of 42 these issues require tuning of the existing protocols and the tables 43 on which they depend. This document provides an overview of a 44 revised system and provides explanatory material for its components. 46 Table of Contents 48 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 49 1.1. Context and Overview . . . . . . . . . . . . . . . . . . . 4 50 1.2. Discussion Forum . . . . . . . . . . . . . . . . . . . . . 4 51 1.3. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 52 1.3.1. Documents and Standards . . . . . . . . . . . . . . . 5 53 1.3.2. DNS "Name" Terminology . . . . . . . . . . . . . . . . 5 54 1.3.3. New Terminology and Restrictions . . . . . . . . . . . 5 55 1.4. Objectives . . . . . . . . . . . . . . . . . . . . . . . . 6 56 1.5. Applicability and Function of IDNA . . . . . . . . . . . . 7 57 1.6. Comprehensibility of IDNA Mechanisms and Processing . . . 8 58 2. Processing in IDNA2008 . . . . . . . . . . . . . . . . . . . . 9 59 3. Permitted Characters: An Inclusion List . . . . . . . . . . . 9 60 3.1. A Tiered Model of Permitted Characters and Labels . . . . 10 61 3.1.1. PROTOCOL-VALID . . . . . . . . . . . . . . . . . . . . 10 62 3.1.1.1. Contextual Rules . . . . . . . . . . . . . . . . . 11 63 3.1.1.2. Rules and Their Application . . . . . . . . . . . 11 64 3.1.2. DISALLOWED . . . . . . . . . . . . . . . . . . . . . . 12 65 3.1.3. UNASSIGNED . . . . . . . . . . . . . . . . . . . . . . 13 66 3.2. Registration Policy . . . . . . . . . . . . . . . . . . . 13 67 3.3. Layered Restrictions: Tables, Context, Registration, 68 Applications . . . . . . . . . . . . . . . . . . . . . . . 13 69 4. Issues that Constrain Possible Solutions . . . . . . . . . . . 14 70 4.1. Display and Network Order . . . . . . . . . . . . . . . . 14 71 4.2. Entry and Display in Applications . . . . . . . . . . . . 15 72 4.3. Linguistic Expectations: Ligatures, Digraphs, and 73 Alternate Character Forms . . . . . . . . . . . . . . . . 16 74 4.4. Case Mapping and Related Issues . . . . . . . . . . . . . 19 75 4.5. Right to Left Text . . . . . . . . . . . . . . . . . . . . 20 76 5. IDNs and the Robustness Principle . . . . . . . . . . . . . . 20 77 6. Front-end and User Interface Processing . . . . . . . . . . . 21 78 7. Migration from IDNA2003 and Unicode Version Synchronization . 24 79 7.1. Design Criteria . . . . . . . . . . . . . . . . . . . . . 24 80 7.1.1. General IDNA Validity Criteria . . . . . . . . . . . . 24 81 7.1.2. Labels in Registration . . . . . . . . . . . . . . . . 26 82 7.1.3. Labels in Lookup . . . . . . . . . . . . . . . . . . . 27 83 7.2. Changes in Character Interpretations . . . . . . . . . . . 28 84 7.3. More Flexibility in User Agents . . . . . . . . . . . . . 29 85 7.4. The Question of Prefix Changes . . . . . . . . . . . . . . 31 86 7.4.1. Conditions Requiring a Prefix Change . . . . . . . . . 31 87 7.4.2. Conditions Not Requiring a Prefix Change . . . . . . . 32 88 7.4.3. Implications of Prefix Changes . . . . . . . . . . . . 32 89 7.5. Stringprep Changes and Compatibility . . . . . . . . . . . 32 90 7.6. The Symbol Question . . . . . . . . . . . . . . . . . . . 33 91 7.7. Migration Between Unicode Versions: Unassigned Code 92 Points . . . . . . . . . . . . . . . . . . . . . . . . . . 34 93 7.8. Other Compatibility Issues . . . . . . . . . . . . . . . . 35 95 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 36 96 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 37 97 10. Internationalization Considerations . . . . . . . . . . . . . 37 98 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 37 99 11.1. IDNA Character Registry . . . . . . . . . . . . . . . . . 37 100 11.2. IDNA Context Registry . . . . . . . . . . . . . . . . . . 38 101 11.3. IANA Repository of IDN Practices of TLDs . . . . . . . . . 38 102 12. Security Considerations . . . . . . . . . . . . . . . . . . . 38 103 12.1. General Security Issues with IDNA . . . . . . . . . . . . 38 104 13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 38 105 13.1. Normative References . . . . . . . . . . . . . . . . . . . 38 106 13.2. Informative References . . . . . . . . . . . . . . . . . . 40 107 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 41 108 A.1. Changes between Version -00 and Version -01 of 109 draft-ietf-idnabis-rationale . . . . . . . . . . . . . . . 41 110 A.2. Version -02 . . . . . . . . . . . . . . . . . . . . . . . 42 111 A.3. Version -03 . . . . . . . . . . . . . . . . . . . . . . . 42 112 A.4. Version -04 . . . . . . . . . . . . . . . . . . . . . . . 43 113 A.5. Version -05 . . . . . . . . . . . . . . . . . . . . . . . 43 114 A.6. Version -06 . . . . . . . . . . . . . . . . . . . . . . . 43 115 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 44 116 Intellectual Property and Copyright Statements . . . . . . . . . . 45 118 1. Introduction 120 1.1. Context and Overview 122 The original standards for Internationalized Domain Names (IDNs) were 123 completed and deployed starting in 2003. Those standards are known 124 as Internationalized Domain Names in Applications (IDNA), taken from 125 the name of the highest level standard within the group, RFC 3490 126 [RFC3490]. After those standards were deployed, a number of issues 127 arose that led to a call for a new version of the IDNA protocol and 128 the associated tables, including a subset of those described in a 129 recent IAB report [RFC4690] and the need to update the system to deal 130 with newer versions of Unicode. This document further explains the 131 issues that have been encountered when they are important to 132 understanding of the revised protocols. It also provides an overview 133 of the new IDNA model and explanatory material for it. Additional 134 explanatory material for the specific components of the proposals 135 appears with the associated documents. 137 A good deal of the background material that appeared in RFC 3490 138 [RFC3490] has been removed from this update. That material is either 139 of historical interest only or has been covered from a more recent 140 perspective in RFC 4690 [RFC4690]. 142 This document is not normative. The information it provides is 143 intended to make the rules, tables, and protocol easier to understand 144 and to provide overview information and suggestions for zone 145 administrators and others who need to make policy, deployment, and 146 similar decisions about IDNs. 148 1.2. Discussion Forum 150 [[ RFC Editor: please remove this section. ]] 152 IDNA2008 is being discussed in the IETF "idnabis" Working Group and 153 on the mailing list idna-update@alvestrand.no 155 1.3. Terminology 157 Terminology that is critical for understanding this document and the 158 rest of the documents that make up IDNA2008, appears in 159 [IDNA2008-Defs]. That document also contains roadmap to the IDNA2008 160 document collection. No attempt should be made to understand this 161 document without the definitions and concepts that appear there. 163 1.3.1. Documents and Standards 165 This document uses the term "IDNA2003" to refer to the set of 166 standards that make up and support the version of IDNA published in 167 2003, i.e., those commonly known as the IDNA base specification 168 [RFC3490], Nameprep [RFC3491], Punycode [RFC3492], and Stringprep 169 [RFC3454]. In this document, those names are used to refer, 170 conceptually, to the individual documents, with the base IDNA 171 specification called just "IDNA". 173 The term "IDNA2008" is used to refer to a new version of IDNA as 174 described in this document and in the documents described in the 175 document listing of [IDNA2008-Defs]. IDNA2008 is not dependent on 176 any of the IDNA2003 specifications other than the one for Punycode 177 encoding. References to "these specifications" or "these documents" 178 are to the entire IDNA2008 set. 180 1.3.2. DNS "Name" Terminology 182 These documents depart from historical DNS terminology and usage in 183 one important respect. Over the years, the community has talked very 184 casually about "names" in the DNS, beginning with calling it "the 185 domain name system". That terminology is fine in the very precise 186 sense that the identifiers of the DNS do provide names for objects 187 and addresses. But, in the context of IDNs, the term has introduced 188 some confusion, confusion that has increased further as people have 189 begun to speak of DNS labels in terms of the words or phrases of 190 various natural languages. 192 Historically, many, perhaps most, of the "names" in the DNS have been 193 mnemonics to identify some particular concept, object, or 194 organization. They are typically derived from, or rooted in, some 195 language because most people think in language-based ways. But, 196 because they are mnemonics, they need not obey the orthographic 197 conventions of any language: it is not a requirement that it be 198 possible for them to be "words". 200 This distinction is important because the reasonable goal of an IDN 201 effort is not to be able to write the great Klingon (or language of 202 one's choice) novel in DNS labels but to be able to form a usefully 203 broad range of mnemonics in ways that are as natural as possible in a 204 very broad range of scripts. 206 1.3.3. New Terminology and Restrictions 208 These documents introduce new terminology, and precise definitions, 209 for the terms "U-labels", "A-labels", labels that are "IDNA-valid", 210 and an "LDH-label" (differing from an LDH-conformant label or fully- 211 qualified domain name). They also introduce a restriction, for IDNA- 212 conformant applications and DNS zones in which IDNA is used, on 213 strings used as labels that contain "--" in the third and fourth 214 positions, essentially requiring that such strings be IDNA-valid. 215 This restriction on strings containing "--" is required for three 216 reasons: 218 o to prevent confusion with pre-IDNA coding forms; 220 o to permit future extensions that would require changing the 221 prefix, no matter how unlikely those might be (see Section 7.4); 222 and 224 o to reduce the opportunities for attacks via the Punycode encoding 225 algorithm itself. 227 Figure 1 of the Definitions Document [IDNA2008-Defs] illustrates the 228 terminology used by IDNA for various types of labels and strings and 229 their relationship. 231 1.4. Objectives 233 The intent of the IDNA revision effort, and hence of this document 234 and the associated ones, is to increase the usability and 235 effectiveness of internationalized domain names (IDNs) while 236 preserving or strengthening the integrity of references that use 237 them. The original "hostname" character definitions (see, e.g., 238 [RFC0810]) struck a balance between the creation of useful mnemonics 239 and the introduction of parsing problems or general confusion in the 240 contexts in which domain names are used. The objective of IDNA2008 241 is to preserve that balance while expanding the character repertoire 242 to include extended versions of Roman-derived scripts and scripts 243 that are not Roman in origin. No work of this sort is able to 244 completely eliminate sources of visual or textual confusion: such 245 confusion is possible even under the original host naming rules where 246 only ASCII characters were permitted. However, through the 247 application of different techniques at different points (see 248 Section 3.3), it should be possible to keep problems to an acceptable 249 minimum. One consequence of this general objective is that the 250 desire of some user or marketing community to use a particular string 251 --whether the reason is to try to write sentences of particular 252 languages in the DNS, to express a facsimile of the symbol for a 253 brand, or for some other purpose-- is not a primary goal within the 254 context of applications in the domain name space. 256 1.5. Applicability and Function of IDNA 258 The IDNA specification solves the problem of extending the repertoire 259 of characters that can be used in domain names to include a large 260 subset of the Unicode repertoire. 262 IDNA does not extend the service offered by DNS to the applications. 263 Instead, the applications (and, by implication, the users) continue 264 to see an exact-match lookup service. Either there is a single 265 exactly-matching name or there is no match. This model has served 266 the existing applications well, but it requires, with or without 267 internationalized domain names, that users know the exact spelling of 268 the domain names that are to be typed into applications such as web 269 browsers and mail user agents. The introduction of the larger 270 repertoire of characters potentially makes the set of misspellings 271 larger, especially given that in some cases the same appearance, for 272 example on a business card, might visually match several Unicode code 273 points or several sequences of code points. 275 The IDNA standard does not require any applications to conform to it, 276 nor does it retroactively change those applications. An application 277 can elect to use IDNA in order to support IDN while maintaining 278 interoperability with existing infrastructure. If an application 279 wants to use non-ASCII characters in domain names, IDNA is the only 280 currently-defined option. Adding IDNA support to an existing 281 application entails changes to the application only, and leaves room 282 for flexibility in front-end processing and more specifically in the 283 user interface (see Section 6). 285 A great deal of the discussion of IDN solutions has focused on 286 transition issues and how IDNs will work in a world where not all of 287 the components have been updated. Proposals that were not chosen by 288 the original IDN Working Group would have depended on updating of 289 user applications, DNS resolvers, and DNS servers in order for a user 290 to apply an internationalized domain name in any form or coding 291 acceptable under that method. While processing must be performed 292 prior to or after access to the DNS, IDNA requires no changes to the 293 DNS protocol or any DNS servers or the resolvers on user's computers. 295 IDNA allows the graceful introduction of IDNs not only by avoiding 296 upgrades to existing infrastructure (such as DNS servers and mail 297 transport agents), but also by allowing some rudimentary use of IDNs 298 in applications by using the ASCII-encoded representation of the 299 labels containing non-ASCII characters. While such names are user- 300 unfriendly to read and type, and hence not optimal for user input, 301 they can be used as a last resort to allow rudimentary IDN usage. 302 For example, they might be the best choice for display if it were 303 known that relevant fonts were not available on the user's computer. 305 In order to allow user-friendly input and output of the IDNs and 306 acceptance of some characters as equivalent to those to be processed 307 according to the protocol, the applications need to be modified to 308 conform to this specification. 310 IDNA uses the Unicode character repertoire, for continuity with the 311 original version of IDNA. 313 1.6. Comprehensibility of IDNA Mechanisms and Processing 315 One of the major goals of this work is to improve the general 316 understanding of how IDNA works and what characters are permitted and 317 what happens to them. Comprehensibility and predictability to users 318 and registrants are themselves important motivations and design goals 319 for this effort. The effort includes some new terminology and a 320 revised and extended model, both covered in this section, and some 321 more specific protocol, processing, and table modifications. Details 322 of the latter appear in other documents (see [IDNA2008-Defs]). 324 Several issues are inherent in the application of IDNs and, indeed, 325 almost any other system that tries to handle international characters 326 and concepts. They range from the apparently trivial --e.g., one 327 cannot display a character for which one does not have a font 328 available locally-- to the more complex and subtle. Many people have 329 observed that internationalization is just a tool to enable effective 330 localization while permitting some global uniformity. Issues of 331 display, of exactly how various strings and characters are entered, 332 and so on are inherently issues about localization and user interface 333 design. 335 A protocol such as IDNA can only assume that such operations as data 336 entry and reconciliation of differences in character forms are 337 possible. It may make some recommendations about how display might 338 work when characters and fonts are not available, but they can only 339 be general recommendations and, because display functions are rarely 340 controlled by the types of applications that would call upon IDNA, 341 will rarely be very effective. 343 However, shifting responsibility for character mapping and other 344 adjustments from the protocol (where it was located in IDNA2003) to 345 the user interface or processing before invoking IDNA raises issues 346 about both what that processing should do and about compatibility for 347 references prepared in an IDNA2003 context. Those issues are 348 discussed in Section 6. 350 Operations for converting between local character sets and normalized 351 Unicode are part of this general set of user interface issues. The 352 conversion is obviously not required at all in a Unicode-native 353 system that maintains all strings in Normalization Form C (NFC). 354 (See [Unicode-UAX15] for precise definitions of NFC and NFKC if 355 needed.) It may, however, involve some complexity in a system that 356 is not Unicode-native, especially if the elements of the local 357 character set do not map exactly and unambiguously into Unicode 358 characters or do so in a way that is not completely stable over time. 359 Perhaps more important, if a label being converted to a local 360 character set contains Unicode characters that have no correspondence 361 in that character set, the application may have to apply special, 362 locally-appropriate, methods to avoid or reduce loss of information. 364 Depending on the system involved, the major difficulty may not lie in 365 the mapping but in accurately identifying the incoming character set 366 and then applying the correct conversion routine. If a local 367 operating system uses one of the ISO 8859 character sets or an 368 extensive national or industrial system such as GB18030 [GB18030] or 369 BIG5 [BIG5], one must correctly identify the character set in use 370 before converting to Unicode even though those character coding 371 systems are substantially or completely Unicode-compatible (i.e., all 372 of the code points in them have an exact and unique mapping to 373 Unicode code points). It may be even more difficult when the 374 character coding system in local use is based on conceptually 375 different assumptions than those used by Unicode about, e.g., about 376 font encodings used for publications in some Indic scripts. Those 377 differences may not easily yield unambiguous conversions or 378 interpretations even if each coding system is internally consistent 379 and adequate to represent the local language and script. 381 2. Processing in IDNA2008 383 These specifications separate Domain Name Registration and Lookup in 384 the protocol specification. Doing so reflects current practice in 385 which per-registry restrictions and special processing are applied at 386 registration time but not during lookup. Even more important in the 387 longer term, it facilitates incremental addition of permitted 388 character groups to avoid freezing on one particular version of 389 Unicode. 391 The actual registration and lookup protocols for IDNA2008 are 392 specified in [IDNA2008-Protocol]. 394 3. Permitted Characters: An Inclusion List 396 This section provides an overview of the model used to establish the 397 algorithm and character lists of [IDNA2008-Tables] and describes the 398 names and applicability of the categories used there. Note that the 399 inclusion of a character in the first category group (Section 3.1.1) 400 does not imply that it can be used indiscriminately; some characters 401 are associated with contextual rules that must be applied as well. 403 The information given in this section is provided to make the rules, 404 tables, and protocol easier to understand. The normative generating 405 rules that correspond to this informal discussion appear in 406 [IDNA2008-Tables] and the rules that actually determine what labels 407 can be registered or looked up are in [IDNA2008-Protocol]. 409 3.1. A Tiered Model of Permitted Characters and Labels 411 Moving to an inclusion model requires respecifying the list of 412 characters that are permitted in IDNs. In IDNA2003, the role and 413 utility of characters are independent of context and fixed forever 414 (or until the standard is replaced). Making completely context- 415 independent rules globally has proven impractical because some 416 characters, especially those that are called "Join_Controls" in 417 Unicode, are needed to make reasonable use of some scripts but have 418 no visible effect(s) in others. IDNA2003 prohibited those types of 419 characters entirely. But the restrictions were much too severe to 420 permit an adequate range of mnemonics for identifiers based on some 421 languages. The requirement to support those characters but limit 422 their use to very specific contexts was reinforced by the observation 423 that handling of particular characters across the languages that use 424 a script, or the use of similar or identical-looking characters in 425 different scripts, is less well understood than many people believed 426 it was several years ago. 428 Independently of the characters chosen (see next subsection), the 429 approach is to divide the characters that appear in Unicode into 430 three categories: 432 3.1.1. PROTOCOL-VALID 434 Characters identified as "PROTOCOL-VALID" (often abbreviated 435 "PVALID") are, in general, permitted by IDNA for all uses in IDNs. 436 Their use may be restricted by rules about the context in which they 437 appear or by other rules that apply to the entire label in which they 438 are to be embedded. For example, any label that contains a character 439 in this category that has a "right-to-left" property must be used in 440 context with the "Bidi" rules (see [IDNA2008-Bidi]). 442 The term "PROTOCOL-VALID" is used to stress the fact that the 443 presence of a character in this category does not imply that a given 444 registry need accept registrations containing any of the characters 445 in the category. Registries are still expected to apply judgment 446 about labels they will accept and to maintain rules consistent with 447 those judgments (see [IDNA2008-Protocol] and Section 3.3). 449 Characters that are placed in the "PROTOCOL-VALID" category are 450 expected to never be removed from it or reclassified. While 451 theoretically characters could be removed from Unicode, such removal 452 would be inconsistent with the Unicode stability principles (see 453 [Unicode51], Appendix F) and hence should never occur. 455 3.1.1.1. Contextual Rules 457 Some characters may be unsuitable for general use in IDNs but 458 necessary for the plausible support of some scripts. The two most 459 commonly-cited examples are the zero-width joiner and non-joiner 460 characters (ZWJ, U+200D and ZWNJ, U+200C), but provisions for 461 unambiguous labels may require that other characters be restricted to 462 particular contexts. For example, the ASCII hyphen is not permitted 463 to start or end a label, whether that label contains non-ASCII 464 characters or not. 466 These characters must not appear in IDNs without additional 467 restrictions, typically because they have no visible consequences in 468 most scripts but affect format or presentation in a few others or 469 because they are combining characters that are safe for use only in 470 conjunction with particular characters or scripts. In order to 471 permit them to be used at all, they are specially identified as 472 "CONTEXTUAL RULE REQUIRED" and, when adequately understood, 473 associated with a rule. In addition, the rule will define whether it 474 is to be applied on lookup as well as registration. A distinction is 475 made between characters that indicate or prohibit joining (known as 476 "CONTEXT-JOINER" or "CONTEXTJ") and other characters requiring 477 contextual treatment ("CONTEXT-OTHER" or "CONTEXTO"). Only the 478 former require full testing at lookup time. 480 3.1.1.2. Rules and Their Application 482 The actual rules may be present or absent. If present, they may have 483 values of "True" (character may be used in any position in any 484 label), "False" (character may not be used in any label), or may be a 485 set of procedural rules that specify the context in which the 486 character is permitted. 488 Examples of descriptions of typical rules, stated informally and in 489 English, include "Must follow a character from Script XYZ", "Must 490 occur only if the entire label is in Script ABC", "Must occur only if 491 the previous and subsequent characters have the DFG property". 493 Because it is easier to identify these characters than to know that 494 they are actually needed in IDNs or how to establish exactly the 495 right rules for each one, a rule may have a null value in a given 496 version of the tables. Characters associated with null rules are not 497 permitted to appear in putative labels for either registration or 498 lookup. Of course, a later version of the tables might contain a 499 non-null rule. 501 The description of the syntax of the rules, and the rules themselves, 502 appears in [IDNA2008-Tables]. 504 3.1.2. DISALLOWED 506 Some characters are inappropriate for use in IDNs and are thus 507 excluded for both registration and lookup (i.e., IDNA-conforming 508 applications performing name lookup should verify that these 509 characters are absent; if they are present, the label strings should 510 be rejected rather than converted to A-labels and looked up. Some of 511 these characters are problematic for use in IDNs (such as the 512 FRACTION SLASH character, U+2044), while some of them (such as the 513 various HEART symbols, e.g., U+2665, U+2661, and U+2765, see 514 Section 7.6) simply fall outside the conventions for typical 515 identifiers (basically letters and numbers). 517 Of course, this category would include code points that had been 518 removed entirely from Unicode should such removals ever occur. 520 Characters that are placed in the "DISALLOWED" category are expected 521 to never be removed from it or reclassified. If a character is 522 classified as "DISALLOWED" in error and the error is sufficiently 523 problematic, the only recourse would be either to introduce a new 524 code point into Unicode and classify it as "PROTOCOL-VALID" or for 525 the IETF to accept the considerable costs of an incompatible change 526 and replace the relevant RFC with one containing appropriate 527 exceptions. 529 There is provision for exception cases but, in general, characters 530 are placed into "DISALLOWED" if they fall into one or more of the 531 following groups: 533 o The character is a compatibility equivalent for another character. 534 In slightly more precise Unicode terms, application of 535 normalization method NFKC to the character yields some other 536 character. 538 o The character is an upper-case form or some other form that is 539 mapped to another character by Unicode casefolding. 541 o The character is a symbol or punctuation form or, more generally, 542 something that is not a letter, digit, or a mark that is used to 543 form a letter or digit. 545 3.1.3. UNASSIGNED 547 For convenience in processing and table-building, code points that do 548 not have assigned values in a given version of Unicode are treated as 549 belonging to a special UNASSIGNED category. Such code points are 550 prohibited in labels to be registered or looked up. The category 551 differs from DISALLOWED in that code points are moved out of it by 552 the simple expedient of being assigned in a later version of Unicode 553 (at which point, they are classified into one of the other categories 554 as appropriate). 556 3.2. Registration Policy 558 While these recommendations cannot and should not define registry 559 policies, registries should develop and apply additional restrictions 560 to reduce confusion and other problems. For example, it is generally 561 believed that labels containing characters from more than one script 562 are a bad practice although there may be some important exceptions to 563 that principle. Some registries may choose to restrict registrations 564 to characters drawn from a very small number of scripts. For many 565 scripts, the use of variant techniques such as those as described in 566 RFC 3843 [RFC3743] and RFC 4290 [RFC4290], and illustrated for 567 Chinese by the tables described in RFC 4713 [RFC4713] may be helpful 568 in reducing problems that might be perceived by users. 570 In general, users will benefit if registries only permit characters 571 from scripts that are well-understood by the registry or its 572 advisers. If a registry decides to reduce opportunities for 573 confusion by constructing policies that disallow characters used in 574 historic writing systems or characters whose use is restricted to 575 specialized, highly technical contexts, some relevant information may 576 be found in Section 2.4 "Specific Character Adjustments", Table 4 577 "Candidate Characters for Exclusion from Identifiers" of 578 [Unicode-UAX31] and Section 3.1. "General Security Profile for 579 Identifiers" in [Unicode-Security]. 581 It is worth stressing that these principles of policy development and 582 application apply at all levels of the DNS, not only, e.g., TLD 583 registrations and that even a trivial, "anything permitted that is 584 valid under the protocol" policy is helpful in that it helps users 585 and application developers know what to expect. 587 3.3. Layered Restrictions: Tables, Context, Registration, Applications 589 The essence of the character rules in IDNA2008 is based on the 590 realization that there is no single magic bullet for any of the 591 issues associated with a multiscript DNS. Instead, the 592 specifications define a variety of approaches that, together, 593 constitute multiple lines of defense against ambiguity in identifiers 594 and loss of referential integrity. The actual character tables are 595 the first mechanism, protocol rules about how those characters are 596 applied or restricted in context are the second, and those two in 597 combination constitute the limits of what can be done by a protocol 598 alone. As discussed in the previous section (Section 3.2), 599 registries are expected to restrict what they permit to be 600 registered, devising and using rules that are designed to optimize 601 the balance between confusion and risk on the one hand and maximum 602 expressiveness in mnemonics on the other. 604 In addition, there is an important role for user agents in warning 605 against label forms that appear unreasonable given their knowledge of 606 local contexts and conventions. Of course, no approach based on 607 naming or identifiers alone can protect against all threats. 609 4. Issues that Constrain Possible Solutions 611 4.1. Display and Network Order 613 The correct treatment of domain names requires a clear distinction 614 between Network Order (the order in which the code points are sent in 615 protocols) and Display Order (the order in which the code points are 616 displayed on a screen or paper). The order of labels in a domain 617 name that contains characters that are normally written right to left 618 is discussed in [IDNA2008-Bidi]. In particular, there are questions 619 about the order in which labels are displayed if left to right and 620 right to left labels are adjacent to each other, especially if there 621 are also multiple consecutive appearances of one of the types. The 622 decision about the display order is ultimately under the control of 623 user agents --including web browsers, mail clients, and the like-- 624 which may be highly localized. Even when formats are specified by 625 protocols, the full composition of an Internationalized Resource 626 Identifier (IRI) [RFC3987] or Internationalized Email address 627 contains elements other than the domain name. For example, IRIs 628 contain protocol identifiers and field delimiter syntax such as 629 "http://" or "mailto:" while email addresses contain the "@" to 630 separate local parts from domain names. User agents are not required 631 to use those protocol-based forms directly but often do so. While 632 display, parsing, and processing within a label is specified by the 633 normative documents in the IDNA2008 collection, the relationship 634 between fully-qualified domain names and internationalized labels is 635 unchanged from the base DNS specifications. Comments in this 636 document about such full domain names are explanatory or examples of 637 what might be done and must not be considered normative. 639 Questions remain about protocol constraints implying that the overall 640 direction of these strings will always be left to right (or right to 641 left) for an IRI or email address, or if they even should conform to 642 such rules. These questions also have several possible answers. 643 Should a domain name abc.def, in which both labels are represented in 644 scripts that are written right to left, be displayed as fed.cba or 645 cba.fed? An IRI for clear text web access would, in network order, 646 begin with "http://" and the characters will appear as 647 "http://abc.def" -- but what does this suggest about the display 648 order? When entering a URI to many browsers, it may be possible to 649 provide only the domain name and leave the "http://" to be filled in 650 by default, assuming no tail (an approach that does not work for 651 protocols other than HTTP or whatever is chosen as the default). The 652 natural display order for the typed domain name on a right to left 653 system is fed.cba. Does this change if a protocol identifier, tail, 654 and the corresponding delimiters are specified? 656 While logic, precedent, and reality suggest that these are questions 657 for user interface design, not IETF protocol specifications, 658 experience in the 1980s and 1990s with mixing systems in which domain 659 name labels were read in network order (left to right) and those in 660 which those labels were read right to left would predict a great deal 661 of confusion, and heuristics that sometimes fail, if each 662 implementation of each application makes its own decisions on these 663 issues. 665 Any version of IDNA, including the current one, must be written in 666 terms of the network (transmission on the wire) order of characters 667 in labels and for the labels in complete (fully-qualified) domain 668 names and must be quite precise about those relationships. While 669 some strong suggestions about display order would be desirable to 670 reduce the chances for inconsistent transcription of domain names 671 from printed form, such suggestions are beyond the scope of these 672 specifications. 674 4.2. Entry and Display in Applications 676 Applications can accept domain names using any character set or sets 677 desired by the application developer, specified by the operating 678 system, or dictated by other constraints, and can display domain 679 names in any character set or character coding system. That is, the 680 IDNA protocol does not affect the interface between users and 681 applications. 683 An IDNA-aware application can accept and display internationalized 684 domain names in two formats: the internationalized character set(s) 685 supported by the application (i.e., an appropriate local 686 representation of a U-label), and as an A-label. Applications may 687 allow the display of A-labels, but are encouraged to not do so except 688 as an interface for special purposes, possibly for debugging, or to 689 cope with display limitations. In general, they should allow, but 690 not encourage, user input of that label form. A-labels are opaque 691 and ugly and malicious variations on them are not easily detected by 692 users. Where possible, they should thus only be exposed to users and 693 in contexts in which they are absolutely needed. Because IDN labels 694 can be rendered either as A-labels or U-labels, the application may 695 reasonably have an option for the user to select the preferred method 696 of display; if it does, rendering the U-label should normally be the 697 default. 699 Domain names are often stored and transported in many places. For 700 example, they are part of documents such as mail messages and web 701 pages. They are transported in many parts of many protocols, such as 702 both the control commands of SMTP and associated the message body 703 parts, and in the headers and the body content in HTTP. It is 704 important to remember that domain names appear both in domain name 705 slots and in the content that is passed over protocols. 707 In protocols and document formats that define how to handle 708 specification or negotiation of charsets, labels can be encoded in 709 any charset allowed by the protocol or document format. If a 710 protocol or document format only allows one charset, the labels must 711 be given in that charset. Of course, not all charsets can properly 712 represent all labels. If a U-label cannot be displayed in its 713 entirety, the only choice (without loss of information) may be to 714 display the A-label. 716 In any place where a protocol or document format allows transmission 717 of the characters in internationalized labels, labels should be 718 transmitted using whatever character encoding and escape mechanism 719 the protocol or document format uses at that place. This provision 720 is intended to prevent situations in which, e.g., UTF-8 domain names 721 appear embedded in text that is otherwise in some other character 722 coding. 724 All protocols that use domain name slots already have the capacity 725 for handling domain names in the ASCII charset. Thus, A-labels can 726 inherently be handled by those protocols. 728 4.3. Linguistic Expectations: Ligatures, Digraphs, and Alternate 729 Character Forms 731 [[anchor13: There is some internal redundancy and repetition in the 732 material in this section. Specific suggestions about to reduce or 733 eliminate redundant text would be appreciated. If no such 734 suggestions are received before -07 is posted, this not will be 735 removed.]] 737 Users often have expectations about character matching or equivalence 738 that are based on their own languages and the orthography of those 739 languages. These expectations may not be consistent with forms or 740 actions that can be naturally accommodated in a character coding 741 system, especially if multiple languages are written using the same 742 script but using different conventions. A Norwegian user might 743 expect a label with the ae-ligature to be treated as the same label 744 as one using the Swedish spelling with a-diaeresis even though 745 applying that mapping to English would be astonishing to users. A 746 user in German might expect a label with an o-umlaut and a label that 747 had "oe" substituted, but was otherwise the same, treated as 748 equivalent even though that substitution would be a clear error in 749 Swedish. A Chinese user might expect automatic matching of 750 Simplified and Traditional Chinese characters, but applying that 751 matching for Korean or Japanese text would create considerable 752 confusion. For that matter, an English user might expect "theater" 753 and "theatre" to match. 755 Related issues arise because there are a number of languages written 756 with alphabetic scripts in which single phonemes are written using 757 two characters, termed a "digraph", for example, the "ph" in 758 "pharmacy" and "telephone". (Note that characters paired in this 759 manner can also appear consecutively without forming a digraph, as in 760 "tophat".) Certain digraphs are normally indicated typographically 761 by setting the two characters closer together than they would be if 762 used consecutively to represent different phonemes. Some digraphs 763 are fully joined as ligatures (strictly designating setting totally 764 without intervening white space, although the term is sometimes 765 applied to close set pairs). An example of this may be seen when the 766 word "encyclopaedia" is set with a U+00E6 LATIN SMALL LIGATURE AE 767 (and some would not consider that word correctly spelled unless the 768 ligature form was used or the "a" was dropped entirely). When these 769 ligature and digraph forms have the same interpretation across all 770 languages that use a given script, application of Unicode 771 normalization generally resolves the differences and causes them to 772 match. When they have different interpretations, any requirements 773 for matching must utilize other methods, presumably at the registry 774 level, or users must be educated to understand that matching will not 775 occur. 777 Difficulties arise from the fact that a given ligature may be a 778 completely optional typographic convenience for representing a 779 digraph in one language (as in the above example with some spelling 780 conventions), while in another language it is a single character that 781 may not always be correctly representable by a two-letter sequence 782 (as in the above example with different spelling conventions). This 783 can be illustrated by many words in the Norwegian language, where the 784 "ae" ligature is the 27th letter of a 29-letter extended Latin 785 alphabet. It is equivalent to the 28th letter of the Swedish 786 alphabet (also containing 29 letters), U+00E4 LATIN SMALL LETTER A 787 WITH DIAERESIS, for which an "ae" cannot be substituted according to 788 current orthographic standards. 790 That character (U+00E4) is also part of the German alphabet where, 791 unlike in the Nordic languages, the two-character sequence "ae" is 792 usually treated as a fully acceptable alternate orthography for the 793 "umlauted a" character. The inverse is however not true, and those 794 two characters cannot necessarily be combined into an "umlauted a". 795 This also applies to another German character, the "umlauted o" 796 (U+00F6 LATIN SMALL LETTER O WITH DIAERESIS) which, for example, 797 cannot be used for writing the name of the author "Goethe". It is 798 also a letter in the Swedish alphabet where, like the "a with 799 diaeresis", it cannot be correctly represented as "oe" and in the 800 Norwegian alphabet, where it is represented, not as "o with 801 diaeresis", but as "slashed o", U+00F8. 803 Some of the ligatures that have explicit code points in Unicode were 804 given special handling in IDNA2003 and now pose additional problems 805 in transition. See Section 7.2. 807 Additional cases with alphabets written right to left are described 808 in Section 4.5. 810 Whether ligatures and digraphs are to be treated as a sequence of 811 characters or as a single standalone one constitute a problem that 812 cannot be resolved solely by operating on scripts. They are, 813 however, a key concern in the IDN context. Their satisfactory 814 resolution will require support in policies set by registries, which 815 therefore need to be particularly mindful not just of this specific 816 issue, but of all other related matters that cannot be dealt with on 817 an exclusively algorithmic and global basis. 819 Just as with the examples of different-looking characters that may be 820 assumed to be the same, it is in general impossible to deal with 821 these situations in a system such as IDNA -- or with Unicode 822 normalization generally -- since determining what to do requires 823 information about the language being used, context, or both. 824 Consequently, these specifications make no attempt to treat these 825 combined characters in any special way. However, their existence 826 provides a prime example of a situation in which a registry that is 827 aware of the language context in which labels are to be registered, 828 and where that language sometimes (or always) treats the two- 829 character sequences as equivalent to the combined form, should give 830 serious consideration to applying a "variant" model [RFC3743] 832 [RFC4290], or to prohibiting registration of one the forms entirely, 833 to reduce the opportunities for user confusion and fraud that would 834 result from the related strings being registered to different 835 parties. 837 [[anchor14: Placeholder: A discussion of the Arabic digit issue 838 shoudl go here once it is resolved in some appropriate way.]] 840 4.4. Case Mapping and Related Issues 842 In the DNS, ASCII letters are stored with their case preserved. 843 Matching during the query process is case-independent, but none of 844 the information that might be represented by choices of case has been 845 lost. That model has been accidentally helpful because, as people 846 have created DNS labels by catenating words (or parts of words) to 847 form labels, case has often been used to distinguish among components 848 and make the labels more memorable. 850 The solution of keeping the characters separate but doing matching 851 independent of case is not feasible with IDNA or any IDNA-like model 852 because the matching would then have to be done on the server rather 853 than have characters mapped on the client. That situation was 854 recognized in IDNA2003 and nothing in these specifications 855 fundamentally changes it or could do so. In IDNA2003, all characters 856 are case-folded and mapped. That results in upper-case characters 857 being mapped to lower-case ones and in some other transformations of 858 alternate forms of characters, especially those that do not have (or 859 did not have) upper-case forms. For example, Greek Final Form Sigma 860 (U+03C2) is mapped to the medial form (U+03C3) and Eszett (German 861 Sharp S, U+00DF) is mapped to "ss". Neither of these mappings is 862 reversible because the upper case of U+03C3 is the Upper Case Sigma 863 (U+03A3) and "ss" is an ASCII string. IDNA2008 permits, at the risk 864 of some incompatibility, slightly more flexibility in this area by 865 avoid case folding and treating these characters as themselves. 866 Approaches to handling that incompatibility are discussed in 867 Section 7.2. Although information is lost in IDNA2003's ToASCII 868 operation so that, in some sense, neither Final Sigma nor Eszett can 869 be represented in an IDN at all, its guarantee of mapping when those 870 characters are used as input can be interpreted as violating one of 871 the conditions discussed in Section 7.4.1 and hence requiring a 872 prefix change. The consensus was to not make a prefix change in 873 spite of this issue. Of course, had a prefix change been made (at 874 the costs discussed in Section 7.4.3) there would have been several 875 options, including, if desired, assignment of the character to the 876 CONTEXTUAL RULE REQUIRED category and requiring that it only be used 877 in carefully-selected contexts. 879 4.5. Right to Left Text 881 In order to be sure that the directionality of right to left text is 882 unambiguous, IDNA2003 required that any label in which right to left 883 characters appear both starts and ends with them, not include any 884 characters with strong left to right properties (which excludes other 885 alphabetic characters but permits European digits), and rejects any 886 other string that contains a right to left character. This is one of 887 the few places where the IDNA algorithms (both in IDNA2003 and in 888 IDAN2008) are required to examine an entire label, not just 889 individual characters. The algorithmic model used in IDNA2003 890 rejects the label when the final character in a right to left string 891 requires a combining mark in order to be correctly represented. 893 That prohibition is not acceptable for writing systems for languages 894 written with consonantal alphabets to which diacritical vocalic 895 systems are applied, and for languages with orthographies derived 896 from them where the combining marks may have different functionality. 897 In both cases the combining marks can be essential components of the 898 orthography. Examples of this are Yiddish, written with an extended 899 Hebrew script, and Dhivehi (the official language of Maldives) which 900 is written in the Thaana script (which is, in turn, derived from the 901 Arabic script). IDNA2008 removes the restriction on final combining 902 characters with a new set of rules for right to left scripts and 903 their characters. Those new rules are specified in [IDNA2008-Bidi]. 905 5. IDNs and the Robustness Principle 907 The model of IDNs described in this document can be seen as a 908 particular instance of the "Robustness Principle" that has been so 909 important to other aspects of Internet protocol design. This 910 principle is often stated as "Be conservative about what you send and 911 liberal in what you accept" (See, e.g., Section 1.2.2 of the 912 applications-layer Host Requirements specification [RFC1123]). For 913 IDNs to work well, not only must the protocol be carefully designed 914 and implemented, but zone administrators (registries) must have and 915 require sensible policies about what is registered -- conservative 916 policies -- and implement and enforce them. 918 Conversely, lookup applications are expected to reject labels that 919 clearly violate global (protocol) rules (no one has ever seriously 920 claimed that being liberal in what is accepted requires being 921 stupid). However, once one gets past such global rules and deals 922 with anything sensitive to script or locale, it is necessary to 923 assume that garbage has not been placed into the DNS, i.e., one must 924 be liberal about what one is willing to look up in the DNS rather 925 than guessing about whether it should have been permitted to be 926 registered. 928 As mentioned elsewhere, if a string cannot be successfully found in 929 the DNS after the lookup processing described here, it makes no 930 difference whether it simply wasn't registered or was prohibited by 931 some rule at the registry. Applications should, however, be 932 sensitive to the fact that, because of the possibility of DNS 933 wildcards, the ability to successfully resolve a name does not 934 guarantee that it was actually registered. 936 If lookup applications, as a user interface (UI) or other local 937 matter, decide to warn about some strings that are valid under the 938 global rules but that they perceive as dangerous, that is their 939 prerogative and we can only hope that the market (and maybe 940 regulators) will reinforce the good choices and discourage the poor 941 ones. In this context, a lookup application that decides a string 942 that is valid under the protocol is dangerous and refuses to look it 943 up is in violation of the protocols; one that is willing to look 944 something up, but warns against it, is exercising a local choice. 946 6. Front-end and User Interface Processing 948 Domain names may be identified and processed in many contexts. They 949 may be typed in by users either by themselves or embedded in an 950 identifier structured for a particular protocol or class of protocols 951 such a email addresses, URIs, or IRIs. They may occur in running 952 text or be processed by one system after being provided in another. 953 Systems may wish to try to normalize URLs so as to determine (or 954 guess) whether a reference is valid or two references point to the 955 same object without actually looking the objects up and comparing 956 them (that is necessary, not just a choice, for URI types that are 957 not intended to be resolved). Some of these goals may be more easily 958 and reliably satisfied than others. While there are strong arguments 959 for any domain name that is placed "on the wire" -- transmitted 960 between systems -- to be in the minimum-ambiguity forms of A-labels, 961 U-labels, or LDH-labels, it is inevitable that programs that process 962 domain names will encounter variant forms. 964 One source of such forms will be labels created under IDNA2003 965 because that protocol allowed labels that were transformed before 966 they were turned from native-character into ACE ("xn--...") format by 967 mapping some characters into other. One consequence of the 968 transformations was that, when the ToUnicode and ToASCII operations 969 of IDNA2003 were applied, ToUnicode(ToASCII(original-label)) often 970 did not produce the original label. IDNA2008 explicitly defines 971 A-labels and U-labels as different forms of the same abstract label, 972 forms that are stable when conversions are performed between them, 973 without mappings. A different way of explaining this is that there 974 are, today, domain names in files on the Internet that use characters 975 that cannot be represented directly in, or recovered from, (A-label) 976 domain names but for which interpretations are provided by IDNA2003. 977 There are two major categories of such characters, those that are 978 removed by NFKC normalization and those upper-case characters that 979 are mapped to lower-case (there are also a few characters that are 980 given special-case mapping treatment in Stringprep including lower- 981 case characters that are case-folded into other lower-case characters 982 or strings). 984 Other issues in domain name identification and processing arise 985 because IDNA2003 specified that several other characters be treated 986 as equivalent to the ASCII period (dot, full stop) character used as 987 a label separator. If a string that might be a domain name appears 988 in an arbitrary context (such as running text), it is difficult, even 989 with only ASCII characters, to know whether an actual domain name (or 990 a protocol parameter like a URI) is present and where it starts and 991 ends. When using Unicode, this gets even more difficult if treatment 992 of certain special characters (like the dot that separates labels in 993 a domain name) depends on context (e.g., prior knowledge of whether 994 the string represents a domain name or not). That knowledge is not 995 available if the primary heuristic for identifying the presence of 996 domain names in strings depends on the presence of dots separating 997 groups of characters with no intervening spaces. 999 As discussed elsewhere in this document, the IDNA2008 model removes 1000 all of these mappings and interpretations, including the equivalence 1001 of different forms of dots, from the protocol, discouraging such 1002 mappings and leaving them, when necessary, to local processing. This 1003 should not be taken to imply that local processing is optional or can 1004 be avoided entirely, even if doing so might have been desirable in a 1005 world without IDNA2003 IDNs in files and archives. Instead, unless 1006 the program context is such that it is known that any IDNs that 1007 appear will contain either U-label or A-label forms, or that other 1008 forms can safely be rejected, some local processing of apparent 1009 domain name strings will be required, both to maintain compatibility 1010 with IDNA2003 and to prevent user astonishment. Such local 1011 processing, while not specified in this document or the associated 1012 ones, will generally take one of two forms: 1014 o Generic Preprocessing. 1015 When the context in which the program or system that processes 1016 domain names operates is global, a reasonable balance must be 1017 found that is sensitive to the broad range of local needs and 1018 assumptions while, at the same time, not sacrificing the needs of 1019 one language, script, or user population to those of another. 1021 For this case, the best practice will usually be to apply NFKC and 1022 case-mapping (or, perhaps better yet, Stringprep itself), plus 1023 dot-mapping where appropriate, to the domain name string prior to 1024 applying IDNA. That practice will not only yield a reasonable 1025 compromise of user experience with protocol requirements but will 1026 be almost completely compatible with the various forms permitted 1027 by IDNA2003. 1029 o Highly Localized Preprocessing. 1030 Unlike the case above, there will be some situations in which 1031 software will be highly localized for a particular environment and 1032 carefully adapted to the expectations of users in that 1033 environment. The many discussions about using the Internet to 1034 preserve and support local cultures suggest that these cases may 1035 be more common in the future than they have been so far. 1037 In these cases, we should avoid trying to tell implementers what 1038 they should do, if only because they are quite likely (and for 1039 good reason) to ignore us. We would assume that they would map 1040 characters that the intuitions of their users would suggest be 1041 mapped and would hope that they would do that mapping as early as 1042 possible, storing A-label or U-label forms in files and 1043 transporting only those forms between systems. One can imagine 1044 switches about whether some sorts of mappings occur, warnings 1045 before applying them or, in a slightly more extreme version of the 1046 approach taken in Internet Explorer version 7 (IE7), systems that 1047 utterly refuse to handle "strange" characters at all if they 1048 appear in U-label form. None of those local decisions are a 1049 threat to interoperability as long as (i) only U-labels and 1050 A-labels are used in interchange with systems outside the local 1051 environment, (ii) no character that would be valid in a U-label as 1052 itself is mapped to something else, (iii) any local mappings are 1053 applied as a preprocessing step (or, for conversions from U-labels 1054 or A-labels to presentation forms, postprocessing), not as part of 1055 IDNA processing proper, and (iv) appropriate consideration is 1056 given to labels that might have entered the environment in 1057 conformance to IDNA2003. 1059 In either case, it is vital that user interface designs and, where 1060 the interfaces are not sufficient, users, be aware that the only 1061 forms of domain names that this protocol anticipates will resolve 1062 globally or compare equal when crude methods (i.e., those not 1063 conforming to the strict definition of label equivalence given in 1064 [IDNA2008-Defs]) are used are those in which all native-script labels 1065 are in U-label form. Forms that assume mapping will occur, 1066 especially forms that were not valid under IDNA2003, may or may not 1067 function in predictable ways across all implementations. 1069 User interfaces involving Latin-based scripts should take special 1070 care when considering how to handle case mapping because small 1071 differences in label strings may cause behavior that is astonishing 1072 to users. Because case-insensitive mapping is done for ASCII strings 1073 by DNS-servers, an all-ASCII label is treated as case-insensitive. 1074 However, if even one of the characters of that string is replaced by 1075 one that requires the label to be given IDN treatment (e.g., by 1076 adding a diacritical mark), then the label immediately becomes case- 1077 sensitive. This suggests that case mapping for Latin-based scripts 1078 (and possibly other scripts with case distinctions) as a 1079 preprocessing matter in applications may be wise to prevent user 1080 astonishment, but, since all applications may not do this and 1081 ambiguity in transport is not desirable, the that case-dependent 1082 forms should not be stored in files. 1084 7. Migration from IDNA2003 and Unicode Version Synchronization 1086 7.1. Design Criteria 1088 As mentioned above and in RFC 4690, two key goals of the IDNA2008 1089 design are to enable applications to be agnostic about whether they 1090 are being run in environments supporting any Unicode version from 3.2 1091 onward and to permit incrementally adding new characters, character 1092 groups, scripts, and other character collections as they are 1093 incorporated into Unicode, without disruption and, in the long term, 1094 without "heavy" processes such as those involving IETF consensus. 1095 (An IETF consensus process is required by the IDNA2008 specifications 1096 and is expected to be required and used until significant experience 1097 accumulates with IDNA operations and new versions of Unicode.) The 1098 mechanisms that support this are outlined above and elsewhere in the 1099 IDNA2008 document set, but this section reviews them in a context 1100 that may be more helpful to those who need to understand the approach 1101 and make plans for it. 1103 7.1.1. General IDNA Validity Criteria 1105 The general criteria for a putative label, and the collection of 1106 characters that make it up, to be considered IDNA-valid are (the 1107 actual rules are rigorously defined in the "Protocol" and "Tables" 1108 documents): 1110 o The characters are "letters", marks needed to form letters, 1111 numerals, or other code points used to write words in some 1112 language. Symbols, drawing characters, and various notational 1113 characters are permanently excluded -- some because they are 1114 actively dangerous in URI, IRI, or similar contexts and others 1115 because there is no evidence that they are important enough to 1116 Internet operations or internationalization to justify expansion 1117 of domain names beyond the general principle of "letters, digits, 1118 and hyphen" and the complexities that would come with it 1119 (additional discussion and rationale for the symbol decision 1120 appears in Section 7.6). 1122 o Other than in very exceptional cases, e.g., where they are needed 1123 to write substantially any word of a given language, punctuation 1124 characters are excluded as well. The fact that a word exists is 1125 not proof that it should be usable in a DNS label and DNS labels 1126 are not expected to be usable for multiple-word phrases (although 1127 they are certainly not prohibited if the conventions and 1128 orthography of a particular language cause that to be possible). 1129 Even for English, very common constructions -- contractions like 1130 "don't" or "it's", names that are written with apostrophes such as 1131 "O'Reilly", or characters for which apostrophes are common 1132 substitutes cannot be represented in DNS labels. Words in English 1133 whose usually-preferred spellings include diacritical marks cannot 1134 be represented under the original hostname rules, but most can be 1135 represented if treated as IDNs. 1137 o Characters that are unassigned (have no character assignment at 1138 all) in the version of Unicode being used by the registry or 1139 application are not permitted, even on lookup. There are at least 1140 two reasons for this. 1142 * Tests involving the context of characters (e.g., some 1143 characters being permitted only adjacent to ones of specific 1144 types but otherwise invisible or very problematic for other 1145 reasons) and integrity tests on complete labels are needed. 1146 Unassigned code points cannot be permitted because one cannot 1147 determine whether particular code points will require 1148 contextual rules (and what those rules should be) before 1149 characters are assigned to them and the properties of those 1150 characters fully understood. 1152 * Unicode specifies that an unassigned code point normalizes (and 1153 case folds) to itself. If the code point is later assigned to 1154 a character, and particularly if the newly-assigned code point 1155 has a combining class that determines its placement relative to 1156 other combining characters, it could normalize to some other 1157 code point or sequence, creating confusion and/or violating 1158 other rules listed here. 1160 o Any character that is mapped to another character by a current 1161 version of NFKC is prohibited as input to IDNA (for either 1162 registration or lookup). With a few exceptions, this principle 1163 excludes any character mapped to another by Nameprep [RFC3491]. 1165 Tables used to identify the characters that are IDNA-valid are 1166 expected to be driven by the principles above, principles that are 1167 specified exactly in [IDNA2008-Tables]). The rules given there are 1168 normative, rather than being just an interpretation of the tables. 1170 7.1.2. Labels in Registration 1172 Anyone entering a label into a DNS zone must properly validate that 1173 label -- i.e., be sure that the criteria for that label are met -- in 1174 order for applications to work as intended. This principle is not 1175 new. For example, since the DNS was first deployed, zone 1176 administrators have been expected to verify that names meet 1177 "hostname" [RFC0952] where necessary for the expected applications. 1178 Later addition of special service location formats [RFC2782] imposed 1179 new requirements on zone administrators for the use of labels that 1180 conform to the requirements of those formats. For zones that will 1181 contain IDNs, support for Unicode version-independence requires 1182 restrictions on all strings placed in the zone. In particular, for 1183 such zones: 1185 o Any label that appears to be an A-label, i.e., any label that 1186 starts in "xn--", must be IDNA-valid, i.e., they must be valid 1187 A-labels, as discussed in Section 2 above. 1189 o The Unicode tables (i.e., tables of code points, character 1190 classes, and properties) and IDNA tables (i.e., tables of 1191 contextual rules such as those that appear in the Tables 1192 document), must be consistent on the systems performing or 1193 validating labels to be registered. Note that this does not 1194 require that tables reflect the latest version of Unicode, only 1195 that all tables used on a given system are consistent with each 1196 other. 1198 Under this model, a registry (or entity communicating with a registry 1199 to accomplish name registrations) will need to update its tables -- 1200 both the Unicode-associated tables and the tables of permitted IDN 1201 characters -- to enable a new script or other set of new characters. 1202 It will not be affected by newer versions of Unicode, or newly- 1203 authorized characters, until and unless it wishes to make those 1204 registrations. The zone administrator is also responsible -- under 1205 the protocol and to registrants and users -- for both checking as 1206 required by the protocol and verification that whatever policies it 1207 develops are complied with, whether those policies are for minimizing 1208 risks due to confusable characters and sequences, for preserving 1209 language or script integrity, or for other purposes. Those checking 1210 and verification procedures are more extensive than those that are is 1211 expected of applications systems that look names up. 1213 Systems looking up or resolving DNS labels, especially IDN DNS 1214 labels, must be able to assume that applicable registration rules 1215 were followed for names entered into the DNS. 1217 7.1.3. Labels in Lookup 1219 Anyone looking up a label in a DNS zone is required to 1221 o Maintain a consistent set of tables, as discussed above. As with 1222 registration, the tables need not reflect the latest version of 1223 Unicode but they must be consistent. 1225 o Validate the characters in labels to be looked up only to the 1226 extent of determining that the U-label does not contain either 1227 code points prohibited by IDNA (categorized as "DISALLOWED") or 1228 code points that are unassigned in its version of Unicode. 1230 o Validate the label itself for conformance with a small number of 1231 whole-label rules, notably verifying that there are no leading 1232 combining marks, that the "bidi" conditions are met if right to 1233 left characters appear, that any required contextual rules are 1234 available and that, if such rules are associated with Joiner 1235 Controls, they are tested. 1237 o Avoid validating other contextual rules about characters, 1238 including mixed-script label prohibitions, although such rules may 1239 be used to influence presentation decisions in the user interface. 1240 [[anchor18: Check this, and all similar statements, against 1241 Protocol when that is finished.]] 1243 By avoiding applying its own interpretation of which labels are valid 1244 as a means of rejecting lookup attempts, the lookup application 1245 becomes less sensitive to version incompatibilities with the 1246 particular zone registry associated with the domain name. 1248 An application or client that processes names according to this 1249 protocol and then resolves them in the DNS will be able to locate any 1250 name that is validly registered, as long as its version of the 1251 Unicode-associated tables is sufficiently up-to-date to interpret all 1252 of the characters in the label. Messages to users should distinguish 1253 between "label contains an unallocated code point" and other types of 1254 lookup failures. A failure on the basis of an old version of Unicode 1255 may lead the user to a desire to upgrade to a newer version, but will 1256 have no other ill effects (this is consistent with behavior in the 1257 transition to the DNS when some hosts could not yet handle some forms 1258 of names or record types). 1260 7.2. Changes in Character Interpretations 1262 [[anchor19: Note in Draft: This subsection is completely new in 1263 version -04 and has been further tuned in -05 and -06 of this 1264 document. It could almost certainly use improvement, although this 1265 note will be removed if there are not significant suggestions about 1266 the -06 version. It also contains some material that is redundant 1267 with material in other sections. I have not tried to remove that 1268 material and will not do so until the WG concludes that this section 1269 is relatively stable, but would appreciate help in identifying what 1270 should be removed or how this might be enhanced to contain more of 1271 that other material. --JcK]] 1273 In those scripts that make case distinctions, there are a few 1274 characters for which an obvious and unique upper case character has 1275 not historically been available to match a lower case one or vice 1276 versa. For those characters, the mappings used in constructing the 1277 Stringprep tables for IDNA2003, performed using the Unicode CaseFold 1278 operation (See Section 5.8 of the Unicode Standard [Unicode51]), 1279 generate different characters or sets of characters. Those 1280 operations are not reversible and lose even more information than 1281 traditional upper case or lower case transformations, but are more 1282 useful than those transformations for comparison purposes. Two 1283 notable characters of this type are the German character Eszett 1284 (Sharp S, U+00DF) and the Greek Final Form Sigma (U+03C2). The 1285 former is case-folded to the ASCII string "ss", the latter to a 1286 medial (Lower Case) Sigma (U+03C3). 1288 The decision to eliminate mappings, including case folding, from the 1289 IDNA2008 protocol in order to make A-labels and U-labels idempotent 1290 made these characters problematic. If they were to be disallowed, 1291 important words and mnemonics could not be written in 1292 orthographically reasonable ways. If they were to be permitted as 1293 characters distinct from the forms produced by case folding, there 1294 would be no information loss and registries would have maximum 1295 flexibility, but labels using those characters that were looked up 1296 according to IDNA2003 rules would be transformed into A-labels using 1297 their case-mapped variations while lookup according to IDNA2008 rules 1298 would be based on different A-labels that represented the actual 1299 characters. 1301 With the understanding that there would be incompatibility either way 1302 but a judgment that the incompatibility was not significant enough to 1303 just a prefix change, the WG concluded that Eszett and Final Form 1304 Sigma should be treated as distinct and Protocol-Valid characters. 1306 The decision faces registries, especially registries maintaining 1307 zones for third parties, with a variation on what has become a 1308 familiar problem: how to introduce a new service in a way that does 1309 not create confusion or significantly weaken or invalidate existing 1310 identifiers. 1312 There have traditionally been several approaches to problems of this 1313 type. Without any preference or claim to completeness, these are: 1315 o Do not permit use of the newly-available character at the registry 1316 level. This might cause lookup failures if a domain name were to 1317 be written with the expectation of the IDNA2003 mapping behavior, 1318 but would eliminate any possibility of false matches. 1320 o Hold a "sunrise"-like arrangement in which holders of the 1321 previously-mapped labels (labels containing "ss" in the Eszett 1322 case or ones containing Lower Case Sigma in the Final Sigma case) 1323 are given priority (and perhaps other benefits) for registering 1324 the corresponding string containing the newly-available 1325 characters. 1327 o Adopt some sort of "variant" approach in which registrants either 1328 obtained labels with both character forms or one of them was 1329 blocked from registration by anyone but the registrant of the 1330 other form. 1332 In principle, lookup applications could also compensate for the 1333 difference in interpretation by looking up the string according to 1334 the interpretation specified in these documents and then, if that 1335 failed, doing the lookup with the mapping, simulating the IDNA2003 1336 interpretation. The risk of false positives is such that this is 1337 generally to be discouraged unless the application is able to engage 1338 in a "is this what you meant" dialogue with the end user. 1340 7.3. More Flexibility in User Agents 1342 These specifications do not include mappings between one character or 1343 code point and others for any reason. Instead, they prohibit the 1344 characters that would be mapped to others by normalization, upper 1345 case to lower case changes, or other rules. As examples, while 1346 mathematical characters based on Latin ones are accepted as input to 1347 IDNA2003, they are prohibited in IDNA2008. Similarly, double-width 1348 characters and other variations are prohibited as IDNA input. 1350 Since the rules in [IDNA2008-Tables] have the effect that only 1351 strings that are not transformed by NFKC are valid, if an application 1352 chooses to perform NFKC normalization before lookup, that operation 1353 is safe since this will never make the application unable to look up 1354 any valid string. However, as discussed above, the application 1355 cannot guarantee that any other application will perform that 1356 mapping, so it should be used only with caution and for informed 1357 users. 1359 In many cases these prohibitions should have no effect on what the 1360 user can type as input to the lookup process. It is perfectly 1361 reasonable for systems that support user interfaces to perform some 1362 character mapping that is appropriate to the local environment. This 1363 would normally be done prior to actual invocation of IDNA. At least 1364 conceptually, the mapping would be part of the Unicode conversions 1365 discussed above and in [IDNA2008-Protocol]. However, those changes 1366 will be local ones only -- local to environments in which users will 1367 clearly understand that the character forms are equivalent. For use 1368 in interchange among systems, it appears to be much more important 1369 that U-labels and A-labels can be mapped back and forth without loss 1370 of information. 1372 One specific, and very important, instance of this strategy arises 1373 with case-folding. In the ASCII-only DNS, names are looked up and 1374 matched in a case-independent way, but no actual case-folding occurs. 1375 Names can be placed in the DNS in either upper or lower case form (or 1376 any mixture of them) and that form is preserved, returned in queries, 1377 and so on. IDNA2003 simulated that behavior for non-ASCII strings by 1378 performing case-folding at registration time (resulting in only 1379 lower-case IDNs in the DNS) and when names were looked up. 1381 As suggested earlier in this section, it appears to be desirable to 1382 do as little character mapping as possible consistent with having 1383 Unicode work correctly (e.g., NFC mapping to resolve different 1384 codings for the same character is still necessary although the 1385 specifications require that it be performed prior to invoking the 1386 protocol) and to make the mapping between A-labels and U-labels 1387 idempotent. Case-mapping is not an exception to this principle. If 1388 only lower case characters can be registered in the DNS (i.e., be 1389 present in a U-label), then IDNA2008 should prohibit upper-case 1390 characters as input (and therefore does so). Some other 1391 considerations reinforce this conclusion. For example, an essential 1392 element of the ASCII case-mapping functions is that, for individual 1393 characters, uppercase(character) must be equal to 1394 uppercase(lowercase(character)). That requirement may not be 1395 satisfied with IDNs. For example, there are some characters in 1396 scripts that use case distinction that do not have counterparts in 1397 one case or the other. The relationship between upper case and lower 1398 case may even be language-dependent, with different languages (or 1399 even the same language in different areas) expecting different 1400 mappings. Of course, the expectations of users who are accustomed to 1401 a case-insensitive DNS environment will probably be well-served if 1402 user agents perform case folding prior to IDNA processing, but the 1403 IDNA procedures themselves should neither require such mapping nor 1404 expect them when they are not natural to the localized environment. 1406 7.4. The Question of Prefix Changes 1408 The conditions that would require a change in the IDNA ACE prefix 1409 ("xn--" for the version of IDNA specified in [RFC3490]) have been a 1410 great concern to the community. A prefix change would clearly be 1411 necessary if the algorithms were modified in a manner that would 1412 create serious ambiguities during subsequent transition in 1413 registrations. This section summarizes our conclusions about the 1414 conditions under which changes in prefix would be necessary and the 1415 implications of such a change. 1417 7.4.1. Conditions Requiring a Prefix Change 1419 An IDN prefix change is needed if a given string would be looked up 1420 or otherwise interpreted differently depending on the version of the 1421 protocol or tables being used. Consequently, work to update IDNs 1422 would require a prefix change if, and only if, one of the following 1423 four conditions were met: 1425 1. The conversion of an A-label to Unicode (i.e., a U-label) yields 1426 one string under IDNA2003 (RFC3490) and a different string under 1427 IDNA2008. 1429 2. An input string that is valid under IDNA2003 and also valid under 1430 IDNA2008 yields two different A-labels with the different 1431 versions of IDNA. This condition is believed to be essentially 1432 equivalent to the one above except for a very small number of 1433 edge cases which may not, pragmatically, justify a prefix change 1434 (See Section 7.2). 1436 Note, however, that if the input string is valid under one 1437 version and not valid under the other, this condition does not 1438 apply. See the first item in Section 7.4.2, below. 1440 3. A fundamental change is made to the semantics of the string that 1441 is inserted in the DNS, e.g., if a decision were made to try to 1442 include language or specific script information in that string, 1443 rather than having it be just a string of characters. 1445 4. A sufficiently large number of characters is added to Unicode so 1446 that the Punycode mechanism for block offsets no longer has 1447 enough capacity to reference the higher-numbered planes and 1448 blocks. This condition is unlikely even in the long term and 1449 certain not to arise in the next few years. 1451 7.4.2. Conditions Not Requiring a Prefix Change 1453 In particular, as a result of the principles described above, none of 1454 the following changes require a new prefix: 1456 1. Prohibition of some characters as input to IDNA. This may make 1457 names that are now registered inaccessible, but does not require 1458 a prefix change. 1460 2. Adjustments in IDNA tables or actions, including normalization 1461 definitions, that affect characters that were already invalid 1462 under IDNA2003. 1464 3. Changes in the style of the IDNA definition that does not alter 1465 the actions performed by IDNA. 1467 7.4.3. Implications of Prefix Changes 1469 While it might be possible to make a prefix change, the costs of such 1470 a change are considerable. Even if they wanted to do so, all 1471 registries could not convert all IDNA2003 ("xn--") registrations to a 1472 new form at the same time and synchronize that change with 1473 applications supporting lookup. Unless all existing registrations 1474 were simply to be declared invalid (and perhaps even then) systems 1475 that needed to support both labels with old prefixes and labels with 1476 new ones would first process a putative label under the IDNA2008 1477 rules and try to look it up and then, if it were not found, would 1478 process the label under IDNA2003 rules and look it up again. That 1479 process could significantly slow down all processing that involved 1480 IDNs in the DNS especially since, in principle, a fully-qualified 1481 name could contain a mixture of labels that were registered with the 1482 old and new prefixes, a situation that would make the use of DNS 1483 caching very difficult. In addition, looking up the same input 1484 string as two separate A-labels would create some potential for 1485 confusion and attacks, since they could, in principle, map to 1486 different targets and then resolve to different entries in the DNS. 1488 Consequently, a prefix change is to be avoided if at all possible, 1489 even if it means accepting some IDNA2003 decisions about character 1490 distinctions as irreversible and/or giving special treatment to edge 1491 cases. 1493 7.5. Stringprep Changes and Compatibility 1495 The Nameprep [RFC3491] specification, a key part of IDNA2003, is a 1496 profile of Stringprep [RFC3454]. While Nameprep is a Stringprep 1497 profile specific to IDNA, Stringprep is used by a number of other 1498 protocols. Concerns have been expressed about problems for non-DNS 1499 uses of Stringprep being caused by changes to the specification 1500 intended to improve the handling of IDNs, most notably as this might 1501 affect identification and authentication protocols. The proposed new 1502 inclusion tables [IDNA2008-Tables], the reduction in the number of 1503 characters permitted as input for registration or lookup (Section 3), 1504 and even the proposed changes in handling of right to left strings 1505 [IDNA2008-Bidi] either give interpretations to strings prohibited 1506 under IDNA2003 or prohibit strings that IDNA2003 permitted. The 1507 IDNA2008 protocol does not use either Nameprep or Stringprep at all, 1508 so there are no side-effect changes to other protocols. 1510 It is particularly important to keep IDNA processing separate from 1511 processing for various security protocols because some of the 1512 constraints that are necessary for smooth and comprehensible use of 1513 IDNs may be unwanted or undesirable in other contexts. For example, 1514 the criteria for good passwords or passphrases are very different 1515 from those for desirable IDNs: passwords should be hard to guess, 1516 while domain names should normally be easily memorable. Similarly, 1517 internationalized SCSI identifiers and other protocol components are 1518 likely to have different requirements than IDNs. 1520 7.6. The Symbol Question 1522 One of the major differences between this specification and the 1523 original version of IDNA is that the original version permitted non- 1524 letter symbols of various sorts, including punctuation and line- 1525 drawing symbols, in the protocol. They were always discouraged in 1526 practice. In particular, both the "IESG Statement" about IDNA and 1527 all versions of the ICANN Guidelines specify that only language 1528 characters be used in labels. This specification disallows symbols 1529 entirely. There are several reasons for this, which include: 1531 o As discussed elsewhere, the original IDNA specification assumed 1532 that as many Unicode characters as possible should be permitted, 1533 directly or via mapping to other characters, in IDNs. This 1534 specification operates on an inclusion model, extrapolating from 1535 the LDH rules -- which have served the Internet very well -- to a 1536 Unicode base rather than an ASCII base. 1538 o Most Unicode names for letters are, in most cases, fairly 1539 intuitive, unambiguous and recognizable to users of the relevant 1540 script. Symbol names are more problematic because there may be no 1541 general agreement on whether a particular glyph matches a symbol; 1542 there are no uniform conventions for naming; variations such as 1543 outline, solid, and shaded forms may or may not exist; and so on. 1544 As just one example, consider a "heart" symbol as it might appear 1545 in a logo that might be read as "I love...". While the user might 1546 read such a logo as "I love..." or "I heart...", considerable 1547 knowledge of the coding distinctions made in Unicode is needed to 1548 know that there more than one "heart" character (e.g., U+2665, 1549 U+2661, and U+2765) and how to describe it. These issues are of 1550 particular importance if strings are expected to be understood or 1551 transcribed by the listener after being read out loud. 1552 [[anchor20: The above paragraph remains controversial as to 1553 whether it is valid. The WG will need to make a decision if this 1554 section is not dropped entirely.]] 1556 o As a simplified example of this, assume one wanted to use a 1557 "heart" or "star" symbol in a label. This is problematic because 1558 those names are ambiguous in the Unicode system of naming (the 1559 actual Unicode names require far more qualification). A user or 1560 would-be registrant has no way to know -- absent careful study of 1561 the code tables -- whether it is ambiguous (e.g., where there are 1562 multiple "heart" characters) or not. Conversely, the user seeing 1563 the hypothetical label doesn't know whether to read it -- try to 1564 transmit it to a colleague by voice -- as "heart", as "love", as 1565 "black heart", or as any of the other examples below. 1567 o The actual situation is even worse than this. There is no 1568 possible way for a normal, casual, user to tell the difference 1569 between the hearts of U+2665 and U+2765 and the stars of U+2606 1570 and U+2729 or the without somehow knowing to look for a 1571 distinction. We have a white heart (U+2661) and few black hearts. 1572 Consequently, describing a label as containing a heart hopelessly 1573 ambiguous: we can only know that it contains one of several 1574 characters that look like hearts or have "heart" in their names. 1575 In cities where "Square" is a popular part of a location name, one 1576 might well want to use a square symbol in a label as well and 1577 there are far more squares of various flavors in Unicode than 1578 there are hearts or stars. 1580 o The consequence of these ambiguities of description and 1581 dependencies on distinctions that were, or were not, made in 1582 Unicode codings is that symbols are a very poor basis for reliable 1583 communication. Consistent with this conclusion, the Unicode 1584 standard recommends that strings used in identifiers not contain 1585 symbols or punctuation [Unicode-UAX31]. Of course, these 1586 difficulties with symbols do not arise with actual pictographic 1587 languages and scripts which would be treated like any other 1588 language characters; the two should not be confused. 1590 7.7. Migration Between Unicode Versions: Unassigned Code Points 1592 In IDNA2003, labels containing unassigned code points are looked up 1593 on the assumption that, if they appear in labels and can be mapped 1594 and then resolved, the relevant standards must have changed and the 1595 registry has properly allocated only assigned values. 1597 In the protocol as described in these documents, strings containing 1598 unassigned code points must not be either looked up or registered. 1599 There are several reasons for this, with the most important ones 1600 being: 1602 o It cannot be known with sufficient reliability in advance that a 1603 code point that was not previously assigned will not be assigned 1604 to a compatibility character or one that would be otherwise 1605 disallowed by the rules in [IDNA2008-Tables]. In IDNA2003, since 1606 there is no direct dependency on NFKC (Stringprep's tables are 1607 based on NFKC, but IDNA2003 depends only on Stringprep), 1608 allocation of a compatibility character might produce some odd 1609 situations, but it would not be a problem. In IDNA2008, where 1610 compatibility characters are generally assigned to DISALLOWED, 1611 permitting strings containing unassigned characters to be looked 1612 up would permit violating the principle that characters in 1613 DISALLOWED are not looked up. 1615 o More generally, the status of an unassigned character with regard 1616 to the DISALLOWED and PROTOCOL-VALID categories, and whether 1617 contextual rules are required with the latter, cannot be evaluated 1618 until a character is actually assigned and known. By contrast, 1619 characters that are actually DISALLOWED are placed in that 1620 category only as a consequence of rules applied to known 1621 properties or per-character evaluation. 1623 It is possible to argue that the issues above are not important and 1624 that, as a consequence, it is better to retain the principle of 1625 looking up labels even if they contain unassigned characters because 1626 all of the important scripts and characters have been coded as of 1627 Unicode 5.1 and hence unassigned code points will be assigned only to 1628 obscure characters or archaic scripts. Unfortunately, that does not 1629 appear to be a safe assumption for at least two reasons. First, much 1630 the same claim of completeness has been made for earlier versions of 1631 Unicode. The reality is that a script that is obscure to much of the 1632 world may still be very important to those who use it. Cultural and 1633 linguistic preservation principles make it inappropriate to declare 1634 the script of no importance in IDNs. Second, we already have 1635 counterexamples in, e.g., the relationships associated with new Han 1636 characters being added (whether in the BMP or in Unicode Plane 2). 1638 7.8. Other Compatibility Issues 1640 The 2003 IDNA model includes several odd artifacts of the context in 1641 which it was developed. Many, if not all, of these are potential 1642 avenues for exploits, especially if the registration process permits 1643 "source" names (names that have not been processed through IDNA and 1644 Nameprep) to be registered. As one example, since the character 1645 Eszett, used in German, is mapped by IDNA2003 into the sequence "ss" 1646 rather than being retained as itself or prohibited, a string 1647 containing that character but that is otherwise in ASCII is not 1648 really an IDN (in the U-label sense defined above) at all. After 1649 Nameprep maps the Eszett out, the result is an ASCII string and so 1650 does not get an xn-- prefix, but the string that can be displayed to 1651 a user appears to be an IDN. The newer version of the protocol 1652 eliminates this artifact. A character is either permitted as itself 1653 or it is prohibited; special cases that make sense only in a 1654 particular linguistic or cultural context can be dealt with as 1655 localization matters where appropriate. 1657 8. Acknowledgments 1659 The editor and contributors would like to express their thanks to 1660 those who contributed significant early (pre-WG) review comments, 1661 sometimes accompanied by text, especially Mark Davis, Paul Hoffman, 1662 Simon Josefsson, and Sam Weiler. In addition, some specific ideas 1663 were incorporated from suggestions, text, or comments about sections 1664 that were unclear supplied by Frank Ellerman, Michael Everson, Asmus 1665 Freytag, Erik van der Poel, Michel Suignard, and Ken Whistler, 1666 although, as usual, they bear little or no responsibility for the 1667 conclusions the editor and contributors reached after receiving their 1668 suggestions. Thanks are also due to Vint Cerf, Debbie Garside, and 1669 Jefsey Morphin for conversations that led to considerable 1670 improvements in the content of this document. 1672 A meeting was held on 30 January 2008 to attempt to reconcile 1673 differences in perspective and terminology about this set of 1674 specifications between the design team and members of the Unicode 1675 Technical Consortium. The discussions at and subsequent to that 1676 meeting were very helpful in focusing the issues and in refining the 1677 specifications. The active participants at that meeting were (in 1678 alphabetic order as usual) Harald Alvestrand, Vint Cerf, Tina Dam, 1679 Mark Davis, Lisa Dusseault, Patrik Faltstrom (by telephone), Cary 1680 Karp, John Klensin, Warren Kumari, Lisa Moore, Erik van der Poel, 1681 Michel Suignard, and Ken Whistler. We express our thanks to Google 1682 for support of that meeting and to the participants for their 1683 contributions. 1685 Useful comments and text on the WG versions of the draft were 1686 received from many participants in the IETF "IDNABIS" WG and a number 1687 of document changes resulted from mailing list discussions made by 1688 that group. Marcos Sanz provided specific analysis and suggestions 1689 that were exceptionally helpful in refining the text, as did Vint 1690 Cerf, Mark Davis, Martin Duerst, Ken Whistler, and Andrew Sullivan. 1692 9. Contributors 1694 While the listed editor held the pen, this core of this document and 1695 the initial WG version represents the joint work and conclusions of 1696 an ad hoc design team consisting of the editor and, in alphabetic 1697 order, Harald Alvestrand, Tina Dam, Patrik Faltstrom, and Cary Karp. 1698 In addition, there were many specific contributions and helpful 1699 comments from those listed in the Acknowledgments section and others 1700 who have contributed to the development and use of the IDNA 1701 protocols. 1703 10. Internationalization Considerations 1705 DNS labels and fully-qualified domain names provide mnemonics that 1706 assist in identifying and referring to resources on the Internet. 1707 IDNs expand the range of those mnemonics to include those based on 1708 languages and character sets other than Western European and Roman- 1709 derived ones. But domain "names" are not, in general, words in any 1710 language. The recommendations of the IETF policy on character sets 1711 and languages, BCP 18 [RFC2277] are applicable to situations in which 1712 language identification is used to provide language-specific 1713 contexts. The DNS is, by contrast, global and international and 1714 ultimately has nothing to do with languages. Adding languages (or 1715 similar context) to IDNs generally, or to DNS matching in particular, 1716 would imply context dependent matching in DNS, which would be a very 1717 significant change to the DNS protocol itself. It would also imply 1718 that users would need to identify the language associated with a 1719 particular label in order to look that label up, a decision that 1720 would be impossible in many or most cases. 1722 11. IANA Considerations 1724 This section gives an overview of registries required for IDNA. The 1725 actual definitions of the first two appear in [IDNA2008-Tables]. 1727 11.1. IDNA Character Registry 1729 The distinction among the three major categories "UNASSIGNED", 1730 "DISALLOWED", and "PROTOCOL-VALID" is made by special categories and 1731 rules that are integral elements of [IDNA2008-Tables]. Convenience 1732 in programming and validation requires a registry of characters and 1733 scripts and their categories, updated for each new version of Unicode 1734 and the characters it contains. The details of this registry are 1735 specified in [IDNA2008-Tables]. 1737 11.2. IDNA Context Registry 1739 For characters that are defined in the IDNA Character Registry list 1740 as PROTOCOL-VALID but requiring a contextual rule (i.e., the types of 1741 rule described in Section 3.1.1.1), IANA will create and maintain a 1742 list of approved contextual rules. The details for those rules 1743 appear in [IDNA2008-Tables]. 1745 11.3. IANA Repository of IDN Practices of TLDs 1747 This registry, historically described as the "IANA Language Character 1748 Set Registry" or "IANA Script Registry" (both somewhat misleading 1749 terms) is maintained by IANA at the request of ICANN. It is used to 1750 provide a central documentation repository of the IDN policies used 1751 by top level domain (TLD) registries who volunteer to contribute to 1752 it and is used in conjunction with ICANN Guidelines for IDN use. 1754 It is not an IETF-managed registry and, while the protocol changes 1755 specified here may call for some revisions to the tables, these 1756 specifications have no direct effect on that registry and no IANA 1757 action is required as a result. 1759 12. Security Considerations 1761 12.1. General Security Issues with IDNA 1763 This document in the IDNA2008 series is purely explanatory and 1764 informational and consequently introduces no new security issues. It 1765 would, of course, be a poor idea for someone to try to implement from 1766 it; such an attempt would almost certainly lead to interoperability 1767 problems and might lead to security ones. A discussion of security 1768 issues with IDNA, including some relevant history, appears in 1769 [IDNA2008-Defs]. 1771 13. References 1773 13.1. Normative References 1775 [ASCII] American National Standards Institute (formerly United 1776 States of America Standards Institute), "USA Code for 1777 Information Interchange", ANSI X3.4-1968, 1968. 1779 ANSI X3.4-1968 has been replaced by newer versions with 1780 slight modifications, but the 1968 version remains 1781 definitive for the Internet. 1783 [IDNA2008-Bidi] 1784 Alvestrand, H. and C. Karp, "An updated IDNA criterion for 1785 right to left scripts", July 2008, . 1788 [IDNA2008-Defs] 1789 Klensin, J., "Internationalized Domain Names for 1790 Applications (IDNA): Definitions and Document Framework", 1791 November 2008, . 1794 [IDNA2008-Protocol] 1795 Klensin, J., "Internationalized Domain Names in 1796 Applications (IDNA): Protocol", November 2008, . 1799 [IDNA2008-Tables] 1800 Faltstrom, P., "The Unicode Code Points and IDNA", 1801 July 2008, . 1804 A version of this document is available in HTML format at 1805 http://stupid.domain.name/idnabis/ 1806 draft-ietf-idnabis-tables-02.html 1808 [RFC3490] Faltstrom, P., Hoffman, P., and A. Costello, 1809 "Internationalizing Domain Names in Applications (IDNA)", 1810 RFC 3490, March 2003. 1812 [RFC3492] Costello, A., "Punycode: A Bootstring encoding of Unicode 1813 for Internationalized Domain Names in Applications 1814 (IDNA)", RFC 3492, March 2003. 1816 [Unicode-UAX15] 1817 The Unicode Consortium, "Unicode Standard Annex #15: 1818 Unicode Normalization Forms", March 2008, 1819 . 1821 [Unicode51] 1822 The Unicode Consortium, "The Unicode Standard, Version 1823 5.1.0", 2008. 1825 defined by: The Unicode Standard, Version 5.0, Boston, MA, 1826 Addison-Wesley, 2007, ISBN 0-321-48091-0, as amended by 1827 Unicode 5.1.0 1828 (http://www.unicode.org/versions/Unicode5.1.0/). 1830 13.2. Informative References 1832 [BIG5] Institute for Information Industry of Taiwan, "Computer 1833 Chinese Glyph and Character Code Mapping Table, Technical 1834 Report C-26", 1984. 1836 There are several forms and variations and a closely- 1837 related standard, CNS 11643. See the discussion in 1838 Chapter 3 of Lunde, K., CJKV Information Processing, 1839 O'Reilly & Associates, 1999 1841 [GB18030] "Chinese National Standard GB 18030-2000: Information 1842 Technology -- Chinese ideograms coded character set for 1843 information interchange -- Extension for the basic set.", 1844 2000. 1846 [RFC0810] Feinler, E., Harrenstien, K., Su, Z., and V. White, "DoD 1847 Internet host table specification", RFC 810, March 1982. 1849 [RFC0952] Harrenstien, K., Stahl, M., and E. Feinler, "DoD Internet 1850 host table specification", RFC 952, October 1985. 1852 [RFC1034] Mockapetris, P., "Domain names - concepts and facilities", 1853 STD 13, RFC 1034, November 1987. 1855 [RFC1035] Mockapetris, P., "Domain names - implementation and 1856 specification", STD 13, RFC 1035, November 1987. 1858 [RFC1123] Braden, R., "Requirements for Internet Hosts - Application 1859 and Support", STD 3, RFC 1123, October 1989. 1861 [RFC2181] Elz, R. and R. Bush, "Clarifications to the DNS 1862 Specification", RFC 2181, July 1997. 1864 [RFC2277] Alvestrand, H., "IETF Policy on Character Sets and 1865 Languages", BCP 18, RFC 2277, January 1998. 1867 [RFC2673] Crawford, M., "Binary Labels in the Domain Name System", 1868 RFC 2673, August 1999. 1870 [RFC2782] Gulbrandsen, A., Vixie, P., and L. Esibov, "A DNS RR for 1871 specifying the location of services (DNS SRV)", RFC 2782, 1872 February 2000. 1874 [RFC3454] Hoffman, P. and M. Blanchet, "Preparation of 1875 Internationalized Strings ("stringprep")", RFC 3454, 1876 December 2002. 1878 [RFC3491] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep 1879 Profile for Internationalized Domain Names (IDN)", 1880 RFC 3491, March 2003. 1882 [RFC3743] Konishi, K., Huang, K., Qian, H., and Y. Ko, "Joint 1883 Engineering Team (JET) Guidelines for Internationalized 1884 Domain Names (IDN) Registration and Administration for 1885 Chinese, Japanese, and Korean", RFC 3743, April 2004. 1887 [RFC3987] Duerst, M. and M. Suignard, "Internationalized Resource 1888 Identifiers (IRIs)", RFC 3987, January 2005. 1890 [RFC4290] Klensin, J., "Suggested Practices for Registration of 1891 Internationalized Domain Names (IDN)", RFC 4290, 1892 December 2005. 1894 [RFC4690] Klensin, J., Faltstrom, P., Karp, C., and IAB, "Review and 1895 Recommendations for Internationalized Domain Names 1896 (IDNs)", RFC 4690, September 2006. 1898 [RFC4713] Lee, X., Mao, W., Chen, E., Hsu, N., and J. Klensin, 1899 "Registration and Administration Recommendations for 1900 Chinese Domain Names", RFC 4713, October 2006. 1902 [Unicode-Security] 1903 The Unicode Consortium, "Unicode Technical Standard #39: 1904 Unicode Security Mechanisms", August 2008, 1905 . 1907 [Unicode-UAX31] 1908 The Unicode Consortium, "Unicode Standard Annex #31: 1909 Unicode Identifier and Pattern Syntax", March 2008, 1910 . 1912 [Unicode-UTR36] 1913 The Unicode Consortium, "Unicode Technical Report #36: 1914 Unicode Security Considerations", July 2008, 1915 . 1917 Appendix A. Change Log 1919 [[ RFC Editor: Please remove this appendix. ]] 1921 A.1. Changes between Version -00 and Version -01 of 1922 draft-ietf-idnabis-rationale 1924 o Clarified the U-label definition to note that U-labels must 1925 contain at least one non-ASCII character. Also clarified the 1926 relationship among label types. 1928 o Rewrote the discussion of Labels in Registration (Section 7.1.2) 1929 and related text about IDNA-validity (in the "Defs" document as of 1930 -04 of this one) to narrow its focus and remove more general 1931 restrictions. Added a temporary note in line to explain the 1932 situation. 1934 o Changed the "IDNA uses Unicode" statement to focus on 1935 compatibility with IDNA2003 and avoid more general or 1936 controversial assertions. 1938 o Added a discussion of examples to Section 7.1 1940 o Made a number of other small editorial changes and corrections 1941 suggested by Mark Davis. 1943 o Added several more discussion anchors and notes and expanded or 1944 updated some existing ones. 1946 A.2. Version -02 1948 o Trimmed change log, removing information about pre-WG drafts. 1950 o Adjusted discussion of Contextual Rules to match the new location 1951 of the tables and some conceptual material. 1953 o Rewrote the material on preprocessing somewhat. 1955 o Moved the material about relationships with IDNA2003 to be part of 1956 a single section on transitions. 1958 o Removed several placeholders and made editorial changes in 1959 accordance with decisions made at IETF 72 in Dublin and not 1960 disputed on the mailing list. 1962 A.3. Version -03 1964 This special update to the Rationale document is intended to try to 1965 get the discussion of what is normative or not under control. While 1966 the IETF does not normally annotate individual sections of documents 1967 with whether they are normative or not, concerns that we don't know 1968 which is which, claims that some material is normative that would be 1969 problematic if so classified, etc., argue that we should at least be 1970 able to have a clear discussion on the subject. 1972 Two annotations have been applied to sections that might reasonably 1973 be considered normative. One annotation is based on the list of 1974 sections in Mark Davis's note of 29 September (http:// 1975 www.alvestrand.no/pipermail/idna-update/2008-September/002667.html). 1976 The other is based on an elaboration of John Klensin's response on 7 1977 October (http://www.alvestrand.no/pipermail/idna-update/2008-October/ 1978 002691.html). These should just be considered two suggestions to 1979 illuminate and, one hopes, advance the Working Group's discussions. 1981 Some additional editorial changes have been made, but they are 1982 basically trivial. In the editor's judgment, it is not possible to 1983 make significantly more progress with this document until the matter 1984 of document organization is settled. 1986 A.4. Version -04 1988 o Definitional and other normative material moved to new document 1989 (draft-ietf-idnabis-defs). Version -03 annotations removed. 1991 o Material on differences between IDNA2003 and IDNA2008 moved to an 1992 appendix in Protocol. 1994 o Material left over from the origins of this document as a 1995 preliminary proposal has been removed or rewritten. 1997 o Changes made to reflect consensus call results, including removing 1998 several placeholder notes for discussion. 2000 o Added more material, including discussion of historic scripts, to 2001 Section 3.2 on registration policies. 2003 o Added a new section (Section 7.2) to contain specific discussion 2004 of handling of characters that are interpreted differently in 2005 input to IDNA2003 and 2008. 2007 o Some material, including this section/appendix, rearranged. 2009 A.5. Version -05 2011 o Many small editorial changes, including changes to eliminate the 2012 last vestiges of what appeared to be 2119 language (upper-case 2013 MUST, SHOULD, or MAY) and small adjustments to terminology. 2015 A.6. Version -06 2017 o Removed Security Considerations material and pointed to Defs, 2018 where it now appears as of version 05. 2020 o Started changing uses of "IDNA2008" in running text to "in these 2021 specifications" or the equivalent. These documents are titled 2022 simply "IDNA"; once they are standardized, "the current version" 2023 may be a more appropriate reference than one containing a year. 2024 As discussed on the mailing list, we can and should discuss how to 2025 refer to these documents at an appropriate time (e.g., when we 2026 know when we will be finished) but, in the interim, it seems 2027 appropriate to simply start getting rid of the version-specific 2028 terminology where it can naturally be removed. 2030 o Additional discussion of mappings, etc., especially for case- 2031 sensitivity. 2033 o More editorial fine-tuning. 2035 Author's Address 2037 John C Klensin 2038 1770 Massachusetts Ave, Ste 322 2039 Cambridge, MA 02140 2040 USA 2042 Phone: +1 617 245 1457 2043 Email: john+ietf@jck.com 2045 Full Copyright Statement 2047 Copyright (C) The IETF Trust (2008). 2049 This document is subject to the rights, licenses and restrictions 2050 contained in BCP 78, and except as set forth therein, the authors 2051 retain all their rights. 2053 This document and the information contained herein are provided on an 2054 "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS 2055 OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND 2056 THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS 2057 OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF 2058 THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED 2059 WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 2061 Intellectual Property 2063 The IETF takes no position regarding the validity or scope of any 2064 Intellectual Property Rights or other rights that might be claimed to 2065 pertain to the implementation or use of the technology described in 2066 this document or the extent to which any license under such rights 2067 might or might not be available; nor does it represent that it has 2068 made any independent effort to identify any such rights. Information 2069 on the procedures with respect to rights in RFC documents can be 2070 found in BCP 78 and BCP 79. 2072 Copies of IPR disclosures made to the IETF Secretariat and any 2073 assurances of licenses to be made available, or the result of an 2074 attempt made to obtain a general license or permission for the use of 2075 such proprietary rights by implementers or users of this 2076 specification can be obtained from the IETF on-line IPR repository at 2077 http://www.ietf.org/ipr. 2079 The IETF invites any interested party to bring to its attention any 2080 copyrights, patents or patent applications, or other proprietary 2081 rights that may cover technology that may be required to implement 2082 this standard. Please address the information to the IETF at 2083 ietf-ipr@ietf.org.