idnits 2.17.00 (12 Aug 2021) /tmp/idnits16085/draft-irtf-panrg-what-not-to-do-19.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == The page length should not exceed 58 lines per page, but there was 1 longer page, the longest (page 1) being 2087 lines Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (26 March 2021) is 414 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC0791' is defined on line 1824, but no explicit reference was found in the text == Outdated reference: draft-iab-use-it-or-lose-it has been published as RFC 9170 == Outdated reference: draft-ietf-quic-transport has been published as RFC 9000 == Outdated reference: A later version (-05) exists of draft-irtf-panrg-path-properties-01 == Outdated reference: draft-irtf-panrg-questions has been published as RFC 9217 -- Obsolete informational reference (is this intentional?): RFC 1190 (Obsoleted by RFC 1819) -- Obsolete informational reference (is this intentional?): RFC 2001 (Obsoleted by RFC 2581) -- Obsolete informational reference (is this intentional?): RFC 2309 (Obsoleted by RFC 7567) -- Obsolete informational reference (is this intentional?): RFC 2460 (Obsoleted by RFC 8200) -- Obsolete informational reference (is this intentional?): RFC 2481 (Obsoleted by RFC 3168) -- Obsolete informational reference (is this intentional?): RFC 3697 (Obsoleted by RFC 6437) -- Obsolete informational reference (is this intentional?): RFC 5575 (Obsoleted by RFC 8955) Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 8 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 PANRG S. Dawkins, Ed. 3 Internet-Draft Tencent America 4 Intended status: Informational 26 March 2021 5 Expires: 27 September 2021 7 Path Aware Networking: Obstacles to Deployment (A Bestiary of Roads Not 8 Taken) 9 draft-irtf-panrg-what-not-to-do-19 11 Abstract 13 At the first meeting of the Path Aware Networking Research Group, the 14 research group agreed to catalog and analyze past efforts to develop 15 and deploy Path Aware techniques, most of which were unsuccessful or 16 at most partially successful, in order to extract insights and 17 lessons for path-aware networking researchers. 19 This document contains that catalog and analysis. 21 Status of This Memo 23 This Internet-Draft is submitted in full conformance with the 24 provisions of BCP 78 and BCP 79. 26 Internet-Drafts are working documents of the Internet Engineering 27 Task Force (IETF). Note that other groups may also distribute 28 working documents as Internet-Drafts. The list of current Internet- 29 Drafts is at https://datatracker.ietf.org/drafts/current/. 31 Internet-Drafts are draft documents valid for a maximum of six months 32 and may be updated, replaced, or obsoleted by other documents at any 33 time. It is inappropriate to use Internet-Drafts as reference 34 material or to cite them other than as "work in progress." 36 This Internet-Draft will expire on 27 September 2021. 38 Copyright Notice 40 Copyright (c) 2021 IETF Trust and the persons identified as the 41 document authors. All rights reserved. 43 This document is subject to BCP 78 and the IETF Trust's Legal 44 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 45 license-info) in effect on the date of publication of this document. 46 Please review these documents carefully, as they describe your rights 47 and restrictions with respect to this document. 49 Table of Contents 51 1. Introduction 52 1.1. What Do "Path" and "Path Awareness" Mean in this Document? 53 2. A Perspective On This Document 54 2.1. Notes for the Reader 55 2.2. A Note About Path-Aware Techniques Included In This 56 Document 57 2.3. Venue for Discussion of this Document 58 2.4. Architectural Guidance 59 2.5. Terminology Used in this Document 60 2.6. Methodology for Contributions 61 3. Applying the Lessons We've Learned 62 4. Summary of Lessons Learned 63 4.1. Justifying Deployment 64 4.2. Providing Benefits for Early Adopters 65 4.3. Providing Benefits During Partial Deployment 66 4.4. Outperforming End-to-end Protocol Mechanisms 67 4.5. Paying for Path Aware Techniques 68 4.6. Impact on Operational Practices 69 4.7. Per-connection State 70 4.8. Keeping Traffic on Fast-paths 71 4.9. Endpoints Trusting Intermediate Nodes 72 4.10. Intermediate Nodes Trusting Endpoints 73 4.11. Reacting to Distant Signals 74 4.12. Support in Endpoint Protocol Stacks 75 4.13. Planning For Failure 76 5. Future Work 77 6. Contributions 78 6.1. Stream Transport (ST, ST2, ST2+) 79 6.1.1. Reasons for Non-deployment 80 6.1.2. Lessons Learned. 81 6.2. Integrated Services (IntServ) 82 6.2.1. Reasons for Non-deployment 83 6.2.2. Lessons Learned. 84 6.3. Quick-Start TCP 85 6.3.1. Reasons for Non-deployment 86 6.3.2. Lessons Learned 87 6.4. ICMP Source Quench 88 6.4.1. Reasons for Non-deployment 89 6.4.2. Lessons Learned 90 6.5. Triggers for Transport (TRIGTRAN) 91 6.5.1. Reasons for Non-deployment 92 6.5.2. Lessons Learned. 93 6.6. Shim6 94 6.6.1. Reasons for Non-deployment 95 6.6.2. Lessons Learned 96 6.6.3. Addendum on MultiPath TCP 97 6.7. Next Steps in Signaling (NSIS) 98 6.7.1. Reasons for Non-deployment 99 6.7.2. Lessons Learned 100 6.8. IPv6 Flow Label 101 6.8.1. Reasons for Non-deployment 102 6.8.2. Lessons Learned 103 6.9. Explicit Congestion Notification (ECN) 104 6.9.1. Reasons for Non-deployment 105 6.9.2. Lessons Learned 106 7. Security Considerations 107 8. IANA Considerations 108 9. Acknowledgments 109 10. Informative References 110 Author's Address 112 1. Introduction 114 This document describes the lessons that IETF participants have 115 learned (and learned the hard way) about Path Aware Networking over a 116 period of several decades, and provides an analysis of reasons why 117 various Path Aware Networking techniques have seen limited or no 118 deployment. 120 1.1. What Do "Path" and "Path Awareness" Mean in this Document? 122 One of the first questions reviewers of this document have asked is 123 "what's the definition of a path, and what's the definition of path 124 awareness?" That is not an easy question to answer for this 125 document. 127 These terms have definitions in other [PANRG] documents, and are 128 still the subject of some discussion in the research group, as of the 129 date of this document. But because this document reflects work 130 performed over several decades, the technologies described in 131 Section 6 significantly predate the current definitions of "path" and 132 "path aware" in use in the Path Aware Networking Research Group, and 133 it is unlikely that all the contributors to Section 6 would have had 134 the same understanding of these terms. Those technologies were 135 considered "path aware" in early PANRG discussions, and so are 136 included in this retrospective document. 138 It is worth noting that the definitions of "path" and "path aware" in 139 [I-D.irtf-panrg-path-properties] would apply to path aware networking 140 techniques at a number of levels of the Internet protocol 141 architecture ([RFC1122], plus several decades of refinements), but 142 the contributions received for this document tended to target the 143 Transport Layer, and to treat a "path" constructed by routers as a 144 "black box". It would be useful to consider how applicable the 145 Lessons Learned cataloged in this document are, at other layers, and 146 that would be a fine topic for follow-on research. 148 The current definition of "Path" in the Path Aware Networking 149 Research Group appears in Section 2 ("Terminology") in 150 [I-D.irtf-panrg-path-properties]. That definition is included here 151 as a convenience to the reader. 153 | Path: A sequence of adjacent path elements over which a packet can 154 | be transmitted, starting and ending with a node. A path is 155 | unidirectional. Paths are time-dependent, i.e., the sequence of 156 | path elements over which packets are sent from one node to another 157 | may change. A path is defined between two nodes. For multicast 158 | or broadcast, a packet may be sent by one node and received by 159 | multiple nodes. In this case, the packet is sent over multiple 160 | paths at once, one path for each combination of sending and 161 | receiving node; these paths do not have to be disjoint. Note that 162 | an entity may have only partial visibility of the path elements 163 | that comprise a path and visibility may change over time. 164 | Different entities may have different visibility of a path and/or 165 | treat path elements at different levels of abstraction. 167 The current definition of "Path Awareness", used by the Path Aware 168 Networking Research Group, appears in Section 1.1 ("Definition") in 169 [I-D.irtf-panrg-questions]. That definition is included here as a 170 convenience to the reader. 172 | For purposes of this document, "path aware networking" describes 173 | endpoint discovery of the properties of paths they use for 174 | communication, and endpoint reaction to these properties that 175 | affects routing and/or transmission; note that this can and 176 | already does happen to some extent in the current Internet 177 | architecture. Expanding on this definition, a "path aware 178 | internetwork" is one in which endpoint discovery of path 179 | properties and endpoint selection of paths used by traffic 180 | exchanged by the endpoint are explicitly supported, regardless of 181 | the specific design of the protocol features which enable this 182 | discovery and selection. 184 2. A Perspective On This Document 186 At the first meeting of the Path Aware Networking Research Group 187 [PANRG], at IETF 99 [PANRG-99], Olivier Bonaventure led a discussion 188 of "A Decade of Path Awareness" [PATH-Decade], on attempts, which 189 were mostly unsuccessful for a variety of reasons, to exploit Path 190 Aware techniques and achieve a variety of goals over the past decade. 191 At the end of that discussion, two things were abundantly clear. 193 * The Internet community has accumulated considerable experience 194 with many Path Aware techniques over a long period of time, and 196 * Although some path aware techniques have been deployed (for 197 example, Differentiated Services, or DiffServ [RFC2475]), most of 198 these techniques haven't seen widespread adoption and deplyment. 199 Even "successful" techniques like DiffServ can face obstacles that 200 prevents wider usage. The reasons for non-adoption and limited 201 adoption and deployment are many, and are worthy of study. 203 The meta-lessons from that experience were 205 * Path aware networking has been more Research than Engineering, so 206 establishing an IRTF Research Group for Path Aware Networking is 207 the right thing to do [RFC7418]. 209 * Analyzing a catalog of past experience to learn the reasons for 210 non-adoption would be a great first step for the Research Group. 212 Allison Mankin, as IRTF Chair, officially chartered the Path Aware 213 Networking Research Group in July, 2018. 215 This document contains the analysis performed by that research group 216 (Section 4), based on that catalog (Section 6). 218 This document represents the consensus of the Path Aware Networking 219 Research Group. 221 2.1. Notes for the Reader 223 This Informational document discusses Path Aware protocol mechanisms 224 considered, and in some cases standardized, by the Internet 225 Engineering Task Force (IETF), and considers Lessons Learned from 226 those mechanisms. The intention is to inform the work of protocol 227 designers, whether in the IRTF, the IETF, or elsewhere in the 228 Internet ecosystem. 230 As an Informational document published in the IRTF stream, this 231 document has no authority beyond the quality of the analysis it 232 contains. 234 2.2. A Note About Path-Aware Techniques Included In This Document 236 This document does not catalog every proposed path aware networking 237 technique that was not adopted and deployed. Instead, we limited our 238 focus to technologies that passed through the IETF community, and 239 still identified enough techniques to provide background for the 240 lessons included in Section 4 to inform researchers and protocol 241 engineers in their work. 243 No shame is intended for the techniques included in this document. 244 As shown in Section 4, the quality of specific techniques had little 245 to do with whether they were deployed or not. Based on the 246 techniques cataloged in this document, it is likely that when these 247 techniques were put forward, the proponents were trying to engineer 248 something that could not be engineered without first carrying out 249 research. Actual shame would be failing to learn from experience, 250 and failing to share that experience with other networking 251 researchers and engineers. 253 2.3. Venue for Discussion of this Document 255 (RFC Editor: please remove this section before publication) 257 Discussion of specific contributed experiences and this document in 258 general should take place on the PANRG mailing list. 260 2.4. Architectural Guidance 262 As background for understanding the Lessons Learned contained in this 263 document, the reader is encouraged to become familiar with the 264 Internet Architecture Board's documents on "What Makes for a 265 Successful Protocol?" [RFC5218] and "Planning for Protocol Adoption 266 and Subsequent Transitions" [RFC8170]. 268 Although these two documents do not specifically target path-aware 269 networking protocols, they are helpful resources for readers seeking 270 to improve their understanding of considerations for successful 271 adoption and deployment of any protocol. For example, the Basic 272 Success Factors described in Setion 2.1 of [RFC5218] are helpful for 273 readers of this document. 275 Because there is an economic aspect to decisions about deployment, 276 the IAB Workshop on Internet Technology Adoption and Transition 277 [ITAT] report [RFC7305] also provides food for thought. 279 Several of the Lessons Learned in Section 4 reflect considerations 280 described in [RFC5218], [RFC7305], and [RFC8170]. 282 2.5. Terminology Used in this Document 284 The terms Node and Element in this document have the meaning defined 285 in [I-D.irtf-panrg-path-properties]. 287 2.6. Methodology for Contributions 289 This document grew out of contributions by various IETF participants 290 with experience with one or more Path Aware Networking techniques. 292 There are many things that could be said about the Path Aware 293 networking techniques that have been developed. For the purposes of 294 this document, contributors were requested to provide 296 * the name of a technique, including an abbreviation if one was used 298 * if available, a long-term pointer to the best reference describing 299 the technique 301 * a short description of the problem the technique was intended to 302 solve 304 * a short description of the reasons why the technique wasn't 305 adopted 307 * a short statement of the lessons that researchers can learn from 308 our experience with this technique. 310 3. Applying the Lessons We've Learned 312 The initial scope for this document was roughly "what mistakes have 313 we made in the decade prior to [PANRG-99], that we shouldn't make 314 again". Some of the contributions in Section 6 predate the initial 315 scope. The earliest Path-Aware Networking technique referred to in 316 Section 6 is Section 6.1, published in the late 1970s. Given that 317 the networking ecosystem has evolved continuously, it seems 318 reasonable to consider how to apply these lessons. 320 The PANRG Research Group reviewed the Lessons Learned (Section 4) 321 contained in the May 23, 2019 version of this document at IETF 105 322 [PANRG-105-Min], and carried out additional discussion at IETF 106 323 [PANRG-106-Min]. Table 1 provides the "sense of the room" about each 324 lesson after those discussions. The intention was to capture whether 325 a specific lesson seems to be 327 * "Invariant" - well-understood and is likely to be applicable for 328 any proposed Path Aware Networking solution. 330 * "Variable" - has impeded deployment in the past, but might not be 331 applicable in a specific technique. Engineering analysis to 332 understand whether the lesson is applicable is prudent. 334 * "Not Now" - this characteristic tends to turn up a minefield full 335 of dragons, and prudent network engineers will wish to avoid 336 gambling on a technique that relies on this, until something 337 significant changes 339 Section 6.9 on ECN was added during the review and approval process, 340 based on a question from Martin Duke. That section, along with its 341 Lessons Learned and place in the "Invariant"/"Variable"/"Not Now" 342 taxonomy, as contained in the March 8, 2021 version of this document, 343 was discussed at [PANRG-110]. 345 +-----------------------------------------------------+-----------+ 346 | Lesson | Category | 347 +=====================================================+===========+ 348 | Justifying Deployment (Section 4.1) | Invariant | 349 +-----------------------------------------------------+-----------+ 350 | Providing Benefits for Early Adopters (Section 4.2) | Invariant | 351 +-----------------------------------------------------+-----------+ 352 | Providing Benefits during Partial Deployment | Invariant | 353 | (Section 4.3) | | 354 +-----------------------------------------------------+-----------+ 355 | Outperforming End-to-end Protocol Mechanisms | Variable | 356 | (Section 4.4) | | 357 +-----------------------------------------------------+-----------+ 358 | Paying for Path Aware Techniques (Section 4.5) | Invariant | 359 +-----------------------------------------------------+-----------+ 360 | Impact on Operational Practices (Section 4.6) | Invariant | 361 +-----------------------------------------------------+-----------+ 362 | Per-connection State (Section 4.7) | Variable | 363 +-----------------------------------------------------+-----------+ 364 | Keeping Traffic on Fast-paths (Section 4.8) | Variable | 365 +-----------------------------------------------------+-----------+ 366 | Endpoints Trusting Intermediate Nodes (Section 4.9) | Not Now | 367 +-----------------------------------------------------+-----------+ 368 | Intermediate Nodes Trusting Endpoints | Not Now | 369 | (Section 4.10) | | 370 +-----------------------------------------------------+-----------+ 371 | Reacting to Distant Signals (Section 4.11) | Variable | 372 +-----------------------------------------------------+-----------+ 373 | Support in Endpoint Protocol Stacks (Section 4.12) | Variable | 374 +-----------------------------------------------------+-----------+ 375 | Planning for Failure (Section 4.13) | Invariant | 376 +-----------------------------------------------------+-----------+ 378 Table 1 380 "Justifying Deployment", "Providing Benefits for Early Adopters", 381 "Paying for Path Aware Techniques", "Impact on Operational Practice", 382 and "Planning for Failure" were considered to be invariant - the 383 sense of the room was that these would always be considerations for 384 any proposed Path Aware Technique. 386 "Providing Benefits During Partial Deployment" was added after IETF 387 105, during research group last call, and is also considered to be 388 invariant. 390 For "Outperforming End-to-end Protocol Mechanisms", there is a trade- 391 off between improved performance from Path Aware Techniques and 392 additional complexity required by some Path Aware Techniques. 394 * For example, if you can obtain the same understanding of path 395 characteristics from measurements obtained over a few more round 396 trips, endpoint implementers are unlikely to be eager to add 397 complexity, and many attributes can be measured from an endpoint, 398 without assistance from intermediate nodes. 400 For "Per-connection State", the key questions discussed in the 401 research group were "how much state" and "where state is maintained". 403 * IntServ (Section 6.2) required state at every intermediate node 404 for every connection between two endpoints. As the Internet 405 ecosystem has evolved, carrying many connections in a tunnel that 406 appears to intermediate nodes as a single connection has become 407 more common, so that additional end-to-end connections don't add 408 additional state to intermediate nodes between tunnel endpoints. 409 If these tunnels are encrypted, intermediate nodes between tunnel 410 endpoints can't distinguish between connections, even if that were 411 desirable. 413 For "Keeping Traffic on Fast-paths", we noted that this was true for 414 many platforms, but not for all. 416 * For backbone routers, this is likely an invariant, but for 417 platforms that rely more on general-purpose computers to make 418 forwarding decisions, this may not be a fatal flaw for Path Aware 419 Networking techniques. 421 For "Endpoints Trusting Intermediate Nodes" and "Intermediate Nodes 422 Trusting Endpoints", these lessons point to the broader need to 423 revisit the Internet Threat Model. 425 * We noted with relief that discussions about this were already 426 underway in the IETF community at IETF 105 (see the Security Area 427 Open Meeting minutes [SAAG-105-Min] for discussion of 428 [I-D.arkko-arch-internet-threat-model] and [I-D.farrell-etm]), and 429 the Internet Architecture Board has created a mailing list for 430 continued discussions ([model-t]), but we recognize that there are 431 Path Aware Networking aspects of this effort, requiring research. 433 For "Reacting to Distant Signals", we noted that not all attributes 434 are equal. 436 * If an attribute is stable over an extended period of time, is 437 difficult to observe via end-to-end mechanisms, and is valuable, 438 Path Aware Techniques that rely on that attribute to provide a 439 significant benefit become more attractive. 441 * Analysis to help identify attributes that are useful enough to 442 justify deployment of Path Aware techniques that make use of those 443 attributes would be helpful. 445 For "Support in Endpoint Protocol Stacks", we noted that Path Aware 446 applications must be able to identify and communicate requirements 447 about path characteristics. 449 * The de-facto sockets API has no way of signaling application 450 expectations for the network path to the protocol stack. 452 4. Summary of Lessons Learned 454 This section summarizes the Lessons Learned from the contributed 455 subsections in Section 6. 457 Each Lesson Learned is tagged with one or more contributions that 458 encountered this obstacle as a significant impediment to deployment. 459 Other contributed techniques may have also encountered this obstacle, 460 but this obstacle may not have been the biggest impediment to 461 deployment for those techniques. 463 It is useful to notice that sometimes an obstacle might impede 464 deployment, while at other times, the same obstacle might prevent 465 adoption and deployment entirely. The research group discussed 466 distinguishing between obstacles that impede and obstacles that 467 prevent, but it appears that the boundary between "impede" and 468 "prevent" can shift over time - some of the Lessons Learned are based 469 on both Path Aware techniques that were not deployed, and Path Aware 470 techniques that were deployed, but were not deployed widely or 471 quickly. See Section 6.6 and Section 6.6.3 as one example of this 472 shifting boundary. 474 4.1. Justifying Deployment 476 The benefit of Path Awareness must be great enough to justify making 477 changes in an operational network. The colloquial U.S. American 478 English expression, "If it ain't broke, don't fix it" is a "best 479 current practice" on today's Internet. (See Section 6.3, 480 Section 6.4, Section 6.5, and Section 6.9, in addition to [RFC5218]). 482 4.2. Providing Benefits for Early Adopters 484 Providing benefits for early adopters can be key - if everyone must 485 deploy a technique in order for the technique to provide benefits, or 486 even to work at all, the technique is unlikely to be adopted widely 487 or quickly. (See Section 6.2 and Section 6.3, in addition to 488 [RFC5218]). 490 4.3. Providing Benefits During Partial Deployment 492 Some proposals require that all path elements along the full length 493 of the path must be upgraded to support a new technique, before any 494 benefits can be seen. This is likely to require coordination between 495 operators who control a subset of path elements, and between 496 operators and end users if endpoint upgrades are required. If a 497 technique provides benefits when only a part of the path has been 498 upgraded, this is likely to encourage adoption and deployment. (See 499 Section 6.2, Section 6.3, and Section 6.9, in addition to [RFC5218]). 501 4.4. Outperforming End-to-end Protocol Mechanisms 503 Adaptive end-to-end protocol mechanisms may respond to feedback 504 quickly enough that the additional realizable benefit from a new Path 505 Aware mechanism that tries to manipulate nodes along a path, or 506 observe the attributes of nodes along a path, may be much smaller 507 than anticipated (See Section 6.3 and Section 6.5). 509 4.5. Paying for Path Aware Techniques 511 "Follow the money." If operators can't charge for a Path Aware 512 technique to recover the costs of deploying it, the benefits to the 513 operator must be really significant. Corollary: If operators charge 514 for a Path Aware technique, the benefits to users of that Path Aware 515 technique must be significant enough to justify the cost. (See 516 Section 6.1, Section 6.2, Section 6.5, and Section 6.9). 518 4.6. Impact on Operational Practices 520 Impact of a Path Aware technique requiring changes to operational 521 practices can affect how quickly or widely a promising technique is 522 deployed. The impacts of these changes may make deployment more 523 likely, but often discourage deployment. (See Section 6.6, including 524 Section 6.6.3). 526 4.7. Per-connection State 528 Per-connection state in intermediate nodes has been an impediment to 529 adoption and deployment in the past, because of added cost and 530 complexity. Often, similar benefits can be achieved with much less 531 finely-grained state. This is especially true as we move from the 532 edge of the network, further into the routing core (See Section 6.1 533 and Section 6.2). 535 4.8. Keeping Traffic on Fast-paths 537 Many modern platforms, especially high-end routers, have been 538 designed with hardware that can make simple per-packet forwarding 539 decisions ("fast-paths"), but have not been designed to make heavy 540 use of in-band mechanisms such as IPv4 and IPv6 Router Alert Options 541 (RAO) that require more processing to make forwarding decisions. 542 Packets carrying in-band mechanisms are diverted to other processors 543 in the router with much lower packet processing rates. Operators can 544 be reluctant to deploy techniques that rely heavily on in-band 545 mechanisms because they may significantly reduce packet throughput. 546 (See Section 6.7). 548 4.9. Endpoints Trusting Intermediate Nodes 550 If intermediate nodes along the path can't be trusted, it's unlikely 551 that endpoints will rely on signals from intermediate nodes to drive 552 changes to endpoint behaviors. We note that "trust" is not binary - 553 one, low, level of trust applies when a node issuing a message can 554 confirm that it has visibility of the packets on the path it is 555 seeking to control [RFC8085] (e.g., an ICMP message included a quoted 556 packet from the source). A higher level of trust can arise when an 557 endpoint has established a short term, or even long term, trust 558 relationship with network nodes. (See Section 6.4 and Section 6.5). 560 4.10. Intermediate Nodes Trusting Endpoints 562 If the endpoints do not have any trust relationship with the 563 intermediate nodes along a path, operators have been reluctant to 564 deploy techniques that rely on endpoints sending unauthenticated 565 control signals to routers. (See Section 6.2 and Section 6.7). (We 566 also note this still remains a factor hindering deployment of 567 DiffServ). 569 4.11. Reacting to Distant Signals 571 Because the Internet is a distributed system, if the distance that 572 information from distant path elements travels to a Path Aware host 573 is sufficiently large, the information may no longer accurately 574 represent the state and situation at the distant host or elements 575 along the path when it is received locally. In this case, the 576 benefit that a Path Aware technique provides will be inconsistent, 577 and may not always be beneficial. (See Section 6.3). 579 4.12. Support in Endpoint Protocol Stacks 581 Just because a protocol stack provides a new feature/signal does not 582 mean that applications will use the feature/signal. Protocol stacks 583 may not know how to effectively utilize Path-Aware techniques, 584 because the protocol stack may require information from applications 585 to permit the technique to work effectively, but applications may not 586 a-priori know that information. Even if the application does know 587 that information, the de-facto sockets API has no way of signaling 588 application expectations for the network path to the protocol stack. 589 In order for applications to provide these expectations to protocol 590 stacks, we need an API that signals more than the packets to be sent. 591 (See Section 6.1 and Section 6.2). 593 4.13. Planning For Failure 595 If early implementers discover severe problems with a new feature, 596 that feature is likely to be disabled, and convincing implementers to 597 re-enable that feature can be very difficult, and can require years 598 or decades. In addition to testing, partial deployment for a subset 599 of users, implementing instrumentation that will detect degraded user 600 experience, and even "failback" to a previous version or "failover" 601 to an entirely different implementation are likely to be helpful. 602 (See Section 6.9). 604 5. Future Work 606 By its nature, this document has been retrospective. In addition to 607 considering how the Lessons Learned to date apply to current and 608 future Path Aware networking proposals, it's also worth considering 609 whether there is deeper investigation left to do. 611 * We note that this work was based on contributions from experts on 612 various Path Aware networking techniques, and all of the 613 contributed techniques involved unicast protocols. We didn't 614 consider how these lessons might apply to multicast, and, given 615 anecdotal reports at the IETF 109 MOPS working group meeting of IP 616 multicast offerings within data centers at one or more cloud 617 providers ([MOPS-109-Min]), it might be useful to think about path 618 awareness in multicast, before we have a history of unsuccessful 619 deployments to document. 621 * The question of whether a mechanism supports admission control, 622 based on either endpoints or applications, is associated with Path 623 Awareness. One of the motivations of IntServ and a number of 624 other architectures (e.g. Deterministic Networking, [RFC8655]) is 625 the ability to "say no" to an application based on resource 626 availability on a path, before the application tries to inject 627 traffic onto that path and discovers the path does not have the 628 capacity to sustain enough utility to meet the application's 629 minimum needs. The question of whether admission control is 630 needed comes up repeatedly, but we have learned a few useful 631 lessons that, while covered implicitly in some of the lessons 632 learned of the document, might be explained explicitly: 634 - We have gained a lot of experience with application-based 635 adaptation since the days where applications just injected 636 traffic in-elastically into the network. Such adaptations seem 637 to work well enough that admission control is of less value to 638 these applications 640 - There are end-to-end measurement techniques that can steer 641 traffic at the application layer (Content Distribution 642 Networks, multi-CDNs like Conviva [Conviva], etc.) 644 - We noted in Section 4.12 that applications often don't know how 645 to utilize Path Aware techniques. This includes not knowing 646 enough about their admission control threshold to be able to 647 ask accurately for the resources they need, whether this is 648 because the application itself doesn't know, or because the 649 application has no way to signal its expectations to the 650 underlying protocol stack. To date, attempts to help them 651 haven't gotten anywhere (e.g. the multiple-TSPEC additions to 652 RSVP to attempt to mirror codec selection by applications 653 [I-D.ietf-tsvwg-intserv-multiple-tspec] expired in 2013). 655 * We note that this work took the then-current IP network 656 architecture as given, at least at the time each technique was 657 proposed. It might be useful to consider aspects of the now- 658 current IP network architecture that ease, or impede, Path Aware 659 networking techniques. For example, there is limited ability in 660 IP to constrain bidirectional paths to be symmetric, and 661 information-centric networking protocols such as Named Data 662 Networking (NDN) and Content-Centric Networking (CCNx) ([RFC8793]) 663 must force bidirectional path symmetry using protocol-specific 664 mechanisms. 666 6. Contributions 668 Contributions on these Path Aware networking techniques were analyzed 669 to arrive at the Lessons Learned captured in Section 4. 671 Our expectation is that most readers will not need to read through 672 this section carefully, but we wanted to record these hard-fought 673 lessons as a service to others who may revisit this document, so 674 they'll have the details close at hand. 676 6.1. Stream Transport (ST, ST2, ST2+) 678 The suggested references for Stream Transport are: 680 * ST - A Proposed Internet Stream Protocol [IEN-119] 682 * Experimental Internet Stream Protocol, Version 2 (ST-II) [RFC1190] 684 * Internet Stream Protocol Version 2 (ST2) Protocol Specification - 685 Version ST2+ [RFC1819] 687 The first version of Stream Transport, ST [IEN-119], was published in 688 the late 1970's and was implemented and deployed on the ARPANET at 689 small scale. It was used throughout the 1980's for experimental 690 transmission of voice, video, and distributed simulation. 692 The second version of the ST specification (ST2) [RFC1190] [RFC1819] 693 was an experimental connection-oriented internetworking protocol that 694 operated at the same layer as connectionless IP. ST2 packets could 695 be distinguished by their IP header version numbers (IP, at that 696 time, used version number 4, while ST2 used version number 5). 698 ST2 used a control plane layered over IP to select routes and reserve 699 capacity for real-time streams across a network path, based on a flow 700 specification communicated by a separate protocol. The flow 701 specification could be associated with QoS state in routers, 702 producing an experimental resource reservation protocol. This 703 allowed ST2 routers along a path to offer end-to-end guarantees, 704 primarily to satisfy the QoS requirements for realtime services over 705 the Internet. 707 6.1.1. Reasons for Non-deployment 709 Although implemented in a range of equipment, ST2 was not widely used 710 after completion of the experiments. It did not offer the 711 scalability and fate-sharing properties that have come to be desired 712 by the Internet community. 714 The ST2 protocol is no longer in use. 716 6.1.2. Lessons Learned. 718 As time passed, the trade-off between router processing and link 719 capacity changed. Links became faster and the cost of router 720 processing became comparatively more expensive. 722 The ST2 control protocol used "hard state" - once a route was 723 established, and resources were reserved, routes and resources 724 existing until they were explicitly released via signaling. A soft- 725 state approach was thought superior to this hard-state approach, and 726 led to development of the IntServ model described in Section 6.2. 728 6.2. Integrated Services (IntServ) 730 The suggested references for IntServ are: 732 * RFC 1633 Integrated Services in the Internet Architecture: an 733 Overview [RFC1633] 735 * RFC 2211 Specification of the Controlled-Load Network Element 736 Service [RFC2211] 738 * RFC 2212 Specification of Guaranteed Quality of Service [RFC2212] 740 * RFC 2215 General Characterization Parameters for Integrated 741 Service Network Elements [RFC2215] 743 * RFC 2205 Resource ReSerVation Protocol (RSVP) [RFC2205] 745 In 1994, when the IntServ architecture document [RFC1633] was 746 published, real-time traffic was first appearing on the Internet. At 747 that time, bandwidth was still a scarce commodity. Internet Service 748 Providers built networks over DS3 (45 Mbps) infrastructure, and sub- 749 rate (< 1 Mpbs) access was common. Therefore, the IETF anticipated a 750 need for a fine-grained QoS mechanism. 752 In the IntServ architecture, some applications can require service 753 guarantees. Therefore, those applications use the Resource 754 Reservation Protocol (RSVP) [RFC2205] to signal QoS reservations 755 across network paths. Every router in the network that participates 756 in IntServ maintains per-flow soft-state to a) perform call admission 757 control and b) deliver guaranteed service. 759 Applications use Flow Specification (Flow Specs) [RFC2210] to 760 describe the traffic that they emit. RSVP reserves capacity for 761 traffic on a per Flow Spec basis. 763 6.2.1. Reasons for Non-deployment 765 Although IntServ has been used in enterprise and government networks, 766 IntServ was never widely deployed on the Internet because of its 767 cost. The following factors contributed to operational cost: 769 * IntServ must be deployed on every router that is on a path where 770 IntServ is to be used. Although it is possible to include a 771 router that does not participate in IntServ along the path being 772 controlled, if that router is likely to become a bottleneck, 773 IntServ cannot be used to avoid that bottleneck along the path 775 * IntServ maintained per flow state 777 As IntServ was being discussed, the following occurred: 779 * For many expected uses, it became more cost effective to solve the 780 QoS problem by adding bandwidth. Between 1994 and 2000, Internet 781 Service Providers upgraded their infrastructures from DS3 (45 782 Mbps) to OC-48 (2.4 Gbps). This meant that even if an endpoint 783 was using IntServ in an IntServ-enabled network, its requests 784 would rarely, if ever, be denied, so endpoints and Internet 785 Service Providers had little reason to enable IntServ. 787 * DiffServ [RFC2475] offered a more cost-effective, albeit less 788 fine-grained, solution to the QoS problem. 790 6.2.2. Lessons Learned. 792 The following lessons were learned: 794 * Any mechanism that requires every participating onpath router to 795 maintain per-flow state is not likely to succeed, unless the 796 additional cost for offering the feature can be recovered from the 797 user. 799 * Any mechanism that requires an operator to upgrade all of its 800 routers is not likely to succeed, unless the additional cost for 801 offering the feature can be recovered from the user. 803 In environments where IntServ has been deployed, trust relationships 804 with endpoints are very different from trust relationships on the 805 Internet itself, and there are often clearly-defined hierarchies in 806 Service Level Agreements (SLAs), and well-defined transport flows 807 operating with pre-determined capacity and latency requirements over 808 paths where capacity or other attributes are constrained. 810 IntServ was never widely deployed to manage capacity across the 811 Internet. However, the technique that it produced was deployed for 812 reasons other than bandwidth management. RSVP is widely deployed as 813 an MPLS signaling mechanism. BGP reuses the RSVP concept of Filter 814 Specs to distribute firewall filters, although they are called Flow 815 Spec Component Types in BGP [RFC5575]. 817 6.3. Quick-Start TCP 819 The suggested references for Quick-Start TCP are: 821 * Quick-Start for TCP and IP [RFC4782] 823 * Determining an appropriate initial sending rate over an 824 underutilized network path [SAF07] 826 * Fast Startup Internet Congestion Control for Broadband Interactive 827 Applications [Sch11] 829 * Using Quick-Start to enhance TCP-friendly rate control performance 830 in bidirectional satellite networks [QS-SAT] 832 Quick-Start [RFC4782] is an Experimental TCP extension that leverages 833 support from the routers on the path to determine an allowed initial 834 sending rate for a path through the Internet, either at the start of 835 data transfers or after idle periods. Without information about the 836 path, a sender cannot easily determine an appropriate initial sending 837 rate. The default TCP congestion control therefore uses the safe but 838 time-consuming slow-start algorithm [RFC5681]. With Quick-Start, 839 connections are allowed to use higher initial sending rates if there 840 is significant unused bandwidth along the path, and if the sender and 841 all of the routers along the path approve the request. 843 By examining the Time To Live (TTL) field in Quick-Start packets, a 844 sender can determine if routers on the path have approved the Quick- 845 Start request. However, this method is unable to take into account 846 the routers hidden by tunnels or other network nodes invisible at the 847 IP layer. 849 The protocol also includes a nonce that provides protection against 850 cheating routers and receivers. If the Quick-Start request is 851 explicitly approved by all routers along the path, the TCP host can 852 send at up to the approved rate; otherwise TCP would use the default 853 congestion control. Quick-Start requires modifications in the 854 involved end-systems as well in routers. Due to the resulting 855 deployment challenges, Quick-Start was only proposed in [RFC4782] for 856 controlled environments. 858 The Quick-Start mechanism is a lightweight, coarse-grained, in-band, 859 network-assisted fast startup mechanism. The benefits are studied by 860 simulation in a research paper [SAF07] that complements the protocol 861 specification. The study confirms that Quick-Start can significantly 862 speed up mid-sized data transfers. That paper also presents router 863 algorithms that do not require keeping per-flow state. Later studies 864 [Sch11] comprehensively analyzes Quick-Start with a full Linux 865 implementation and with a router fast path prototype using a network 866 processor. In both cases, Quick-Start could be implemented with 867 limited additional complexity. 869 6.3.1. Reasons for Non-deployment 871 However, experiments with Quick-Start in [Sch11] revealed several 872 challenges: 874 * Having information from the routers along the path can reduce the 875 risk of congestion, but cannot avoid it entirely. Determining 876 whether there is unused capacity is not trivial in actual router 877 and host implementations. Data about available capacity visible 878 at the IP layer may be imprecise, and due to the propagation 879 delay, information can already be outdated when it reaches a 880 sender. There is a trade-off between the speedup of data 881 transfers and the risk of congestion even with Quick-Start. This 882 could be mitigated by only allowing Quick-Start to access a 883 proportion of the unused capacity along a path. 885 * For scalable router fast path implementation, it is important to 886 enable parallel processing of packets, as this is a widely used 887 method e.g. in network processors. One challenge is 888 synchronization of information between packets that are processed 889 in parallel, which should be avoided as much as possible. 891 * Only some types of application traffic can benefit from Quick- 892 Start. Capacity needs to be requested and discovered. The 893 discovered capacity needs to be utilized by the flow, or it 894 implicitly becomes available for other flows. Failing to use the 895 requested capacity may have already reduced the pool of Quick- 896 Start capacity that was made available to other competing Quick- 897 Start requests. The benefit is greatest when senders use this 898 only for bulk flows and avoid sending unnecessary Quick-Start 899 requests, e.g. for flows that only send a small amount of data. 900 Choosing an appropriate request size requires application-internal 901 knowledge that is not commonly expressed by the transport API. 902 How a sender can determine the rate for an initial Quick-Start 903 request is still a largely unsolved problem. 905 There is no known deployment of Quick-Start for TCP or other IETF 906 transports. 908 6.3.2. Lessons Learned 910 Some lessons can be learned from Quick-Start. Despite being a very 911 light-weight protocol, Quick-Start suffers from poor incremental 912 deployment properties, both regarding the required modifications in 913 network infrastructure as well as its interactions with applications. 914 Except for corner cases, congestion control can be quite efficiently 915 performed end-to-end in the Internet, and in modern stacks there is 916 not much room for significant improvement by additional network 917 support. 919 After publication of the Quick-Start specification, there have been 920 large-scale experiments with an initial window of up to 10 MSS 921 [RFC6928]. This alternative "IW10" approach can also ramp-up data 922 transfers faster than the standard congestion control, but it only 923 requires sender-side modifications. As a result, this approach can 924 be easier and incrementally deployed in the Internet. While 925 theoretically Quick-Start can outperform "IW10", the improvement in 926 completion time for data transfer times can, in many cases, be small. 927 After publication of [RFC6928], most modern TCP stacks have increased 928 their default initial window. 930 6.4. ICMP Source Quench 932 The suggested references for ICMP Source Quench are: 934 * INTERNET CONTROL MESSAGE PROTOCOL [RFC0792] 936 The ICMP Source Quench message [RFC0792] allowed an on-path router to 937 request the source of a flow to reduce its sending rate. This method 938 allowed a router to provide an early indication of impending 939 congestion on a path to the sources that contribute to that 940 congestion. 942 6.4.1. Reasons for Non-deployment 944 This method was deployed in Internet routers over a period of time, 945 the reaction of endpoints to receiving this signal has varied. For 946 low speed links, with low multiplexing of flows the method could be 947 used to regulate (momentarily reduce) the transmission rate. 948 However, the simple signal does not scale with link speed, or the 949 number of flows sharing a link. 951 The approach was overtaken by the evolution of congestion control 952 methods in TCP [RFC2001], and later also by other IETF transports. 953 Because these methods were based upon measurement of the end-to-end 954 path and an algorithm in the endpoint, they were able to evolve and 955 mature more rapidly than methods relying on interactions between 956 operational routers and endpoint stacks. 958 After ICMP Source Quench was specified, the IETF began to recommend 959 that transports provide end-to-end congestion control [RFC2001]. The 960 Source Quench method has been obsoleted by the IETF [RFC6633], and 961 both hosts and routers must now silently discard this message. 963 6.4.2. Lessons Learned 965 This method had several problems: 967 First, [RFC0792] did not sufficiently specify how the sender would 968 react to the ICMP Source Quench signal from the path (e.g., 969 [RFC1016]). There was ambiguity in how the sender should utilize 970 this additional information. This could lead to unfairness in the 971 way that receivers (or routers) responded to this message. 973 Second, while the message did provide additional information, the 974 Explicit Congestion Notification (ECN) mechanism [RFC3168] provided a 975 more robust and informative signal for network nodes to provide early 976 indication that a path has become congested. 978 The mechanism originated at a time when the Internet trust model was 979 very different. Most endpoint implementations did not attempt to 980 verify that the message originated from an on-path node before they 981 utilized the message. This made it vulnerable to denial of service 982 attacks. In theory, routers might have chosen to use the quoted 983 packet contained in the ICMP payload to validate that the message 984 originated from an on-path node, but this would have increased per- 985 packet processing overhead for each router along the path, would have 986 required transport functionality in the router to verify whether the 987 quoted packet header corresponded to a packet the router had sent. 988 In addition, section 5.2 of [RFC4443] noted ICMPv6-based attacks on 989 hosts that would also have threatened routers processing ICMPv6 990 Source Quench payloads. As time passed, it became increasingly 991 obvious that the lack of validation of the messages exposed receivers 992 to a security vulnerability where the messages could be forged to 993 create a tangible denial of service opportunity. 995 6.5. Triggers for Transport (TRIGTRAN) 997 The suggested references for TRIGTRAN are: 999 * TRIGTRAN BOF at IETF 55 [TRIGTRAN-55] 1001 * TRIGTRAN BOF at IETF 56 [TRIGTRAN-56] 1003 TCP [RFC0793] has a well-known weakness - the end-to-end flow control 1004 mechanism has only a single signal, the loss of a segment, and TCP 1005 implementations since the late 1980s have interpreted the loss of a 1006 segment as evidence that the path between two endpoints may have 1007 become congested enough to exhaust buffers on intermediate hops, so 1008 that the TCP sender should "back off" - reduce its sending rate until 1009 it knows that its segments are now being delivered without loss 1010 [RFC5681]. More modern TCP stacks have added a growing array of 1011 strategies about how to establish the sending rate [RFC5681], but 1012 when a path is no longer operational, TCP would continue to retry 1013 transmissions, which would fail, again, and double their 1014 Retransmission Time Out (RTO) timers with each failed transmission, 1015 with the result that TCP would wait many seconds before retrying a 1016 segment, even if the path becomes operational while the sender is 1017 waiting for its next retry. 1019 The thinking behind TRIGTRAN was that if a path completely stopped 1020 working because a link along the path was "down", somehow something 1021 along the path could signal TCP when that link returned to service, 1022 and the sending TCP could retry immediately, without waiting for a 1023 full retransmission timeout (RTO) period. 1025 6.5.1. Reasons for Non-deployment 1027 The early dreams for TRIGTRAN were dashed because of an assumption 1028 that TRIGTRAN triggers would be unauthenticated. This meant that any 1029 "safe" TRIGTRAN mechanism would have relied on a mechanism such as 1030 setting the IPv4 TTL or IPv6 Hop Count to 255 at a sender and testing 1031 that it was 254 upon receipt, so that a receiver could verify that a 1032 signal was generated by an adjacent sender known to be on the path 1033 being used, and not some unknown sender which might not even be on 1034 the path (e.g., "The Generalized TTL Security Mechanism (GTSM)" 1035 [RFC5082]). This situation is very similar to the case for ICMP 1036 Source Quench messages as described in Section 6.4, which were also 1037 unauthenticated, and could be sent by an off-path attacker, resulting 1038 in deprecation of ICMP Source Quench message processing [RFC6633]. 1040 TRIGTRAN's scope shrunk from "the path is down" to "the first-hop 1041 link is down". 1043 But things got worse. 1045 Because TRIGTRAN triggers would only be provided when the first-hop 1046 link was "down", TRIGTRAN triggers couldn't replace normal TCP 1047 retransmission behavior if the path failed because some link further 1048 along the network path was "down". So TRIGTRAN triggers added 1049 complexity to an already complex TCP state machine, and did not allow 1050 any existing complexity to be removed. 1052 There was also an issue that the TRIGTRAN signal was not sent in 1053 response to a specific host that had been sending packets, and was 1054 instead a signal that stimulated a response by any sender on the 1055 link. This needs to scale when there are multiple flows trying to 1056 use the same resource, yet the sender of a trigger has no 1057 understanding how many of the potential traffic sources will respond 1058 by sending packets - if recipients of the signal back-off their 1059 responses to a trigger to improve scaling, then that immediately 1060 mitigates the benefit of the signal. 1062 Finally, intermediate forwarding nodes required modification to 1063 provide TRIGTRAN triggers, but operators couldn't charge for TRIGTRAN 1064 triggers, so there was no way to recover the cost of modifying, 1065 testing, and deploying updated intermediate nodes. 1067 Two TRIGTRAN BOFs were held, at IETF 55 [TRIGTRAN-55] and IETF 56 1068 [TRIGTRAN-56], but this work was not chartered, and there was no 1069 interest in deploying TRIGTRAN unless it was chartered and 1070 standardized in the IETF. 1072 6.5.2. Lessons Learned. 1074 The reasons why this work was not chartered, much less deployed, 1075 provide several useful lessons for researchers. 1077 * TRIGTRAN started with a plausible value proposition, but 1078 networking realities in the early 2000s forced reductions in scope 1079 that led directly to reductions in potential benefits, but no 1080 corresponding reductions in costs and complexity. 1082 * These reductions in scope were the direct result of an inability 1083 for hosts to trust or authenticate TRIGTRAN signals they received 1084 from the network. 1086 * Operators did not believe they could charge for TRIGTRAN 1087 signaling, because first-hop links didn't fail frequently, and 1088 TRIGTRAN provided no reduction in operating expenses, so there was 1089 little incentive to purchase and deploy TRIGTRAN-capable network 1090 equipment. 1092 It is also worth noting that the targeted environment for TRIGTRAN in 1093 the late 1990s contained links with a relatively small number of 1094 directly-connected hosts - for instance, cellular or satellite links. 1095 The transport community was well aware of the dangers of sender 1096 synchronization based on multiple senders receiving the same stimulus 1097 at the same time, but the working assumption for TRIGTRAN was that 1098 there wouldn't be enough senders for this to be a meaningful problem. 1099 In the 2010s, it is common for a single "link" to support many 1100 senders and receivers on a single link, likely requiring TRIGTRAN 1101 senders to wait some random amount of time before sending after 1102 receiving a TRIGTRAN signal, which would have reduced the benefits of 1103 TRIGTRAN even more. 1105 6.6. Shim6 1107 The suggested references for Shim6 are: 1109 * Shim6: Level 3 Multihoming Shim Protocol for IPv6 [RFC5533] 1111 The IPv6 routing architecture [RFC1887] assumed that most sites on 1112 the Internet would be identified by Provider Assigned IPv6 prefixes, 1113 so that Default-Free Zone routers only contained routes to other 1114 providers, resulting in a very small IPv6 global routing table. 1116 For a single-homed site, this could work well. A multihomed site 1117 with only one upstream provider could also work well, although BGP 1118 multihoming from a single upstream provider was often a premium 1119 service (costing more than twice as much as two single-homed sites), 1120 and if the single upstream provider went out of service, all of the 1121 multihomed paths could fail simultaneously. 1123 IPv4 sites often multihomed by obtaining Provider Independent 1124 prefixes, and advertising these prefixes through multiple upstream 1125 providers. With the assumption that any multihomed IPv4 site would 1126 also multihome in IPv6, it seemed likely that IPv6 routing would be 1127 subject to the same pressures to announce Provider Independent 1128 prefixes, resulting in a global IPv6 routing table that exhibited the 1129 same explosive growth as the global IPv4 routing table. During the 1130 early 2000s, work began on a protocol that would provide multihoming 1131 for IPv6 sites without requiring sites to advertise Provider 1132 Independent prefixes into the IPv6 global routing table. 1134 This protocol, called Shim6, allowed two endpoints to exchange 1135 multiple addresses ("Locators") that all mapped to the same endpoint 1136 ("Identity"). After an endpoint learned multiple Locators for the 1137 other endpoint, it could send to any of those Locators with the 1138 expectation that those packets would all be delivered to the endpoint 1139 with the same Identity. Shim6 was an example of an "Identity/Locator 1140 Split" protocol. 1142 Shim6, as defined in [RFC5533] and related RFCs, provided a workable 1143 solution for IPv6 multihoming using Provider Assigned prefixes, 1144 including capability discovery and negotiation, and allowing end-to- 1145 end application communication to continue even in the face of path 1146 failure, because applications don't see Locator failures, and 1147 continue to communicate with the same Identity using a different 1148 Locator. 1150 6.6.1. Reasons for Non-deployment 1152 Note that the problem being addressed was "site multihoming", but 1153 Shim6 was providing "host multihoming". That meant that the decision 1154 about what path would be used was under host control, not under edge 1155 router control. 1157 Although more work could have been done to provide a better technical 1158 solution, the biggest impediments to Shim6 deployment were 1159 operational and business considerations. These impediments were 1160 discussed at multiple network operator group meetings, including 1161 [Shim6-35] at [NANOG-35]. 1163 The technical issues centered around concerns that Shim6 relied on 1164 the host to track all the connections, while also tracking Identity/ 1165 Locator mappings in the kernel, and tracking failures to recognize 1166 that an available path has failed. 1168 The operational issues centered around concerns that operators were 1169 performing traffic engineering on traffic aggregates. With Shim6, 1170 these operator traffic engineering policies must be pushed down to 1171 individual hosts. 1173 In addition, operators would have no visibility or control over the 1174 decision of hosts choosing to switch to another path. They expressed 1175 concerns that relying on hosts to steer traffic exposed operator 1176 networks to oscillation based on feedback loops, if hosts moved from 1177 path to path frequently. Given that Shim6 was intended to support 1178 multihoming across operators, operators providing only one of the 1179 paths would have even less visibility as traffic suddenly appeared 1180 and disappeared on their networks. 1182 In addition, firewalls that expected to find a TCP or UDP transport- 1183 level protocol header in the IP payload would see a Shim6 Identity 1184 header instead, and would not perform transport-protocol-based 1185 firewalling functions because the firewall's normal processing logic 1186 would not look past the Identity header. 1188 The business issues centered on reducing or removing the ability to 1189 sell BGP multihoming service to their own customers, which is often 1190 more expensive than two single-homed connectivity services. 1192 6.6.2. Lessons Learned 1194 It is extremely important to take operational concerns into account 1195 when a path-aware protocol is making decisions about path selection 1196 that may conflict with existing operational practices and business 1197 considerations. 1199 6.6.3. Addendum on MultiPath TCP 1201 During discussions in the PANRG session at IETF 103 [PANRG-103-Min], 1202 Lars Eggert, past Transport Area Director, pointed out that during 1203 charter discussions for the Multipath TCP working group [MP-TCP], 1204 operators expressed concerns that customers could use Multipath TCP 1205 to loadshare TCP connections across operators simultaneously and 1206 compare passive performance measurements across network paths in real 1207 time, changing the balance of power in those business relationships. 1208 Although the Multipath TCP working group was chartered, this concern 1209 could have acted as an obstacle to deployment. 1211 Operator objections to Shim6 were focused on technical concerns, but 1212 this concern could have also been an obstacle to Shim6 deployment if 1213 the technical concerns had been overcome. 1215 6.7. Next Steps in Signaling (NSIS) 1217 The suggested references for Next Steps in Signaling (NSIS) are: 1219 * the concluded working group charter [NSIS-CHARTER-2001] 1221 * GIST: General Internet Signalling Transport [RFC5971] 1223 * NAT/Firewall NSIS Signaling Layer Protocol (NSLP) [RFC5973] 1225 * NSIS Signaling Layer Protocol (NSLP) for Quality-of-Service 1226 Signaling [RFC5974] 1228 * Authorization for NSIS Signaling Layer Protocols [RFC5981] 1230 The NSIS Working Group worked on signaling techniques for network 1231 layer resources (e.g., QoS resource reservations, Firewall and NAT 1232 traversal). 1234 When RSVP [RFC2205] was used in deployments, a number of questions 1235 came up about its perceived limitations and potential missing 1236 features. The issues noted in the NSIS Working Group charter 1237 [NSIS-CHARTER-2001] include interworking between domains with 1238 different QoS architectures, mobility and roaming for IP interfaces, 1239 and complexity. Later, the lack of security in RSVP was also 1240 recognized ([RFC4094]). 1242 The NSIS Working Group was chartered to tackle those issues and 1243 initially focused on QoS signaling as its primary use case. However, 1244 over time a new approach evolved that introduced a modular 1245 architecture using application-specific signaling protocols (the NSIS 1246 Signaling Layer Protocol (NSLP)) on top of a generic signaling 1247 transport protocol (the NSIS Transport Layer Protocol (NTLP)). 1249 The NTLP is defined in [RFC5971]. Two NSLPs are defined: the NSIS 1250 Signaling Layer Protocol (NSLP) for Quality-of-Service Signaling 1251 [RFC5974] as well as the NAT/Firewall NSIS Signaling Layer Protocol 1252 (NSLP) [RFC5973]. 1254 6.7.1. Reasons for Non-deployment 1256 The obstacles for deployment can be grouped into implementation- 1257 related aspects and operational aspects. 1259 * Implementation-related aspects: 1261 Although NSIS provides benefits with respect to flexibility, 1262 mobility, and security compared to other network signaling 1263 techniques, hardware vendors were reluctant to deploy this solution, 1264 because it would require additional implementation effort and would 1265 result in additional complexity for router implementations. 1267 The NTLP mainly operates as path-coupled signaling protocol, i.e, its 1268 messages are processed at the intermediate node's control plane that 1269 are also forwarding the data flows. This requires a mechanism to 1270 intercept signaling packets while they are forwarded in the same 1271 manner (especially along the same path) as data packets. NSIS uses 1272 the IPv4 and IPv6 Router Alert Option (RAO) to allow for interception 1273 of those path-coupled signaling messages, and this technique requires 1274 router implementations to correctly understand and implement the 1275 handling of RAOs, e.g., to only process packet with RAOs of interest 1276 and to leave packets with irrelevant RAOs in the fast forwarding 1277 processing path (a comprehensive discussion of these issues can be 1278 found in [RFC6398]). The latter was an issue with some router 1279 implementations at the time of standardization. 1281 Another reason is that path-coupled signaling protocols that interact 1282 with routers and request manipulation of state at these routers (or 1283 any other network element in general) are under scrutiny: a packet 1284 (or sequence of packets) out of the mainly untrusted data path is 1285 requesting creation and manipulation of network state. This is seen 1286 as potentially dangerous (e.g., opens up a Denial of Service (DoS) 1287 threat to a router's control plane) and difficult for an operator to 1288 control. Path-coupled signaling approaches were considered 1289 problematic (see also section 3 of [RFC6398]). There are 1290 recommendations on how to secure NSIS nodes and deployments (e.g., 1291 [RFC5981]). 1293 * Operational Aspects: 1295 NSIS not only required trust between customers and their provider, 1296 but also among different providers. Especially, QoS signaling 1297 techniques would require some kind of dynamic service level agreement 1298 support that would imply (potentially quite complex) bilateral 1299 negotiations between different Internet service providers. This 1300 complexity was not considered to be justified and increasing the 1301 bandwidth (and thus avoiding bottlenecks) was cheaper than actively 1302 managing network resource bottlenecks by using path-coupled QoS 1303 signaling techniques. Furthermore, an end-to-end path typically 1304 involves several provider domains and these providers need to closely 1305 cooperate in cases of failures. 1307 6.7.2. Lessons Learned 1309 One goal of NSIS was to decrease the complexity of the signaling 1310 protocol, but a path-coupled signaling protocol comes with the 1311 intrinsic complexity of IP-based networks, beyond the complexity of 1312 the signaling protocol itself. Sources of intrinsic complexity 1313 include: 1315 * the presence of asymmetric routes between endpoints and routers 1317 * the lack of security and trust at large in the Internet 1318 infrastructure 1320 * the presence of different trust boundaries 1322 * the effects of best-effort networks (e.g., robustness to packet 1323 loss) 1325 * divergence from the fate sharing principle (e.g., state within the 1326 network). 1328 Any path-coupled signaling protocol has to deal with these realities. 1330 Operators view the use of IPv4 and IPv6 Router Alert Option (RAO) to 1331 signal routers along the path from end systems with suspicion, 1332 because these end systems are usually not authenticated and heavy use 1333 of RAOs can easily increase the CPU load on routers that are designed 1334 to process most packets using a hardware "fast path" and diverting 1335 packets containing RAO to a slower, more capable processor. 1337 6.8. IPv6 Flow Label 1339 The suggested references for IPv6 Flow Label are: 1341 * IPv6 Flow Label Specification [RFC6437] 1343 IPv6 specifies a 20-bit field Flow Label field [RFC6437], included in 1344 the fixed part of the IPv6 header and hence present in every IPv6 1345 packet. An endpoint sets the value in this field to one of a set of 1346 pseudo-randomly assigned values. If a packet is not part of any 1347 flow, the flow label value is set to zero [RFC3697]. A number of 1348 Standards Track and Best Current Practice RFCs (e.g., [RFC8085], 1349 [RFC6437], [RFC6438]) encourage IPv6 endpoints to set a non-zero 1350 value in this field. A multiplexing transport could choose to use 1351 multiple flow labels to allow the network to independently forward 1352 its subflows, or to use one common value for the traffic aggregate. 1353 The flow label is present in all fragments. IPsec was originally put 1354 forward as one important use-case for this mechanism and does encrypt 1355 the field [RFC6438]. 1357 Once set, the flow label can provide information that can help inform 1358 network nodes about subflows present at the transport layer, without 1359 needing to interpret the setting of upper layer protocol fields 1360 [RFC6294]. This information can also be used to coordinate how 1361 aggregates of transport subflows are grouped when queued in the 1362 network and to select appropriate per-flow forwarding when choosing 1363 between alternate paths [RFC6438] (e.g. for Equal Cost Multipath 1364 Routing (ECMP) and Link Aggregation (LAG)). 1366 6.8.1. Reasons for Non-deployment 1368 Despite the field being present in every IPv6 packet, the mechanism 1369 did not receive as much use as originally envisioned. One reason is 1370 that to be useful it requires engagement by two different 1371 stakeholders: 1373 * Endpoint Implementation: 1375 For network nodes along a path to utilize the flow label there needs 1376 to be a non-zero value inserted in the field [RFC6437] at the sending 1377 endpoint. There needs to be an incentive for an endpoint to set an 1378 appropriate non-zero value. The value should appropriately reflect 1379 the level of aggregation the traffic expects to be provided by the 1380 network. However, this requires the stack to know granularity at 1381 which flows should be identified (or conversely which flows should 1382 receive aggregated treatment), i.e., which packets carry the same 1383 flow label. Therefore, setting a non-zero value may result in 1384 additional choices that need to be made by an application developer. 1386 Although the standard [RFC3697] forbids any encoding of meaning into 1387 the flow label value, the opportunity to use the flow label as a 1388 covert channel or to signal other meta-information may have raised 1389 concerns about setting a non-zero value [RFC6437]. 1391 Before methods are widely deployed to use this method, there could be 1392 no incentive for an endpoint to set the field. 1394 * Operational support in network nodes: 1396 A benefit can only be realized when a network node along the path 1397 also uses this information to inform its decisions. Network 1398 equipment (routers and/or middleboxes) need to include appropriate 1399 support so they can utilize the field when making decisions about how 1400 to classify flows, or to inform forwarding choices. Use of any 1401 optional feature in a network node also requires corresponding 1402 updates to operational procedures, and therefore is normally only 1403 introduced when the cost can be justified. 1405 A benefit from utilizing the flow label is expected to be increased 1406 quality of experience for applications - but this comes at some 1407 operational cost to an operator, and requires endpoints to set the 1408 field. 1410 6.8.2. Lessons Learned 1412 The flow label is a general purpose header field for use by the path. 1413 Multiple uses have been proposed. One candidate use was to reduce 1414 the complexity of forwarding decisions. However, modern routers can 1415 use a "fast path", often taking advantage of hardware to accelerate 1416 processing. The method can assist in more complex forwarding, such 1417 as ECMP and load balancing. 1419 Although [RFC6437] recommended that endpoints should by default 1420 choose uniformly-distributed labels for their traffic, the 1421 specification permitted an endpoint to choose to set a zero value. 1422 This ability of endpoints to choose to set a flow label of zero has 1423 had consequences on deployability: 1425 * Before wide-scale support by endpoints, it would be impossible to 1426 rely on a non-zero flow label being set. Network nodes therefore 1427 would need to also employ other techniques to realize equivalent 1428 functions. An example of a method is one assuming semantics of 1429 the source port field to provide entropy input to a network-layer 1430 hash. This use of a 5-tuple to classify a packet represents a 1431 layering violation [RFC6294]. When other methods have been 1432 deployed, they increase the cost of deploying standards-based 1433 methods, even though they may offer less control to endpoints and 1434 result in potential interaction with other uses/interpretation of 1435 the field. 1437 * Even though the flow label is specified as an end-to-end field, 1438 some network paths have been observed to not transparently forward 1439 the flow label. This could result from non-conformant equipment, 1440 or could indicate that some operational networks have chosen to 1441 re-use the protocol field for other (e.g. internal purposes). 1442 This results in lack of transparency, and a deployment hurdle to 1443 endpoints expecting that they can set a flow label that is 1444 utilized by the network. The more recent practice of "greasing" 1445 [GREASE] would suggest that a different outcome could have been 1446 achieved if endpoints were always required to set a non-zero 1447 value. 1449 * [RFC1809] noted that setting the choice of the flow label value 1450 can depend on the expectations of the traffic generated by an 1451 application, which suggests an API should be presented to control 1452 the setting or policy that is used. However, many currently 1453 available APIs do not have this support. 1455 A growth in the use of encrypted transports, (e.g. QUIC [QUIC-WG]) 1456 seems likely to raise similar issues to those discussed above and 1457 could motivate renewed interest in utilizing the flow label. 1459 6.9. Explicit Congestion Notification (ECN) 1461 The suggested references for Explicit Congestion Notification (ECN) 1462 are: 1464 * Recommendations on Queue Management and Congestion Avoidance in 1465 the Internet [RFC2309] 1467 * A Proposal to add Explicit Congestion Notification (ECN) to IP 1468 [RFC2481] 1470 * The Addition of Explicit Congestion Notification (ECN) to IP 1471 [RFC3168] 1473 * Implementation Report on Experiences with Various TCP RFCs 1474 [vista-impl], slides 6 and 7 1476 * Implementation and Deployment of ECN [SallyFloyd] 1478 In the early 1990s, the large majority of Internet traffic used TCP 1479 as its transport protocol, but TCP had no way to detect path 1480 congestion before the path was so congested that packets were being 1481 dropped, and these congestion events could affect all senders using a 1482 path, either by "lockout", where long-lived flows monopolized the 1483 queues along a path, or by "full queues", where queues remain full, 1484 or almost full, for a long period of time. 1486 In response to this situation, "Active Queue Management" (AQM) was 1487 deployed in the network. A number of AQM disciplines have been 1488 deployed, but one common approach was that routers dropped packets 1489 when a threshold buffer length was reached, so that transport 1490 protocols like TCP that were responsive to loss would detect this 1491 loss and reduce their sending rates. Random Early Detection (RED) 1492 was one such proposal in the IETF. As the name suggests, a router 1493 using RED as its AQM discipline that detected time-averaged queue 1494 lengths passing a threshold would choose incoming packets 1495 probabilistically to be dropped [RFC2309]. In response to this 1496 situation, "Active Queue Management" (AQM) was deployed in the 1497 network. A number of AQM disciplines have been deployed, but one 1498 common approach was that routers dropped packets when a threshold 1499 buffer length was reached, so that transport protocols like TCP that 1500 were responsive to loss would detect this loss and reduce their 1501 sending rates. Random Early Detection (RED) was one such proposal in 1502 the IETF. As the name suggests, a router using RED as its AQM 1503 discipline that detected time-averaged queue lengths passing a 1504 threshold would choose incoming packets probabilistically to be 1505 dropped [RFC2309]. 1507 Researchers suggested that providing "explicit congestion 1508 notifications" to senders when routers along the path detected their 1509 queues were building, so that some senders would "slow down" as if a 1510 loss had occurred, so that the path queues had time to drain, and the 1511 path still had sufficient buffer capacity to accommodate bursty 1512 arrivals of packets from other senders. This was proposed as an 1513 Experiment in [RFC2481], and standardized in [RFC3168]. 1515 A key aspect of ECN was the use of IP header fields rather than IP 1516 options to carry explicit congestion notifications, since the 1517 proponents recognized that 1519 Many routers process the "regular" headers in IP packets more 1520 efficiently than they process the header information in IP 1521 options. 1523 Unlike most of the Path Aware technologies included in this document, 1524 the story of ECN continues to the present day, and encountered a 1525 large number of Lessons Learned during that time. The early history 1526 of ECN (non-)deployment provides Lessons Learned that were not 1527 captured by other contributions in Section 6, so that is the emphasis 1528 in this section of the document. 1530 6.9.1. Reasons for Non-deployment 1532 There are at least three sub-stories - ECN deployment in clients, ECN 1533 deployment in routers, and AQM deployment in operational networks. 1534 All three sub-stories mattered. 1536 The proponents of ECN did so much right, anticipating many of the 1537 Lessons Learned now recognized in Section 4. They recognized the 1538 need to support incremental deployment (Section 4.2). They 1539 considered the impact on router throughput (Section 4.8). They even 1540 considered trust issues between end nodes and the network, both for 1541 non-compliant end nodes (Section 4.10) and non-compliant routers 1542 (Section 4.9). 1544 They were rewarded with ECN being implemented in major operating 1545 systems, both for end nodes and for routers. A number of 1546 implementations are listed under "Implementation and Deployment of 1547 ECN" at [SallyFloyd]. 1549 What they did not anticipate, was routers that would crash, when they 1550 saw bits 6 and 7 in the IPv4 TOS octet [RFC0791]/IPv6 Traffic Class 1551 field [RFC2460], which [RFC2481] redefined to be "currently unused", 1552 being set to a non-zero value. 1554 As described in [vista-impl], 1556 Intermediate Gateway Device problem #1: one of the most popular 1557 versions from one of the most popular vendors. When a data packet 1558 arrives with either ECT(0) or ECT(1) (indicating successful ECN 1559 capability negotiation) indicated, router crashed. Cannot be 1560 recovered at TCP layer (sic) 1562 This implementation, which would be run on a significant percentage 1563 of Internet end nodes, was shipped with ECN disabled, as was true for 1564 several of the other implementations listed under "Implementation and 1565 Deployment of ECN" at [SallyFloyd]. Even if subsequent router 1566 vendors fixed these implementations, ECN was still disabled on end 1567 nodes, and given the tradeoff between the benefits of enabling ECN 1568 (somewhat better behavior during congestion) and the risks of 1569 enabling ECN (possibly crashing a router somewhere along the path), 1570 ECN tended to stay disabled on implementations that supported ECN for 1571 decades afterwards. 1573 6.9.2. Lessons Learned 1575 Of the contributions included in Section 6, ECN may be unique in 1576 providing these lessons: 1578 * Even if you do everything right, you may trip over implementation 1579 bugs in devices you know nothing about, that will cause severe 1580 problems that prevent successful deployment of your path aware 1581 technology. 1583 * After implementations disable your Path Aware technology, it may 1584 take years, or even decades, to convince implementers to re-enable 1585 it by default. 1587 These two lessons, taken together, could be summarized as "you get 1588 one chance to get it right". 1590 During discussion of ECN at [PANRG-110], we noted that "you get one 1591 chance to get it right" isn't quite correct today, because operating 1592 systems on so many host systems are frequently updated, and transport 1593 protocols like QUIC [I-D.ietf-quic-transport] are being implemented 1594 in user space, and can be updated without touching installed 1595 operating systems. Neither of these factors were true in the early 1596 2000s. 1598 We think that these restatements of the ECN Lessons Learned are more 1599 useful for current implementers: 1601 * Even if you do everything right, you may trip over implementation 1602 bugs in devices you know nothing about, that will cause severe 1603 problems that prevent successful deployment of your path aware 1604 technology. Testing before deployment isn't enough to ensure 1605 successful deployment. It is also necessary to "deploy gently", 1606 which often means deploying for a small subset of users to gain 1607 experience, and implementing feedback mechanisms to detect that 1608 user experience is being degraded. 1610 * After implementations disable your Path Aware technology, it may 1611 take years, or even decades, to convince implementers to re-enable 1612 it by default. This might be based on the difficulty of 1613 distributing implementations that enable it by default, but are 1614 just as likely to be based on the "bad taste in the mouth" that 1615 implementers have after an unsuccessful deployment attempt that 1616 degraded user experience. 1618 With these expansions, the two lessons, taken together, could be more 1619 helpfully summarized as "plan for failure" - anticipate what your 1620 next step will be, if initial deployment is unsuccessful. 1622 ECN deployment was also hindered by non-deployment of AQM in many 1623 devices, because of operator interest in QoS features provided in the 1624 network, rather than using the network to assist end systems in 1625 providing for themselves. But that's another story, and the AQM 1626 Lessons Learned are already covered in other contributions in 1627 Section 6. 1629 7. Security Considerations 1631 This document describes Path Aware techniques that were not adopted 1632 and widely deployed on the Internet, so it doesn't affect the 1633 security of the Internet. 1635 If this document meets its goals, we may develop new techniques for 1636 Path Aware Networking that would affect the security of the Internet, 1637 but security considerations for those techniques will be described in 1638 the corresponding RFCs that specify them. 1640 8. IANA Considerations 1642 This document makes no requests of IANA. 1644 9. Acknowledgments 1646 Initial material for Section 6.1 on ST2 was provided by Gorry 1647 Fairhurst. 1649 Initial material for Section 6.2 on IntServ was provided by Ron 1650 Bonica. 1652 Initial material for Section 6.3 on Quick-Start TCP was provided by 1653 Michael Scharf, who also provided suggestions to improve this section 1654 after it was edited. 1656 Initial material for Section 6.4 on ICMP Source Quench was provided 1657 by Gorry Fairhurst. 1659 Initial material for Section 6.5 on Triggers for Transport (TRIGTRAN) 1660 was provided by Spencer Dawkins. 1662 Section 6.6 on Shim6 builds on initial material describing obstacles 1663 provided by Erik Nordmark, with background added by Spencer Dawkins. 1665 Initial material for Section 6.7 on Next Steps In Signaling (NSIS) 1666 was provided by Roland Bless and Martin Stiemerling. 1668 Initial material for Section 6.8 on IPv6 Flow Labels was provided by 1669 Gorry Fairhurst. 1671 Initial material for Section 6.9 on Explicit Congestion Notification 1672 was provided by Spencer Dawkins. 1674 Our thanks to Adrian Farrel, Bob Briscoe, C.M. Heard, David Black, 1675 Eric Kinnear, Erik Auerswald, Gorry Fairhurst, Jake Holland, Joe 1676 Touch, Joeri de Ruiter, Kireeti Kompella, Mohamed Boucadair, Roland 1677 Bless, Ruediger Geib, Theresa Enghardt, and Wes Eddy, who provided 1678 review comments on this document as a "work in process". 1680 Mallory Knodel reviewed this document for the Internet Research 1681 Steering Group, and provided many helpful suggestions. 1683 David Oran also provided helpful comments and text suggestions on 1684 this document during Internet Research Steering Group balloting. In 1685 particular, Section 5 reflects his review. 1687 Benjamin Kaduk and Rob Wilton provided helpful comments during 1688 Internet Engineering Steering Group conflict review. 1690 Special thanks to Adrian Farrel for helping Spencer navigate the 1691 twisty little passages of Flow Specs and Filter Specs in IntServ, 1692 RSVP, MPLS, and BGP. They are all alike, except when they are 1693 different [Colossal-Cave]. 1695 10. Informative References 1697 [Colossal-Cave] 1698 "Wikipedia Page for Colossal Cave Adventure", January 1699 2019, 1700 . 1702 [Conviva] "Conviva Precision : Data Sheet", December 2020, 1703 . 1706 [GREASE] Thomson, M., "Long-term Viability of Protocol Extension 1707 Mechanisms", July 2019, . 1710 [I-D.arkko-arch-internet-threat-model] 1711 Arkko, J., "Changes in the Internet Threat Model", Work in 1712 Progress, Internet-Draft, draft-arkko-arch-internet- 1713 threat-model-01, 8 July 2019, . 1717 [I-D.farrell-etm] 1718 Farrell, S., "We're gonna need a bigger threat model", 1719 Work in Progress, Internet-Draft, draft-farrell-etm-03, 6 1720 July 2019, . 1723 [I-D.ietf-quic-transport] 1724 Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed 1725 and Secure Transport", Work in Progress, Internet-Draft, 1726 draft-ietf-quic-transport-34, 14 January 2021, 1727 . 1730 [I-D.ietf-tsvwg-intserv-multiple-tspec] 1731 Polk, J. and S. Dhesikan, "Integrated Services (IntServ) 1732 Extension to Allow Signaling of Multiple Traffic 1733 Specifications and Multiple Flow Specifications in 1734 RSVPv1", Work in Progress, Internet-Draft, draft-ietf- 1735 tsvwg-intserv-multiple-tspec-02, 25 February 2013, 1736 . 1739 [I-D.irtf-panrg-path-properties] 1740 Enghardt, T. and C. Krahenbuhl, "A Vocabulary of Path 1741 Properties", Work in Progress, Internet-Draft, draft-irtf- 1742 panrg-path-properties-01, 7 September 2020, 1743 . 1746 [I-D.irtf-panrg-questions] 1747 Trammell, B., "Current Open Questions in Path Aware 1748 Networking", Work in Progress, Internet-Draft, draft-irtf- 1749 panrg-questions-08, 23 December 2020, 1750 . 1753 [IEN-119] Forgie, J., "ST - A Proposed Internet Stream Protocol", 1754 September 1979, 1755 . 1757 [ITAT] "IAB Workshop on Internet Technology Adoption and 1758 Transition (ITAT)", December 2013, 1759 . 1761 [model-t] "Model-t -- Discussions of changes in Internet deployment 1762 patterns and their impact on the Internet threat model", 1763 n.d., . 1765 [MOPS-109-Min] 1766 "Media Operations Working Group - IETF-109 Minutes", 1767 November 2020, 1768 . 1771 [MP-TCP] "Multipath TCP Working Group Home Page", n.d., 1772 . 1774 [NANOG-35] "North American Network Operators Group NANOG-35 Agenda", 1775 October 2005, 1776 . 1778 [NSIS-CHARTER-2001] 1779 "Next Steps In Signaling Working Group Charter", March 1780 2011, 1781 . 1783 [PANRG] "Path Aware Networking Research Group (Home Page)", n.d., 1784 . 1786 [PANRG-103-Min] 1787 "Path Aware Networking Research Group - IETF-103 Minutes", 1788 November 2018, 1789 . 1791 [PANRG-105-Min] 1792 "Path Aware Networking Research Group - IETF-105 Minutes", 1793 July 2019, 1794 . 1796 [PANRG-106-Min] 1797 "Path Aware Networking Research Group - IETF-106 Minutes", 1798 November 2019, 1799 . 1801 [PANRG-110] 1802 "Path Aware Networking Research Group - IETF-110", July 1803 2017, 1804 . 1806 [PANRG-99] "Path Aware Networking Research Group - IETF-99", July 1807 2017, 1808 . 1810 [PATH-Decade] 1811 Bonaventure, O., "A Decade of Path Awareness", July 2017, 1812 . 1815 [QS-SAT] Secchi, R., Sathiaseelan, A., Potorti, F., Gotta, A., and 1816 G. Fairhurst, "Using Quick-Start to enhance TCP-friendly 1817 rate control performance in bidirectional satellite 1818 networks", 2009, 1819 . 1821 [QUIC-WG] "QUIC Working Group Home Page", n.d., 1822 . 1824 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1825 DOI 10.17487/RFC0791, September 1981, 1826 . 1828 [RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, 1829 RFC 792, DOI 10.17487/RFC0792, September 1981, 1830 . 1832 [RFC0793] Postel, J., "Transmission Control Protocol", STD 7, 1833 RFC 793, DOI 10.17487/RFC0793, September 1981, 1834 . 1836 [RFC1016] Prue, W. and J. Postel, "Something a Host Could Do with 1837 Source Quench: The Source Quench Introduced Delay 1838 (SQuID)", RFC 1016, DOI 10.17487/RFC1016, July 1987, 1839 . 1841 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1842 Communication Layers", STD 3, RFC 1122, 1843 DOI 10.17487/RFC1122, October 1989, 1844 . 1846 [RFC1190] Topolcic, C., "Experimental Internet Stream Protocol: 1847 Version 2 (ST-II)", RFC 1190, DOI 10.17487/RFC1190, 1848 October 1990, . 1850 [RFC1633] Braden, R., Clark, D., and S. Shenker, "Integrated 1851 Services in the Internet Architecture: an Overview", 1852 RFC 1633, DOI 10.17487/RFC1633, June 1994, 1853 . 1855 [RFC1809] Partridge, C., "Using the Flow Label Field in IPv6", 1856 RFC 1809, DOI 10.17487/RFC1809, June 1995, 1857 . 1859 [RFC1819] Delgrossi, L., Ed. and L. Berger, Ed., "Internet Stream 1860 Protocol Version 2 (ST2) Protocol Specification - Version 1861 ST2+", RFC 1819, DOI 10.17487/RFC1819, August 1995, 1862 . 1864 [RFC1887] Rekhter, Y., Ed. and T. Li, Ed., "An Architecture for IPv6 1865 Unicast Address Allocation", RFC 1887, 1866 DOI 10.17487/RFC1887, December 1995, 1867 . 1869 [RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast 1870 Retransmit, and Fast Recovery Algorithms", RFC 2001, 1871 DOI 10.17487/RFC2001, January 1997, 1872 . 1874 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1875 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1876 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1877 September 1997, . 1879 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated 1880 Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, 1881 . 1883 [RFC2211] Wroclawski, J., "Specification of the Controlled-Load 1884 Network Element Service", RFC 2211, DOI 10.17487/RFC2211, 1885 September 1997, . 1887 [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification 1888 of Guaranteed Quality of Service", RFC 2212, 1889 DOI 10.17487/RFC2212, September 1997, 1890 . 1892 [RFC2215] Shenker, S. and J. Wroclawski, "General Characterization 1893 Parameters for Integrated Service Network Elements", 1894 RFC 2215, DOI 10.17487/RFC2215, September 1997, 1895 . 1897 [RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, 1898 S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., 1899 Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, 1900 S., Wroclawski, J., and L. Zhang, "Recommendations on 1901 Queue Management and Congestion Avoidance in the 1902 Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, 1903 . 1905 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1906 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 1907 December 1998, . 1909 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1910 and W. Weiss, "An Architecture for Differentiated 1911 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1912 . 1914 [RFC2481] Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit 1915 Congestion Notification (ECN) to IP", RFC 2481, 1916 DOI 10.17487/RFC2481, January 1999, 1917 . 1919 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1920 of Explicit Congestion Notification (ECN) to IP", 1921 RFC 3168, DOI 10.17487/RFC3168, September 2001, 1922 . 1924 [RFC3697] Rajahalme, J., Conta, A., Carpenter, B., and S. Deering, 1925 "IPv6 Flow Label Specification", RFC 3697, 1926 DOI 10.17487/RFC3697, March 2004, 1927 . 1929 [RFC4094] Manner, J. and X. Fu, "Analysis of Existing Quality-of- 1930 Service Signaling Protocols", RFC 4094, 1931 DOI 10.17487/RFC4094, May 2005, 1932 . 1934 [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet 1935 Control Message Protocol (ICMPv6) for the Internet 1936 Protocol Version 6 (IPv6) Specification", STD 89, 1937 RFC 4443, DOI 10.17487/RFC4443, March 2006, 1938 . 1940 [RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick- 1941 Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782, 1942 January 2007, . 1944 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C. 1945 Pignataro, "The Generalized TTL Security Mechanism 1946 (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007, 1947 . 1949 [RFC5218] Thaler, D. and B. Aboba, "What Makes for a Successful 1950 Protocol?", RFC 5218, DOI 10.17487/RFC5218, July 2008, 1951 . 1953 [RFC5533] Nordmark, E. and M. Bagnulo, "Shim6: Level 3 Multihoming 1954 Shim Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, 1955 June 2009, . 1957 [RFC5575] Marques, P., Sheth, N., Raszuk, R., Greene, B., Mauch, J., 1958 and D. McPherson, "Dissemination of Flow Specification 1959 Rules", RFC 5575, DOI 10.17487/RFC5575, August 2009, 1960 . 1962 [RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion 1963 Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, 1964 . 1966 [RFC5971] Schulzrinne, H. and R. Hancock, "GIST: General Internet 1967 Signalling Transport", RFC 5971, DOI 10.17487/RFC5971, 1968 October 2010, . 1970 [RFC5973] Stiemerling, M., Tschofenig, H., Aoun, C., and E. Davies, 1971 "NAT/Firewall NSIS Signaling Layer Protocol (NSLP)", 1972 RFC 5973, DOI 10.17487/RFC5973, October 2010, 1973 . 1975 [RFC5974] Manner, J., Karagiannis, G., and A. McDonald, "NSIS 1976 Signaling Layer Protocol (NSLP) for Quality-of-Service 1977 Signaling", RFC 5974, DOI 10.17487/RFC5974, October 2010, 1978 . 1980 [RFC5981] Manner, J., Stiemerling, M., Tschofenig, H., and R. Bless, 1981 Ed., "Authorization for NSIS Signaling Layer Protocols", 1982 RFC 5981, DOI 10.17487/RFC5981, February 2011, 1983 . 1985 [RFC6294] Hu, Q. and B. Carpenter, "Survey of Proposed Use Cases for 1986 the IPv6 Flow Label", RFC 6294, DOI 10.17487/RFC6294, June 1987 2011, . 1989 [RFC6398] Le Faucheur, F., Ed., "IP Router Alert Considerations and 1990 Usage", BCP 168, RFC 6398, DOI 10.17487/RFC6398, October 1991 2011, . 1993 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 1994 "IPv6 Flow Label Specification", RFC 6437, 1995 DOI 10.17487/RFC6437, November 2011, 1996 . 1998 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 1999 for Equal Cost Multipath Routing and Link Aggregation in 2000 Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, 2001 . 2003 [RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", 2004 RFC 6633, DOI 10.17487/RFC6633, May 2012, 2005 . 2007 [RFC6928] Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis, 2008 "Increasing TCP's Initial Window", RFC 6928, 2009 DOI 10.17487/RFC6928, April 2013, 2010 . 2012 [RFC7305] Lear, E., Ed., "Report from the IAB Workshop on Internet 2013 Technology Adoption and Transition (ITAT)", RFC 7305, 2014 DOI 10.17487/RFC7305, July 2014, 2015 . 2017 [RFC7418] Dawkins, S., Ed., "An IRTF Primer for IETF Participants", 2018 RFC 7418, DOI 10.17487/RFC7418, December 2014, 2019 . 2021 [RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage 2022 Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085, 2023 March 2017, . 2025 [RFC8170] Thaler, D., Ed., "Planning for Protocol Adoption and 2026 Subsequent Transitions", RFC 8170, DOI 10.17487/RFC8170, 2027 May 2017, . 2029 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 2030 "Deterministic Networking Architecture", RFC 8655, 2031 DOI 10.17487/RFC8655, October 2019, 2032 . 2034 [RFC8793] Wissingh, B., Wood, C., Afanasyev, A., Zhang, L., Oran, 2035 D., and C. Tschudin, "Information-Centric Networking 2036 (ICN): Content-Centric Networking (CCNx) and Named Data 2037 Networking (NDN) Terminology", RFC 8793, 2038 DOI 10.17487/RFC8793, June 2020, 2039 . 2041 [SAAG-105-Min] 2042 "Security Area Open Meeting - IETF-105 Minutes", July 2043 2019, . 2046 [SAF07] Sarolahti, P., Allman, M., and S. Floyd, "Determining an 2047 appropriate sending rate over an underutilized network 2048 path", Computer Networking Volume 51, Number 7, May 2007. 2050 [SallyFloyd] 2051 Floyd, S., "ECN (Explicit Congestion Notification) in TCP/ 2052 IP", n.d., . 2054 [Sch11] Scharf, M., "Fast Startup Internet Congestion Control for 2055 Broadband Interactive Applications", Ph.D. Thesis, 2056 University of Stuttgart, April 2011. 2058 [Shim6-35] Meyer, D., Huston, G., Schiller, J., and V. Gill, "IAB 2059 IPv6 Multihoming Panel at NANOG 35", NANOG North American 2060 Network Operator Group, October 2005, 2061 . 2063 [TRIGTRAN-55] 2064 "Triggers for Transport BOF at IETF 55", July 2003, 2065 . 2067 [TRIGTRAN-56] 2068 "Triggers for Transport BOF at IETF 56", November 2003, 2069 . 2071 [vista-impl] 2072 Sridharan, M., Bansal, D., and D. Thaler, "Implementation 2073 Report on Experiences with Various TCP RFCs", November 2074 2003, . 2077 Author's Address 2079 Spencer Dawkins (editor) 2080 Tencent America 2081 United States of America 2083 Email: spencerdawkins.ietf@gmail.com