idnits 2.17.00 (12 Aug 2021) /tmp/idnits15383/draft-ietf-teas-rfc3272bis-16.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document date (24 March 2022) is 51 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-17) exists of draft-ietf-idr-segment-routing-te-policy-16 == Outdated reference: A later version (-19) exists of draft-ietf-lsr-flex-algo-18 == Outdated reference: A later version (-05) exists of draft-ietf-lsr-ip-flexalgo-04 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-21 == Outdated reference: A later version (-10) exists of draft-ietf-teas-ietf-network-slices-08 == Outdated reference: A later version (-09) exists of draft-irtf-nmrg-ibn-concepts-definitions-06 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group A. Farrel, Ed. 3 Internet-Draft Old Dog Consulting 4 Obsoletes: 3272 (if approved) 24 March 2022 5 Intended status: Informational 6 Expires: 25 September 2022 8 Overview and Principles of Internet Traffic Engineering 9 draft-ietf-teas-rfc3272bis-16 11 Abstract 13 This document describes the principles of traffic engineering (TE) in 14 the Internet. The document is intended to promote better 15 understanding of the issues surrounding traffic engineering in IP 16 networks and the networks that support IP networking, and to provide 17 a common basis for the development of traffic engineering 18 capabilities for the Internet. The principles, architectures, and 19 methodologies for performance evaluation and performance optimization 20 of operational networks are also discussed. 22 This work was first published as RFC 3272 in May 2002. This document 23 obsoletes RFC 3272 by making a complete update to bring the text in 24 line with best current practices for Internet traffic engineering and 25 to include references to the latest relevant work in the IETF. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at https://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on 25 September 2022. 44 Copyright Notice 46 Copyright (c) 2022 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 51 license-info) in effect on the date of publication of this document. 52 Please review these documents carefully, as they describe your rights 53 and restrictions with respect to this document. Code Components 54 extracted from this document must include Revised BSD License text as 55 described in Section 4.e of the Trust Legal Provisions and are 56 provided without warranty as described in the Revised BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 61 1.1. What is Internet Traffic Engineering? . . . . . . . . . . 4 62 1.2. Components of Traffic Engineering . . . . . . . . . . . . 6 63 1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8 64 1.4. Terminology . . . . . . . . . . . . . . . . . . . . . . . 8 65 2. Background . . . . . . . . . . . . . . . . . . . . . . . . . 11 66 2.1. Context of Internet Traffic Engineering . . . . . . . . . 11 67 2.2. Network Domain Context . . . . . . . . . . . . . . . . . 12 68 2.3. Problem Context . . . . . . . . . . . . . . . . . . . . . 14 69 2.3.1. Congestion and its Ramifications . . . . . . . . . . 15 70 2.4. Solution Context . . . . . . . . . . . . . . . . . . . . 15 71 2.4.1. Combating the Congestion Problem . . . . . . . . . . 17 72 2.5. Implementation and Operational Context . . . . . . . . . 20 73 3. Traffic Engineering Process Models . . . . . . . . . . . . . 21 74 3.1. Components of the Traffic Engineering Process Model . . . 21 75 4. Taxonomy of Traffic Engineering Systems . . . . . . . . . . . 22 76 4.1. Time-Dependent Versus State-Dependent Versus 77 Event-Dependent . . . . . . . . . . . . . . . . . . . . . 22 78 4.2. Offline Versus Online . . . . . . . . . . . . . . . . . . 24 79 4.3. Centralized Versus Distributed . . . . . . . . . . . . . 24 80 4.3.1. Hybrid Systems . . . . . . . . . . . . . . . . . . . 25 81 4.3.2. Considerations for Software Defined Networking . . . 25 82 4.4. Local Versus Global . . . . . . . . . . . . . . . . . . . 26 83 4.5. Prescriptive Versus Descriptive . . . . . . . . . . . . . 26 84 4.5.1. Intent-Based Networking . . . . . . . . . . . . . . . 27 85 4.6. Open-Loop Versus Closed-Loop . . . . . . . . . . . . . . 27 86 4.7. Tactical versus Strategic . . . . . . . . . . . . . . . . 27 87 5. Review of TE Techniques . . . . . . . . . . . . . . . . . . . 28 88 5.1. Overview of IETF Projects Related to Traffic 89 Engineering . . . . . . . . . . . . . . . . . . . . . . . 28 90 5.1.1. IETF TE Mechanisms . . . . . . . . . . . . . . . . . 28 91 5.1.2. IETF Approaches Relying on TE Mechanisms . . . . . . 33 92 5.1.3. IETF Techniques Used by TE Mechanisms . . . . . . . . 35 93 5.2. Content Distribution . . . . . . . . . . . . . . . . . . 44 94 6. Recommendations for Internet Traffic Engineering . . . . . . 45 95 6.1. Generic Non-functional Recommendations . . . . . . . . . 45 96 6.2. Routing Recommendations . . . . . . . . . . . . . . . . . 47 97 6.3. Traffic Mapping Recommendations . . . . . . . . . . . . . 50 98 6.4. Measurement Recommendations . . . . . . . . . . . . . . . 51 99 6.5. Policing, Planning, and Access Control . . . . . . . . . 51 100 6.6. Network Survivability . . . . . . . . . . . . . . . . . . 52 101 6.6.1. Survivability in MPLS Based Networks . . . . . . . . 54 102 6.6.2. Protection Options . . . . . . . . . . . . . . . . . 55 103 6.7. Multi-Layer Traffic Engineering . . . . . . . . . . . . . 56 104 6.8. Traffic Engineering in Diffserv Environments . . . . . . 57 105 6.9. Network Controllability . . . . . . . . . . . . . . . . . 58 106 7. Inter-Domain Considerations . . . . . . . . . . . . . . . . . 59 107 8. Overview of Contemporary TE Practices in Operational IP 108 Networks . . . . . . . . . . . . . . . . . . . . . . . . 60 109 9. Security Considerations . . . . . . . . . . . . . . . . . . . 63 110 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 64 111 11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 64 112 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 66 113 13. Informative References . . . . . . . . . . . . . . . . . . . 66 114 Appendix A. Historic Overview . . . . . . . . . . . . . . . . . 81 115 A.1. Traffic Engineering in Classical Telephone Networks . . . 81 116 A.2. Evolution of Traffic Engineering in Packet Networks . . . 82 117 A.2.1. Adaptive Routing in the ARPANET . . . . . . . . . . . 83 118 A.2.2. Dynamic Routing in the Internet . . . . . . . . . . . 83 119 A.2.3. ToS Routing . . . . . . . . . . . . . . . . . . . . . 84 120 A.2.4. Equal Cost Multi-Path . . . . . . . . . . . . . . . . 84 121 A.2.5. Nimrod . . . . . . . . . . . . . . . . . . . . . . . 85 122 A.3. Development of Internet Traffic Engineering . . . . . . . 85 123 A.3.1. Overlay Model . . . . . . . . . . . . . . . . . . . . 85 124 Appendix B. Overview of Traffic Engineering Related Work in Other 125 SDOs . . . . . . . . . . . . . . . . . . . . . . . . . . 86 126 B.1. Overview of ITU Activities Related to Traffic 127 Engineering . . . . . . . . . . . . . . . . . . . . . . . 86 128 Appendix C. Summary of Changes Since RFC 3272 . . . . . . . . . 87 129 C.1. RFC 3272 . . . . . . . . . . . . . . . . . . . . . . . . 87 130 C.2. This Document . . . . . . . . . . . . . . . . . . . . . . 90 131 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 94 133 1. Introduction 135 This document describes the principles of Internet traffic 136 engineering (TE). The objective of the document is to articulate the 137 general issues and principles for Internet TE, and where appropriate 138 to provide recommendations, guidelines, and options for the 139 development of preplanned (offline) and dynamic (online) Internet TE 140 capabilities and support systems. 142 Even though Internet TE is most effective when applied end-to-end, 143 the focus of this document is TE within a given domain (such as an 144 autonomous system). However, because a preponderance of Internet 145 traffic tends to originate in one autonomous system and terminate in 146 another, this document also provides an overview of aspects 147 pertaining to inter-domain TE. 149 This document provides a terminology and taxonomy for describing and 150 understanding common Internet TE concepts. 152 This work was first published as [RFC3272] in May 2002. This 153 document obsoletes [RFC3272] by making a complete update to bring the 154 text in line with best current practices for Internet TE and to 155 include references to the latest relevant work in the IETF. It is 156 worth noting around three fifths of the RFCs referenced in this 157 document post-date the publication of RFC 3272. Appendix C provides 158 a summary of changes between RFC 3272 and this document. 160 1.1. What is Internet Traffic Engineering? 162 One of the most significant functions performed in the Internet is 163 the routing and forwarding of traffic from ingress nodes to egress 164 nodes. Therefore, one of the most distinctive functions performed by 165 Internet traffic engineering is the control and optimization of these 166 routing and forwarding functions, to steer traffic through the 167 network. 169 Internet traffic engineering is defined as that aspect of Internet 170 network engineering dealing with the issues of performance evaluation 171 and performance optimization of operational IP networks. Traffic 172 engineering encompasses the application of technology and scientific 173 principles to the measurement, characterization, modeling, and 174 control of Internet traffic [RFC2702], [AWD2]. 176 It is the performance of the network as seen by end users of network 177 services that is paramount. The characteristics visible to end users 178 are the emergent properties of the network, which are the 179 characteristics of the network when viewed as a whole. A central 180 goal of the service provider, therefore, is to enhance the emergent 181 properties of the network while taking economic considerations into 182 account. This is accomplished by addressing traffic oriented 183 performance requirements while utilizing network resources without 184 waste and in a reliable way. Traffic oriented performance measures 185 include delay, delay variation, packet loss, and throughput. 187 Internet TE responds to network events (such as link or node 188 failures, reported or predicted congestion, planned maintenance, 189 service degradation, planned changes in the traffic matrix, etc.). 190 Aspects of capacity management respond at intervals ranging from days 191 to years. Routing control functions operate at intervals ranging 192 from milliseconds to days. Packet level processing functions operate 193 at very fine levels of temporal resolution, ranging from picoseconds 194 to milliseconds while reacting to the real-time statistical behavior 195 of traffic. 197 Thus, the optimization aspects of TE can be viewed from a control 198 perspective, and can be both pro-active and reactive. In the pro- 199 active case, the TE control system takes preventive action to protect 200 against predicted unfavorable future network states, for example, by 201 engineering backup paths. It may also take action that will lead to 202 a more desirable future network state. In the reactive case, the 203 control system responds to correct issues and adapt to network 204 events, such as routing after failure. 206 Another important objective of Internet TE is to facilitate reliable 207 network operations [RFC2702]. Reliable network operations can be 208 facilitated by providing mechanisms that enhance network integrity 209 and by embracing policies emphasizing network survivability. This 210 reduces the vulnerability of services to outages arising from errors, 211 faults, and failures occurring within the network infrastructure. 213 The optimization aspects of TE can be achieved through capacity 214 management and traffic management. In this document, capacity 215 management includes capacity planning, routing control, and resource 216 management. Network resources of particular interest include link 217 bandwidth, buffer space, and computational resources. In this 218 document, traffic management includes: 220 1. Nodal traffic control functions such as traffic conditioning, 221 queue management, and scheduling. 223 2. Other functions that regulate the flow of traffic through the 224 network or that arbitrate access to network resources between 225 different packets or between different traffic flows. 227 One major challenge of Internet TE is the realization of automated 228 control capabilities that adapt quickly and cost effectively to 229 significant changes in network state, while still maintaining 230 stability of the network. Performance evaluation can assess the 231 effectiveness of TE methods, and the results of this evaluation can 232 be used to identify existing problems, guide network re-optimization, 233 and aid in the prediction of potential future problems. However, 234 this process can also be time consuming and may not be suitable to 235 act on short-lived changes in the network. 237 Performance evaluation can be achieved in many different ways. The 238 most notable techniques include analytic methods, simulation, and 239 empirical methods based on measurements. 241 Traffic engineering comes in two flavors: 243 * A background process that constantly monitors traffic and network 244 conditions, and optimizes the use of resources to improve 245 performance. 247 * A form of a pre-planned optimized traffic distribution that is 248 considered optimal. 250 In the later case, any deviation from the optimum distribution (e.g., 251 caused by a fiber cut) is reverted upon repair without further 252 optimization. However, this form of TE relies upon the notion that 253 the planned state of the network is optimal. Hence, in such a mode 254 there are two levels of TE: the TE-planning task to enable optimum 255 traffic distribution, and the routing and forwarding tasks that keep 256 traffic flows attached to the pre-planned distribution. 258 As a general rule, TE concepts and mechanisms must be sufficiently 259 specific and well-defined to address known requirements, but 260 simultaneously flexible and extensible to accommodate unforeseen 261 future demands (see Section 6.1). 263 1.2. Components of Traffic Engineering 265 As mentioned in Section 1.1, Internet traffic engineering provides 266 performance optimization of IP networks while utilizing network 267 resources economically and reliably. Such optimization is supported 268 at the control/controller level and within the data/forwarding plane. 270 The key elements required in any TE solution are as follows: 272 1. Policy 274 2. Path steering 276 3. Resource management 278 Some TE solutions rely on these elements to a lesser or greater 279 extent. Debate remains about whether a solution can truly be called 280 TE if it does not include all of these elements. For the sake of 281 this document, we assert that all TE solutions must include some 282 aspects of all of these elements. Other solutions can be classed as 283 "partial TE" and also fall in scope of this document. 285 Policy allows for the selection of paths (including next hops) based 286 on information beyond basic reachability. Early definitions of 287 routing policy, e.g., [RFC1102] and [RFC1104], discuss routing policy 288 being applied to restrict access to network resources at an aggregate 289 level. BGP is an example of a commonly used mechanism for applying 290 such policies, see [RFC4271] and [RFC8955]. In the TE context, 291 policy decisions are made within the control plane or by controllers 292 in the management plane, and govern the selection of paths. Examples 293 can be found in [RFC4655] and [RFC5394]. Standard TE solutions may 294 cover the mechanisms to distribute and/or enforce polices, but 295 specific policy definition is generally unspecified. 297 Path steering is the ability to forward packets using more 298 information than just knowledge of the next hop. Examples of path 299 steering include IPv4 source routes [RFC0791], RSVP-TE explicit 300 routes [RFC3209], Segment Routing [RFC8402], and Service Function 301 Chaining [RFC7665]. Path steering for TE can be supported via 302 control plane protocols, by encoding in the data plane headers, or by 303 a combination of the two. This includes when control is provided by 304 a controller using a network-facing control protocol. 306 Resource management provides resource-aware control and forwarding. 307 Examples of resources are bandwidth, buffers, and queues, all of 308 which can be managed to control loss and latency. 310 * Resource reservation is the control aspect of resource management. 311 It provides for domain-wide consensus about which network 312 resources are used by a particular flow. This determination may 313 be made at a very course or very fine level. Note that this 314 consensus exists at the network control or controller level, not 315 within the data plane. It may be composed purely of accounting/ 316 bookkeeping, but it typically includes an ability to admit, 317 reject, or reclassify a flow based on policy. Such accounting can 318 be done based on any combination of a static understanding of 319 resource requirements, and the use of dynamic mechanisms to 320 collect requirements (e.g., via [RFC3209]) and resource 321 availability (e.g., via [RFC4203]). 323 * Resource allocation is the data plane aspect of resource 324 management. It provides for the allocation of specific node and 325 link resources to specific flows. Example resources include 326 buffers, policing, and rate-shaping mechanisms that are typically 327 supported via queuing. It also includes the matching of a flow 328 (i.e., flow classification) to a particular set of allocated 329 resources. The method of flow classification and granularity of 330 resource management is technology specific. Examples include 331 Diffserv with dropping and remarking [RFC4594], MPLS-TE [RFC3209], 332 and GMPLS based label switched paths [RFC3945], as well as 333 controller-based solutions [RFC8453]. This level of resource 334 control, while optional, is important in networks that wish to 335 support congestion management policies to control or regulate the 336 offered traffic to deliver different levels of service and 337 alleviate congestion problems, or those networks that wish to 338 control the latency experienced by specific traffic flows. 340 1.3. Scope 342 The scope of this document is intra-domain TE. That is, TE within a 343 given autonomous system in the Internet. This document discusses 344 concepts pertaining to intra- domain traffic control, including such 345 issues as routing control, micro and macro resource allocation, and 346 the control coordination problems that arise consequently. 348 This document describes and characterizes techniques already in use 349 or in advanced development for Internet TE. The way these techniques 350 fit together is discussed and scenarios in which they are useful will 351 be identified. 353 Although the emphasis in this document is on intra-domain traffic 354 engineering, an overview of the high level considerations pertaining 355 to inter-domain TE is provided in Section 7. Inter-domain Internet 356 TE is crucial to the performance enhancement of the global Internet 357 infrastructure. 359 Whenever possible, relevant requirements from existing IETF documents 360 and other sources are incorporated by reference. 362 1.4. Terminology 364 This section provides terminology which is useful for Internet TE. 365 The definitions presented apply to this document. These terms may 366 have other meanings elsewhere. 368 Busy hour: A one hour period within a specified interval of time 369 (typically 24 hours) in which the traffic load in a network or 370 sub-network is greatest. 372 Congestion: A state of a network resource in which the traffic 373 incident on the resource exceeds its output capacity over an 374 interval of time. 376 Congestion avoidance: An approach to congestion management that 377 attempts to obviate the occurrence of congestion. 379 Congestion control: An approach to congestion management that 380 attempts to remedy congestion problems that have already occurred. 382 Constraint-based routing: A class of routing protocols that take 383 specified traffic attributes, network constraints, and policy 384 constraints into account when making routing decisions. 385 Constraint-based routing is applicable to traffic aggregates as 386 well as flows. It is a generalization of QoS-based routing. 388 Demand side congestion management: A congestion management scheme 389 that addresses congestion problems by regulating or conditioning 390 offered load. 392 Effective bandwidth: The minimum amount of bandwidth that can be 393 assigned to a flow or traffic aggregate in order to deliver 394 'acceptable service quality' to the flow or traffic aggregate. 396 Hot-spot: A network element or subsystem which is in a state of 397 congestion. 399 Inter-domain traffic: Traffic that originates in one Autonomous 400 system and terminates in another. 402 Metric: A parameter defined in terms of standard units of 403 measurement. 405 Measurement methodology: A repeatable measurement technique used to 406 derive one or more metrics of interest. 408 Network survivability: The capability to provide a prescribed level 409 of QoS for existing services after a given number of failures 410 occur within the network. 412 Offline traffic engineering: A traffic engineering system that 413 exists outside of the network. 415 Online traffic engineering: A traffic engineering system that exists 416 within the network, typically implemented on or as adjuncts to 417 operational network elements. 419 Performance measures: Metrics that provide quantitative or 420 qualitative measures of the performance of systems or subsystems 421 of interest. 423 Performance metric: A performance parameter defined in terms of 424 standard units of measurement. 426 Provisioning: The process of assigning or configuring network 427 resources to meet certain requests. 429 QoS routing: Class of routing systems that selects paths to be used 430 by a flow based on the QoS requirements of the flow. 432 Service Level Agreement (SLA): A contract between a provider and a 433 customer that guarantees specific levels of performance and 434 reliability at a certain cost. 436 Service Level Objective (SLO): A key element of an SLA between a 437 provider and a customer. SLOs are agreed upon as a means of 438 measuring the performance of the Service Provider and are outlined 439 as a way of avoiding disputes between the two parties based on 440 misunderstanding. 442 Stability: An operational state in which a network does not 443 oscillate in a disruptive manner from one mode to another mode. 445 Supply-side congestion management: A congestion management scheme 446 that provisions additional network resources to address existing 447 and/or anticipated congestion problems. 449 Traffic characteristic: A description of the temporal behavior or a 450 description of the attributes of a given traffic flow or traffic 451 aggregate. 453 Traffic engineering system: A collection of objects, mechanisms, and 454 protocols that are used together to accomplish traffic engineering 455 objectives. 457 Traffic flow: A stream of packets between two end-points that can be 458 characterized in a certain way. A common classification for a 459 traffic flow selects packets with the "five-tuple" of source and 460 destination addresses, source and destination ports, and protocol 461 ID. 463 Traffic matrix: A representation of the traffic demand between a set 464 of origin and destination abstract nodes. An abstract node can 465 consist of one or more network elements. 467 Traffic monitoring: The process of observing traffic characteristics 468 at a given point in a network and collecting the traffic 469 information for analysis and further action. 471 Traffic trunk: An aggregation of traffic flows belonging to the same 472 class which are forwarded through a common path. A traffic trunk 473 may be characterized by an ingress and egress node, and a set of 474 attributes which determine its behavioral characteristics and 475 requirements from the network. 477 2. Background 479 The Internet aims to convey IP packets from ingress nodes to egress 480 nodes efficiently, expeditiously, and economically. Furthermore, in 481 a multiclass service environment (e.g., Diffserv capable networks - 482 see Section 5.1.1.2), the resource sharing parameters of the network 483 must be appropriately determined and configured according to 484 prevailing policies and service models to resolve resource contention 485 issues arising from mutual interference between packets traversing 486 through the network. Thus, consideration must be given to resolving 487 competition for network resources between traffic flows belonging to 488 the same service class (intra-class contention resolution) and 489 traffic flows belonging to different classes (inter-class contention 490 resolution). 492 2.1. Context of Internet Traffic Engineering 494 The context of Internet traffic engineering includes: 496 1. A network domain context that defines the scope under 497 consideration, and in particular the situations in which the TE 498 problems occur. The network domain context includes network 499 structure, network policies, network characteristics, network 500 constraints, network quality attributes, and network optimization 501 criteria. 503 2. A problem context defining the general and concrete issues that 504 TE addresses. The problem context includes identification, 505 abstraction of relevant features, representation, formulation, 506 specification of the requirements on the solution space, and 507 specification of the desirable features of acceptable solutions. 509 3. A solution context suggesting how to address the issues 510 identified by the problem context. The solution context includes 511 analysis, evaluation of alternatives, prescription, and 512 resolution. 514 4. An implementation and operational context in which the solutions 515 are instantiated. The implementation and operational context 516 includes planning, organization, and execution. 518 The context of Internet TE and the different problem scenarios are 519 discussed in the following subsections. 521 2.2. Network Domain Context 523 IP networks range in size from small clusters of routers situated 524 within a given location, to thousands of interconnected routers, 525 switches, and other components distributed all over the world. 527 At the most basic level of abstraction, an IP network can be 528 represented as a distributed dynamic system consisting of: 530 * a set of interconnected resources which provide transport services 531 for IP traffic subject to certain constraints 533 * a demand system representing the offered load to be transported 534 through the network 536 * a response system consisting of network processes, protocols, and 537 related mechanisms which facilitate the movement of traffic 538 through the network (see also [AWD2]). 540 The network elements and resources may have specific characteristics 541 restricting the manner in which the traffic demand is handled. 542 Additionally, network resources may be equipped with traffic control 543 mechanisms managing the way in which the demand is serviced. Traffic 544 control mechanisms may be used to: 546 * control packet processing activities within a given resource 548 * arbitrate contention for access to the resource by different 549 packets 551 * regulate traffic behavior through the resource. 553 A configuration management and provisioning system may allow the 554 settings of the traffic control mechanisms to be manipulated by 555 external or internal entities in order to exercise control over the 556 way in which the network elements respond to internal and external 557 stimuli. 559 The details of how the network carries packets are specified in the 560 policies of the network administrators and are installed through 561 network configuration management and policy based provisioning 562 systems. Generally, the types of service provided by the network 563 also depend upon the technology and characteristics of the network 564 elements and protocols, the prevailing service and utility models, 565 and the ability of the network administrators to translate policies 566 into network configurations. 568 Internet networks have two significant characteristics: 570 * they provide real-time services 572 * their operating environments are very dynamic. 574 The dynamic characteristics of IP and IP/MPLS networks can be 575 attributed in part to fluctuations in demand, to the interaction 576 between various network protocols and processes, to the rapid 577 evolution of the infrastructure which demands the constant inclusion 578 of new technologies and new network elements, and to transient and 579 persistent faults which occur within the system. 581 Packets contend for the use of network resources as they are conveyed 582 through the network. A network resource is considered to be 583 congested if, for an interval of time, the arrival rate of packets 584 exceed the output capacity of the resource. Congestion may result in 585 some of the arriving packets being delayed or even dropped. 587 Congestion increases transit delay, delay variation, may lead to 588 packet loss, and reduces the predictability of network services. 589 Clearly, congestion is highly undesirable. Combating congestion at a 590 reasonable cost is a major objective of Internet TE. 592 Efficient sharing of network resources by multiple traffic flows is a 593 basic operational premise for the Internet. A fundamental challenge 594 in network operation is to increase resource utilization while 595 minimizing the possibility of congestion. 597 The Internet has to function in the presence of different classes of 598 traffic with different service requirements. This requirement is 599 clarified in [RFC2475] which also provides an architecture for 600 Differentiated Services (Diffserv). That document describes how 601 packets can be grouped into behavior aggregates such that each 602 aggregate has a common set of behavioral characteristics or a common 603 set of delivery requirements. Delivery requirements of a specific 604 set of packets may be specified explicitly or implicitly. Two of the 605 most important traffic delivery requirements are: 607 * Capacity constraints can be expressed statistically as peak rates, 608 mean rates, burst sizes, or as some deterministic notion of 609 effective bandwidth. 611 * QoS requirements can be expressed in terms of: 613 - integrity constraints such as packet loss 615 - temporal constraints such as timing restrictions for the 616 delivery of each packet (delay) and timing restrictions for the 617 delivery of consecutive packets belonging to the same traffic 618 stream (delay variation). 620 2.3. Problem Context 622 There are several problems associated with operating a network 623 described in the previous section. This section analyzes the problem 624 context in relation to TE. The identification, abstraction, 625 representation, and measurement of network features relevant to TE 626 are significant issues. 628 A particular challenge is to formulate the problems that traffic 629 engineering attempts to solve. For example: 631 * How to identify the requirements on the solution space? 633 * How to specify the desirable features of solutions? 635 * How to actually solve the problems? 637 * How to measure and characterize the effectiveness of solutions? 639 Another class of problems is how to measure and estimate relevant 640 network state parameters. Effective TE relies on a good estimate of 641 the offered traffic load as well as a view of the underlying topology 642 and associated resource constraints. A network-wide view of the 643 topology is also a must for offline planning. 645 Still another class of problem is how to characterize the state of 646 the network and how to evaluate its performance. The performance 647 evaluation problem is two-fold: one aspect relates to the evaluation 648 of the system-level performance of the network; the other aspect 649 relates to the evaluation of resource-level performance, which 650 restricts attention to the performance analysis of individual network 651 resources. 653 In this document, we refer to the system-level characteristics of the 654 network as the "macro-states" and the resource-level characteristics 655 as the "micro-states." The system-level characteristics are also 656 known as the emergent properties of the network. Correspondingly, we 657 refer to the TE schemes dealing with network performance optimization 658 at the systems level as "macro-TE" and the schemes that optimize at 659 the individual resource level as "micro-TE." Under certain 660 circumstances, the system-level performance can be derived from the 661 resource-level performance using appropriate rules of composition, 662 depending upon the particular performance measures of interest. 664 Another fundamental class of problem concerns how to effectively 665 optimize network performance. Performance optimization may entail 666 translating solutions for specific TE problems into network 667 configurations. Optimization may also entail some degree of resource 668 management control, routing control, and capacity augmentation. 670 2.3.1. Congestion and its Ramifications 672 Congestion is one of the most significant problems in an operational 673 IP context. A network element is said to be congested if it 674 experiences sustained overload over an interval of time. Congestion 675 almost always results in degradation of service quality to end users. 676 Congestion control schemes can include demand-side policies and 677 supply-side policies. Demand-side policies may restrict access to 678 congested resources or dynamically regulate the demand to alleviate 679 the overload situation. Supply-side policies may expand or augment 680 network capacity to better accommodate offered traffic. Supply-side 681 policies may also re-allocate network resources by redistributing 682 traffic over the infrastructure. Traffic redistribution and resource 683 re-allocation serve to increase the 'effective capacity' of the 684 network. 686 The emphasis of this document is primarily on congestion management 687 schemes falling within the scope of the network, rather than on 688 congestion management systems dependent upon sensitivity and 689 adaptivity from end-systems. That is, the aspects that are 690 considered in this document with respect to congestion management are 691 those solutions that can be provided by control entities operating on 692 the network and by the actions of network administrators and network 693 operations systems. 695 2.4. Solution Context 697 The solution context for Internet TE involves analysis, evaluation of 698 alternatives, and choice between alternative courses of action. 699 Generally, the solution context is based on making inferences about 700 the current or future state of the network, and making decisions that 701 may involve a preference between alternative sets of action. More 702 specifically, the solution context demands reasonable estimates of 703 traffic workload, characterization of network state, derivation of 704 solutions which may be implicitly or explicitly formulated, and 705 possibly instantiating a set of control actions. Control actions may 706 involve the manipulation of parameters associated with routing, 707 control over tactical capacity acquisition, and control over the 708 traffic management functions. 710 The following list of instruments may be applicable to the solution 711 context of Internet TE. 713 * A set of policies, objectives, and requirements (which may be 714 context dependent) for network performance evaluation and 715 performance optimization. 717 * A collection of online and possibly offline tools and mechanisms 718 for measurement, characterization, modeling, and control traffic, 719 and control over the placement and allocation of network 720 resources, as well as control over the mapping or distribution of 721 traffic onto the infrastructure. 723 * A set of constraints on the operating environment, the network 724 protocols, and the TE system itself. 726 * A set of quantitative and qualitative techniques and methodologies 727 for abstracting, formulating, and solving TE problems. 729 * A set of administrative control parameters which may be 730 manipulated through a configuration management system. Such 731 system itself may include a configuration control subsystem, a 732 configuration repository, a configuration accounting subsystem, 733 and a configuration auditing subsystem. 735 * A set of guidelines for network performance evaluation, 736 performance optimization, and performance improvement. 738 Determining traffic characteristics through measurement or estimation 739 is very useful within the realm the TE solution space. Traffic 740 estimates can be derived from customer subscription information, 741 traffic projections, traffic models, and from actual measurements. 742 The measurements may be performed at different levels, e.g., at the 743 traffic-aggregate level or at the flow level. Measurements at the 744 flow level or on small traffic aggregates may be performed at edge 745 nodes, when traffic enters and leaves the network. Measurements for 746 large traffic-aggregates may be performed within the core of the 747 network. 749 To conduct performance studies and to support planning of existing 750 and future networks, a routing analysis may be performed to determine 751 the paths the routing protocols will choose for various traffic 752 demands, and to ascertain the utilization of network resources as 753 traffic is routed through the network. Routing analysis captures the 754 selection of paths through the network, the assignment of traffic 755 across multiple feasible routes, and the multiplexing of IP traffic 756 over traffic trunks (if such constructs exist) and over the 757 underlying network infrastructure. A model of network topology is 758 necessary to perform routing analysis. A network topology model may 759 be extracted from: 761 * network architecture documents 763 * network designs 765 * information contained in router configuration files 767 * routing databases 769 * routing tables 771 * automated tools that discover and collate network topology 772 information. 774 Topology information may also be derived from servers that monitor 775 network state, and from servers that perform provisioning functions. 777 Routing in operational IP networks can be administratively controlled 778 at various levels of abstraction including the manipulation of BGP 779 attributes and interior gateway protocol (IGP) metrics. For path 780 oriented technologies such as MPLS, routing can be further controlled 781 by the manipulation of relevant TE parameters, resource parameters, 782 and administrative policy constraints. Within the context of MPLS, 783 the path of an explicitly routed label switched path (LSP) can be 784 computed and established in various ways including: 786 * manually 788 * automatically, online using constraint-based routing processes 789 implemented on label switching routers 791 * automatically, offline using constraint-based routing entities 792 implemented on external TE support systems. 794 2.4.1. Combating the Congestion Problem 796 Minimizing congestion is a significant aspect of Internet traffic 797 engineering. This subsection gives an overview of the general 798 approaches that have been used or proposed to combat congestion. 800 Congestion management policies can be categorized based upon the 801 following criteria (see [YARE95] for a more detailed taxonomy of 802 congestion control schemes): 804 1. Congestion Management Based on Response Time Scales 806 * Long (weeks to months): Expanding network capacity by adding 807 new equipment, routers, and links takes time and is 808 comparatively costly. Capacity planning needs to take this 809 into consideration. Network capacity is expanded based on 810 estimates or forecasts of future traffic development and 811 traffic distribution. These upgrades are typically carried 812 out over weeks or months, or maybe even years. 814 * Medium (minutes to days): Several control policies fall within 815 the medium timescale category. Examples include: 817 a. Adjusting routing protocol parameters to route traffic 818 away or towards certain segments of the network. 820 b. Setting up or adjusting explicitly routed LSPs in MPLS 821 networks to route traffic trunks away from possibly 822 congested resources or toward possibly more favorable 823 routes. 825 c. Re-configuring the logical topology of the network to make 826 it correlate more closely with the spatial traffic 827 distribution using, for example, an underlying path- 828 oriented technology such as MPLS LSPs or optical channel 829 trails. 831 Many of these adaptive schemes rely on measurement systems. A 832 measurement system monitors changes in traffic distribution, 833 traffic loads, and network resource utilization and then 834 provides feedback to the online or offline TE mechanisms and 835 tools so that they can trigger control actions within the 836 network. The TE mechanisms and tools can be implemented in a 837 distributed or centralized fashion. A centralized scheme may 838 have global visibility into the network state and may produce 839 more optimal solutions. However, centralized schemes are 840 prone to single points of failure and may not scale as well as 841 distributed schemes. Moreover, the information utilized by a 842 centralized scheme may be stale and might not reflect the 843 actual state of the network. It is not an objective of this 844 document to make a recommendation between distributed and 845 centralized schemes: that is a choice that network 846 administrators must make based on their specific needs. 848 * Short (picoseconds to minutes): This category includes packet 849 level processing functions and events that are recorded on the 850 order of several round trip times. It also includes router 851 mechanisms such as passive and active buffer management. All 852 of these mechanisms are used to control congestion or signal 853 congestion to end systems so that they can adaptively regulate 854 the rate at which traffic is injected into the network. One 855 of the most popular active queue management schemes, 856 especially for TCP traffic, is Random Early Detection (RED) 857 [FLJA93]. During congestion (but before the queue is filled), 858 the RED scheme chooses arriving packets to "mark" according to 859 a probabilistic algorithm which takes into account the average 860 queue size. A router that does not utilize explicit 861 congestion notification (ECN) [FLOY94] can simply drop marked 862 packets to alleviate congestion and implicitly notify the 863 receiver about the congestion. On the other hand, if the 864 router supports ECN, it can set the ECN field in the packet 865 header. Several variations of RED have been proposed to 866 support different drop precedence levels in multi-class 867 environments [RFC2597]. RED provides congestion avoidance 868 which is not worse than traditional Tail-Drop (TD) queue 869 management (drop arriving packets only when the queue is 870 full). Importantly, RED reduces the possibility of global 871 synchronization where retransmission burst become synchronized 872 across the whole network, and improves fairness among 873 different TCP sessions. However, RED by itself cannot prevent 874 congestion and unfairness caused by sources unresponsive to 875 RED, e.g., UDP traffic and some misbehaved greedy connections. 876 Other schemes have been proposed to improve the performance 877 and fairness in the presence of unresponsive traffic. Some of 878 those schemes (such as Longest Queue Drop (LQD) and Dynamic 879 Soft Partitioning with Random Drop (RND) [SLDC98]) were 880 proposed as theoretical frameworks and are typically not 881 available in existing commercial products. Advice on the use 882 of Active Queue Management (AQM) schemes is provided in 883 [RFC7567]. 885 2. Reactive Versus Preventive Congestion Management Schemes 887 * Reactive (recovery) congestion management policies react to 888 existing congestion problems. All the policies described 889 above for the long and medium time scales can be categorized 890 as being reactive. They are based on monitoring and 891 identifying congestion problems that exist in the network, and 892 on the initiation of relevant actions to ease a situation. 894 * Preventive (predictive/avoidance) policies take proactive 895 action to prevent congestion based on estimates and 896 predictions of future congestion problems (e.g., traffic 897 matrix forecasts). Some of the policies described for the 898 long and medium time scales fall into this category. 899 Preventive policies do not necessarily respond immediately to 900 existing congestion problems. Instead, forecasts of traffic 901 demand and workload distribution are considered, and action 902 may be taken to prevent potential future congestion problems. 903 The schemes described for the short time scale can also be 904 used for congestion avoidance because dropping or marking 905 packets before queues actually overflow would trigger 906 corresponding TCP sources to slow down. 908 3. Supply-Side Versus Demand-Side Congestion Management Schemes 910 * Supply-side congestion management policies increase the 911 effective capacity available to traffic in order to control or 912 reduce congestion. This can be accomplished by increasing 913 capacity or by balancing distribution of traffic over the 914 network. Capacity planning aims to provide a physical 915 topology and associated link bandwidths that match or exceed 916 estimated traffic workload and traffic distribution subject to 917 traffic forecasts and budgetary or other constraints. If the 918 actual traffic distribution does not fit the topology derived 919 from capacity panning, then the traffic can be mapped onto the 920 topology by using routing control mechanisms, by applying path 921 oriented technologies (e.g., MPLS LSPs and optical channel 922 trails) to modify the logical topology, or by employing some 923 other load redistribution mechanisms. 925 * Demand-side congestion management policies control or regulate 926 the offered traffic to alleviate congestion problems. For 927 example, some of the short time scale mechanisms described 928 earlier as well as policing and rate-shaping mechanisms 929 attempt to regulate the offered load in various ways. 931 2.5. Implementation and Operational Context 933 The operational context of Internet TE is characterized by constant 934 changes that occur at multiple levels of abstraction. The 935 implementation context demands effective planning, organization, and 936 execution. The planning aspects may involve determining prior sets 937 of actions to achieve desired objectives. Organizing involves 938 arranging and assigning responsibility to the various components of 939 the TE system and coordinating the activities to accomplish the 940 desired TE objectives. Execution involves measuring and applying 941 corrective or perfective actions to attain and maintain desired TE 942 goals. 944 3. Traffic Engineering Process Models 946 This section describes a generic process model that captures the 947 high-level practical aspects of Internet traffic engineering in an 948 operational context. The process model is described as a sequence of 949 actions that must be carried out to optimize the performance of an 950 operational network (see also [RFC2702], [AWD2]). This process model 951 may be enacted explicitly or implicitly, by a software process or by 952 a human. 954 The TE process model is iterative [AWD2]. The four phases of the 955 process model described below are repeated as a continual sequence. 957 * Define the relevant control policies that govern the operation of 958 the network. 960 * Acquire measurement data from the operational network. 962 * Analyze the network state and characterize the traffic workload. 963 Proactive analysis identifies potential problems that could 964 manifest in the future. Reactive analysis identifies existing 965 problems and determines their causes. 967 * Optimize the performance of the network. This involves a decision 968 process which selects and implements a set of actions from a set 969 of alternatives given the results of the three previous steps. 970 Optimization actions may include the use of techniques to control 971 the offered traffic and to control the distribution of traffic 972 across the network. 974 3.1. Components of the Traffic Engineering Process Model 976 The key components of the traffic engineering process model are as 977 follows. 979 1. Measurement is crucial to the TE function. The operational state 980 of a network can only be conclusively determined through 981 measurement. Measurement is also critical to the optimization 982 function because it provides feedback data which is used by TE 983 control subsystems. This data is used to adaptively optimize 984 network performance in response to events and stimuli originating 985 within and outside the network. Measurement in support of the TE 986 function can occur at different levels of abstraction. For 987 example, measurement can be used to derive packet level 988 characteristics, flow level characteristics, user or customer 989 level characteristics, traffic aggregate characteristics, 990 component level characteristics, and network wide 991 characteristics. 993 2. Modeling, analysis, and simulation are important aspects of 994 Internet TE. Modeling involves constructing an abstract or 995 physical representation which depicts relevant traffic 996 characteristics and network attributes. A network model is an 997 abstract representation of the network which captures relevant 998 network features, attributes, and characteristic. Network 999 simulation tools are extremely useful for TE. Because of the 1000 complexity of realistic quantitative analysis of network 1001 behavior, certain aspects of network performance studies can only 1002 be conducted effectively using simulation. 1004 3. Network performance optimization involves resolving network 1005 issues by transforming such issues into concepts that enable a 1006 solution, identification of a solution, and implementation of the 1007 solution. Network performance optimization can be corrective or 1008 perfective. In corrective optimization, the goal is to remedy a 1009 problem that has occurred or that is incipient. In perfective 1010 optimization, the goal is to improve network performance even 1011 when explicit problems do not exist and are not anticipated. 1013 4. Taxonomy of Traffic Engineering Systems 1015 This section presents a short taxonomy of traffic engineering systems 1016 constructed based on TE styles and views as listed below and 1017 described in greater detail in the following subsections of this 1018 document. 1020 * Time-dependent versus State-dependent versus Event-dependent 1022 * Offline versus Online 1024 * Centralized versus Distributed 1026 * Local versus Global Information 1028 * Prescriptive versus Descriptive 1030 * Open Loop versus Closed Loop 1032 * Tactical versus Strategic 1034 4.1. Time-Dependent Versus State-Dependent Versus Event-Dependent 1036 Traffic engineering methodologies can be classified as time- 1037 dependent, state-dependent, or event-dependent. All TE schemes are 1038 considered to be dynamic in this document. Static TE implies that no 1039 TE methodology or algorithm is being applied - it is a feature of 1040 network planning, but lacks the reactive and flexible nature of TE. 1042 In time-dependent TE, historical information based on periodic 1043 variations in traffic (such as time of day) is used to pre-program 1044 routing and other TE control mechanisms. Additionally, customer 1045 subscription or traffic projection may be used. Pre-programmed 1046 routing plans typically change on a relatively long time scale (e.g., 1047 daily). Time-dependent algorithms do not attempt to adapt to short- 1048 term variations in traffic or changing network conditions. An 1049 example of a time-dependent algorithm is a global centralized 1050 optimizer where the input to the system is a traffic matrix and 1051 multi-class QoS requirements as described [MR99]. Another example of 1052 such a methodology is the application of data mining to Internet 1053 traffic [AJ19] which enables the use of various machine learning 1054 algorithms to identify patterns within historically collected 1055 datasets about Internet traffic, and to extract information in order 1056 to guide decision-making, and to improve efficiency and productivity 1057 of operational processes. 1059 State-dependent TE adapts the routing plans based on the current 1060 state of the network which provides additional information on 1061 variations in actual traffic (i.e., perturbations from regular 1062 variations) that could not be predicted using historical information. 1063 Constraint-based routing is an example of state-dependent TE 1064 operating in a relatively long time scale. An example operating in a 1065 relatively short timescale is a load-balancing algorithm described in 1066 [MATE]. The state of the network can be based on parameters flooded 1067 by the routers. Another approach is for a particular router 1068 performing adaptive TE to send probe packets along a path to gather 1069 the state of that path. [RFC6374] defines protocol extensions to 1070 collect performance measurements from MPLS networks. Another 1071 approach is for a management system to gather the relevant 1072 information directly from network elements using telemetry data 1073 collection "publication/subscription" techniques [RFC7923]. Timely 1074 gathering and distribution of state information is critical for 1075 adaptive TE. While time-dependent algorithms are suitable for 1076 predictable traffic variations, state-dependent algorithms may be 1077 applied to increase network efficiency and resilience to adapt to the 1078 prevailing network state. 1080 Event-dependent TE methods can also be used for TE path selection. 1081 Event-dependent TE methods are distinct from time-dependent and 1082 state-dependent TE methods in the manner in which paths are selected. 1083 These algorithms are adaptive and distributed in nature and typically 1084 use learning models to find good paths for TE in a network. While 1085 state-dependent TE models typically use available-link-bandwidth 1086 (ALB) flooding for TE path selection, event-dependent TE methods do 1087 not require ALB flooding. Rather, event-dependent TE methods 1088 typically search out capacity by learning models, as in the success- 1089 to-the-top (STT) method. ALB flooding can be resource intensive, 1090 since it requires link bandwidth to carry LSAs, processor capacity to 1091 process LSAs, and the overhead can limit area/Autonomous System (AS) 1092 size. Modeling results suggest that event-dependent TE methods could 1093 lead to a reduction in ALB flooding overhead without loss of network 1094 throughput performance [I-D.ietf-tewg-qos-routing]. 1096 4.2. Offline Versus Online 1098 Traffic engineering requires the computation of routing plans. The 1099 computation may be performed offline or online. The computation can 1100 be done offline for scenarios where routing plans need not be 1101 executed in real-time. For example, routing plans computed from 1102 forecast information may be computed offline. Typically, offline 1103 computation is also used to perform extensive searches on multi- 1104 dimensional solution spaces. 1106 Online computation is required when the routing plans must adapt to 1107 changing network conditions as in state-dependent algorithms. Unlike 1108 offline computation (which can be computationally demanding), online 1109 computation is geared toward relative simple and fast calculations to 1110 select routes, fine-tune the allocations of resources, and perform 1111 load balancing. 1113 4.3. Centralized Versus Distributed 1115 Under centralized control there is a central authority which 1116 determines routing plans and perhaps other TE control parameters on 1117 behalf of each router. The central authority periodically collects 1118 network-state information from all routers, and sends routing 1119 information to the routers. The update cycle for information 1120 exchange in both directions is a critical parameter directly 1121 impacting the performance of the network being controlled. 1122 Centralized control may need high processing power and high bandwidth 1123 control channels. 1125 Distributed control determines route selection by each router 1126 autonomously based on the router's view of the state of the network. 1127 The network state information may be obtained by the router using a 1128 probing method or distributed by other routers on a periodic basis 1129 using link state advertisements. Network state information may also 1130 be disseminated under exception conditions. Examples of protocol 1131 extensions used to advertise network link state information are 1132 defined in [RFC5305], [RFC6119], [RFC7471], [RFC8570], and [RFC8571]. 1133 See also Section 5.1.3.8. 1135 4.3.1. Hybrid Systems 1137 In practice, most TE systems will be a hybrid of central and 1138 distributed control. For example, a popular MPLS approach to TE is 1139 to use a central controller based on an active, stateful PCE, but to 1140 use routing and signaling protocols to make local decisions at 1141 routers within the network. Local decisions may be able to respond 1142 more quickly to network events, but may result in conflicts with 1143 decisions made by other routers. 1145 Network operations for TE systems may also use a hybrid of offline 1146 and online computation. TE paths may be precomputed based on stable- 1147 state network information and planned traffic demands, but may then 1148 be modified in the active network depending on variations in network 1149 state and traffic load. Furthermore, responses to network events may 1150 be precomputed offline to allow rapid reactions without further 1151 computation, or may be derived online depending on the nature of the 1152 events. 1154 Lastly, note that a fully functional TE system is likely to use all 1155 aspects of time-dependent, state-dependent, and event-dependent 1156 methodologies as described in Section 4.1. 1158 4.3.2. Considerations for Software Defined Networking 1160 As discussed in Section 5.1.2.2, one of the main drivers for SDN is a 1161 decoupling of the network control plane from the data plane 1162 [RFC7149]. However, SDN may also combine centralized control of 1163 resources, and facilitate application-to-network interaction via an 1164 application programming interface (API) such as [RFC8040]. Combining 1165 these features provides a flexible network architecture that can 1166 adapt to network requirements of a variety of higher-layer 1167 applications, a concept often referred to as the "programmable 1168 network" [RFC7426]. 1170 The centralized control aspect of SDN helps improve global network 1171 resource utilization compared with distributed network control, where 1172 local policy may often override global optimization goals. In an SDN 1173 environment, the data plane forwards traffic to its desired 1174 destination. However, before traffic reaches the data plane, the 1175 logically centralized SDN control plane often determines the end-to- 1176 end path the application traffic will take in the network. 1177 Therefore, the SDN control plane needs to be aware of the underlying 1178 network topology, capabilities and current node and link resource 1179 state. 1181 Using a PCE-based SDN control framework [RFC7491], the available 1182 network topology may be discovered by running a passive instance of 1183 OSPF or IS-IS, or via BGP-LS [RFC7752], to generate a TED (see 1184 Section 5.1.3.12). The PCE is used to compute a path (see 1185 Section 5.1.3.10) based on the TED and available bandwidth, and 1186 further path optimization may be based on requested objective 1187 functions [RFC5541]. When a suitable path has been computed the 1188 programming of the explicit network path may be performed using 1189 either end-to-end signaling protocol [RFC3209] or per-hop with each 1190 node being directly programmed [RFC8283] by the SDN controller. 1192 By utilizing a centralized approach to network control, additional 1193 network benefits are also available, including Global Concurrent 1194 Optimization (GCO) [RFC5557]. A GCO path computation request will 1195 simultaneously use the network topology and set of new end-to-end 1196 path requests, along with their respective constraints, for optimal 1197 placement in the network. Correspondingly, a GCO-based computation 1198 may be applied to recompute existing network paths to groom traffic 1199 and to mitigate congestion. 1201 4.4. Local Versus Global 1203 Traffic engineering algorithms may require local and global network- 1204 state information. 1206 Local information is the state of a portion of the domain. Examples 1207 include the bandwidth and packet loss rate of a particular path, or 1208 the state and capabilities of a network link. Local state 1209 information may be sufficient for certain instances of distributed 1210 control TE. 1212 Global information is the state of the entire TE domain. Examples 1213 include a global traffic matrix, and loading information on each link 1214 throughout the domain of interest. Global state information is 1215 typically required with centralized control. Distributed TE systems 1216 may also need global information in some cases. 1218 4.5. Prescriptive Versus Descriptive 1220 TE systems may also be classified as prescriptive or descriptive. 1222 Prescriptive traffic engineering evaluates alternatives and 1223 recommends a course of action. Prescriptive TE can be further 1224 categorized as either corrective or perfective. Corrective TE 1225 prescribes a course of action to address an existing or predicted 1226 anomaly. Perfective TE prescribes a course of action to evolve and 1227 improve network performance even when no anomalies are evident. 1229 Descriptive traffic engineering, on the other hand, characterizes the 1230 state of the network and assesses the impact of various policies 1231 without recommending any particular course of action. 1233 4.5.1. Intent-Based Networking 1235 One way to express a service request is through "intent". Intent- 1236 Based Networking aims to produce networks that are simpler to manage 1237 and operate, requiring only minimal intervention. Intent is defined 1238 in [I-D.irtf-nmrg-ibn-concepts-definitions] as a set of operational 1239 goals (that a network should meet) and outcomes (that a network is 1240 supposed to deliver), defined in a declarative manner without 1241 specifying how to achieve or implement them. 1243 Intent provides data and functional abstraction so that users and 1244 operators do not need to be concerned with low-level device 1245 configuration or the mechanisms used to achieve a given intent. This 1246 approach can be conceptually easier for a user, but may be less 1247 expressive in terms of constraints and guidelines. 1249 Intent-Based Networking is applicable to TE because many of the high- 1250 level objectives may be expressed as "intent." For example, load 1251 balancing, delivery of services, and robustness against failures. 1252 The intent is converted by the management system into TE actions 1253 within the network. 1255 4.6. Open-Loop Versus Closed-Loop 1257 Open-loop traffic engineering control is where control action does 1258 not use feedback information from the current network state. The 1259 control action may use its own local information for accounting 1260 purposes, however. 1262 Closed-loop traffic engineering control is where control action 1263 utilizes feedback information from the network state. The feedback 1264 information may be in the form of historical information or current 1265 measurement. 1267 4.7. Tactical versus Strategic 1269 Tactical traffic engineering aims to address specific performance 1270 problems (such as hot-spots) that occur in the network from a 1271 tactical perspective, without consideration of overall strategic 1272 imperatives. Without proper planning and insights, tactical TE tends 1273 to be ad hoc in nature. 1275 Strategic traffic engineering approaches the TE problem from a more 1276 organized and systematic perspective, taking into consideration the 1277 immediate and longer term consequences of specific policies and 1278 actions. 1280 5. Review of TE Techniques 1282 This section briefly reviews different TE-related approaches proposed 1283 and implemented in telecommunications and computer networks using 1284 IETF protocols and architectures. These approaches are organized 1285 into three categories: 1287 * TE mechanisms which adhere to the definition provided in 1288 Section 1.2. 1290 * Approaches that rely upon those TE mechanisms. 1292 * Techniques that are used by those TE mechanisms and approaches. 1294 The discussion is not intended to be comprehensive. It is primarily 1295 intended to illuminate existing approaches to TE in the Internet. A 1296 historic overview of TE in telecommunications networks is provided in 1297 Appendix A, while Appendix B describes approaches in other standards 1298 bodies. 1300 5.1. Overview of IETF Projects Related to Traffic Engineering 1302 This subsection reviews a number of IETF activities pertinent to 1303 Internet traffic engineering. 1305 5.1.1. IETF TE Mechanisms 1307 5.1.1.1. Integrated Services 1309 The IETF developed the Integrated Services (Intserv) model that 1310 requires resources, such as bandwidth and buffers, to be reserved a 1311 priori for a given traffic flow to ensure that the quality of service 1312 requested by the traffic flow is satisfied. The Integrated Services 1313 model includes additional components beyond those used in the best- 1314 effort model such as packet classifiers, packet schedulers, and 1315 admission control. A packet classifier is used to identify flows 1316 that are to receive a certain level of service. A packet scheduler 1317 handles the scheduling of service to different packet flows to ensure 1318 that QoS commitments are met. Admission control is used to determine 1319 whether a router has the necessary resources to accept a new flow. 1321 The main issue with the Integrated Services model has been 1322 scalability [RFC2998], especially in large public IP networks which 1323 may potentially have millions of active traffic flows in transit 1324 concurrently. 1326 A notable feature of the Integrated Services model is that it 1327 requires explicit signaling of QoS requirements from end systems to 1328 routers [RFC2753]. The Resource Reservation Protocol (RSVP) performs 1329 this signaling function and is a critical component of the Integrated 1330 Services model. RSVP is described in Section 5.1.3.2. 1332 5.1.1.2. Differentiated Services 1334 The goal of Differentiated Services (Diffserv) within the IETF was to 1335 devise scalable mechanisms for categorization of traffic into 1336 behavior aggregates, which ultimately allows each behavior aggregate 1337 to be treated differently, especially when there is a shortage of 1338 resources such as link bandwidth and buffer space [RFC2475]. One of 1339 the primary motivations for Diffserv was to devise alternative 1340 mechanisms for service differentiation in the Internet that mitigate 1341 the scalability issues encountered with the Intserv model. 1343 Diffserv uses the Differentiated Services field in the IP header (the 1344 DS field) consisting of six bits in what was formerly known as the 1345 Type of Service (TOS) octet. The DS field is used to indicate the 1346 forwarding treatment that a packet should receive at a transit node 1347 [RFC2474]. Diffserv includes the concept of Per-Hop Behavior (PHB) 1348 groups. Using the PHBs, several classes of services can be defined 1349 using different classification, policing, shaping, and scheduling 1350 rules. 1352 For an end-user of network services to utilize Differentiated 1353 Services provided by its Internet Service Provider (ISP), it may be 1354 necessary for the user to have an SLA with the ISP. An SLA may 1355 explicitly or implicitly specify a Traffic Conditioning Agreement 1356 (TCA) which defines classifier rules as well as metering, marking, 1357 discarding, and shaping rules. 1359 Packets are classified, and possibly policed and shaped at the 1360 ingress to a Diffserv network. When a packet traverses the boundary 1361 between different Diffserv domains, the DS field of the packet may be 1362 re-marked according to existing agreements between the domains. 1364 Differentiated Services allows only a finite number of service 1365 classes to be specified by the DS field. The main advantage of the 1366 Diffserv approach relative to the Intserv model is scalability. 1367 Resources are allocated on a per-class basis and the amount of state 1368 information is proportional to the number of classes rather than to 1369 the number of application flows. 1371 The Diffserv model deals with traffic management issues on a per hop 1372 basis. The Diffserv control model consists of a collection of micro- 1373 TE control mechanisms. Other TE capabilities, such as capacity 1374 management (including routing control), are also required in order to 1375 deliver acceptable service quality in Diffserv networks. The concept 1376 of Per Domain Behaviors has been introduced to better capture the 1377 notion of Differentiated Services across a complete domain [RFC3086]. 1379 Diffserv procedures can also be applied in an MPLS context. See 1380 Section 6.8 for more information. 1382 5.1.1.3. Segment Routing Policy 1384 SR Policy [I-D.ietf-spring-segment-routing-policy] is an evolution of 1385 Segment Routing (see Section 5.1.3.11) to enhance the TE capabilities 1386 of SR. It is a framework that enables instantiation of an ordered 1387 list of segments on a node for implementing a source routing policy 1388 with a specific intent for traffic steering from that node. 1390 An SR Policy is identified through the tuple . The head-end is the IP address of the node where the policy 1392 is instantiated. The endpoint is the IP address of the destination 1393 of the policy. The color is an index that associates the SR Policy 1394 with an intent (e.g., low-latency). 1396 The head-end node is notified of SR Policies and associated SR paths 1397 via configuration or by a extensions to protocols such as PCEP 1398 [RFC8664] or BGP [I-D.ietf-idr-segment-routing-te-policy]. Each SR 1399 path consists of a Segment-List (an SR source-routed path), and the 1400 head-end uses the endpoint and color parameters to classify packets 1401 to match the SR policy and so determine along which path to forward 1402 them. If an SR Policy is associated with a set of SR paths, each is 1403 associated with a weight for weighted load balancing. Furthermore, 1404 multiple SR Policies may be associated with a set of SR paths to 1405 allow multiple traffic flows to be placed on the same paths. 1407 An SR Binding SID (BSID) are also be associated with each candidate 1408 path associated with an SR Policy, or with the SR Policy itself. The 1409 head-end node installs a BSID-keyed entry in the forwarding plane and 1410 assigns it the action of steering packets that match the entry to the 1411 selected path of the SR Policy. This steering can be done in various 1412 ways: 1414 * SID Steering: Incoming packets have an active SID matching a local 1415 BSID at the head-end. 1417 * Per-destination Steering: Incoming packets match a BGP/Service 1418 route which indicates an SR Policy. 1420 * Per-flow Steering: Incoming packets match a forwarding array (for 1421 example, the classic 5-tuple) which indicates an SR Policies. 1423 * Policy-based Steering: Incoming packets match a routing policy 1424 which directs them to an SR Policy. 1426 5.1.1.4. Transport-Based TE 1428 In addition to IP-based TE mechanisms, transport-based TE approaches 1429 can be considered in specific deployment contexts (e.g., data 1430 centers, multi-homing). For example, the 3GPP defines the Access 1431 Traffic Steering, Switching, and Splitting (ATSSS) [ATSSS] service 1432 functions as follows. 1434 Access Traffic Steering: This is the selection of an access network 1435 for a new flow and the transfer of the traffic of that flow over 1436 the selected access network. 1438 Access Traffic Switching: This is the migration of all packets of an 1439 ongoing flow from one access network to another access network. 1440 Only one access network is in use at a time. 1442 Access Traffic Splitting: This is about forwarding the packets of a 1443 flow across multiple access networks simultaneously. 1445 The control plane is used to provide hosts and specific network 1446 devices with a set or policies that specify which flows are eligible 1447 to use the ATSSS service. The traffic that matches an ATSSS policy 1448 can be distributed among the available access networks following one 1449 of the following four modes. 1451 Active-Standby: The traffic is forwarded via a specific access 1452 (called "active access") and switched to another access (called 1453 "standby access") when the active access is unavailable. 1455 Priority-based: Network accesses are assigned priority levels that 1456 indicate which network access is to be used first. The traffic 1457 associated with the matching flow will be steered onto the network 1458 access with the highest priority until congestion is detected, 1459 then the overflow will be forwarded over the next highest priority 1460 access. 1462 Load-Balancing: The traffic is distributed among the available 1463 access networks following a distribution ratio (e.g., 75% - 25%). 1465 Smallest Delay: The traffic is forwarded via the access that 1466 presents the smallest round-trip-time (RTT). 1468 For resource management purposes, hosts and network devices support 1469 means such as congestion control, RTT measurement, and packet 1470 scheduling. 1472 For TCP traffic, Multipath TCP [RFC8684] and the 0-RTT Convert 1473 Protocol [RFC8803] are used to provide the ATSSS service. 1475 QUIC [RFC9000] is a UDP-based multiplexed and secure transport 1476 protocol. QUIC provides applications with flow-controlled streams 1477 for structured communication, low-latency connection establishment, 1478 and network path migration. 1480 QUIC is a connection-oriented protocol that creates a stateful 1481 interaction between a client and server. QUIC uses a handshake 1482 procedure that combines negotiation of cryptographic and transport 1483 parameters. This is a key differentiation from other transport 1484 protocols. 1486 With QUIC it is possible to support the ATSSS switching and steering 1487 functions, but splitting is not yet supported. Indeed, QUIC supports 1488 a connection migration procedure that allows peers to change their 1489 transport coordinates (IP addresses, port numbers) without breaking 1490 the underlying QUIC connection. 1492 5.1.1.5. Deterministic Networking 1494 Deterministic Networking (DetNet) [RFC8655] is an architecture for 1495 applications with critical timing and reliability requirements. The 1496 layered architecture particularly focuses on developing DetNet 1497 service capabilities in the data plane [RFC8938]. The DetNet service 1498 sub-layer provides a set of Packet Replication, Elimination, and 1499 Ordering Functions (PREOF) functions to provide end-to-end service 1500 assurance. The DetNet forwarding sub-layer provides corresponding 1501 forwarding assurance (low packet loss, bounded latency, and in-order 1502 delivery) functions using resource allocations and explicit route 1503 mechanisms. 1505 The separation into two sub-layers allows a greater flexibility to 1506 adapt Detnet capability over a number of TE data plane mechanisms 1507 such as IP, MPLS, and Segment Routing. More importantly it 1508 interconnects IEEE 802.1 Time Sensitive Networking (TSN) [RFC9023] 1509 deployed in Industry Control and Automation Systems (ICAS). 1511 DetNet can be seen as a specialized branch of TE, since it sets up 1512 explicit optimized paths with allocation of resources as requested. 1513 A DetNet application can express its QoS attributes or traffic 1514 behavior using any combination of DetNet functions described in sub- 1515 layers. They are then distributed and provisioned using well- 1516 established control and provisioning mechanisms adopted for traffic- 1517 engineering. 1519 In DetNet, a considerable state information is required to maintain 1520 per flow queuing disciplines and resource reservation for a large 1521 number of individual flows. This can be quite challenging for 1522 network operations during network events such as faults, change in 1523 traffic volume or re-provisioning. Therefore, DetNet recommends 1524 support for aggregated flows, however, it still requires large amount 1525 of control signaling to establish and maintain DetNet flows. 1527 5.1.2. IETF Approaches Relying on TE Mechanisms 1529 5.1.2.1. Application-Layer Traffic Optimization 1531 This document describes various TE mechanisms available in the 1532 network. However, distributed applications in general and, in 1533 particular, bandwidth-greedy P2P applications that are used, for 1534 example, for file sharing, cannot directly use those techniques. As 1535 per [RFC5693], applications could greatly improve traffic 1536 distribution and quality by cooperating with external services that 1537 are aware of the network topology. Addressing the Application-Layer 1538 Traffic Optimization (ALTO) problem means, on the one hand, deploying 1539 an ALTO service to provide applications with information regarding 1540 the underlying network (e.g., basic network location structure and 1541 preferences of network paths) and, on the other hand, enhancing 1542 applications in order to use such information to perform better-than- 1543 random selection of the endpoints with which they establish 1544 connections. 1546 The basic function of ALTO is based on abstract maps of a network. 1547 These maps provide a simplified view, yet enough information about a 1548 network for applications to effectively utilize them. Additional 1549 services are built on top of the maps. [RFC7285] describes a 1550 protocol implementing the ALTO services as an information-publishing 1551 interface that allows a network to publish its network information 1552 such as network locations, costs between them at configurable 1553 granularities, and end-host properties to network applications. The 1554 information published by the ALTO Protocol should benefit both the 1555 network and the applications. The ALTO Protocol uses a REST-ful 1556 design and encodes its requests and responses using JSON [RFC8259] 1557 with a modular design by dividing ALTO information publication into 1558 multiple ALTO services (e.g., the Map service, the Map-Filtering 1559 Service, the Endpoint Property Service, and the Endpoint Cost 1560 Service). 1562 [RFC8189] defines a new service that allows an ALTO Client to 1563 retrieve several cost metrics in a single request for an ALTO 1564 filtered cost map and endpoint cost map. [RFC8896] extends the ALTO 1565 cost information service so that applications decide not only 'where' 1566 to connect, but also 'when'. This is useful for applications that 1567 need to perform bulk data transfer and would like to schedule these 1568 transfers during an off-peak hour, for example. 1569 [I-D.ietf-alto-performance-metrics] introducing network performance 1570 metrics, including network delay, jitter, packet loss rate, hop 1571 count, and bandwidth. The ALTO server may derive and aggregate such 1572 performance metrics from BGP-LS (see Section 5.1.3.9) or IGP-TE (see 1573 Section 5.1.3.8), or management tools, and then expose the 1574 information to allow applications to determine 'where' to connect 1575 based on network performance criteria. ALTO WG is evaluating the use 1576 of network TE properties while making application decisions for new 1577 use-cases such as Edge computing and Datacenter interconnect. 1579 5.1.2.2. Network Virtualization and Abstraction 1581 One of the main drivers for Software Defined Networking (SDN) 1582 [RFC7149] is a decoupling of the network control plane from the data 1583 plane. This separation has been achieved for TE networks with the 1584 development of MPLS/GMPLS (see Section 5.1.3.3 and Section 5.1.3.4) 1585 and the Path Computation Element (PCE) (Section 5.1.3.10). One of 1586 the advantages of SDN is its logically centralized control regime 1587 that allows a global view of the underlying networks. Centralized 1588 control in SDN helps improve network resource utilization compared 1589 with distributed network control. 1591 Abstraction and Control of TE Networks (ACTN) [RFC8453] defines a 1592 hierarchical SDN architecture which describes the functional entities 1593 and methods for the coordination of resources across multiple 1594 domains, to provide end-to-end traffic engineered services. ACTN 1595 facilitates end-to-end connections and provides them to the user. 1596 ACTN is focused on: 1598 * Abstraction of the underlying network resources and how they are 1599 provided to higher-layer applications and customers. 1601 * Virtualization of underlying resources for use by the customer, 1602 application, or service. The creation of a virtualized 1603 environment allows operators to view and control multi-domain 1604 networks as a single virtualized network. 1606 * Presentation to customers of networks as a virtual network via 1607 open and programmable interfaces. 1609 The ACTN managed infrastructure is built from traffic engineered 1610 network resources, which may include statistical packet bandwidth, 1611 physical forwarding plane sources (such as wavelengths and time 1612 slots), forwarding and cross-connect capabilities. The type of 1613 network virtualization seen in ACTN allows customers and applications 1614 (tenants) to utilize and independently control allocated virtual 1615 network resources as if resources as if they were physically their 1616 own resource. The ACTN network is "sliced", with tenants being given 1617 a different partial and abstracted topology view of the physical 1618 underlying network. 1620 5.1.2.3. Network Slicing 1622 An IETF Network Slice is a logical network topology connecting a 1623 number of endpoints using a set of shared or dedicated network 1624 resources [I-D.ietf-teas-ietf-network-slices]. The resources are 1625 used to satisfy specific Service Level Objectives (SLOs) specified by 1626 the consumer. 1628 IETF network slices are not, of themselves, TE constructs. However, 1629 a network operator that offers IETF network slices is likely to use 1630 many TE tools in order to manage their network and provide the 1631 services. 1633 IETF Network Slices are defined such that they are independent of the 1634 underlying infrastructure connectivity and technologies used. From a 1635 customer's perspective an IETF Network Slice looks like a VPN 1636 connectivity matrix with additional information about the level of 1637 service required between endpoints. From an operator's perspective 1638 the IETF Network Slice looks like a set of routing or tunneling 1639 instructions with the network resource reservations necessary to 1640 provide the required service levels as specified by the SLOs. The 1641 concept of an IETF network slice is consistent with an enhanced VPN 1642 (VPN+) [I-D.ietf-teas-enhanced-vpn]. 1644 5.1.3. IETF Techniques Used by TE Mechanisms 1645 5.1.3.1. Constraint-Based Routing 1647 Constraint-based routing refers to a class of routing systems that 1648 compute routes through a network subject to the satisfaction of a set 1649 of constraints and requirements. In the most general case, 1650 constraint-based routing may also seek to optimize overall network 1651 performance while minimizing costs. 1653 The constraints and requirements may be imposed by the network itself 1654 or by administrative policies. Constraints may include bandwidth, 1655 hop count, delay, and policy instruments such as resource class 1656 attributes. Constraints may also include domain specific attributes 1657 of certain network technologies and contexts which impose 1658 restrictions on the solution space of the routing function. Path 1659 oriented technologies such as MPLS have made constraint-based routing 1660 feasible and attractive in public IP networks. 1662 The concept of constraint-based routing within the context of MPLS TE 1663 requirements in IP networks was first described in [RFC2702] and led 1664 to developments such as MPLS-TE [RFC3209] as described in 1665 Section 5.1.3.3. 1667 Unlike QoS-based routing (for example, see [RFC2386], [MA], and 1668 [I-D.ietf-idr-performance-routing]) which generally addresses the 1669 issue of routing individual traffic flows to satisfy prescribed flow- 1670 based QoS requirements subject to network resource availability, 1671 constraint-based routing is applicable to traffic aggregates as well 1672 as flows and may be subject to a wide variety of constraints which 1673 may include policy restrictions. 1675 5.1.3.1.1. IGP Flexible Algorithms (Flex-Algos) 1677 The traditional approach to routing in an IGP network relies on the 1678 IGPs deriving "shortest paths" over the network based solely on the 1679 IGP metric assigned to the links. Such an approach is often limited: 1680 traffic may tend to converge toward the destination, possibly causing 1681 congestion; and it is not possible to steer traffic onto paths 1682 depending on the end-to-end qualities demanded by the applications. 1684 To overcome this limitation, various sorts of TE have been widely 1685 deployed (as described in this document), where the TE component is 1686 responsible for computing the path based on additional metrics and/or 1687 constraints. Such paths (or tunnels) need to be installed in the 1688 routers' forwarding tables in addition to, or as a replacement for 1689 the original paths computed by IGPs. The main drawback of these TE 1690 approaches is the additional complexity of protocols and management, 1691 and the state that may need to be maintained within the network. 1693 IGP flexible algorithms (flex-algos) [I-D.ietf-lsr-flex-algo] allow 1694 IGPs to construct constraint-based paths over the network by 1695 computing constraint- based next hops. The intent of flex-algos is 1696 to reduce TE complexity by letting an IGP perform some basic TE 1697 computation capabilities. Flex-algo includes a set of extensions to 1698 the IGPs that enable a router to send TLVs that: 1700 * describe a set of constraints on the topology 1702 * identify calculation-type 1704 * describe a metric-type that is to be used to compute the best 1705 paths through the constrained topology. 1707 A given combination of calculation-type, metric-type, and constraints 1708 is known as a "Flexible Algorithm Definition" (or FAD). A router 1709 that sends such a set of TLVs also assigns a specific identifier (the 1710 Flexible Algorithm) to the specified combination of calculation-type, 1711 metric-type, and constraints. 1713 There are two use cases for flex-algo: in IP networks 1714 [I-D.ietf-lsr-ip-flexalgo] and in segment routing networks 1715 [I-D.ietf-lsr-flex-algo]. In the first case, flex-algo computes 1716 paths to an IPv4 or IPv6 address, in the second case, flex-algo 1717 computes paths to a prefix SID (see Section 5.1.3.11). 1719 There are many use cases where flex-algo can bring big value, such 1720 as: 1722 * Expansion of functionality of IP Performance metrics [RFC5664] 1723 when points of interest could instantiate specific constraint- 1724 based routing (flex-algo) based on the measurement results. 1726 * Nested usage of flex-algo and TE extensions for IGP (see 1727 Section 5.1.3.8) when we can form 'underlay' by means of flex-algo 1728 and 'overlay' by TE. 1730 * Flex-algo in SR-MPLS (Section 5.1.3.11) is a base use case when we 1731 can easily benefit from TE-like topology that will be built 1732 without external TE component on routers or PCE (see 1733 Section 5.1.3.10). 1735 * Building of network slices [I-D.ietf-teas-ietf-network-slices] 1736 where particular IETF network slice SLO can be guaranteed by flex- 1737 algo. 1739 5.1.3.2. RSVP 1741 RSVP is a soft state signaling protocol [RFC2205]. It supports 1742 receiver initiated establishment of resource reservations for both 1743 multicast and unicast flows. RSVP was originally developed as a 1744 signaling protocol within the Integrated Services framework (see 1745 Section 5.1.1.1) for applications to communicate QoS requirements to 1746 the network and for the network to reserve relevant resources to 1747 satisfy the QoS requirements [RFC2205]. 1749 In RSVP, the traffic sender or source node sends a PATH message to 1750 the traffic receiver with the same source and destination addresses 1751 as the traffic which the sender will generate. The PATH message 1752 contains: (1) a sender traffic specification describing the 1753 characteristics of the traffic, (2) a sender template specifying the 1754 format of the traffic, and (3) an optional advertisement 1755 specification which is used to support the concept of One Pass With 1756 Advertising (OPWA) [RFC2205]. Every intermediate router along the 1757 path forwards the PATH message to the next hop determined by the 1758 routing protocol. Upon receiving a PATH message, the receiver 1759 responds with a RESV message which includes a flow descriptor used to 1760 request resource reservations. The RESV message travels to the 1761 sender or source node in the opposite direction along the path that 1762 the PATH message traversed. Every intermediate router along the path 1763 can reject or accept the reservation request of the RESV message. If 1764 the request is rejected, the rejecting router will send an error 1765 message to the receiver and the signaling process will terminate. If 1766 the request is accepted, link bandwidth and buffer space are 1767 allocated for the flow and the related flow state information is 1768 installed in the router. 1770 One of the issues with the original RSVP specification was 1771 Scalability. This is because reservations were required for micro- 1772 flows, so that the amount of state maintained by network elements 1773 tends to increase linearly with the number of traffic flows. These 1774 issues are described in [RFC2961] which also modifies and extends 1775 RSVP to mitigate the scaling problems to make RSVP a versatile 1776 signaling protocol for the Internet. For example, RSVP has been 1777 extended to reserve resources for aggregation of flows, to set up 1778 MPLS explicit label switched paths (see Section 5.1.3.3), and to 1779 perform other signaling functions within the Internet. [RFC2961] 1780 also describes a mechanism to reduce the amount of Refresh messages 1781 required to maintain established RSVP sessions. 1783 5.1.3.3. Multiprotocol Label Switching (MPLS) 1785 MPLS is an advanced forwarding scheme which also includes extensions 1786 to conventional IP control plane protocols. MPLS extends the 1787 Internet routing model and enhances packet forwarding and path 1788 control [RFC3031]. 1790 At the ingress to an MPLS domain, Label Switching Routers (LSRs) 1791 classify IP packets into Forwarding Equivalence Classes (FECs) based 1792 on a variety of factors, including, e.g., a combination of the 1793 information carried in the IP header of the packets and the local 1794 routing information maintained by the LSRs. An MPLS label stack 1795 entry is then prepended to each packet according to their forwarding 1796 equivalence classes. The MPLS label stack entry is 32 bits long and 1797 contains a 20-bit label field. 1799 An LSR makes forwarding decisions by using the label prepended to 1800 packets as the index into a local next hop label forwarding entry 1801 (NHLFE). The packet is then processed as specified in the NHLFE. 1802 The incoming label may be replaced by an outgoing label (label swap), 1803 and the packet may be forwarded to the next LSR. Before a packet 1804 leaves an MPLS domain, its MPLS label may be removed (label pop). A 1805 Label Switched Path (LSP) is the path between an ingress LSRs and an 1806 egress LSRs through which a labeled packet traverses. The path of an 1807 explicit LSP is defined at the originating (ingress) node of the LSP. 1808 MPLS can use a signaling protocol such as RSVP or LDP to set up LSPs. 1810 MPLS is a very powerful technology for Internet TE because it 1811 supports explicit LSPs which allow constraint-based routing to be 1812 implemented efficiently in IP networks [AWD2]. The requirements for 1813 TE over MPLS are described in [RFC2702]. Extensions to RSVP to 1814 support instantiation of explicit LSP are discussed in [RFC3209]. 1816 5.1.3.4. Generalized MPLS (GMPLS) 1818 GMPLS extends MPLS control protocols to encompass time-division 1819 (e.g., Synchronous Optical Network / Synchronous Digital Hierarchy 1820 (SONET/SDH), Plesiochronous Digital Hierarchy (PDH), Optical 1821 Transport Network (OTN)), wavelength (lambdas), and spatial switching 1822 (e.g., incoming port or fiber to outgoing port or fiber) as well as 1823 continuing to support packet switching. GMPLS provides a common set 1824 of control protocols for all of these layers (including some 1825 technology-specific extensions) each of which has a diverse data or 1826 forwarding plane. GMPLS covers both the signaling and the routing 1827 part of that control plane and is based on the TE extensions to MPLS 1828 (see Section 5.1.3.3). 1830 In GMPLS, the original MPLS architecture is extended to include LSRs 1831 whose forwarding planes rely on circuit switching, and therefore 1832 cannot forward data based on the information carried in either packet 1833 or cell headers. Specifically, such LSRs include devices where the 1834 switching is based on time slots, wavelengths, or physical ports. 1835 These additions impact basic LSP properties: how labels are requested 1836 and communicated, the unidirectional nature of MPLS LSPs, how errors 1837 are propagated, and information provided for synchronizing the 1838 ingress and egress LSRs. 1840 5.1.3.5. IP Performance Metrics 1842 The IETF IP Performance Metrics (IPPM) working group has developed a 1843 set of standard metrics that can be used to monitor the quality, 1844 performance, and reliability of Internet services. These metrics can 1845 be applied by network operators, end-users, and independent testing 1846 groups to provide users and service providers with a common 1847 understanding of the performance and reliability of the Internet 1848 component 'clouds' they use/provide [RFC2330]. The criteria for 1849 performance metrics developed by the IPPM working group are described 1850 in [RFC2330]. Examples of performance metrics include one-way packet 1851 loss [RFC7680], one-way delay [RFC7679], and connectivity measures 1852 between two nodes [RFC2678]. Other metrics include second-order 1853 measures of packet loss and delay. 1855 Some of the performance metrics specified by the IPPM working group 1856 are useful for specifying SLAs. SLAs are sets of service level 1857 objectives negotiated between users and service providers, wherein 1858 each objective is a combination of one or more performance metrics, 1859 possibly subject to certain constraints. 1861 5.1.3.6. Flow Measurement 1863 The IETF Real Time Flow Measurement (RTFM) working group produced an 1864 architecture that defines a method to specify traffic flows as well 1865 as a number of components for flow measurement (meters, meter 1866 readers, manager) [RFC2722]. A flow measurement system enables 1867 network traffic flows to be measured and analyzed at the flow level 1868 for a variety of purposes. As noted in RFC 2722, a flow measurement 1869 system can be very useful in the following contexts: 1871 * understanding the behavior of existing networks 1873 * planning for network development and expansion 1875 * quantification of network performance 1877 * verifying the quality of network service 1878 * attribution of network usage to users. 1880 A flow measurement system consists of meters, meter readers, and 1881 managers. A meter observes packets passing through a measurement 1882 point, classifies them into groups, accumulates usage data (such as 1883 the number of packets and bytes for each group), and stores the usage 1884 data in a flow table. A group may represent any collection of user 1885 applications, hosts, networks, etc. A meter reader gathers usage 1886 data from various meters so it can be made available for analysis. A 1887 manager is responsible for configuring and controlling meters and 1888 meter readers. The instructions received by a meter from a manager 1889 include flow specifications, meter control parameters, and sampling 1890 techniques. The instructions received by a meter reader from a 1891 manager include the address of the meter whose date is to be 1892 collected, the frequency of data collection, and the types of flows 1893 to be collected. 1895 5.1.3.7. Endpoint Congestion Management 1897 [RFC3124] provides a set of congestion control mechanisms for the use 1898 of transport protocols. It is also allows the development of 1899 mechanisms for unifying congestion control across a subset of an 1900 endpoint's active unicast connections (called a congestion group). A 1901 congestion manager continuously monitors the state of the path for 1902 each congestion group under its control. The manager uses that 1903 information to instruct a scheduler on how to partition bandwidth 1904 among the connections of that congestion group. 1906 5.1.3.8. TE Extensions to the IGPs 1908 [RFC5305] describes the extensions to the Intermediate System to 1909 Intermediate System (IS-IS) protocol to support TE, similarly 1910 [RFC3630] specifies TE extensions for OSPFv2 ([RFC5329] has the same 1911 description for OSPFv3). 1913 The idea of redistribution of TE extensions such as link type and ID, 1914 local and remote IP addresses, TE metric, maximum bandwidth, maximum 1915 reservable bandwidth and unreserved bandwidth, admin group in IGP is 1916 a common for both IS-IS and OSPF. The information distributed by the 1917 IGPs in this way can be used to build a view of the state and 1918 capabilities of a TE network (see Section 5.1.3.12). 1920 The difference is in the details of their transmission: IS-IS uses 1921 the Extended IS Reachability TLV (type 22) and Sub-TLVs for those TE 1922 parameters, OSPFv2 uses Opaque LSA [RFC5250] type 10 (OSPFv3 uses 1923 Intra-Area-TE-LSA) with two top-level TLV (Router Address and Link) 1924 also with Sub-TLVs for that purpose. 1926 IS-IS also uses the Extended IP Reachability TLV (type 135, which 1927 have the new 32 bit metric) and the TE Router ID TLV (type 134). 1928 Those Sub-TLV details are described in [RFC8570] for IS-IS and in 1929 [RFC7471] for OSPFv2 ([RFC5329] for OSPFv3). 1931 5.1.3.9. Link-State BGP 1933 In a number of environments, a component external to a network is 1934 called upon to perform computations based on the network topology and 1935 current state of the connections within the network, including TE 1936 information. This is information typically distributed by IGP 1937 routing protocols within the network (see Section 5.1.3.8. 1939 The Border Gateway Protocol (BGP) (see also Section 7) is one of the 1940 essential routing protocols that glue the Internet together. BGP 1941 Link State (BGP-LS) [RFC7752] is a mechanism by which link-state and 1942 TE information can be collected from networks and shared with 1943 external components using the BGP routing protocol. The mechanism is 1944 applicable to physical and virtual IGP links, and is subject to 1945 policy control. 1947 Information collected by BGP-LS can be used to construct the Traffic 1948 Engineering Database (TED, see Section 5.1.3.12) for use by the Path 1949 Computation Element (PCE, see Section 5.1.3.10), or may be used by 1950 Application-Layer Traffic Optimization (ALTO) servers (see 1951 Section 5.1.2.1). 1953 5.1.3.10. Path Computation Element 1955 Constraint-based path computation is a fundamental building block for 1956 TE in MPLS and GMPLS networks. Path computation in large, multi- 1957 domain networks is complex and may require special computational 1958 components and cooperation between the elements in different domains. 1959 The Path Computation Element (PCE) [RFC4655] is an entity (component, 1960 application, or network node) that is capable of computing a network 1961 path or route based on a network graph and applying computational 1962 constraints. 1964 Thus, a PCE can provide a central component in a TE system operating 1965 on the TE Database (TED, see Section 5.1.3.12) with delegated 1966 responsibility for determining paths in MPLS, GMPLS, or Segment 1967 Routing networks. The PCE uses the Path Computation Element 1968 Communication Protocol (PCEP) [RFC5440] to communicate with Path 1969 Computation Clients (PCCs), such as MPLS LSRs, to answer their 1970 requests for computed paths or to instruct them to initiate new paths 1971 [RFC8281] and maintain state about paths already installed in the 1972 network [RFC8231]. 1974 PCEs form key components of a number of TE systems. More information 1975 about the applicability of PCE can be found in [RFC8051], while 1976 [RFC6805] describes the application of PCE to determining paths 1977 across multiple domains. PCE also has potential use in Abstraction 1978 and Control of TE Networks (ACTN) (see Section 5.1.2.2), Centralized 1979 Network Control [RFC8283], and Software Defined Networking (SDN) (see 1980 Section 4.3.2). 1982 5.1.3.11. Segment Routing 1984 Segment Routing (SR) [RFC8402] leverages the source routing and 1985 tunneling paradigms. The path a packet takes is defined at the 1986 ingress and the packet is tunneled to the egress. A node steers a 1987 packet through a controlled set of instructions, called segments, by 1988 prepending the packet with an SR header: a label stack in MPLS case; 1989 a series of 128-bit segment identifiers in the IPv6 case. 1991 Segments are identified by Segment Identifiers (SIDs). There are 1992 four types of SID that are relevant for TE. 1994 Prefix SID: Unique within the routing domain used to identify a 1995 prefix. 1997 Node SID: A Prefix SID with the 'N' bit set to identify a node. 1999 Adjacency SID: Identifies a unidirectional adjacency. 2001 Binding SID: A Binding SID has two purposes: 2003 1. Used to advertise the mappings of prefixes to SIDs/Labels. 2005 2. Used to advertise a path available for a Forwarding 2006 Equivalence Class. 2008 A segment can represent any instruction, topological or service- 2009 based, thanks to the MPLS architecture [RFC3031]. Labels can be 2010 looked up in a global context (platform wide) as well as in some 2011 other context (see "context labels" in Section 3 of [RFC5331]). 2013 The application of "policy" to segment routing can make SR into a TE 2014 mechanism as described in Section 5.1.1.3. 2016 5.1.3.12. Network TE State Definition and Presentation 2018 The network states that are relevant to the TE need to be stored in 2019 the system and presented to the user. The Traffic Engineering 2020 Database (TED) is a collection of all TE information about all TE 2021 nodes and TE links in the network, which is an essential component of 2022 a TE system, such as MPLS-TE [RFC2702] and GMPLS [RFC3945]. In order 2023 to formally define the data in the TED and to present the data to the 2024 user with high usability, the data modeling language YANG [RFC7950] 2025 can be used as described in [RFC8795]. 2027 5.1.3.13. System Management and Control Interfaces 2029 The TE control system needs to have a management interface that is 2030 human-friendly and a control interfaces that is programmable for 2031 automation. The Network Configuration Protocol (NETCONF) [RFC6241] 2032 or the RESTCONF Protocol [RFC8040] provide programmable interfaces 2033 that are also human-friendly. These protocols use XML or JSON 2034 encoded messages. When message compactness or protocol bandwidth 2035 consumption needs to be optimized for the control interface, other 2036 protocols, such as Group Communication for the Constrained 2037 Application Protocol (CoAP) [RFC7390] or gRPC, are available, 2038 especially when the protocol messages are encoded in a binary format. 2039 Along with any of these protocols, the data modeling language YANG 2040 [RFC7950] can be used to formally and precisely define the interface 2041 data. 2043 The Path Computation Element Communication Protocol (PCEP) [RFC5440] 2044 is another protocol that has evolved to be an option for the TE 2045 system control interface. The messages of PCEP are TLV-based, not 2046 defined by a data modeling language such as YANG. 2048 5.2. Content Distribution 2050 The Internet is dominated by client-server interactions, principally 2051 Web traffic although in the future, more sophisticated media servers 2052 may become dominant. The location and performance of major 2053 information servers has a significant impact on the traffic patterns 2054 within the Internet as well as on the perception of service quality 2055 by end users. 2057 A number of dynamic load balancing techniques have been devised to 2058 improve the performance of replicated information servers. These 2059 techniques can cause spatial traffic characteristics to become more 2060 dynamic in the Internet because information servers can be 2061 dynamically picked based upon the location of the clients, the 2062 location of the servers, the relative utilization of the servers, the 2063 relative performance of different networks, and the relative 2064 performance of different parts of a network. This process of 2065 assignment of distributed servers to clients is called traffic 2066 directing. It is an application layer function. 2068 Traffic directing schemes that allocate servers in multiple 2069 geographically dispersed locations to clients may require empirical 2070 network performance statistics to make more effective decisions. In 2071 the future, network measurement systems may need to provide this type 2072 of information. 2074 When congestion exists in the network, traffic directing and traffic 2075 engineering systems should act in a coordinated manner. This topic 2076 is for further study. 2078 The issues related to location and replication of information 2079 servers, particularly web servers, are important for Internet traffic 2080 engineering because these servers contribute a substantial proportion 2081 of Internet traffic. 2083 6. Recommendations for Internet Traffic Engineering 2085 This section describes high-level recommendations for traffic 2086 engineering in the Internet in general terms. 2088 The recommendations describe the capabilities needed to solve a TE 2089 problem or to achieve a TE objective. Broadly speaking, these 2090 recommendations can be categorized as either functional or non- 2091 functional recommendations. 2093 * Functional recommendations describe the functions that a traffic 2094 engineering system should perform. These functions are needed to 2095 realize TE objectives by addressing traffic engineering problems. 2097 * Non-functional recommendations relate to the quality attributes or 2098 state characteristics of a TE system. These recommendations may 2099 contain conflicting assertions and may sometimes be difficult to 2100 quantify precisely. 2102 6.1. Generic Non-functional Recommendations 2104 The generic non-functional recommendations for Internet traffic 2105 engineering are listed in the paragraphs that follow. In a given 2106 context, some of these recommendations may be critical while others 2107 may be optional. Therefore, prioritization may be required during 2108 the development phase of a TE system to tailor it to a specific 2109 operational context. 2111 Usability: Usability is a human aspect of TE systems. It refers to 2112 the ease with which a TE system can be deployed and operated. In 2113 general, it is desirable to have a TE system that can be readily 2114 deployed in an existing network. It is also desirable to have a 2115 TE system that is easy to operate and maintain. 2117 Automation: Whenever feasible, a TE system should automate as many 2118 TE functions as possible to minimize the amount of human effort 2119 needed to analyze and control operational networks. Automation is 2120 particularly important in large-scale public networks because of 2121 the high cost of the human aspects of network operations and the 2122 high risk of network problems caused by human errors. Automation 2123 may entail the incorporation of automatic feedback and 2124 intelligence into some components of the TE system. 2126 Scalability: Public networks continue to grow rapidly with respect 2127 to network size and traffic volume. Therefore, to remain 2128 applicable as the network evolves, a TE system should be scalable. 2129 In particular, a TE system should remain functional as the network 2130 expands with regard to the number of routers and links, and with 2131 respect to the traffic volume. A TE system should have a scalable 2132 architecture, should not adversely impair other functions and 2133 processes in a network element, and should not consume too many 2134 network resources when collecting and distributing state 2135 information, or when exerting control. 2137 Stability: Stability is a very important consideration in TE systems 2138 that respond to changes in the state of the network. State- 2139 dependent TE methodologies typically include a trade-off between 2140 responsiveness and stability. It is strongly recommended that 2141 when a trade-off between responsiveness and stability is needed, 2142 it should be made in favor of stability (especially in public IP 2143 backbone networks). 2145 Flexibility: A TE system should allow for changes in optimization 2146 policy. In particular, a TE system should provide sufficient 2147 configuration options so that a network administrator can tailor 2148 the system to a particular environment. It may also be desirable 2149 to have both online and offline TE subsystems which can be 2150 independently enabled and disabled. TE systems that are used in 2151 multi-class networks should also have options to support class 2152 based performance evaluation and optimization. 2154 Visibility: Mechanisms should exist as part of the TE system to 2155 collect statistics from the network and to analyze these 2156 statistics to determine how well the network is functioning. 2157 Derived statistics such as traffic matrices, link utilization, 2158 latency, packet loss, and other performance measures of interest 2159 which are determined from network measurements can be used as 2160 indicators of prevailing network conditions. The capabilities of 2161 the various components of the routing system are other examples of 2162 status information which should be observable. 2164 Simplicity: A TE system should be as simple as possible. Simplicity 2165 in user interface does not necessarily imply that the TE system 2166 will use naive algorithms. When complex algorithms and internal 2167 structures are used, the user interface should hide such 2168 complexities from the network administrator as much as possible. 2170 Interoperability: Whenever feasible, TE systems and their components 2171 should be developed with open standards-based interfaces to allow 2172 interoperation with other systems and components. 2174 Security: Security is a critical consideration in TE systems. Such 2175 systems typically exert control over functional aspects of the 2176 network to achieve the desired performance objectives. Therefore, 2177 adequate measures must be taken to safeguard the integrity of the 2178 TE system. Adequate measures must also be taken to protect the 2179 network from vulnerabilities that originate from security breaches 2180 and other impairments within the TE system. 2182 The remaining subsections of this section focus on some of the high- 2183 level functional recommendations for TE. 2185 6.2. Routing Recommendations 2187 Routing control is a significant aspect of Internet traffic 2188 engineering. Routing impacts many of the key performance measures 2189 associated with networks, such as throughput, delay, and utilization. 2190 Generally, it is very difficult to provide good service quality in a 2191 wide area network without effective routing control. A desirable TE 2192 routing system is one that takes traffic characteristics and network 2193 constraints into account during route selection while maintaining 2194 stability. 2196 Shortest path first (SPF) IGPs are based on shortest path algorithms 2197 and have limited control capabilities for TE [RFC2702], [AWD2]. 2198 These limitations include: 2200 1. Pure SPF protocols do not take network constraints and traffic 2201 characteristics into account during route selection. For 2202 example, IGPs always select the shortest paths based on link 2203 metrics assigned by administrators) so load sharing cannot be 2204 performed across paths of different costs. Using shortest paths 2205 to forward traffic may cause the following problems: 2207 * If traffic from a source to a destination exceeds the capacity 2208 of a link along the shortest path, the link (and hence the 2209 shortest path) becomes congested while a longer path between 2210 these two nodes may be under-utilized 2212 * The shortest paths from different sources can overlap at some 2213 links. If the total traffic from the sources exceeds the 2214 capacity of any of these links, congestion will occur. 2216 * Problems can also occur because traffic demand changes over 2217 time, but network topology and routing configuration cannot be 2218 changed as rapidly. This causes the network topology and 2219 routing configuration to become sub-optimal over time, which 2220 may result in persistent congestion problems. 2222 2. The Equal-Cost Multi-Path (ECMP) capability of SPF IGPs supports 2223 sharing of traffic among equal cost paths between two nodes. 2224 However, ECMP attempts to divide the traffic as equally as 2225 possible among the equal cost shortest paths. Generally, ECMP 2226 does not support configurable load sharing ratios among equal 2227 cost paths. The result is that one of the paths may carry 2228 significantly more traffic than other paths because it may also 2229 carry traffic from other sources. This situation can result in 2230 congestion along the path that carries more traffic. Weighted 2231 ECMP (WECMP) (see, for example, [I-D.ietf-bess-evpn-unequal-lb]) 2232 provides some mitigation. 2234 3. Modifying IGP metrics to control traffic routing tends to have 2235 network-wide effects. Consequently, undesirable and 2236 unanticipated traffic shifts can be triggered as a result. Work 2237 described in Section 8 may be capable of better control [FT00], 2238 [FT01]. 2240 Because of these limitations, new capabilities are needed to enhance 2241 the routing function in IP networks. Some of these capabilities are 2242 summarized below. 2244 * Constraint-based routing computes routes to fulfill requirements 2245 subject to constraints. This can be useful in public IP backbones 2246 with complex topologies. Constraints may include bandwidth, hop 2247 count, delay, and administrative policy instruments such as 2248 resource class attributes [RFC2702], [RFC2386]. This makes it 2249 possible to select routes that satisfy a given set of 2250 requirements. Routes computed by constraint-based routing are not 2251 necessarily the shortest paths. Constraint-based routing works 2252 best with path-oriented technologies that support explicit 2253 routing, such as MPLS. 2255 * Constraint-based routing can also be used as a way to distribute 2256 traffic onto the infrastructure, including for best effort 2257 traffic. For example, congestion problems caused by uneven 2258 traffic distribution may be avoided or reduced by knowing the 2259 reservable bandwidth attributes of the network links and by 2260 specifying the bandwidth requirements for path selection. 2262 * A number of enhancements to the link state IGPs are needed to 2263 allow them to distribute additional state information required for 2264 constraint-based routing. The extensions to OSPF are described in 2265 [RFC3630], and to IS-IS in [RFC5305]. Some of the additional 2266 topology state information includes link attributes such as 2267 reservable bandwidth and link resource class attribute (an 2268 administratively specified property of the link). The resource 2269 class attribute concept is defined in [RFC2702]. The additional 2270 topology state information is carried in new TLVs and sub-TLVs in 2271 IS-IS, or in the Opaque LSA in OSPF [RFC5305], [RFC3630]. 2273 * An enhanced link-state IGP may flood information more frequently 2274 than a normal IGP. This is because even without changes in 2275 topology, changes in reservable bandwidth or link affinity can 2276 trigger the enhanced IGP to initiate flooding. A trade-off 2277 between the timeliness of the information flooded and the flooding 2278 frequency is typically implemented using a threshold based on the 2279 percentage change of the advertised resources to avoid excessive 2280 consumption of link bandwidth and computational resources, and to 2281 avoid instability in the TED. 2283 * In a TE system, it is also desirable for the routing subsystem to 2284 make the load splitting ratio among multiple paths (with equal 2285 cost or different cost) configurable. This capability gives 2286 network administrators more flexibility in the control of traffic 2287 distribution across the network. It can be very useful for 2288 avoiding/relieving congestion in certain situations. Examples can 2289 be found in [XIAO] and [I-D.ietf-bess-evpn-unequal-lb]. 2291 * The routing system should also have the capability to control the 2292 routes of subsets of traffic without affecting the routes of other 2293 traffic if sufficient resources exist for this purpose. This 2294 capability allows a more refined control over the distribution of 2295 traffic across the network. For example, the ability to move 2296 traffic away from its original path to another path (without 2297 affecting other traffic paths) allows the traffic to be moved from 2298 resource-poor network segments to resource-rich segments. Path 2299 oriented technologies such as MPLS-TE inherently support this 2300 capability as discussed in [AWD2]. 2302 * Additionally, the routing subsystem should be able to select 2303 different paths for different classes of traffic (or for different 2304 traffic behavior aggregates) if the network supports multiple 2305 classes of service (different behavior aggregates). 2307 6.3. Traffic Mapping Recommendations 2309 Traffic mapping is the assignment of traffic workload onto (pre- 2310 established) paths to meet certain requirements. Thus, while 2311 constraint-based routing deals with path selection, traffic mapping 2312 deals with the assignment of traffic to established paths which may 2313 have been generated by constraint-based routing or by some other 2314 means. Traffic mapping can be performed by time-dependent or state- 2315 dependent mechanisms, as described in Section 4.1. 2317 An important aspect of the traffic mapping function is the ability to 2318 establish multiple paths between an originating node and a 2319 destination node, and the capability to distribute the traffic 2320 between the two nodes across the paths according to some policies. A 2321 pre-condition for this scheme is the existence of flexible mechanisms 2322 to partition traffic and then assign the traffic partitions onto the 2323 parallel paths as noted in [RFC2702]. When traffic is assigned to 2324 multiple parallel paths, it is recommended that special care should 2325 be taken to ensure proper ordering of packets belonging to the same 2326 application (or traffic flow) at the destination node of the parallel 2327 paths. 2329 Mechanisms that perform the traffic mapping functions should aim to 2330 map the traffic onto the network infrastructure to minimize 2331 congestion. If the total traffic load cannot be accommodated, or if 2332 the routing and mapping functions cannot react fast enough to 2333 changing traffic conditions, then a traffic mapping system may use 2334 short time scale congestion control mechanisms (such as queue 2335 management, scheduling, etc.) to mitigate congestion. Thus, 2336 mechanisms that perform the traffic mapping functions complement 2337 existing congestion control mechanisms. In an operational network, 2338 traffic should be mapped onto the infrastructure such that intra- 2339 class and inter-class resource contention are minimized (see 2340 Section 2). 2342 When traffic mapping techniques that depend on dynamic state feedback 2343 (e.g., MATE [MATE] and such like) are used, special care must be 2344 taken to guarantee network stability. 2346 6.4. Measurement Recommendations 2348 The importance of measurement in TE has been discussed throughout 2349 this document. A TE system should include mechanisms to measure and 2350 collect statistics from the network to support the TE function. 2351 Additional capabilities may be needed to help in the analysis of the 2352 statistics. The actions of these mechanisms should not adversely 2353 affect the accuracy and integrity of the statistics collected. The 2354 mechanisms for statistical data acquisition should also be able to 2355 scale as the network evolves. 2357 Traffic statistics may be classified according to long-term or short- 2358 term timescales. Long-term traffic statistics are very useful for 2359 traffic engineering. Long-term traffic statistics may periodicity 2360 record network workload (such as hourly, daily, and weekly variations 2361 in traffic profiles) as well as traffic trends. Aspects of the 2362 traffic statistics may also describe class of service characteristics 2363 for a network supporting multiple classes of service. Analysis of 2364 the long-term traffic statistics may yield other information such as 2365 busy hour characteristics, traffic growth patterns, persistent 2366 congestion problems, hot-spot, and imbalances in link utilization 2367 caused by routing anomalies. 2369 A mechanism for constructing traffic matrices for both long-term and 2370 short-term traffic statistics should be in place. In multi-service 2371 IP networks, the traffic matrices may be constructed for different 2372 service classes. Each element of a traffic matrix represents a 2373 statistic about the traffic flow between a pair of abstract nodes. 2374 An abstract node may represent a router, a collection of routers, or 2375 a site in a VPN. 2377 Traffic statistics should provide reasonable and reliable indicators 2378 of the current state of the network on the short-term scale. Some 2379 short term traffic statistics may reflect link utilization and link 2380 congestion status. Examples of congestion indicators include 2381 excessive packet delay, packet loss, and high resource utilization. 2382 Examples of mechanisms for distributing this kind of information 2383 include SNMP, probing tools, FTP, IGP link state advertisements, and 2384 NETCONF/RESTCONF, etc. 2386 6.5. Policing, Planning, and Access Control 2388 The recommendations in Section 6.2 and Section 6.3 may be sub-optimal 2389 or even ineffective if the amount of traffic flowing on a route or 2390 path exceeds the capacity of the resource on that route or path. 2391 Several approaches can be used to increase the performance of TE 2392 systems. 2394 * The fundamental approach is some form of planning where traffic is 2395 steered onto paths so that it is distributed across the available 2396 resources. This planning may be centralized or distributed, and 2397 must be aware of the planned traffic volumes and available 2398 resources. However, this approach is only of value if the traffic 2399 is presented conformant to the planned traffic volumes. 2401 * Traffic flows may be policed at the edges of a network. This is a 2402 simple way to check that the actual traffic volumes are consistent 2403 with the planned volumes. Some form of measurement (see 2404 Section 6.4) is used to determine the rate of arrival of traffic 2405 and excess traffic could be discarded. Alternatively, excess 2406 traffic could be forwarded as best-effort within the network. 2407 However, this approach is only completely effective if the 2408 planning is stringent and network-wide, and if a harsh approach is 2409 taken to disposing of excess traffic. 2411 * Resource-based admission control is the process whereby network 2412 nodes decide whether to grant access to resources. The basis for 2413 the decision on a packet-by-packet basis is determination of the 2414 flow to which the packet belongs. This information is combined 2415 with policy instructions that have been locally configured, or 2416 installed through the management or control planes. The end 2417 result is that a packet may be allowed to access (or use) specific 2418 resources on the node if and only if the policy is conformed with 2419 for the flow to which the packet belongs. 2421 Combining some element of all three of these measures is advisable to 2422 achieve a better TE system. 2424 6.6. Network Survivability 2426 Network survivability refers to the capability of a network to 2427 maintain service continuity in the presence of faults. This can be 2428 accomplished by promptly recovering from network impairments and 2429 maintaining the required QoS for existing services after recovery. 2430 Survivability is an issue of great concern within the Internet 2431 community due to the demand to carry mission critical traffic, real- 2432 time traffic, and other high priority traffic over the Internet. 2433 Survivability can be addressed at the device level by developing 2434 network elements that are more reliable; and at the network level by 2435 incorporating redundancy into the architecture, design, and operation 2436 of networks. It is recommended that a philosophy of robustness and 2437 survivability should be adopted in the architecture, design, and 2438 operation of TE that control IP networks (especially public IP 2439 networks). Because different contexts may demand different levels of 2440 survivability, the mechanisms developed to support network 2441 survivability should be flexible so that they can be tailored to 2442 different needs. A number of tools and techniques have been 2443 developed to enable network survivability including MPLS Fast Reroute 2444 [RFC4090], Topology Independent Loop-free Alternate Fast Re-route for 2445 Segment Routing [I-D.ietf-rtgwg-segment-routing-ti-lfa] RSVP-TE 2446 Extensions in Support of End-to-End GMPLS Recovery [RFC4872], and 2447 GMPLS Segment Recovery [RFC4873]. 2449 The impact of service outages varies significantly for different 2450 service classes depending on the duration of the outage which can 2451 vary from milliseconds (with minor service impact) to seconds (with 2452 possible call drops for IP telephony and session time-outs for 2453 connection oriented transactions) to minutes and hours (with 2454 potentially considerable social and business impact). Different 2455 duration outages have different impacts depending on the throughput 2456 of the traffic flows that are interrupted. 2458 Failure protection and restoration capabilities are available in 2459 multiple layers as network technologies have continued to evolve. 2460 Optical networks are capable of providing dynamic ring and mesh 2461 restoration functionality at the wavelength level. At the SONET/SDH 2462 layer survivability capability is provided with Automatic Protection 2463 Switching (APS) as well as self-healing ring and mesh architectures. 2464 Similar functionality is provided by layer 2 technologies such as 2465 Ethernet. 2467 Rerouting is used at the IP layer to restore service following link 2468 and node outages. Rerouting at the IP layer occurs after a period of 2469 routing convergence which may require seconds to minutes to complete. 2470 Path-oriented technologies such as MPLS ([RFC3469]) can be used to 2471 enhance the survivability of IP networks in a potentially cost 2472 effective manner. 2474 An important aspect of multi-layer survivability is that technologies 2475 at different layers may provide protection and restoration 2476 capabilities at different granularities in terms of time scales and 2477 at different bandwidth granularity (from packet-level to wavelength 2478 level). Protection and restoration capabilities can also be 2479 sensitive to different service classes and different network utility 2480 models. Coordinating different protection and restoration 2481 capabilities across multiple layers in a cohesive manner to ensure 2482 network survivability is maintained at reasonable cost is a 2483 challenging task. Protection and restoration coordination across 2484 layers may not always be feasible, because networks at different 2485 layers may belong to different administrative domains. 2487 The following paragraphs present some of the general recommendations 2488 for protection and restoration coordination. 2490 * Protection and restoration capabilities from different layers 2491 should be coordinated to provide network survivability in a 2492 flexible and cost effective manner. Avoiding duplication of 2493 functions in different layers is one way to achieve the 2494 coordination. Escalation of alarms and other fault indicators 2495 from lower to higher layers may also be performed in a coordinated 2496 manner. The order of timing of restoration triggers from 2497 different layers is another way to coordinate multi-layer 2498 protection/restoration. 2500 * Network capacity reserved in one layer to provide protection and 2501 restoration is not available to carry traffic in a higher layer: 2502 it is not visible as spare capacity in the higher layer. Placing 2503 protection/restoration functions in many layers may increase 2504 redundancy and robustness, but it can result in significant 2505 inefficiencies in network resource utilization. Careful planning 2506 is needed to balance the trade-off between the desire for 2507 survivability and the optimal use of resources. 2509 * It is generally desirable to have protection and restoration 2510 schemes that are intrinsically bandwidth efficient. 2512 * Failure notifications throughout the network should be timely and 2513 reliable if they are to be acted on as triggers for effective 2514 protection and restoration actions. 2516 * Alarms and other fault monitoring and reporting capabilities 2517 should be provided at the right network layers so that the 2518 protection and restoration actions can be taken in those layers. 2520 6.6.1. Survivability in MPLS Based Networks 2522 Because MPLS is path-oriented, it has the potential to provide faster 2523 and more predictable protection and restoration capabilities than 2524 conventional hop by hop routed IP systems. Protection types for MPLS 2525 networks can be divided into four categories. 2527 * Link Protection: The objective of link protection is to protect an 2528 LSP from the failure of a given link. Under link protection, a 2529 protection or backup LSP (the secondary LSP) follows a path that 2530 is disjoint from the path of the working or operational LSP (the 2531 primary LSP) at the particular link where link protection is 2532 required. When the protected link fails, traffic on the working 2533 LSP is switched to the protection LSP at the head-end of the 2534 failed link. As a local repair method, link protection can be 2535 fast. This form of protection may be most appropriate in 2536 situations where some network elements along a given path are 2537 known to be less reliable than others. 2539 * Node Protection: The objective of node protection is to protect an 2540 LSP from the failure of a given node. Under node protection, the 2541 secondary LSP follows a path that is disjoint from the path of the 2542 primary LSP at the particular node where node protection is 2543 required. The secondary LSP is also disjoint from the primary LSP 2544 at all links attached to the node to be protected. When the 2545 protected node fails, traffic on the working LSP is switched over 2546 to the protection LSP at the upstream LSR directly connected to 2547 the failed node. Node protection covers a slightly larger part of 2548 the network compared to link protection, but is otherwise 2549 fundamentally the same. 2551 * Path Protection: The goal of LSP path protection (or end-to-end 2552 protection) is to protect an LSP from any failure along its routed 2553 path. Under path protection, the path of the protection LSP is 2554 completely disjoint from the path of the working LSP. The 2555 advantage of path protection is that the backup LSP protects the 2556 working LSP from all possible link and node failures along the 2557 path, except for failures of ingress or egress LSR. Additionally, 2558 path protection may be more efficient in terms of resource usage 2559 than link or node protection applied at every jop along the path. 2560 However, path protection may be slower than link and node 2561 protection because the fault notifications have to be propagated 2562 further. 2564 * Segment Protection: An MPLS domain may be partitioned into 2565 multiple subdomains (protection domains). Path protection is 2566 applied to the path of each LSP as it crosses the domain from its 2567 ingress to the domain to where it egresses the domain. In cases 2568 where an LSP traverses multiple protection domains, a protection 2569 mechanism within a domain only needs to protect the segment of the 2570 LSP that lies within the domain. Segment protection will 2571 generally be faster than end-to-end path protection because 2572 recovery generally occurs closer to the fault and the notification 2573 doesn't have to propagate as far. 2575 See [RFC3469] and [RFC6372] for a more comprehensive discussion of 2576 MPLS based recovery. 2578 6.6.2. Protection Options 2580 Another issue to consider is the concept of protection options. We 2581 use notation such as "m:n protection", where m is the number of 2582 protection LSPs used to protect n working LSPs. In all cases except 2583 1+1 protection, the resources associated with the protection LSPs can 2584 be used to carry preemptable best-effort traffic when the working LSP 2585 is functioning correctly. 2587 * 1:1 protection: One working LSP is protected/restored by one 2588 protection LSP. 2590 * 1:n protection: One protection LSP is used to protect/restore n 2591 working LSPs. Only one failed LSP can be restored at any time. 2593 * n:1 protection: One working LSP is protected/restored by n 2594 protection LSPs, possibly with load splitting across the 2595 protection LSPs. This may be especially useful when it is not 2596 feasible to find one path for the backup that can satisfy the 2597 bandwidth requirement of the primary LSP. 2599 * 1+1 protection: Traffic is sent concurrently on both the working 2600 LSP and a protection LSP. The egress LSR selects one of the two 2601 LSPs based on local policy (usually based on traffic integrity). 2602 When a fault disrupts the traffic on one LSP, the egress switches 2603 to receive traffic from the other LSP. This approach is expensive 2604 in how it consumes network but recovers from failures most 2605 rapidly. 2607 6.7. Multi-Layer Traffic Engineering 2609 Networks are often arranged as layers. A layer relationship may 2610 represent the interaction between technologies (for example, an IP 2611 network operated over an optical network), or the relationship 2612 between different network operators (for example, a customer network 2613 operated over a service provider's network). Note that a multi-layer 2614 network does not imply the use of multiple technologies, although 2615 some form of encapsulation is often applied. 2617 Multi-layer traffic engineering presents a number of challenges 2618 associated with scalability and confidentiality. These issues are 2619 addressed in [RFC7926] which discusses the sharing of information 2620 between domains through policy filters, aggregation, abstraction, and 2621 virtualization. That document also discusses how existing protocols 2622 can support this scenario with special reference to BGP-LS (see 2623 Section 5.1.3.9). 2625 PCE (see Section 5.1.3.10) is also a useful tool for multi-layer 2626 networks as described in [RFC6805], [RFC8685], and [RFC5623]. 2627 Signaling techniques for multi-layer TE are described in [RFC6107]. 2629 See also Appendix A.3.1 for a discussion of how the overlay model has 2630 been important in the development of TE, and Section 6.6 for 2631 examination of multi-layer network survivability. 2633 6.8. Traffic Engineering in Diffserv Environments 2635 Increasing requirements to support multiple classes of traffic in the 2636 Internet, such as best effort and mission critical data, calls for IP 2637 networks to differentiate traffic according to some criteria and to 2638 give preferential treatment to certain types of traffic. Large 2639 numbers of flows can be aggregated into a few behavior aggregates 2640 based on some criteria based on common performance requirements in 2641 terms of packet loss ratio, delay, and jitter, or in terms of common 2642 fields within the IP packet headers. 2644 Differentiated Services (Diffserv) [RFC2475] can be used to ensure 2645 that SLAs defined to differentiate between traffic flows are met. 2646 Classes of service (CoS) can be supported in a Diffserv environment 2647 by concatenating per-hop behaviors (PHBs) along the routing path. A 2648 PHB is the forwarding behavior that a packet receives at a Diffserv- 2649 compliant node, and it can be configured at each router. PHBs are 2650 delivered using buffer management and packet scheduling mechanisms 2651 and require that the ingress nodes use traffic classification, 2652 marking, policing, and shaping. 2654 TE can complement Diffserv to improve utilization of network 2655 resources. TE can be operated on an aggregated basis across all 2656 service classes [RFC3270], or on a per service class basis. The 2657 former is used to provide better distribution of the traffic load 2658 over the network resources (see [RFC3270] for detailed mechanisms to 2659 support aggregate TE). The latter case is discussed below since it 2660 is specific to the Diffserv environment, with so called Diffserv- 2661 aware traffic engineering [RFC4124]. 2663 For some Diffserv networks, it may be desirable to control the 2664 performance of some service classes by enforcing relationships 2665 between the traffic workload contributed by each service class and 2666 the amount of network resources allocated or provisioned for that 2667 service class. Such relationships between demand and resource 2668 allocation can be enforced using a combination of, for example: 2670 * TE mechanisms on a per service class basis that enforce the 2671 relationship between the amount of traffic contributed by a given 2672 service class and the resources allocated to that class. 2674 * Mechanisms that dynamically adjust the resources allocated to a 2675 given service class to relate to the amount of traffic contributed 2676 by that service class. 2678 It may also be desirable to limit the performance impact of high 2679 priority traffic on relatively low priority traffic. This can be 2680 achieved, for example, by controlling the percentage of high priority 2681 traffic that is routed through a given link. Another way to 2682 accomplish this is to increase link capacities appropriately so that 2683 lower priority traffic can still enjoy adequate service quality. 2684 When the ratio of traffic workload contributed by different service 2685 classes varies significantly from router to router, it may not be 2686 enough to rely on conventional IGP routing protocols or on TE 2687 mechanisms that are not sensitive to different service classes. 2688 Instead, it may be desirable to perform TE, especially routing 2689 control and mapping functions, on a per service class basis. One way 2690 to accomplish this in a domain that supports both MPLS and Diffserv 2691 is to define class specific LSPs and to map traffic from each class 2692 onto one or more LSPs that correspond to that service class. An LSP 2693 corresponding to a given service class can then be routed and 2694 protected/restored in a class dependent manner, according to specific 2695 policies. 2697 Performing TE on a per class basis may require per-class parameters 2698 to be distributed. It is common to have some classes share some 2699 aggregate constraints (e.g., maximum bandwidth requirement) without 2700 enforcing the constraint on each individual class. These classes can 2701 be grouped into class-types, and per-class-type parameters can be 2702 distributed to improve scalability. This also allows better 2703 bandwidth sharing between classes in the same class-type. A class- 2704 type is a set of classes that satisfy the following two conditions: 2706 * Classes in the same class-type have common aggregate requirements 2707 to satisfy required performance levels. 2709 * There is no requirement to be enforced at the level of an 2710 individual class in the class-type. Note that it is, 2711 nevertheless, still possible to implement some priority policies 2712 for classes in the same class-type to permit preferential access 2713 to the class-type bandwidth through the use of preemption 2714 priorities. 2716 See [RFC4124] for detailed requirements on Diffserv-aware TE. 2718 6.9. Network Controllability 2720 Offline and online (see Section 4.2) TE considerations are of limited 2721 utility if the network cannot be controlled effectively to implement 2722 the results of TE decisions and to achieve the desired network 2723 performance objectives. 2725 Capacity augmentation is a coarse-grained solution to TE issues. 2726 However, it is simple and may be advantageous if bandwidth is 2727 abundant and cheap. However, bandwidth is not always abundant and 2728 cheap, and additional capacity might not always be the best solution. 2730 Adjustments of administrative weights and other parameters associated 2731 with routing protocols provide finer-grained control, but this 2732 approach is difficult to use and imprecise because of the the way the 2733 routing protocols interact occur across the network. 2735 Control mechanisms can be manual (e.g., static configuration), 2736 partially-automated (e.g., scripts), or fully-automated (e.g., policy 2737 based management systems). Automated mechanisms are particularly 2738 useful in large scale networks. Multi-vendor interoperability can be 2739 facilitated by standardized management systems (e.g., YANG models) to 2740 support the control functions required to address TE objectives. 2742 Network control functions should be secure, reliable, and stable as 2743 these are often needed to operate correctly in times of network 2744 impairments (e.g., during network congestion or security attacks). 2746 7. Inter-Domain Considerations 2748 Inter-domain TE is concerned with performance optimization for 2749 traffic that originates in one administrative domain and terminates 2750 in a different one. 2752 BGP [RFC4271] is the standard exterior gateway protocol used to 2753 exchange routing information between autonomous systems (ASes) in the 2754 Internet. BGP includes a sequential decision process that calculates 2755 the preference for routes to a given destination network. There are 2756 two fundamental aspects to inter-domain TE using BGP: 2758 * Route Redistribution: Controlling the import and export of routes 2759 between ASes, and controlling the redistribution of routes between 2760 BGP and other protocols within an AS. 2762 * Best path selection: Selecting the best path when there are 2763 multiple candidate paths to a given destination network. This is 2764 performed by the BGP decision process, selecting preferred exit 2765 points out of an AS towards specific destination networks taking a 2766 number of different considerations into account. The BGP path 2767 selection process can be influenced by manipulating the attributes 2768 associated with the process, including NEXT-HOP, WEIGHT, LOCAL- 2769 PREFERENCE, AS-PATH, ROUTE-ORIGIN, MULTI-EXIT-DESCRIMINATOR (MED), 2770 IGP METRIC, etc. 2772 Route-maps provide the flexibility to implement complex BGP policies 2773 based on pre-configured logical conditions. They can be used to 2774 control import and export policies for incoming and outgoing routes, 2775 control the redistribution of routes between BGP and other protocols, 2776 and influence the selection of best paths by manipulating the 2777 attributes associated with the BGP decision process. Very complex 2778 logical expressions that implement various types of policies can be 2779 implemented using a combination of Route-maps, BGP-attributes, 2780 Access-lists, and Community attributes. 2782 When considering inter-domain TE with BGP, note that the outbound 2783 traffic exit point is controllable, whereas the interconnection point 2784 where inbound traffic is received typically is not. Therefore, it is 2785 up to each individual network to implement TE strategies that deal 2786 with the efficient delivery of outbound traffic from its customers to 2787 its peering points. The vast majority of TE policy is based on a 2788 "closest exit" strategy, which offloads inter-domain traffic at the 2789 nearest outbound peering point towards the destination AS. Most 2790 methods of manipulating the point at which inbound traffic enters a 2791 are either ineffective, or not accepted in the peering community. 2793 Inter-domain TE with BGP is generally effective, but it is usually 2794 applied in a trial-and-error fashion because a TE system usually only 2795 has a view of the available network resources within one domain (an 2796 AS in this case). A systematic approach for inter-domain TE requires 2797 cooperation between the domains. Further, what may be considered a 2798 good solution in one domain may not necessarily be a good solution in 2799 another. Moreover, it is generally considered inadvisable for one 2800 domain to permit a control process from another domain to influence 2801 the routing and management of traffic in its network. 2803 MPLS TE-tunnels (LSPs) can add a degree of flexibility in the 2804 selection of exit points for inter-domain routing by applying rhe 2805 concept of relative and absolute metrics. If BGP attributes are 2806 defined such that the BGP decision process depends on IGP metrics to 2807 select exit points for inter-domain traffic, then some inter-domain 2808 traffic destined to a given peer network can be made to prefer a 2809 specific exit point by establishing a TE-tunnel between the router 2810 making the selection and the peering point via a TE-tunnel and 2811 assigning the TE-tunnel a metric which is smaller than the IGP cost 2812 to all other peering points. RSVP-TE protocol extensions for inter- 2813 domain MPLS and GMPLS are described in [RFC5151]. 2815 Similarly to intra-domain TE, inter-domain TE is best accomplished 2816 when a traffic matrix can be derived to depict the volume of traffic 2817 from one AS to another. 2819 8. Overview of Contemporary TE Practices in Operational IP Networks 2821 This section provides an overview of some TE practices in IP 2822 networks. The focus is on aspects of control of the routing function 2823 in operational contexts. The intent here is to provide an overview 2824 of the commonly used practices: the discussion is not intended to be 2825 exhaustive. 2827 Service providers apply many of the TE mechanisms described in this 2828 document to optimize the performance of their IP networks. These 2829 techniques include capacity planning for long timescales; routing 2830 control using IGP metrics and MPLS, as well as path planning and path 2831 control using MPLS and Segment Routing for medium timescales; and 2832 traffic management mechanisms for short timescale. 2834 Capacity planning is an important component of how a service provider 2835 plans an effective IP network. These plans may take the following 2836 aspects into account: location of and new links or nodes, existing 2837 and predicted traffic patterns, costs, link capacity, topology, 2838 routing design, and survivability. 2840 Performance optimization of operational networks is usually an 2841 ongoing process in which traffic statistics, performance parameters, 2842 and fault indicators are continually collected from the network. 2843 This empirical data is analyzed and used to trigger TE mechanisms. 2844 Tools that perform what-if analysis can also be used to assist the TE 2845 process by reviewing scenarios before a new set of configurations are 2846 implemented in the operational network. 2848 Real-time intra-domain TE using the IGP is done by increasing the 2849 OSPF or IS-IS metric of a congested link until enough traffic has 2850 been diverted away from that link. This approach has some 2851 limitations as discussed in Section 6.2. Intra-domain TE approaches 2852 ([RR94] [FT00] [FT01] [WANG]) take traffic matrix, network topology, 2853 and network performance objectives as input, and produce link metrics 2854 and load-sharing ratios. These processes open the possibility for 2855 intra-domain TE with IGP to be done in a more systematic way. 2857 Administrators of MPLS-TE networks specify and configure link 2858 attributes and resource constraints such as maximum reservable 2859 bandwidth and resource class attributes for the links in the domain. 2860 A link state IGP that supports TE extensions (IS-IS-TE or OSPF-TE) is 2861 used to propagate information about network topology and link 2862 attributes to all routers in the domain. Network administrators 2863 specify the LSPs that are to originate at each router. For each LSP, 2864 the network administrator specifies the destination node and the 2865 attributes of the LSP which indicate the requirements that are to be 2866 satisfied during the path selection process. The attributes may 2867 include and explicit path for the LSP to follow, or originating 2868 router uses a local constraint-based routing process to compute the 2869 path of the LSP. RSVP-TE is used as a signaling protocol to 2870 instantiate the LSPs. By assigning proper bandwidth values to links 2871 and LSPs, congestion caused by uneven traffic distribution can be 2872 avoided or mitigated. 2874 The bandwidth attributes of an LSP relates to the bandwidth 2875 requirements of traffic that flows through the LSP. The traffic 2876 attribute of an LSP can be modified to accommodate persistent shifts 2877 in demand (traffic growth or reduction). If network congestion 2878 occurs due to some unexpected events, existing LSPs can be rerouted 2879 to alleviate the situation or network administrator can configure new 2880 LSPs to divert some traffic to alternative paths. The reservable 2881 bandwidth of the congested links can also be reduced to force some 2882 LSPs to be rerouted to other paths. A traffic matrix in an MPLS 2883 domain can also be estimated by monitoring the traffic on LSPs. Such 2884 traffic statistics can be used for a variety of purposes including 2885 network planning and network optimization. 2887 Network management and planning systems have evolved and taken over a 2888 lot of the responsibility for determining traffic paths in TE 2889 networks. This allows a network-wide view of resources, and 2890 facilitates coordination of the use of resources for all traffic 2891 flows in the network. Initial solutions using a PCE to perform path 2892 computation on behalf of network routers have given way to an 2893 approach that follows the SDN architecture. A stateful PCE is able 2894 to track all of the LSPs in the network and can redistribute them to 2895 make better use of the available resources. Such a PCE can forms 2896 part of a network orchestrator that uses PCEP or some other 2897 southbound interface to instruct the signaling protocol or directly 2898 program the routers. 2900 Segment routing leverages a centralized TE controller and either an 2901 MPLS or IPv6 forwarding plane, but does not need to use a signaling 2902 protocol or management plane protocol to reserve resources in the 2903 routers. All resource reservation is logical within the controller, 2904 and not distributed to the routers. Packets are steered through the 2905 network using segment routing, and this may have configuration and 2906 operational scaling benefits. 2908 As mentioned in Section 7, there is usually no direct control over 2909 the distribution of inbound traffic to a domain. Therefore, the main 2910 goal of inter-domain TE is to optimize the distribution of outbound 2911 traffic between multiple inter-domain links. When operating a global 2912 network, maintaining the ability to operate the network in a regional 2913 fashion where desired, while continuing to take advantage of the 2914 benefits of a global network, also becomes an important objective. 2916 Inter-domain TE with BGP begins with the placement of multiple 2917 peering interconnection points that are in close proximity to traffic 2918 sources/destination, and offer lowest cost paths across the network 2919 between the peering points and the sources/destinations. Some 2920 location-decision problems that arise in association with inter- 2921 domain routing are discussed in [AWD5]. 2923 Once the locations of the peering interconnects have been determined 2924 and implemented, the network operator decides how best to handle the 2925 routes advertised by the peer, as well as how to propagate the peer's 2926 routes within their network. One way to engineer outbound traffic 2927 flows in a network with many peering interconnects is to create a 2928 hierarchy of peers. Generally, the shortest AS paths will be chosen 2929 to forward traffic but BGP metrics can be used to prefer some peers 2930 and so favor particular paths. Preferred peers are those peers 2931 attached through peering interconnects with the most available 2932 capacity. Changes may be needed, for example, to deal with a 2933 "problem peer" who is difficult to work with on upgrades or is 2934 charging high prices for connectivity to their network. In that 2935 case, the peer may be given a reduced preference. This type of 2936 change can affect a large amount of traffic, and is only used after 2937 other methods have failed to provide the desired results. 2939 When there are multiple exit points toward a given peer, and only one 2940 of them is congested, it is not necessary to shift traffic away from 2941 the peer entirely, but only from the one congested connections. This 2942 can be achieved by using passive IGP-metrics, AS-path filtering, or 2943 prefix filtering. 2945 9. Security Considerations 2947 This document does not introduce new security issues. 2949 Network security is, of course, an important issue. In general, TE 2950 mechanisms are security neutral: they may use tunnels which can 2951 slightly help protect traffic from inspection and which, in some 2952 cases, can be secured using encryption; they put traffic onto 2953 predictable paths within the network that may make it easier to find 2954 and attack; they increase the complexity or operation and management 2955 of the network; and they enable traffic to be steered onto more 2956 secure links or to more secure parts of the network. 2958 The consequences of attacks on the control and management protocols 2959 used to operate TE networks can be significant: traffic can be 2960 hijacked to pass through specific nodes that perform inspection, or 2961 even to be delivered to the wrong place; traffic can be steered onto 2962 paths that deliver quality that is below the desired quality; and, 2963 networks can be congested or have resources on key links consumed. 2964 Thus, it is important to use adequate protection mechanisms on all 2965 protocols used to deliver TE. 2967 Certain aspects of a network may be deduced from the details of the 2968 TE paths that are used. For example, the link connectivity of the 2969 network, and the quality and load on individual links may be assumed 2970 from knowing the paths of traffic and the requirements they place on 2971 the network (for example, by seeing the control messages or through 2972 path- trace techniques). Such knowledge can be used to launch 2973 targeted attacks (for example, taking down critical links) or can 2974 reveal commercially sensitive information (for example, whether a 2975 network is close to capacity). Network operators may, therefore, 2976 choose techniques that mask or hide information from within the 2977 network. 2979 10. IANA Considerations 2981 This draft makes no requests for IANA action. 2983 11. Acknowledgments 2985 Much of the text in this document is derived from RFC 3272. The 2986 authors of this document would like to express their gratitude to all 2987 involved in that work. Although the source text has been edited in 2988 the production of this document, the original authors should be 2989 considered as Contributors to this work. They were: 2991 Daniel O. Awduche 2992 Movaz Networks 2994 Angela Chiu 2995 Celion Networks 2997 Anwar Elwalid 2998 Lucent Technologies 3000 Indra Widjaja 3001 Bell Labs, Lucent Technologies 3003 XiPeng Xiao 3004 Redback Networks 3006 The acknowledgements in RFC3272 were as below. All people who helped 3007 in the production of that document also need to be thanked for the 3008 carry-over into this new document. 3010 The authors would like to thank Jim Boyle for inputs on the 3011 recommendations section, Francois Le Faucheur for inputs on 3012 Diffserv aspects, Blaine Christian for inputs on measurement, 3013 Gerald Ash for inputs on routing in telephone networks and for 3014 text on event-dependent TE methods, Steven Wright for inputs 3015 on network controllability, and Jonathan Aufderheide for 3016 inputs on inter-domain TE with BGP. Special thanks to 3017 Randy Bush for proposing the TE taxonomy based on "tactical versus 3018 strategic" methods. The subsection describing an "Overview of 3019 ITU Activities Related to Traffic Engineering" was adapted from 3020 a contribution by Waisum Lai. Useful feedback and pointers to 3021 relevant materials were provided by J. Noel Chiappa. 3022 Additional comments were provided by Glenn Grotefeld during 3023 the working last call process. Finally, the authors would like 3024 to thank Ed Kern, the TEWG co-chair, for his comments and 3025 support. 3027 The early versions of this document were produced by the TEAS Working 3028 Group's RFC3272bis Design Team. The full list of members of this 3029 team is: 3031 Acee Lindem 3032 Adrian Farrel 3033 Aijun Wang 3034 Daniele Ceccarelli 3035 Dieter Beller 3036 Jeff Tantsura 3037 Julien Meuric 3038 Liu Hua 3039 Loa Andersson 3040 Luis Miguel Contreras 3041 Martin Horneffer 3042 Tarek Saad 3043 Xufeng Liu 3045 The production of this document includes a fix to the original text 3046 resulting from an Errata Report by Jean-Michel Grimaldi. 3048 The author of this document would also like to thank Dhurv Dhody, 3049 Gyan Mishra, and Dave Taht for review comments. 3051 This work is partially supported by the European Commission under 3052 Horizon 2020 grant agreement number 101015857 Secured autonomic 3053 traffic management for a Tera of SDN flows (Teraflow). 3055 12. Contributors 3057 The following people contributed substantive text to this document: 3059 Gert Grammel 3060 EMail: ggrammel@juniper.net 3062 Loa Andersson 3063 EMail: loa@pi.nu 3065 Xufeng Liu 3066 EMail: xufeng.liu.ietf@gmail.com 3068 Lou Berger 3069 EMail: lberger@labn.net 3071 Jeff Tantsura 3072 EMail: jefftant.ietf@gmail.com 3074 Daniel King 3075 EMail: daniel@olddog.co.uk 3077 Boris Hassanov 3078 EMail: bhassanov@yandex-team.ru 3080 Kiran Makhijani 3081 Email: kiranm@futurewei.com 3083 Dhruv Dhody 3084 Email: dhruv.ietf@gmail.com 3086 Mohamed Boucadair 3087 Email: mohamed.boucadair@orange.com 3089 13. Informative References 3091 [AJ19] Adekitan, A., Abolade, J., and O. Shobayo, "Data mining 3092 approach for predicting the daily Internet data traffic of 3093 a smart university", Article Journal of Big Data, 2019, 3094 Volume 6, Number 1, Page 1, 1998, 3095 . 3098 [ASH2] Ash, J., "Dynamic Routing in Telecommunications Networks", 3099 Book McGraw Hill, 1998, 3100 . 3102 [ATSSS] "Study on access traffic steering, switch and splitting 3103 support in the 5G System (5GS) architecture", 3104 Specification 3GPP Technical Report 23.793 Release 16, 3105 December 2018, . 3108 [AWD2] Awduche, D., "MPLS and Traffic Engineering in IP 3109 Networks", Article IEEE Communications Magazine, December 3110 1999, . 3112 [AWD5] Awduche, D., "An Approach to Optimal Peering Between 3113 Autonomous Systems in the Internet", Paper International 3114 Conference on Computer Communications and Networks 3115 (ICCCN'98), October 1998, 3116 . 3118 [FLJA93] Floyd, S. and V. Jacobson, "Random Early Detection 3119 Gateways for Congestion Avoidance", Article IEEE/ACM 3120 Transactions on Networking, Vol. 1, p. 387-413, November 3121 1993, 3122 . 3124 [FLOY94] Floyd, S., "TCP and Explicit Congestion Notification", 3125 Article ACM Computer Communication Review, V. 24, No. 5, 3126 p. 10-23, October 1994, 3127 . 3129 [FT00] Fortz, B. and M. Thorup, "Internet Traffic Engineering by 3130 Optimizing OSPF Weights", Article IEEE INFOCOM 2000, March 3131 2000, . 3134 [FT01] Fortz, B. and M. Thorup, "Optimizing OSPF/IS-IS Weights in 3135 a Changing World", n.d., 3136 . 3138 [HUSS87] Hurley, B.R., Seidl, C.J.R., and W.F. Sewel, "A Survey of 3139 Dynamic Routing Methods for Circuit-Switched Traffic", 3140 Article IEEE Communication Magazine, September 1987, 3141 . 3143 [I-D.ietf-alto-performance-metrics] 3144 Wu, Q., Yang, Y. R., Lee, Y., Dhody, D., Randriamasy, S., 3145 and L. M. C. Murillo, "ALTO Performance Cost Metrics", 3146 Work in Progress, Internet-Draft, draft-ietf-alto- 3147 performance-metrics-28, 21 March 2022, 3148 . 3151 [I-D.ietf-bess-evpn-unequal-lb] 3152 Malhotra, N., Sajassi, A., Rabadan, J., Drake, J., 3153 Lingala, A., and S. Thoria, "Weighted Multi-Path 3154 Procedures for EVPN Multi-Homing", Work in Progress, 3155 Internet-Draft, draft-ietf-bess-evpn-unequal-lb-15, 17 3156 November 2021, . 3159 [I-D.ietf-idr-performance-routing] 3160 Xu, X., Hegde, S., Talaulikar, K., Boucadair, M., and C. 3161 Jacquenet, "Performance-based BGP Routing Mechanism", Work 3162 in Progress, Internet-Draft, draft-ietf-idr-performance- 3163 routing-03, 22 December 2020, 3164 . 3167 [I-D.ietf-idr-segment-routing-te-policy] 3168 Previdi, S., Filsfils, C., Talaulikar, K., Mattes, P., 3169 Jain, D., and S. Lin, "Advertising Segment Routing 3170 Policies in BGP", Work in Progress, Internet-Draft, draft- 3171 ietf-idr-segment-routing-te-policy-16, 7 March 2022, 3172 . 3175 [I-D.ietf-lsr-flex-algo] 3176 Psenak, P., Hegde, S., Filsfils, C., Talaulikar, K., and 3177 A. Gulko, "IGP Flexible Algorithm", Work in Progress, 3178 Internet-Draft, draft-ietf-lsr-flex-algo-18, 25 October 3179 2021, . 3182 [I-D.ietf-lsr-ip-flexalgo] 3183 Britto, W., Hegde, S., Kaneriya, P., Shetty, R., Bonica, 3184 R., and P. Psenak, "IGP Flexible Algorithms (Flex- 3185 Algorithm) In IP Networks", Work in Progress, Internet- 3186 Draft, draft-ietf-lsr-ip-flexalgo-04, 19 December 2021, 3187 . 3190 [I-D.ietf-rtgwg-segment-routing-ti-lfa] 3191 Litkowski, S., Bashandy, A., Filsfils, C., Francois, P., 3192 Decraene, B., and D. Voyer, "Topology Independent Fast 3193 Reroute using Segment Routing", Work in Progress, 3194 Internet-Draft, draft-ietf-rtgwg-segment-routing-ti-lfa- 3195 08, 21 January 2022, . 3198 [I-D.ietf-spring-segment-routing-policy] 3199 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 3200 P. Mattes, "Segment Routing Policy Architecture", Work in 3201 Progress, Internet-Draft, draft-ietf-spring-segment- 3202 routing-policy-21, 19 March 2022, 3203 . 3206 [I-D.ietf-teas-enhanced-vpn] 3207 Dong, J., Bryant, S., Li, Z., Miyasaka, T., and Y. Lee, "A 3208 Framework for Enhanced Virtual Private Network (VPN+) 3209 Services", Work in Progress, Internet-Draft, draft-ietf- 3210 teas-enhanced-vpn-10, 6 March 2022, 3211 . 3214 [I-D.ietf-teas-ietf-network-slices] 3215 Farrel, A., Drake, J., Rokui, R., Homma, S., Makhijani, 3216 K., Contreras, L. M., and J. Tantsura, "Framework for IETF 3217 Network Slices", Work in Progress, Internet-Draft, draft- 3218 ietf-teas-ietf-network-slices-08, 6 March 2022, 3219 . 3222 [I-D.ietf-tewg-qos-routing] 3223 Ash, G., "Traffic Engineering & QoS Methods for IP-, ATM-, 3224 & Based Multiservice Networks", Work in Progress, 3225 Internet-Draft, draft-ietf-tewg-qos-routing-04, 15 October 3226 2001, . 3229 [I-D.irtf-nmrg-ibn-concepts-definitions] 3230 Clemm, A., Ciavaglia, L., Granville, L. Z., and J. 3231 Tantsura, "Intent-Based Networking - Concepts and 3232 Definitions", Work in Progress, Internet-Draft, draft- 3233 irtf-nmrg-ibn-concepts-definitions-06, 15 December 2021, 3234 . 3237 [ITU-E600] "Terms and Definitions of Traffic Engineering", 3238 Recommendation ITU-T Recommendation E.600, March 1993, 3239 . 3241 [ITU-E701] "Reference Connections for Traffic Engineering", 3242 Recommendation ITU-T Recommendation E.701, October 1993, 3243 . 3245 [ITU-E801] "Framework for Service Quality Agreement", 3246 Recommendation ITU-T Recommendation E.801, October 1996, 3247 . 3249 [MA] Ma, Q., "Quality of Service Routing in Integrated Services 3250 Networks", Ph.D. PhD Dissertation, CMU-CS-98-138, CMU, 3251 1998, . 3253 [MATE] Elwalid, A., Jin, C., Low, S., and I. Widjaja, "MATE - 3254 MPLS Adaptive Traffic Engineering", 3255 Proceedings INFOCOM'01, April 2001, 3256 . 3259 [MCQ80] McQuillan, J.M., Richer, I., and E.C. Rosen, "The New 3260 Routing Algorithm for the ARPANET", Transaction IEEE 3261 Transactions on Communications, vol. 28, no. 5, p. 3262 711-719, May 1980, . 3265 [MR99] Mitra, D. and K.G. Ramakrishnan, "A Case Study of 3266 Multiservice, Multipriority Traffic Engineering Design for 3267 Data Networks", Proceedings Globecom'99, December 1999, 3268 . 3270 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 3271 DOI 10.17487/RFC0791, September 1981, 3272 . 3274 [RFC1102] Clark, D., "Policy routing in Internet protocols", 3275 RFC 1102, DOI 10.17487/RFC1102, May 1989, 3276 . 3278 [RFC1104] Braun, H., "Models of policy based routing", RFC 1104, 3279 DOI 10.17487/RFC1104, June 1989, 3280 . 3282 [RFC1992] Castineyra, I., Chiappa, N., and M. Steenstrup, "The 3283 Nimrod Routing Architecture", RFC 1992, 3284 DOI 10.17487/RFC1992, August 1996, 3285 . 3287 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 3288 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 3289 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 3290 September 1997, . 3292 [RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, 3293 DOI 10.17487/RFC2328, April 1998, 3294 . 3296 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 3297 "Framework for IP Performance Metrics", RFC 2330, 3298 DOI 10.17487/RFC2330, May 1998, 3299 . 3301 [RFC2386] Crawley, E., Nair, R., Rajagopalan, B., and H. Sandick, "A 3302 Framework for QoS-based Routing in the Internet", 3303 RFC 2386, DOI 10.17487/RFC2386, August 1998, 3304 . 3306 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 3307 "Definition of the Differentiated Services Field (DS 3308 Field) in the IPv4 and IPv6 Headers", RFC 2474, 3309 DOI 10.17487/RFC2474, December 1998, 3310 . 3312 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 3313 and W. Weiss, "An Architecture for Differentiated 3314 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 3315 . 3317 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 3318 "Assured Forwarding PHB Group", RFC 2597, 3319 DOI 10.17487/RFC2597, June 1999, 3320 . 3322 [RFC2678] Mahdavi, J. and V. Paxson, "IPPM Metrics for Measuring 3323 Connectivity", RFC 2678, DOI 10.17487/RFC2678, September 3324 1999, . 3326 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 3327 McManus, "Requirements for Traffic Engineering Over MPLS", 3328 RFC 2702, DOI 10.17487/RFC2702, September 1999, 3329 . 3331 [RFC2722] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow 3332 Measurement: Architecture", RFC 2722, 3333 DOI 10.17487/RFC2722, October 1999, 3334 . 3336 [RFC2753] Yavatkar, R., Pendarakis, D., and R. Guerin, "A Framework 3337 for Policy-based Admission Control", RFC 2753, 3338 DOI 10.17487/RFC2753, January 2000, 3339 . 3341 [RFC2961] Berger, L., Gan, D., Swallow, G., Pan, P., Tommasi, F., 3342 and S. Molendini, "RSVP Refresh Overhead Reduction 3343 Extensions", RFC 2961, DOI 10.17487/RFC2961, April 2001, 3344 . 3346 [RFC2998] Bernet, Y., Ford, P., Yavatkar, R., Baker, F., Zhang, L., 3347 Speer, M., Braden, R., Davie, B., Wroclawski, J., and E. 3348 Felstaine, "A Framework for Integrated Services Operation 3349 over Diffserv Networks", RFC 2998, DOI 10.17487/RFC2998, 3350 November 2000, . 3352 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 3353 Label Switching Architecture", RFC 3031, 3354 DOI 10.17487/RFC3031, January 2001, 3355 . 3357 [RFC3086] Nichols, K. and B. Carpenter, "Definition of 3358 Differentiated Services Per Domain Behaviors and Rules for 3359 their Specification", RFC 3086, DOI 10.17487/RFC3086, 3360 April 2001, . 3362 [RFC3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager", 3363 RFC 3124, DOI 10.17487/RFC3124, June 2001, 3364 . 3366 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 3367 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 3368 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 3369 . 3371 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 3372 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 3373 Protocol Label Switching (MPLS) Support of Differentiated 3374 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 3375 . 3377 [RFC3272] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 3378 Xiao, "Overview and Principles of Internet Traffic 3379 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 3380 . 3382 [RFC3469] Sharma, V., Ed. and F. Hellstrand, Ed., "Framework for 3383 Multi-Protocol Label Switching (MPLS)-based Recovery", 3384 RFC 3469, DOI 10.17487/RFC3469, February 2003, 3385 . 3387 [RFC3630] Katz, D., Kompella, K., and D. Yeung, "Traffic Engineering 3388 (TE) Extensions to OSPF Version 2", RFC 3630, 3389 DOI 10.17487/RFC3630, September 2003, 3390 . 3392 [RFC3945] Mannie, E., Ed., "Generalized Multi-Protocol Label 3393 Switching (GMPLS) Architecture", RFC 3945, 3394 DOI 10.17487/RFC3945, October 2004, 3395 . 3397 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 3398 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 3399 DOI 10.17487/RFC4090, May 2005, 3400 . 3402 [RFC4124] Le Faucheur, F., Ed., "Protocol Extensions for Support of 3403 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 3404 DOI 10.17487/RFC4124, June 2005, 3405 . 3407 [RFC4203] Kompella, K., Ed. and Y. Rekhter, Ed., "OSPF Extensions in 3408 Support of Generalized Multi-Protocol Label Switching 3409 (GMPLS)", RFC 4203, DOI 10.17487/RFC4203, October 2005, 3410 . 3412 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 3413 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 3414 DOI 10.17487/RFC4271, January 2006, 3415 . 3417 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 3418 Guidelines for DiffServ Service Classes", RFC 4594, 3419 DOI 10.17487/RFC4594, August 2006, 3420 . 3422 [RFC4655] Farrel, A., Vasseur, J.-P., and J. Ash, "A Path 3423 Computation Element (PCE)-Based Architecture", RFC 4655, 3424 DOI 10.17487/RFC4655, August 2006, 3425 . 3427 [RFC4872] Lang, J.P., Ed., Rekhter, Y., Ed., and D. Papadimitriou, 3428 Ed., "RSVP-TE Extensions in Support of End-to-End 3429 Generalized Multi-Protocol Label Switching (GMPLS) 3430 Recovery", RFC 4872, DOI 10.17487/RFC4872, May 2007, 3431 . 3433 [RFC4873] Berger, L., Bryskin, I., Papadimitriou, D., and A. Farrel, 3434 "GMPLS Segment Recovery", RFC 4873, DOI 10.17487/RFC4873, 3435 May 2007, . 3437 [RFC4920] Farrel, A., Ed., Satyanarayana, A., Iwata, A., Fujita, N., 3438 and G. Ash, "Crankback Signaling Extensions for MPLS and 3439 GMPLS RSVP-TE", RFC 4920, DOI 10.17487/RFC4920, July 2007, 3440 . 3442 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 3443 Domain MPLS and GMPLS Traffic Engineering -- Resource 3444 Reservation Protocol-Traffic Engineering (RSVP-TE) 3445 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 3446 2008, . 3448 [RFC5250] Berger, L., Bryskin, I., Zinin, A., and R. Coltun, "The 3449 OSPF Opaque LSA Option", RFC 5250, DOI 10.17487/RFC5250, 3450 July 2008, . 3452 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 3453 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 3454 2008, . 3456 [RFC5329] Ishiguro, K., Manral, V., Davey, A., and A. Lindem, Ed., 3457 "Traffic Engineering Extensions to OSPF Version 3", 3458 RFC 5329, DOI 10.17487/RFC5329, September 2008, 3459 . 3461 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 3462 Label Assignment and Context-Specific Label Space", 3463 RFC 5331, DOI 10.17487/RFC5331, August 2008, 3464 . 3466 [RFC5394] Bryskin, I., Papadimitriou, D., Berger, L., and J. Ash, 3467 "Policy-Enabled Path Computation Framework", RFC 5394, 3468 DOI 10.17487/RFC5394, December 2008, 3469 . 3471 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 3472 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 3473 DOI 10.17487/RFC5440, March 2009, 3474 . 3476 [RFC5541] Le Roux, JL., Vasseur, JP., and Y. Lee, "Encoding of 3477 Objective Functions in the Path Computation Element 3478 Communication Protocol (PCEP)", RFC 5541, 3479 DOI 10.17487/RFC5541, June 2009, 3480 . 3482 [RFC5557] Lee, Y., Le Roux, JL., King, D., and E. Oki, "Path 3483 Computation Element Communication Protocol (PCEP) 3484 Requirements and Protocol Extensions in Support of Global 3485 Concurrent Optimization", RFC 5557, DOI 10.17487/RFC5557, 3486 July 2009, . 3488 [RFC5623] Oki, E., Takeda, T., Le Roux, JL., and A. Farrel, 3489 "Framework for PCE-Based Inter-Layer MPLS and GMPLS 3490 Traffic Engineering", RFC 5623, DOI 10.17487/RFC5623, 3491 September 2009, . 3493 [RFC5664] Halevy, B., Welch, B., and J. Zelenka, "Object-Based 3494 Parallel NFS (pNFS) Operations", RFC 5664, 3495 DOI 10.17487/RFC5664, January 2010, 3496 . 3498 [RFC5693] Seedorf, J. and E. Burger, "Application-Layer Traffic 3499 Optimization (ALTO) Problem Statement", RFC 5693, 3500 DOI 10.17487/RFC5693, October 2009, 3501 . 3503 [RFC6107] Shiomoto, K., Ed. and A. Farrel, Ed., "Procedures for 3504 Dynamically Signaled Hierarchical Label Switched Paths", 3505 RFC 6107, DOI 10.17487/RFC6107, February 2011, 3506 . 3508 [RFC6119] Harrison, J., Berger, J., and M. Bartlett, "IPv6 Traffic 3509 Engineering in IS-IS", RFC 6119, DOI 10.17487/RFC6119, 3510 February 2011, . 3512 [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., 3513 and A. Bierman, Ed., "Network Configuration Protocol 3514 (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, 3515 . 3517 [RFC6372] Sprecher, N., Ed. and A. Farrel, Ed., "MPLS Transport 3518 Profile (MPLS-TP) Survivability Framework", RFC 6372, 3519 DOI 10.17487/RFC6372, September 2011, 3520 . 3522 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 3523 Measurement for MPLS Networks", RFC 6374, 3524 DOI 10.17487/RFC6374, September 2011, 3525 . 3527 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 3528 "IPv6 Flow Label Specification", RFC 6437, 3529 DOI 10.17487/RFC6437, November 2011, 3530 . 3532 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 3533 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 3534 RFC 6790, DOI 10.17487/RFC6790, November 2012, 3535 . 3537 [RFC6805] King, D., Ed. and A. Farrel, Ed., "The Application of the 3538 Path Computation Element Architecture to the Determination 3539 of a Sequence of Domains in MPLS and GMPLS", RFC 6805, 3540 DOI 10.17487/RFC6805, November 2012, 3541 . 3543 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 3544 Networking: A Perspective from within a Service Provider 3545 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 3546 . 3548 [RFC7285] Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S., 3549 Previdi, S., Roome, W., Shalunov, S., and R. Woundy, 3550 "Application-Layer Traffic Optimization (ALTO) Protocol", 3551 RFC 7285, DOI 10.17487/RFC7285, September 2014, 3552 . 3554 [RFC7390] Rahman, A., Ed. and E. Dijk, Ed., "Group Communication for 3555 the Constrained Application Protocol (CoAP)", RFC 7390, 3556 DOI 10.17487/RFC7390, October 2014, 3557 . 3559 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 3560 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 3561 Defined Networking (SDN): Layers and Architecture 3562 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 3563 2015, . 3565 [RFC7471] Giacalone, S., Ward, D., Drake, J., Atlas, A., and S. 3566 Previdi, "OSPF Traffic Engineering (TE) Metric 3567 Extensions", RFC 7471, DOI 10.17487/RFC7471, March 2015, 3568 . 3570 [RFC7491] King, D. and A. Farrel, "A PCE-Based Architecture for 3571 Application-Based Network Operations", RFC 7491, 3572 DOI 10.17487/RFC7491, March 2015, 3573 . 3575 [RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF 3576 Recommendations Regarding Active Queue Management", 3577 BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, 3578 . 3580 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 3581 Chaining (SFC) Architecture", RFC 7665, 3582 DOI 10.17487/RFC7665, October 2015, 3583 . 3585 [RFC7679] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3586 Ed., "A One-Way Delay Metric for IP Performance Metrics 3587 (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 3588 2016, . 3590 [RFC7680] Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, 3591 Ed., "A One-Way Loss Metric for IP Performance Metrics 3592 (IPPM)", STD 82, RFC 7680, DOI 10.17487/RFC7680, January 3593 2016, . 3595 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 3596 S. Ray, "North-Bound Distribution of Link-State and 3597 Traffic Engineering (TE) Information Using BGP", RFC 7752, 3598 DOI 10.17487/RFC7752, March 2016, 3599 . 3601 [RFC7923] Voit, E., Clemm, A., and A. Gonzalez Prieto, "Requirements 3602 for Subscription to YANG Datastores", RFC 7923, 3603 DOI 10.17487/RFC7923, June 2016, 3604 . 3606 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 3607 Ceccarelli, D., and X. Zhang, "Problem Statement and 3608 Architecture for Information Exchange between 3609 Interconnected Traffic-Engineered Networks", BCP 206, 3610 RFC 7926, DOI 10.17487/RFC7926, July 2016, 3611 . 3613 [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", 3614 RFC 7950, DOI 10.17487/RFC7950, August 2016, 3615 . 3617 [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF 3618 Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, 3619 . 3621 [RFC8051] Zhang, X., Ed. and I. Minei, Ed., "Applicability of a 3622 Stateful Path Computation Element (PCE)", RFC 8051, 3623 DOI 10.17487/RFC8051, January 2017, 3624 . 3626 [RFC8189] Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost 3627 Application-Layer Traffic Optimization (ALTO)", RFC 8189, 3628 DOI 10.17487/RFC8189, October 2017, 3629 . 3631 [RFC8231] Crabbe, E., Minei, I., Medved, J., and R. Varga, "Path 3632 Computation Element Communication Protocol (PCEP) 3633 Extensions for Stateful PCE", RFC 8231, 3634 DOI 10.17487/RFC8231, September 2017, 3635 . 3637 [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data 3638 Interchange Format", STD 90, RFC 8259, 3639 DOI 10.17487/RFC8259, December 2017, 3640 . 3642 [RFC8281] Crabbe, E., Minei, I., Sivabalan, S., and R. Varga, "Path 3643 Computation Element Communication Protocol (PCEP) 3644 Extensions for PCE-Initiated LSP Setup in a Stateful PCE 3645 Model", RFC 8281, DOI 10.17487/RFC8281, December 2017, 3646 . 3648 [RFC8283] Farrel, A., Ed., Zhao, Q., Ed., Li, Z., and C. Zhou, "An 3649 Architecture for Use of PCE and the PCE Communication 3650 Protocol (PCEP) in a Network with Central Control", 3651 RFC 8283, DOI 10.17487/RFC8283, December 2017, 3652 . 3654 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 3655 Decraene, B., Litkowski, S., and R. Shakir, "Segment 3656 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 3657 July 2018, . 3659 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 3660 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 3661 DOI 10.17487/RFC8453, August 2018, 3662 . 3664 [RFC8570] Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, 3665 D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) 3666 Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 3667 2019, . 3669 [RFC8571] Ginsberg, L., Ed., Previdi, S., Wu, Q., Tantsura, J., and 3670 C. Filsfils, "BGP - Link State (BGP-LS) Advertisement of 3671 IGP Traffic Engineering Performance Metric Extensions", 3672 RFC 8571, DOI 10.17487/RFC8571, March 2019, 3673 . 3675 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 3676 "Deterministic Networking Architecture", RFC 8655, 3677 DOI 10.17487/RFC8655, October 2019, 3678 . 3680 [RFC8664] Sivabalan, S., Filsfils, C., Tantsura, J., Henderickx, W., 3681 and J. Hardwick, "Path Computation Element Communication 3682 Protocol (PCEP) Extensions for Segment Routing", RFC 8664, 3683 DOI 10.17487/RFC8664, December 2019, 3684 . 3686 [RFC8684] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. 3687 Paasch, "TCP Extensions for Multipath Operation with 3688 Multiple Addresses", RFC 8684, DOI 10.17487/RFC8684, March 3689 2020, . 3691 [RFC8685] Zhang, F., Zhao, Q., Gonzalez de Dios, O., Casellas, R., 3692 and D. King, "Path Computation Element Communication 3693 Protocol (PCEP) Extensions for the Hierarchical Path 3694 Computation Element (H-PCE) Architecture", RFC 8685, 3695 DOI 10.17487/RFC8685, December 2019, 3696 . 3698 [RFC8795] Liu, X., Bryskin, I., Beeram, V., Saad, T., Shah, H., and 3699 O. Gonzalez de Dios, "YANG Data Model for Traffic 3700 Engineering (TE) Topologies", RFC 8795, 3701 DOI 10.17487/RFC8795, August 2020, 3702 . 3704 [RFC8803] Bonaventure, O., Ed., Boucadair, M., Ed., Gundavelli, S., 3705 Seo, S., and B. Hesmans, "0-RTT TCP Convert Protocol", 3706 RFC 8803, DOI 10.17487/RFC8803, July 2020, 3707 . 3709 [RFC8896] Randriamasy, S., Yang, R., Wu, Q., Deng, L., and N. 3710 Schwan, "Application-Layer Traffic Optimization (ALTO) 3711 Cost Calendar", RFC 8896, DOI 10.17487/RFC8896, November 3712 2020, . 3714 [RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 3715 Bryant, "Deterministic Networking (DetNet) Data Plane 3716 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 3717 . 3719 [RFC8955] Loibl, C., Hares, S., Raszuk, R., McPherson, D., and M. 3720 Bacher, "Dissemination of Flow Specification Rules", 3721 RFC 8955, DOI 10.17487/RFC8955, December 2020, 3722 . 3724 [RFC9000] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based 3725 Multiplexed and Secure Transport", RFC 9000, 3726 DOI 10.17487/RFC9000, May 2021, 3727 . 3729 [RFC9023] Varga, B., Ed., Farkas, J., Malis, A., and S. Bryant, 3730 "Deterministic Networking (DetNet) Data Plane: IP over 3731 IEEE 802.1 Time-Sensitive Networking (TSN)", RFC 9023, 3732 DOI 10.17487/RFC9023, June 2021, 3733 . 3735 [RR94] Rodrigues, M.A. and K.G. Ramakrishnan, "Optimal Routing in 3736 Shortest Path Data Networks", Proceedings ITS'94, Rio de 3737 Janeiro, Brazil, 1994, 3738 . 3741 [SLDC98] Suter, B., Lakshman, T., Stiliadis, D., and A. Choudhury, 3742 "Design Considerations for Supporting TCP with Per-flow 3743 Queueing", Proceedings INFOCOM'98, p. 299-306, 1998, 3744 . 3746 [WANG] Wang, Y., Wang, Z., and L. Zhang, "Internet traffic 3747 engineering without full mesh overlaying", 3748 Proceedings INFOCOM'2001, April 2001, 3749 . 3751 [XIAO] Xiao, X., Hannan, A., Bailey, B., and L. Ni, "Traffic 3752 Engineering with MPLS in the Internet", Article IEEE 3753 Network Magazine, March 2000, 3754 . 3757 [YARE95] Yang, C. and A. Reddy, "A Taxonomy for Congestion Control 3758 Algorithms in Packet Switching Networks", Article IEEE 3759 Network Magazine, p. 34-45, 1995, 3760 . 3763 Appendix A. Historic Overview 3765 A.1. Traffic Engineering in Classical Telephone Networks 3767 This subsection presents a brief overview of traffic engineering in 3768 telephone networks which often relates to the way user traffic is 3769 steered from an originating node to the terminating node. This 3770 subsection presents a brief overview of this topic. A detailed 3771 description of the various routing strategies applied in telephone 3772 networks is included in the book by G. Ash [ASH2]. 3774 The early telephone network relied on static hierarchical routing, 3775 whereby routing patterns remained fixed independent of the state of 3776 the network or time of day. The hierarchy was intended to 3777 accommodate overflow traffic, improve network reliability via 3778 alternate routes, and prevent call looping by employing strict 3779 hierarchical rules. The network was typically over-provisioned since 3780 a given fixed route had to be dimensioned so that it could carry user 3781 traffic during a busy hour of any busy day. Hierarchical routing in 3782 the telephony network was found to be too rigid upon the advent of 3783 digital switches and stored program control which were able to manage 3784 more complicated TE rules. 3786 Dynamic routing was introduced to alleviate the routing inflexibility 3787 in the static hierarchical routing so that the network would operate 3788 more efficiently. This resulted in significant economic gains 3789 [HUSS87]. Dynamic routing typically reduces the overall loss 3790 probability by 10 to 20 percent (compared to static hierarchical 3791 routing). Dynamic routing can also improve network resilience by 3792 recalculating routes on a per-call basis and periodically updating 3793 routes. 3795 There are three main types of dynamic routing in the telephone 3796 network. They are time-dependent routing, state-dependent routing 3797 (SDR), and event dependent routing (EDR). 3799 In time-dependent routing, regular variations in traffic loads (such 3800 as time of day or day of week) are exploited in pre-planned routing 3801 tables. In state-dependent routing, routing tables are updated 3802 online according to the current state of the network (e.g., traffic 3803 demand, utilization, etc.). In event dependent routing, routing 3804 changes are triggers by events (such as call setups encountering 3805 congested or blocked links) whereupon new paths are searched out 3806 using learning models. EDR methods are real-time adaptive, but they 3807 do not require global state information as does SDR. Examples of EDR 3808 schemes include the dynamic alternate routing (DAR) from BT, the 3809 state-and-time dependent routing (STR) from NTT, and the success-to- 3810 the-top (STT) routing from AT&T. 3812 Dynamic non-hierarchical routing (DNHR) is an example of dynamic 3813 routing that was introduced in the AT&T toll network in the 1980's to 3814 respond to time-dependent information such as regular load variations 3815 as a function of time. Time-dependent information in terms of load 3816 may be divided into three timescales: hourly, weekly, and yearly. 3817 Correspondingly, three algorithms are defined to pre-plan the routing 3818 tables. The network design algorithm operates over a year-long 3819 interval while the demand servicing algorithm operates on a weekly 3820 basis to fine tune link sizes and routing tables to correct forecast 3821 errors on the yearly basis. At the smallest timescale, the routing 3822 algorithm is used to make limited adjustments based on daily traffic 3823 variations. Network design and demand servicing are computed using 3824 offline calculations. Typically, the calculations require extensive 3825 searches on possible routes. On the other hand, routing may need 3826 online calculations to handle crankback. DNHR adopts a "two-link" 3827 approach whereby a path can consist of two links at most. The 3828 routing algorithm presents an ordered list of route choices between 3829 an originating switch and a terminating switch. If a call overflows, 3830 a via switch (a tandem exchange between the originating switch and 3831 the terminating switch) would send a crankback signal to the 3832 originating switch. This switch would then select the next route, 3833 and so on, until there are no alternative routes available in which 3834 the call is blocked. Note that the concept of crankback found its 3835 way into GMPLS as [RFC4920]. 3837 A.2. Evolution of Traffic Engineering in Packet Networks 3839 This subsection reviews related prior work that was intended to 3840 improve the performance of data networks. Indeed, optimization of 3841 the performance of data networks started in the early days of the 3842 ARPANET. Other early commercial networks such as SNA also recognized 3843 the importance of performance optimization and service 3844 differentiation. 3846 In terms of traffic management, the Internet has been a best effort 3847 service environment until recently. In particular, very limited 3848 traffic management capabilities existed in IP networks to provide 3849 differentiated queue management and scheduling services to packets 3850 belonging to different classes. 3852 In terms of routing control, the Internet has employed distributed 3853 protocols for intra-domain routing. These protocols are highly 3854 scalable and resilient. However, they are based on simple algorithms 3855 for path selection which have very limited functionality to allow 3856 flexible control of the path selection process. 3858 In the following subsections, the evolution of practical traffic 3859 engineering mechanisms in IP networks and its predecessors are 3860 reviewed. 3862 A.2.1. Adaptive Routing in the ARPANET 3864 The early ARPANET recognized the importance of adaptive routing where 3865 routing decisions were based on the current state of the network 3866 [MCQ80]. Early minimum delay routing approaches forwarded each 3867 packet to its destination along a path for which the total estimated 3868 transit time was the smallest. Each node maintained a table of 3869 network delays, representing the estimated delay that a packet would 3870 experience along a given path toward its destination. The minimum 3871 delay table was periodically transmitted by a node to its neighbors. 3872 The shortest path, in terms of hop count, was also propagated to give 3873 the connectivity information. 3875 One drawback to this approach is that dynamic link metrics tend to 3876 create "traffic magnets" causing congestion to be shifted from one 3877 location of a network to another location, resulting in oscillation 3878 and network instability. 3880 A.2.2. Dynamic Routing in the Internet 3882 The Internet evolved from the ARPANET and adopted dynamic routing 3883 algorithms with distributed control to determine the paths that 3884 packets should take en-route to their destinations. The routing 3885 algorithms are adaptations of shortest path algorithms where costs 3886 are based on link metrics. The link metric can be based on static or 3887 dynamic quantities. The link metric based on static quantities may 3888 be assigned administratively according to local criteria. The link 3889 metric based on dynamic quantities may be a function of a network 3890 congestion measure such as delay or packet loss. 3892 It was apparent early that static link metric assignment was 3893 inadequate because it can easily lead to unfavorable scenarios in 3894 which some links become congested while others remain lightly loaded. 3895 One of the many reasons for the inadequacy of static link metrics is 3896 that link metric assignment was often done without considering the 3897 traffic matrix in the network. Also, the routing protocols did not 3898 take traffic attributes and capacity constraints into account when 3899 making routing decisions. This results in traffic concentration 3900 being localized in subsets of the network infrastructure and 3901 potentially causing congestion. Even if link metrics are assigned in 3902 accordance with the traffic matrix, unbalanced loads in the network 3903 can still occur due to a number factors including: 3905 * Resources may not be deployed in the most optimal locations from a 3906 routing perspective. 3908 * Forecasting errors in traffic volume and/or traffic distribution. 3910 * Dynamics in traffic matrix due to the temporal nature of traffic 3911 patterns, BGP policy change from peers, etc. 3913 The inadequacy of the legacy Internet interior gateway routing system 3914 is one of the factors motivating the interest in path oriented 3915 technology with explicit routing and constraint-based routing 3916 capability such as MPLS. 3918 A.2.3. ToS Routing 3920 Type-of-Service (ToS) routing involves different routes going to the 3921 same destination with selection dependent upon the ToS field of an IP 3922 packet [RFC2474]. The ToS classes may be classified as low delay and 3923 high throughput. Each link is associated with multiple link costs 3924 and each link cost is used to compute routes for a particular ToS. A 3925 separate shortest path tree is computed for each ToS. The shortest 3926 path algorithm must be run for each ToS resulting in very expensive 3927 computation. Classical ToS-based routing is now outdated as the IP 3928 header field has been replaced by a Diffserv field. Effective TE is 3929 difficult to perform in classical ToS-based routing because each 3930 class still relies exclusively on shortest path routing which results 3931 in localization of traffic concentration within the network. 3933 A.2.4. Equal Cost Multi-Path 3935 Equal Cost Multi-Path (ECMP) is another technique that attempts to 3936 address the deficiency in the Shortest Path First (SPF) interior 3937 gateway routing systems [RFC2328]. In the classical SPF algorithm, 3938 if two or more shortest paths exist to a given destination, the 3939 algorithm will choose one of them. The algorithm is modified 3940 slightly in ECMP so that if two or more equal cost shortest paths 3941 exist between two nodes, the traffic between the nodes is distributed 3942 among the multiple equal-cost paths. Traffic distribution across the 3943 equal-cost paths is usually performed in one of two ways: (1) packet- 3944 based in a round-robin fashion, or (2) flow-based using hashing on 3945 source and destination IP addresses and possibly other fields of the 3946 IP header. The first approach can easily cause out- of-order packets 3947 while the second approach is dependent upon the number and 3948 distribution of flows. Flow-based load sharing may be unpredictable 3949 in an enterprise network where the number of flows is relatively 3950 small and less heterogeneous (for example, hashing may not be 3951 uniform), but it is generally effective in core public networks where 3952 the number of flows is large and heterogeneous. 3954 In ECMP, link costs are static and bandwidth constraints are not 3955 considered, so ECMP attempts to distribute the traffic as equally as 3956 possible among the equal-cost paths independent of the congestion 3957 status of each path. As a result, given two equal-cost paths, it is 3958 possible that one of the paths will be more congested than the other. 3959 Another drawback of ECMP is that load sharing cannot be achieved on 3960 multiple paths which have non-identical costs. 3962 MPLS (through the Entropy Label [RFC6790]) and IPv6 (through the Flow 3963 Label [RFC6437]) allow the ingress node to set a hash value that can 3964 be used in the network for ECMP. These mechanisms can be used to 3965 take into account other constraints and, if there is sufficient 3966 visibility into the network (perhaps through a central controller), 3967 these mechanisms can be used to achieve for more evenly distributed 3968 load balancing giving a modicum of traffic steering. 3970 A.2.5. Nimrod 3972 Nimrod was a routing system developed to provide heterogeneous 3973 service specific routing in the Internet, while taking multiple 3974 constraints into account [RFC1992]. Essentially, Nimrod was a link 3975 state routing protocol to support path oriented packet forwarding. 3976 It used the concept of maps to represent network connectivity and 3977 services at multiple levels of abstraction. Mechanisms allowed 3978 restriction of the distribution of routing information. 3980 Even though Nimrod did not enjoy deployment in the public Internet, a 3981 number of key concepts incorporated into the Nimrod architecture, 3982 such as explicit routing which allows selection of paths at 3983 originating nodes, are beginning to find applications in some recent 3984 constraint-based routing initiatives. 3986 A.3. Development of Internet Traffic Engineering 3988 A.3.1. Overlay Model 3990 In the overlay model, a virtual-circuit network, such as Synchronous 3991 Optical Network / Synchronous Digital Hierarchy (SONET/SDH), Optical 3992 Transport Network (OTN), or Wavelength Division Multiplexing (WDM), 3993 provides virtual-circuit connectivity between routers that are 3994 located at the edges of a virtual-circuit cloud. In this mode, two 3995 routers that are connected through a virtual circuit see a direct 3996 adjacency between themselves independent of the physical route taken 3997 by the virtual circuit through the ATM, frame relay, or WDM network. 3998 Thus, the overlay model essentially decouples the logical topology 3999 that routers see from the physical topology that the ATM, frame 4000 relay, or WDM network manages. The overlay model based on ATM or 4001 frame relay enables a network administrator or an automaton to employ 4002 TE concepts to perform path optimization by re-configuring or 4003 rearranging the virtual circuits so that a virtual circuit on a 4004 congested or sub-optimal physical link can be re-routed to a less 4005 congested or more optimal one. In the overlay model, TE is also 4006 employed to establish relationships between the traffic management 4007 parameters (e.g., Peak Cell Rate, Sustained Cell Rate, and Maximum 4008 Burst Size for ATM) of the virtual- circuit technology and the actual 4009 traffic that traverses each circuit. These relationships can be 4010 established based upon known or projected traffic profiles, and some 4011 other factors. 4013 Appendix B. Overview of Traffic Engineering Related Work in Other SDOs 4015 B.1. Overview of ITU Activities Related to Traffic Engineering 4017 This section provides an overview of prior work within the ITU-T 4018 pertaining to traffic engineering in traditional telecommunications 4019 networks. 4021 ITU-T Recommendations E.600 [ITU-E600], E.701 [ITU-E701], and E.801 4022 [ITU-E801] address TE issues in traditional telecommunications 4023 networks. Recommendation E.600 provides a vocabulary for describing 4024 TE concepts, while E.701 defines reference connections, Grade of 4025 Service (GoS), and traffic parameters for ISDN. Recommendation E.701 4026 uses the concept of a reference connection to identify representative 4027 cases of different types of connections without describing the 4028 specifics of their actual realizations by different physical means. 4029 As defined in Recommendation E.600, "a connection is an association 4030 of resources providing means for communication between two or more 4031 devices in, or attached to, a telecommunication network." Also, 4032 E.600 defines "a resource as any set of physically or conceptually 4033 identifiable entities within a telecommunication network, the use of 4034 which can be unambiguously determined" [ITU-E600]. There can be 4035 different types of connections as the number and types of resources 4036 in a connection may vary. 4038 Typically, different network segments are involved in the path of a 4039 connection. For example, a connection may be local, national, or 4040 international. The purposes of reference connections are to clarify 4041 and specify traffic performance issues at various interfaces between 4042 different network domains. Each domain may consist of one or more 4043 service provider networks. 4045 Reference connections provide a basis to define grade of service 4046 (GoS) parameters related to TE within the ITU-T framework. As 4047 defined in E.600, "GoS refers to a number of traffic engineering 4048 variables which are used to provide a measure of the adequacy of a 4049 group of resources under specified conditions." These GoS variables 4050 may be probability of loss, dial tone, delay, etc. They are 4051 essential for network internal design and operation as well as for 4052 component performance specification. 4054 GoS is different from quality of service (QoS) in the ITU framework. 4055 QoS is the performance perceivable by a telecommunication service 4056 user and expresses the user's degree of satisfaction of the service. 4057 QoS parameters focus on performance aspects observable at the service 4058 access points and network interfaces, rather than their causes within 4059 the network. GoS, on the other hand, is a set of network oriented 4060 measures which characterize the adequacy of a group of resources 4061 under specified conditions. For a network to be effective in serving 4062 its users, the values of both GoS and QoS parameters must be related, 4063 with GoS parameters typically making a major contribution to the QoS. 4065 Recommendation E.600 stipulates that a set of GoS parameters must be 4066 selected and defined on an end-to-end basis for each major service 4067 category provided by a network to assist the network provider with 4068 improving efficiency and effectiveness of the network. Based on a 4069 selected set of reference connections, suitable target values are 4070 assigned to the selected GoS parameters under normal and high load 4071 conditions. These end-to-end GoS target values are then apportioned 4072 to individual resource components of the reference connections for 4073 dimensioning purposes. 4075 Appendix C. Summary of Changes Since RFC 3272 4077 The changes to this document since RFC 3272 are substantial and not 4078 easily summarized as section-by-section changes. The material in the 4079 document has been moved around considerably, some of it removed, and 4080 new text added. 4082 The approach taken here is to list the table of content of both the 4083 previous RFC and this document saying, respectively, where the text 4084 has been placed and where the text came from. 4086 C.1. RFC 3272 4088 1.0 Introduction: Edited in place in Section 1. 4090 1.1 What is Internet Traffic Engineering?: Edited in place in 4091 Section 1.1. 4093 1.2 Scope: Moved to Section 1.3. 4095 1.3 Terminology: Moved to Section 1.4 with some obsolete terms 4096 removed and a little editing. 4098 2.0 Background: Retained as Section 2 with some text removed. 4100 2.1 Context of Internet Traffic Engineering: Retained as 4101 Section 2.1. 4103 2.2 Network Context: Rewritten as Section 2.2. 4105 2.3 Problem Context: Rewritten as Section 2.3. 4107 2.3.1 Congestion and its Ramifications: Retained as 4108 Section 2.3.1. 4110 2.4 Solution Context: Edited as Section 2.4. 4112 2.4.1 Combating the Congestion Problem: Reformatted as 4113 Section 2.4.1. 4115 2.5 Implementation and Operational Context: Retained as 4116 Section 2.5. 4118 3.0 Traffic Engineering Process Model: Retained as Section 3. 4120 3.1 Components of the Traffic Engineering Process Model: Retained 4121 as Section 3.1. 4123 3.2 Measurement: Merged into Section 3.1. 4125 3.3 Modeling, Analysis, and Simulation: Merged into Section 3.1. 4127 3.4 Optimization: Merged into Section 3.1. 4129 4.0 Historical Review and Recent Developments: Retained as 4130 Section 5, but the very historic aspects moved to Appendix A. 4132 4.1 Traffic Engineering in Classical Telephone Networks: Moved to 4133 Appendix A.1. 4135 4.2 Evolution of Traffic Engineering in the Internet: Moved to Ap 4136 pendix A.2. 4138 4.2.1 Adaptive Routing in ARPANET: Moved to Appendix A.2.1. 4140 4.2.2 Dynamic Routing in the Internet: Moved to 4141 Appendix A.2.2. 4143 4.2.3 ToS Routing: Moved to Appendix A.2.3. 4145 4.2.4 Equal Cost Multi-Path: Moved to Appendix A.2.4. 4147 4.2.5 Nimrod: Moved to Appendix A.2.5. 4149 4.3 Overlay Model: Moved to Appendix A.3.1. 4151 4.4 Constraint-Based Routing: Retained as Section 5.1.3.1, but 4152 moved into Section 5.1. 4154 4.5 Overview of Other IETF Projects Related to Traffic 4155 Engineering: Retained as Section 5.1 with many new subsections. 4157 4.5.1 Integrated Services: Retained as Section 5.1.1.1. 4159 4.5.2 RSVP: Retained as Section 5.1.3.2 with some edits. 4161 4.5.3 Differentiated Services: Retained as Section 5.1.1.2. 4163 4.5.4 MPLS: Retained as Section 5.1.3.3. 4165 4.5.5 IP Performance Metrics: Retained as Section 5.1.3.5. 4167 4.5.6 Flow Measurement: Retained as Section 5.1.3.6 with some 4168 reformatting. 4170 4.5.7 Endpoint Congestion Management: Retained as 4171 Section 5.1.3.7. 4173 4.6 Overview of ITU Activities Related to Traffic Engineering: Mo 4174 ved to Appendix B.1. 4176 4.7 Content Distribution: Retained as Section 5.2. 4178 5.0 Taxonomy of Traffic Engineering Systems: Retained as Section 4. 4180 5.1 Time-Dependent Versus State-Dependent: Retained as 4181 Section 4.1. 4183 5.2 Offline Versus Online: Retained as Section 4.2. 4185 5.3 Centralized Versus Distributed: Retained as Section 4.3 with 4186 additions. 4188 5.4 Local Versus Global: Retained as Section 4.4. 4190 5.5 Prescriptive Versus Descriptive: Retained as Section 4.5 with 4191 additions. 4193 5.6 Open-Loop Versus Closed-Loop: Retained as Section 4.6. 4195 5.7 Tactical vs Strategic: Retained as Section 4.7. 4197 6.0 Recommendations for Internet Traffic Engineering: Retained as 4198 Section 6. 4200 6.1 Generic Non-functional Recommendations: Retained as 4201 Section 6.1. 4203 6.2 Routing Recommendations: Retained as Section 6.2 with edits. 4205 6.3 Traffic Mapping Recommendations: Retained as Section 6.3. 4207 6.4 Measurement Recommendations: Retained as Section 6.4. 4209 6.5 Network Survivability: Retained as Section 6.6. 4211 6.5.1 Survivability in MPLS Based Networks: Retained as 4212 Section 6.6.1. 4214 6.5.2 Protection Option: Retained as Section 6.6.2. 4216 6.6 Traffic Engineering in Diffserv Environments: Retained as 4217 Section 6.8 with edits. 4219 6.7 Network Controllability: Retained as Section 6.9. 4221 7.0 Inter-Domain Considerations: Retained as Section 7. 4223 8.0 Overview of Contemporary TE Practices in Operational IP 4224 Networks: Retained as Section 8. 4226 9.0 Conclusion: Removed. 4228 10.0 Security Considerations: Retained as Section 9 with 4229 considerable new text. 4231 C.2. This Document 4233 * Section 1: Based on Section 1 of RFC 3272. 4235 - Section 1.1: Based on Section 1.1 of RFC 3272. 4237 - Section 1.2: New for this document. 4239 - Section 1.3: Based on Section 1.2 of RFC 3272. 4241 - Section 1.4: Based on Section 1.3 of RFC 3272. 4243 * Section 2: Based on Section 2. of RFC 3272. 4245 - Section 2.1: Based on Section 2.1 of RFC 3272. 4247 - Section 2.2: Based on Section 2.2 of RFC 3272. 4249 - Section 2.3: Based on Section 2.3 of RFC 3272. 4251 o Section 2.3.1: Based on Section 2.3.1 of RFC 3272. 4253 - Section 2.4: Based on Section 2.4 of RFC 3272. 4255 o Section 2.4.1: Based on Section 2.4.1 of RFC 327 4257 - Section 2.5: Based on Section 2.5 of RFC 3272. 4259 * Section 3: Based on Section 3 of RFC 3272. 4261 - Section 3.1: Based on Sections 3.1, 3.2, 3.3, and 3.4 of RFC 4262 3272. 4264 * Section 4: Based on Section 5 of RFC 3272. 4266 - Section 4.1: Based on Section 5.1 of RFC 3272. 4268 - Section 4.2: Based on Section 5.2 of RFC 3272. 4270 - Section 4.3: Based on Section 5.3 of RFC 3272. 4272 o Section 4.3.1: New for this document. 4274 o Section 4.3.2: New for this document. 4276 - Section 4.4: Based on Section 5.4 of RFC 3272. 4278 - Section 4.5: Based on Section 5.5 of RFC 3272. 4280 o Section 4.5.1: New for this document. 4282 - Section 4.6: Based on Section 5.6 of RFC 3272. 4284 - Section 4.7: Based on Section 5.7 of RFC 3272. 4286 * Section 5: Based on Section 4 of RFC 3272. 4288 - Section 5.1: Based on Section 4.5 of RFC 3272. 4290 o Section 5.1.1.1: Based on Section 4.5.1 of RFC 3272. 4292 o Section 5.1.1.2: Based on Section 4.5.3 of RFC 3272. 4294 o Section 5.1.1.3: New for this document. 4296 o Section 5.1.1.4: New for this document. 4298 o Section 5.1.1.5: New for this document. 4300 o Section 5.1.2.1: New for this document. 4302 o Section 5.1.2.2: New for this document. 4304 o Section 5.1.2.3: New for this document. 4306 o Section 5.1.3.1: Based on Section 4.4 of RFC 3272. 4308 + Section 5.1.3.1.1: New for this document. 4310 o Section 5.1.3.2: Based on Section 4.5.2 of RFC 3272. 4312 o Section 5.1.3.3: Based on Section 4.5.4 of RFC 3272. 4314 o Section 5.1.3.4: New for this document. 4316 o Section 5.1.3.5: Based on Section 4.5.5 of RFC 3272. 4318 o Section 5.1.3.6: Based on Section 4.5.6 of RFC 3272. 4320 o Section 5.1.3.7: Based on Section 4.5.7 of RFC 3272. 4322 o Section 5.1.3.8: New for this document. 4324 o Section 5.1.3.9: New for this document. 4326 o Section 5.1.3.10: New for this document. 4328 o Section 5.1.3.11: New for this document. 4330 o Section 5.1.3.12: New for this document. 4332 o Section 5.1.3.13: New for this document. 4334 - Section 5.2: Based on Section 4.7 of RFC 3272. 4336 * Section 6: Based on Section 6 of RFC 3272. 4338 - Section 6.1: Based on Section 6.1 of RFC 3272. 4340 - Section 6.2: Based on Section 6.2 of RFC 3272. 4342 - Section 6.3: Based on Section 6.3 of RFC 3272. 4344 - Section 6.4: Based on Section 6.4 of RFC 3272. 4346 - Section 6.5: New for this document. 4348 - Section 6.6: Based on Section 6.5 of RFC 3272. 4350 o Section 6.6.1: Based on Section 6.5.1 of RFC 3272. 4352 o Section 6.6.2: Based on Section 6.5.2 of RFC 3272. 4354 - Section 6.7: New for this document. 4356 - Section 6.8: Based on Section 6.6. of RFC 3272. 4358 - Section 6.9: Based on Section 6.7 of RFC 3272. 4360 * Section 7: Based on Section 7 of RFC 3272. 4362 * Section 8: Based on Section 8 of RFC 3272. 4364 * Section 9: Based on Section 10 of RFC 3272. 4366 * Appendix A: New for this document. 4368 - Appendix A.1: Based on Section 4.1 of RFC 3272. 4370 - Appendix A.2: Based on Section 4.2 of RFC 3272. 4372 o Appendix A.2.1: Based on Section 4.2.1 of RFC 3272. 4374 o Appendix A.2.2: Based on Section 4.2.2 of RFC 3272. 4376 o Appendix A.2.3: Based on Section 4.2.3 of RFC 3272. 4378 o Appendix A.2.4: Based on Section 4.2.4 of RFC 3272. 4380 o Appendix A.2.5: Based on Section 4.2.5 of RFC 3272. 4382 - Appendix A.3: New for this document. 4384 o Appendix A.3.1: Based on Section 4.3 of RFC 3272. 4386 * Appendix B: New for this document. 4388 - Appendix B.1: Based on Section 4.7 of RFC 3272. 4390 Author's Address 4392 Adrian Farrel (editor) 4393 Old Dog Consulting 4394 Email: adrian@olddog.co.uk