idnits 2.17.00 (12 Aug 2021) /tmp/idnits31212/draft-ietf-teas-enhanced-vpn-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (25 October 2021) is 208 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'I-D.ietf-teas-actn-yang' is defined on line 1597, but no explicit reference was found in the text == Outdated reference: A later version (-06) exists of draft-dong-6man-enhanced-vpn-vtn-id-05 == Outdated reference: A later version (-04) exists of draft-dong-teas-enhanced-vpn-vtn-scalability-03 == Outdated reference: A later version (-16) exists of draft-ietf-opsawg-l2nm-09 == Outdated reference: draft-ietf-opsawg-l3sm-l3nm has been published as RFC 9182 == Outdated reference: A later version (-13) exists of draft-ietf-opsawg-ntf-09 == Outdated reference: A later version (-04) exists of draft-ietf-spring-resource-aware-segments-03 == Outdated reference: A later version (-22) exists of draft-ietf-spring-segment-routing-policy-13 == Outdated reference: A later version (-14) exists of draft-ietf-teas-actn-vn-yang-13 == Outdated reference: A later version (-09) exists of draft-ietf-teas-actn-yang-08 == Outdated reference: A later version (-01) exists of draft-ietf-teas-applicability-actn-slicing-00 == Outdated reference: A later version (-01) exists of draft-ietf-teas-ietf-network-slice-nbi-yang-00 == Outdated reference: A later version (-10) exists of draft-ietf-teas-ietf-network-slices-04 Summary: 0 errors (**), 0 flaws (~~), 14 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TEAS Working Group J. Dong 3 Internet-Draft Huawei 4 Intended status: Informational S. Bryant 5 Expires: 28 April 2022 University of Surrey 6 Z. Li 7 China Mobile 8 T. Miyasaka 9 KDDI Corporation 10 Y. Lee 11 Samsung 12 25 October 2021 14 A Framework for Enhanced Virtual Private Network (VPN+) Services 15 draft-ietf-teas-enhanced-vpn-09 17 Abstract 19 This document describes the framework for Enhanced Virtual Private 20 Network (VPN+) services. The purpose of enhanced VPNs is to support 21 the needs of new applications, particularly applications that are 22 associated with 5G services, by utilizing an approach that is based 23 on existing VPN and Traffic Engineering (TE) technologies and adds 24 characteristics that specific services require over those provided by 25 traditional VPNs. 27 Typically, VPN+ will be used to underpin network slicing, but could 28 also be of use in its own right providing enhanced connectivity 29 services between customer sites. 31 It is envisaged that enhanced VPNs will be delivered using a 32 combination of existing, modified, and new networking technologies. 33 This document provides an overview of relevant technologies and 34 identifies some areas for potential new work. 36 Status of This Memo 38 This Internet-Draft is submitted in full conformance with the 39 provisions of BCP 78 and BCP 79. 41 Internet-Drafts are working documents of the Internet Engineering 42 Task Force (IETF). Note that other groups may also distribute 43 working documents as Internet-Drafts. The list of current Internet- 44 Drafts is at https://datatracker.ietf.org/drafts/current/. 46 Internet-Drafts are draft documents valid for a maximum of six months 47 and may be updated, replaced, or obsoleted by other documents at any 48 time. It is inappropriate to use Internet-Drafts as reference 49 material or to cite them other than as "work in progress." 51 This Internet-Draft will expire on 28 April 2022. 53 Copyright Notice 55 Copyright (c) 2021 IETF Trust and the persons identified as the 56 document authors. All rights reserved. 58 This document is subject to BCP 78 and the IETF Trust's Legal 59 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 60 license-info) in effect on the date of publication of this document. 61 Please review these documents carefully, as they describe your rights 62 and restrictions with respect to this document. Code Components 63 extracted from this document must include Simplified BSD License text 64 as described in Section 4.e of the Trust Legal Provisions and are 65 provided without warranty as described in the Simplified BSD License. 67 Table of Contents 69 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 70 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 6 71 3. Overview of the Requirements . . . . . . . . . . . . . . . . 7 72 3.1. Performance Guarantees . . . . . . . . . . . . . . . . . 7 73 3.2. Isolation between Enhanced VPN Services . . . . . . . . . 9 74 3.2.1. A Pragmatic Approach to Isolation . . . . . . . . . . 11 75 3.3. Integration . . . . . . . . . . . . . . . . . . . . . . . 11 76 3.3.1. Abstraction . . . . . . . . . . . . . . . . . . . . . 12 77 3.4. Dynamic Changes . . . . . . . . . . . . . . . . . . . . . 12 78 3.5. Customized Control . . . . . . . . . . . . . . . . . . . 13 79 3.6. Applicability to Overlay Technologies . . . . . . . . . . 13 80 3.7. Inter-Domain and Inter-Layer Network . . . . . . . . . . 14 81 4. Architecture of Enhanced VPNs . . . . . . . . . . . . . . . . 14 82 4.1. Layered Architecture . . . . . . . . . . . . . . . . . . 16 83 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity . . . . . 18 84 4.3. Application Specific Data Types . . . . . . . . . . . . . 19 85 4.4. Scaling Considerations . . . . . . . . . . . . . . . . . 19 86 5. Candidate Technologies . . . . . . . . . . . . . . . . . . . 20 87 5.1. Packet Forwarding Plane Technologies . . . . . . . . . . 20 88 5.1.1. Flexible Ethernet . . . . . . . . . . . . . . . . . . 20 89 5.1.2. Dedicated Queues . . . . . . . . . . . . . . . . . . 21 90 5.1.3. Time Sensitive Networking . . . . . . . . . . . . . . 21 91 5.2. Layer Three Data Plane . . . . . . . . . . . . . . . . . 21 92 5.2.1. Deterministic Networking . . . . . . . . . . . . . . 22 93 5.2.2. MPLS Traffic Engineering (MPLS-TE) . . . . . . . . . 22 94 5.2.3. Segment Routing . . . . . . . . . . . . . . . . . . . 22 95 5.3. Non-Packet Data Plane . . . . . . . . . . . . . . . . . . 23 96 5.4. Control Plane . . . . . . . . . . . . . . . . . . . . . . 23 97 5.5. Management Plane . . . . . . . . . . . . . . . . . . . . 24 98 5.6. Applicability of Service Data Models to Enhanced VPN . . 25 99 6. Applicability to Network Slice Realization . . . . . . . . . 26 100 6.1. VTN Planning and Instantiation . . . . . . . . . . . . . 26 101 6.2. Enhanced VPN Provisioning . . . . . . . . . . . . . . . . 27 102 6.3. Network Slice Traffic Steering and Forwarding . . . . . . 27 103 7. Scalability Considerations . . . . . . . . . . . . . . . . . 28 104 7.1. Maximum Stack Depth of SR . . . . . . . . . . . . . . . . 29 105 7.2. RSVP-TE Scalability . . . . . . . . . . . . . . . . . . . 29 106 7.3. SDN Scaling . . . . . . . . . . . . . . . . . . . . . . . 29 107 8. OAM Considerations . . . . . . . . . . . . . . . . . . . . . 29 108 9. Telemetry Considerations . . . . . . . . . . . . . . . . . . 30 109 10. Enhanced Resiliency . . . . . . . . . . . . . . . . . . . . . 30 110 11. Operational Considerations . . . . . . . . . . . . . . . . . 32 111 12. Security Considerations . . . . . . . . . . . . . . . . . . . 32 112 13. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 113 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 33 114 15. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 33 115 16. References . . . . . . . . . . . . . . . . . . . . . . . . . 33 116 16.1. Normative References . . . . . . . . . . . . . . . . . . 33 117 16.2. Informative References . . . . . . . . . . . . . . . . . 34 118 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 40 120 1. Introduction 122 Virtual private networks (VPNs) have served the industry well as a 123 means of providing different groups of users with logically isolated 124 connectivity over a common network. The common or base network that 125 is used to provide the VPNs is often referred to as the underlay, and 126 the VPN is often called an overlay. 128 Customers of a network operator may request a connectivity services 129 with advanced characteristics such as low latency guarantees, bounded 130 jitter, or isolation from other services or customers so that changes 131 in some other service (such as changes in network load, or events 132 such as congestion or outages) have no or only acceptable effect on 133 the throughput or latency of the services provided to the customer. 134 These services are referred to as "enhanced VPNs" (known as VPN+) in 135 that they are similar to VPN services providing the customer with the 136 required connectivity, but in addition they have enhanced 137 characteristics. 139 The concept of network slicing has gained traction driven largely by 140 needs surfacing from 5G [NGMN-NS-Concept] [TS23501] [TS28530] 141 [BBF-SD406]. According to [TS28530], a 5G end-to-end network slice 142 consists of three major types of network segments: Radio Access 143 Network (RAN), Transport Network (TN), and Mobile Core Network (CN). 144 The transport network provides the connectivity between different 145 entities in RAN and CN segments of a 5G end-to-end network slice, 146 with specific performance commitment. 148 [I-D.ietf-teas-ietf-network-slices] introduces the concept and the 149 general framework of IETF network slices. An IETF Network Slice is a 150 logical network topology connecting a number of endpoints using a set 151 of shared or dedicated network resources that are used to satisfy 152 specific Service Level Objectives (SLOs) and Service Level 153 Expectations (SLEs). In this document (which is solely about IETF 154 technologies) we refer to an "IETF network slice" simply as a 155 "network slice": a network slice is considered one possible use case 156 of an enhanced VPN. 158 A network slice could span multiple technologies (such as IP or 159 Optical) and multiple administrative domains. Depending on the 160 customer's requirement, a network slice could be isolated from other 161 network slices in terms of data plane, control plane, and management 162 plane resources. 164 Network slicing builds on the concepts of resource management, 165 network virtualization, and abstraction to provide performance 166 assurance, flexibility, programmability, and modularity. It may use 167 techniques such as Software Defined Networking (SDN) [RFC7149], 168 network abstraction [RFC7926] and Network Function Virtualization 169 (NFV) [RFC8172] [RFC8568] to create multiple logical (virtual) 170 networks, each tailored for use by a set of services or by a 171 particular tenant or a group of tenants that share the same or 172 similar requirements. These logical networks are created on top of a 173 common underlay network. How the network slices are engineered can 174 be deployment-specific. 176 VPN+ can be used to instantiate a network slice, but the technique 177 can also be of use in general cases to provide enhanced connectivity 178 services between customer sites. 180 The requirements of enhanced VPN services cannot be met by simple 181 overlay networks, as these services require tighter coordination and 182 integration between the underlay and the overlay network. VPN+ is 183 built from a VPN overlay and an underlying Virtual Transport Network 184 (VTN) which has a customized network topology and a set of dedicated 185 or shared resources in the underlay network. The enhanced VPN may 186 also include a set of invoked service functions located within the 187 underlay network. Thus, an enhanced VPN can achieve greater 188 isolation with strict performance guarantees. These new properties, 189 which have general applicability, are also of interest as part of a 190 network slicing solution. 192 It is not envisaged that VPN+ services will replace traditional VPN 193 services. Traditional VPN services will continue to be delivered 194 using pre-existing mechanisms and can co-exist with VPN+ services. 195 In fact, compared to traditional VPNs, it is not envisaged that large 196 numbers of VPN+ services will be deployed in a network. In other 197 words, it is not intended that all existing VPNs supported by a 198 network will use VPN+ techniques. 200 This document describes a framework for using existing, modified, and 201 potential new technologies as components to provide a VPN+ service. 202 Specifically, we are concerned with: 204 * The functional requirements and service characteristics of an 205 enhanced VPN. 207 * The design of the enhanced VPN data plane. 209 * The necessary control and management protocols in both the 210 underlay and the overlay of the enhanced VPN. 212 * The mechanisms to achieve integration between overlay and 213 underlay. 215 * The necessary Operation, Administration, and Management (OAM) 216 methods to instrument an enhanced VPN to make sure that the 217 required Service Level Agreement (SLA) between the customer and 218 the network operator is met, and to take any corrective action 219 (such as switching traffic to an alternate path) to avoid SLA 220 violation. 222 The required layered network structure to achieve this is shown in 223 Section 4.1. 225 2. Terminology 227 In this document, the relationship of the four terms "VPN", "VPN+", 228 "VTN", and "Network Slice" are as follows: 230 * A Virtual Private Network (VPN) refers to the overlay network 231 service that provides the connectivity between different customer 232 sites, and that maintains traffic separation between different 233 customers. IPVPN is defined in [RFC2764], L2VPN is defined in 234 [RFC4664], L3VPN is defined in [RFC4364], and EVPN is defined in 235 [RFC7209]. 237 * An enhanced VPN (VPN+) is an evolution of the VPN service that 238 makes additional service-specific commitments. An enhanced VPN is 239 made by integrating an overlay VPN with a set of network resources 240 allocated in the underlay network. 242 * A Virtual Transport Network (VTN) is a virtual underlay network 243 which consists of a set of dedicated or shared network resources 244 allocated from the physical underlay network, and is associated 245 with a customized logical network topology. VTN has the 246 capability to deliver the performance characteristics required by 247 the VPN+ customers and to provide isolation between different VPN+ 248 services. 250 * A network slice could be provided by provisioning an enhanced VPN 251 in the network. Other mechanisms for delivering network slices 252 may exist but are not in scope for this document. 254 The term "tenant" is used in this document to refer to the customers 255 and all of their associated enhanced VPNs. 257 The following terms are also used in this document. Some of them are 258 newly defined, some others reference existing definitions. 260 ACTN: Abstraction and Control of Traffic Engineered Networks 261 [RFC8453] 263 DetNet: Deterministic Networking. See [DETNET] and [RFC8655] 265 FlexE: Flexible Ethernet [FLEXE] 267 TSN: Time Sensitive Networking [TSN] 269 VN: Virtual Network [I-D.ietf-teas-actn-vn-yang] 271 VTP: Virtual Transport Path. A VTP is a path through the VTN which 272 provides the required connectivity and performance between two or 273 more customer sites. 275 3. Overview of the Requirements 277 This section provides an overview of the requirements of an enhanced 278 VPN service. 280 3.1. Performance Guarantees 282 Performance guarantees are made by network operators to their 283 customers in relation to the services provided to the customers. 284 They are usually expressed in SLAs as a set of SLOs. 286 There are several kinds of performance guarantee, including 287 guaranteed maximum packet loss, guaranteed maximum delay, and 288 guaranteed delay variation. Note that these guarantees apply to 289 conformance traffic, out-of-profile traffic will be handled according 290 to a separate agreement with the customer. 292 Guaranteed maximum packet loss is usually addressed by setting packet 293 priorities, queue size, and discard policy. However this becomes 294 more difficult when the requirement is combined with latency 295 requirements. The limiting case is zero congestion loss, and that is 296 the goal of DetNet [DETNET] and TSN [TSN]. In modern optical 297 networks, loss due to transmission errors already approaches zero, 298 but there is the possibility of failure of the interface or the fiber 299 itself. This type of fault can only be addressed by some form of 300 signal duplication and transmission over diverse paths. 302 Guaranteed maximum latency is required by a number of applications 303 particularly real-time control applications and some types of virtual 304 reality applications. DetNet [DETNET] is relevant, however 305 additional methods of enhancing the underlay to better support the 306 delay guarantees may be needed, and these methods will need to be 307 integrated with the overall service provisioning mechanisms. 309 Guaranteed maximum delay variation is a performance guarantee that 310 may also be needed. [RFC8578] calls up a number of cases that need 311 this guarantee, for example in electrical utilities. Time transfer 312 is an example service that needs a performance guarantee, although it 313 is in the nature of time that the service might be delivered by the 314 underlay as a shared service and not provided through different 315 enhanced VPNs. Alternatively, a dedicated enhanced VPN might be used 316 to provide this as a shared service. 318 This suggests that a spectrum of service guarantees need to be 319 considered when deploying an enhanced VPN. As a guide to 320 understanding the design requirements we can consider four types of 321 service: 323 * Best effort 325 * Assured bandwidth 327 * Guaranteed latency 329 * Enhanced delivery 331 The best effort service is the basic service as provided by current 332 VPNs. 334 An assured bandwidth service is one in which the bandwidth over some 335 period of time is assured. This can be achieved either simply based 336 on a best effort service with over-capacity provisioning, or it can 337 be based on MPLS traffic engineered label switching paths (TE-LSPs) 338 with bandwidth reservations. Depending on the technique used, 339 however, the bandwidth is not necessarily assured at any instant. 340 Providing assured bandwidth to VPNs, for example by using per-VPN TE- 341 LSPs, is not widely deployed at least partially due to scalability 342 concerns. VPN+ aims to provide a more scalable approach for such 343 services. 345 A guaranteed latency service has an upper bound to edge-to-edge 346 latency. Assuring the upper bound is sometimes more important than 347 minimizing latency. There are several new technologies that provide 348 some assistance with this performance guarantee. Firstly, the IEEE 349 TSN project [TSN] introduces the concept of scheduling of delay- and 350 loss-sensitive packets. The DetNet work [DETNET] is also of 351 relevance in assuring an upper bound of end-to-end packet latency. 352 FlexE [FLEXE] is also useful to help provide these guarantees. The 353 use of such underlying technologies to deliver VPN+ services needs to 354 be considered. 356 An enhanced delivery service is one in which the underlay network (at 357 Layer 3) attempts to deliver the packet through multiple paths in the 358 hope of eliminating packet loss due to equipment or media failures. 359 Such a mechanism may need to be used for VPN+ service. 361 3.2. Isolation between Enhanced VPN Services 363 One element of the SLA demanded for an enhanced VPN may be a 364 guarantee that the service offered to the customer will not be 365 affected by any other traffic flows in the network. This is termed 366 "isolation" and a customer may express the requirement for isolation 367 as an SLE [I-D.ietf-teas-ietf-network-slices]. 369 One way for a network operator to meet the requirement for isolation 370 is simply by setting and conforming to all the SLOs. For example, 371 traffic congestion (interference from other services) might impact on 372 the latency experienced by a VPN+ customer. Thus, in this example, 373 conformance to a latency SLO would be the primary requirement for 374 delivery of the VPN+ service, and isolation from other services might 375 be only a means to that end. 377 Another way for a service provider to meet this SLE is to control the 378 degree to which traffic from one service is isolated from other 379 services in the network. 381 There is a fine distinction between how isolation is requested by a 382 customer and how it is delivered by the service provider. In 383 general, the customer is interested in service performance and not 384 how it is delivered. Thus, for example, the customer wants specific 385 quality guarantees and is not concerned about how the service 386 provider delivers them. However, it should be noted that some 387 aspects of isolation might be directly measurable by a customer if 388 they have information about the traffic patterns on a number services 389 supported by the same service provider. Furthermore, a customer may 390 be nervous about disruption caused by other services, contamination 391 by other traffic, or delivery of their traffic to the wrong 392 destinations. In this way, the customer may want to specify (and pay 393 for) the level of isolation provided by the service provider. 395 Isolation is achieved in the realization of a VPN+ through existing 396 technologies that may be supplemented by new mechanisms. The service 397 provider chooses which processes to use to meet this SLE just as they 398 choose how to meet all other SLOs and SLEs. Isolation may be 399 achieved in the network by various forms of resource partitioning 400 ranging from simple separation of service traffic on delivery 401 (ensuring that traffic is not delivered to the wrong customer), 402 through sharing of resources with some form of safeguards, to 403 dedicated allocation of resources for a specific enhanced VPN. For 404 example, interference avoidance may be achieved by network capacity 405 planning, allocating dedicated network resources, traffic policing or 406 shaping, prioritizing in using shared network resources, etc. 408 The terms hard and soft isolation are used to indicate different 409 levels of isolation. A service has soft isolation if the traffic of 410 one service cannot be received by the customers of another service. 411 The existing IP and MPLS VPNs are examples of services with soft 412 isolation: the network delivers the traffic only to the required 413 customer endpoints. However, with soft isolation, as the network 414 resources are shared, traffic from some services may congest the 415 network, resulting in packet loss and delay for other services. The 416 ability for a service or a group of services to be sheltered from 417 this effect is called hard isolation. Hard isolation may be needed 418 so that applications with exacting requirements can function 419 correctly, despite other demands (perhaps a burst of traffic in 420 another service) competing for the underlying resources. A customer 421 may request different degrees of isolation ranging from soft 422 isolation to hard isolation. In practice isolation may be delivered 423 on a spectrum between soft and hard, and in some cases soft and hard 424 isolation may be used in a hierarchical manner with one enhanced VPN 425 being built on another. 427 To provide the required level of isolation, resources may need to be 428 reserved in the data plane of the underlay network and dedicated to 429 traffic from a specific enhanced VPN or a specific group of enhanced 430 VPNs. This may introduce scalability concerns both in the 431 implementation (as each enhanced VPN would need to be tracked in the 432 network) and in how many resources need to be reserved and may be 433 under-used (see Section 4.4). Thus, some trade-off needs to be 434 considered to provide the isolation between enhanced VPNs while still 435 allowing reasonable resource sharing. 437 An optical underlay can offer a high degree of isolation, at the cost 438 of allocating resources on a long-term and end-to-end basis. On the 439 other hand, where adequate isolation can be achieved at the packet 440 layer, this permits the resources to be shared amongst a group of 441 services and only dedicated to a service on a temporary basis. 443 The next section explores a pragmatic approach to isolation in packet 444 networks. 446 3.2.1. A Pragmatic Approach to Isolation 448 A key question is whether it is possible to achieve hard isolation in 449 packet networks that were designed to provide statistical 450 multiplexing through sharing of data plane resources, a significant 451 economic advantage when compared to a dedicated, or a Time Division 452 Multiplexing (TDM) network. Clearly, there is no need to provide 453 more isolation than is required by the applications, and an 454 approximation to full hard isolation is sufficient in most cases. 455 For example, pseudowires [RFC3985] emulate services that would have 456 had hard isolation in their native form. 458 O=================================================O 459 | \---------------v---------------/ 460 Statistical Pragmatic Absolute 461 Multiplexing Isolation Isolation 462 (Traditional VPNs) (Enhanced VPN) (Dedicated Network) 464 Figure 1: The Spectrum of Isolation 466 Figure 1 shows a spectrum of isolation that may be delivered by a 467 network. At one end of the spectrum, we see statistical multiplexing 468 technologies that support traditional VPNs. This is a service type 469 that has served the industry well and will continue to do so. At the 470 opposite end of the spectrum, we have the absolute isolation provided 471 by dedicated transport networks. The goal of enhanced VPNs is 472 "pragmatic isolation". This is isolation that is better than what is 473 obtainable from pure statistical multiplexing, more cost effective 474 and flexible than a dedicated network, but is a practical solution 475 that is good enough for the majority of applications. Mechanisms for 476 both soft isolation and hard isolation are needed to meet different 477 levels of service requirement. 479 3.3. Integration 481 The way to achieve the characteristics demanded by an enhanced VPN 482 (such as guaranteed or predictable performance) is by integrating the 483 overlay VPN with a particular set of resources in the underlay 484 network which are allocated to meet the service requirement. This 485 needs be done in a flexible and scalable way so that it can be widely 486 deployed in operators' networks to support a reasonable number of 487 enhanced VPN customers. 489 Taking mobile networks and in particular 5G into consideration, the 490 integration of the network with service functions is likely a 491 requirement. The IETF's work on service function chaining (SFC) 492 [SFC] provides a foundation for this. Service functions can be 493 considered as part of enhanced VPN services. The detailed mechanisms 494 about the integration between service functions and enhanced VPNs are 495 out of the scope of this document. 497 3.3.1. Abstraction 499 Integration of the overlay VPN and the underlay network resources 500 does not need to be a tight mapping. As described in [RFC7926], 501 abstraction is the process of applying policy to a set of information 502 about a traffic engineered (TE) network to produce selective 503 information that represents the potential ability to connect across 504 the network. The process of abstraction presents the connectivity 505 graph in a way that is independent of the underlying network 506 technologies, capabilities, and topology so that the graph can be 507 used to plan and deliver network services in a uniform way. 509 Virtual networks can be built on top of an abstracted topology that 510 represents the connectivity capabilities of the underlay network as 511 described in the framework for Abstraction and Control of TE Networks 512 (ACTN) [RFC8453] as discussed further in Section 5.5. 513 [I-D.ietf-teas-applicability-actn-slicing] describes the 514 applicability of ACTN to network slicing and is, therefore, relevant 515 to the consideration of using ACTN to enable enhanced VPNs. 517 3.4. Dynamic Changes 519 Enhanced VPNs need to be created, modified, and removed from the 520 network according to service demands. An enhanced VPN that requires 521 hard isolation (Section 3.2) must not be disrupted by the 522 instantiation or modification of another enhanced VPN. Determining 523 whether modification of an enhanced VPN can be disruptive to that 524 VPN, and whether the traffic in flight will be disrupted can be a 525 difficult problem. 527 The data plane aspects of this problem are discussed further in 528 Section 5.1,Section 5.2, and Section 5.3. 530 The control plane aspects of this problem are discussed further in 531 Section 5.4. 533 The management plane aspects of this problem are discussed further in 534 Section 5.5. 536 Dynamic changes both to the enhanced VPN and to the underlay 537 transport network need to be managed to avoid disruption to services 538 that are sensitive to changes in network performance. 540 In addition to non-disruptively managing the network during changes 541 such as the inclusion of a new VPN endpoint or a change to a link, 542 VPN traffic might need to be moved because of changes to traffic 543 patterns and volumes. 545 3.5. Customized Control 547 In many cases the customers are delivered with enhanced VPN services 548 without knowing the information about the underlying VTNs. However, 549 depends on the agreement between the operator and the customer, in 550 some cases the customer may also be provided with some information 551 about the underlying VTNs. Such information can be filtered or 552 aggregated according to the operator's policy. This allows the 553 customer of the enhanced VPN to have some visibility and even control 554 over how the underlying topology and resources of the VTN are used. 555 For example, the customers may be able to specify the service paths 556 within the VTN for specific traffic flows of their enhanced VPNs. 557 Depending on the requirements, an enhanced VPN customer may have his 558 own network controller, which may be provided with an interface to 559 the control or management system run by the network operator. Note 560 that such control is within the scope of the customer's enhanced VPN, 561 any additional changes beyond this would require some intervention by 562 the network operator. 564 A description of the control plane aspects of this problem are 565 discussed further in Section 5.4. A description of the management 566 plane aspects of this feature can be found in Section 5.5. 568 3.6. Applicability to Overlay Technologies 570 The concept of enhanced VPN can be applied to any existing and future 571 multi-tenancy overlay technologies including but not limited to : 573 * Layer-2 point-to-point services such as pseudowires [RFC3985] 575 * Layer-2 VPNs [RFC4664] 577 * Ethernet VPNs [RFC7209] 579 * Layer-3 VPNs [RFC4364], [RFC2764] 581 Where such VPN service types need enhanced isolation and delivery 582 characteristics, the technologies described in Section 5 can be used 583 to provide an underlay with the required enhanced performance. 585 3.7. Inter-Domain and Inter-Layer Network 587 In some scenarios, an enhanced VPN service may span multiple network 588 domains. A domain is considered to be any collection of network 589 elements within a common realm of address space or path computation 590 responsibility [RFC5151] for example, an Autonomous System. In some 591 domains the network operator may manage a multi-layered network, for 592 example, a packet network over an optical network. When enhanced 593 VPNs are provisioned in such network scenarios, the technologies used 594 in different network planes (data plane, control plane, and 595 management plane) need to provide mechanisms to support multi-domain 596 and multi-layer coordination and integration, so as to provide the 597 required service characteristics for different enhanced VPNs, and 598 improve network efficiency and operational simplicity. 600 4. Architecture of Enhanced VPNs 602 A number of enhanced VPN services will typically be provided by a 603 common network infrastructure. Each enhanced VPN consists of both 604 the overlay and a VTN with a specific set of network resources and 605 service functions allocated in the underlay to satisfy the needs of 606 the VPN customer. One VTN may support one of more enhanced VPNs. 607 The integration between overlay and various underlay resources 608 ensures the required isolation between different enhanced VPNs, and 609 achieves the guaranteed performance for different services. 611 An enhanced VPN needs to be designed with consideration given to: 613 * An enhanced data plane. 615 * A control plane to create enhanced VPNs, making use of the data 616 plane isolation and performance guarantee techniques. 618 * A management plane for enhanced VPN service life-cycle management. 620 These topics are expanded below. 622 * The enhanced data plane: 624 - Provides the required packet latency and jitter 625 characteristics. 627 - Provides the required packet loss characteristics. 629 - Provides the required resource isolation capability, e.g., 630 bandwidth guarantee. 632 - Provides the mechanism to associate a packet with the set of 633 resources allocated to the enhanced VPN to which the packet 634 belongs. 636 * The control plane: 638 - Collects information about the underlying network topology and 639 available resources, and exports this to nodes in the network 640 and/or a centralized controller as required. 642 - Creates the required VTNs with the resources and properties 643 needed by the enhanced VPN services that are they support. 645 - Determines the risk of SLA violation and takes appropriate 646 avoiding action. 648 - Determines the right balance of per-packet and per-node state 649 according to the needs of the enhanced VPN services to scale to 650 the required size. 652 * The management plane: 654 - Provides an interface between the enhanced VPN provider (e.g., 655 operator's network management system ) and the enhanced VPN 656 customer (e.g., an organization or a service with enhanced VPN 657 requirement) such that some of the operation requests can be 658 met without interfering with the enhanced VPN of other 659 customers. 661 - Provides an interface between the enhanced VPN provider and the 662 enhanced VPN customers to expose the network capability 663 information toward the enhanced VPN customer. 665 - Provides the service life-cycle management and operation of 666 enhanced VPNs (e.g., creation, modification, assurance/ 667 monitoring, and decommissioning). 669 * Operations, Administration, and Maintenance (OAM) 671 - Provides the tools to verify the connectivity and performance 672 of the enhanced VPN. 674 - Provides the tools to verify whether the underlay network 675 resources are correctly allocated and operating properly. 677 * Telemetry 678 - Provides the mechanisms to collect network information about 679 the operation of the data plane, control plane, and management 680 plane. More specifically, telemetry provides the mechanisms to 681 collect network data: 683 o from the underlay network for overall performance evaluation 684 and for the planning enhanced VPN services. 686 o from each enhanced VPN and for monitoring and analytics of 687 the characteristics and SLA fulfillment of the enhanced VPN 688 services. 690 4.1. Layered Architecture 692 The layered architecture of an enhanced VPN is shown in Figure 2. 694 Underpinning everything is the physical network infrastructure layer 695 which provide the underlying resources used to provision the 696 separated virtual transport networks (VTNs). This includes the 697 partitioning of link and/or node resources. Each subset of link or 698 node resource can be considered as a virtual link or virtual node 699 used to build the VTNs. 701 /\ 702 || 703 +-------------------+ Centralized 704 | Network Controller| Control & Management 705 +-------------------+ 706 || 707 \/ 708 o---------------------------o VPN Service 1 709 /-------------o 710 o____________/______________o VPN Service 2 711 _________________o 712 _____/ 713 o___/ \_________________o VPN Service 3 714 \_______________________o 715 ...... ... 716 o-----------\ /-------------o 717 o____________X______________o VPN Service n 719 __________________________ 720 / o----o-----o / 721 / / / / VTN-1 722 / o-----o-----o----o----o / 723 /_________________________/ 724 __________________________ 725 / o----o / 727 / / / \ / VTN-2 728 / o-----o----o---o------o / 729 /_________________________/ 730 ...... ... 731 ___________________________ 732 / o----o / 733 / / / / VTN-m 734 / o-----o----o----o-----o / 735 /__________________________/ 737 ++++ ++++ ++++ 738 +--+===+--+===+--+ 739 +--+===+--+===+--+ 740 ++++ +++\\ ++++ 741 || || \\ || Physical 742 || || \\ || Network 743 ++++ ++++ ++++ \\+++ ++++ Infrastructure 744 +--+===+--+===+--+===+--+===+--+ 745 +--+===+--+===+--+===+--+===+--+ 746 ++++ ++++ ++++ ++++ ++++ 748 o Virtual Node ++++ 749 +--+ Physical Node with resource partition 750 -- Virtual Link +--+ 751 ++++ 752 == Physical Link with resource partition 754 Figure 2: The Layered Architecture of VPN+ 756 Various components and techniques discussed in Section 5 can be used 757 to enable resource partitioning, such as FlexE, TSN, DetNet, 758 dedicated queues, etc. These partitions may be physical or virtual 759 so long as the SLA required by the higher layers is met. 761 Based on the network resources provided by the physical network 762 infrastructure, multiple VTNs can be provisioned, each with 763 customized topology and other attributes to meet the requirements of 764 different enhanced VPNs or different groups of enhanced VPNs. To get 765 the required characteristics, each VTN needs to be mapped to a set of 766 network nodes and links in the network infrastructure. And on each 767 node or link, the VTN is associated with a set of resources which are 768 allocated for the processing of traffic in the VTN. The VTN provides 769 the integration between the virtual network topology and the required 770 underlying network resources. The VTN is an essential scaling 771 technique, as it has the potential of eliminating per-path state from 772 the network. In addition, when a group of enhanced VPNs is supported 773 by a single VTN, there is need only to maintain network state for the 774 single VTN (see Section 4.4 for more information). 776 The centralized network controller is used to create the VTN, and to 777 instruct the network nodes to allocate the required resources to each 778 VTN and to provision the enhanced VPN services on the VTNs. A 779 distributed control plane may also be used for the distribution of 780 the VTN topology and attribute information between nodes within the 781 VTNs. 783 The process used to create VTNs and to allocate network resources for 784 use by VTNs needs to take a holistic view of the needs of all of its 785 customers and to partition the resources accordingly. However, 786 within a VTN these resources can, if required, be managed via a 787 dynamic control plane. This provides the required scalability and 788 isolation. 790 4.2. Multi-Point to Multi-Point (MP2MP) Connectivity 792 At the level of a overlay VPN service, the required connectivity for 793 an MP2MP service is usually full or partial mesh. To support such 794 VPN services, the corresponding VTN connectivity also needs to have 795 an abstracted MP2MP connectivity. 797 Other service requirements may be expressed at different 798 granularities, some of which can be applicable to the whole service, 799 while some others may only be applicable to some pairs of end points. 800 For example, when a particular level of performance guarantee is 801 required, the point-to-point path through the underlying VTN of the 802 enhanced VPN may need to be specifically engineered to meet the 803 required performance guarantee. 805 4.3. Application Specific Data Types 807 Although a lot of the traffic that will be carried over the enhanced 808 VPN will likely be IPv4 or IPv6, the design must be capable of 809 carrying other traffic types, in particular Ethernet traffic. This 810 is easily accomplished through the various pseudowire (PW) techniques 811 [RFC3985]. Where the underlay is MPLS, Ethernet can be carried over 812 the enhanced VPN encapsulated according to the method specified in 813 [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - 814 Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic 815 carried according to [RFC4719]. Encapsulations have been defined for 816 most of the common Layer-2 types for both PW over MPLS and for 817 L2TPv3. 819 4.4. Scaling Considerations 821 VPNs are instantiated as overlays on top of an operator's network and 822 offered as services to the operator's customers. An important 823 feature of overlays is that they can deliver services without placing 824 per-service state in the core of the underlay network. 826 Enhanced VPNs may need to install some additional state within the 827 network to achieve the features that they require. Solutions must 828 consider minimizing and controlling the scale of such state, and 829 deployment architectures should constrain the number of enhanced VPNs 830 that would exist where such services would place additional state in 831 the network. It is expected that the number of enhanced VPNs will be 832 small at the beginning, and even in future the number of enhanced 833 VPNs will be much fewer than traditional VPNs because pre-existing 834 VPN techniques are good enough to meet the needs of most existing 835 VPN-type services. 837 In general, it is not required that the state in the network be 838 maintained in a 1:1 relationship with the VPN+ services. It will 839 usually be possible to aggregate a set or group of VPN+ services so 840 that they share the same VTN and the same set of network resources 841 (much in the same way that current VPNs are aggregated over transport 842 tunnels) so that collections of enhanced VPNs that require the same 843 behavior from the network in terms of resource reservation, latency 844 bounds, resiliency, etc. can be grouped together. This is an 845 important feature to assist with the scaling characteristics of VPN+ 846 deployments. 848 [I-D.dong-teas-enhanced-vpn-vtn-scalability] provides more details of 849 scalability considerations for enhanced VPNs, and Section 7 includes 850 a greater discussion of scalability considerations. 852 5. Candidate Technologies 854 A VPN is a network created by applying a demultiplexing technique to 855 the underlying network (the underlay) to distinguish the traffic of 856 one VPN from that of another. A VPN path that travels by other than 857 the shortest path through the underlay normally requires state in the 858 underlay to specify that path. State is normally applied to the 859 underlay through the use of the RSVP-TE signaling protocol, or 860 directly through the use of an SDN controller, although other 861 techniques may emerge as this problem is studied. This state gets 862 harder to manage as the number of VPN paths increases. Furthermore, 863 as we increase the coupling between the underlay and the overlay to 864 support the enhanced VPN service, this state will increase further. 865 Thus, an enhanced VPN solution needs tighter coupling with the 866 underlay than is the case with existing VPN techniques. We cannot, 867 for example, share the network resource between enhanced VPNs which 868 require hard isolation. 870 In an enhanced VPN, different subsets of the underlay resources can 871 be dedicated to different enhanced VPNs or different groups of 872 enhanced VPNs through the use of VTNs. 874 5.1. Packet Forwarding Plane Technologies 876 Several candidate Layer 2 packet- or frame-based data plane solutions 877 which provide the required isolation and guarantees are described in 878 the following sections. 880 5.1.1. Flexible Ethernet 882 FlexE [FLEXE] provides the ability to multiplex channels over an 883 Ethernet link to create point-to-point fixed- bandwidth connections 884 in a way that provides hard isolation. FlexE also supports bonding 885 links to create larger links out of multiple low capacity links. 887 However, FlexE is only a link level technology. When packets are 888 received by the downstream node, they need to be processed in a way 889 that preserves that isolation in the downstream node. This in turn 890 requires a queuing and forwarding implementation that preserves the 891 end-to-end isolation. 893 If different FlexE channels are used for different services, then no 894 sharing is possible between the FlexE channels. This means that it 895 may be difficult to dynamically redistribute unused bandwidth to 896 lower priority services in another FlexE channel. If one FlexE 897 channel is used by one customer, the customer can use some methods to 898 manage the relative priority of their own traffic in the FlexE 899 channel. 901 5.1.2. Dedicated Queues 903 DiffServ based queuing systems are described in [RFC2475] and 904 [RFC4594]. This approach is not sufficient to provide isolation for 905 enhanced VPNs because DiffServ does not provide enough markers to 906 differentiate between traffic of a large number of enhanced VPNs. 907 Nor does DiffServ offer the range of service classes that each VPN 908 needs to provide to its tenants. This problem is particularly acute 909 with an MPLS underlay, because MPLS only provides eight traffic 910 classes. 912 In addition, DiffServ, as currently implemented, mainly provides per- 913 hop priority-based scheduling, and it is difficult to use it to 914 achieve quantitative resource reservation. 916 To address these problems and to reduce the potential interference 917 between enhanced VPNs, it would be necessary to steer traffic to 918 dedicated input and output queues per enhanced VPN: some routers have 919 a large number of queues and sophisticated queuing systems which 920 could support this, while some routers may struggle to provide the 921 granularity and level of isolation required by the applications of 922 enhanced VPN. 924 5.1.3. Time Sensitive Networking 926 Time Sensitive Networking (TSN) [TSN] is an IEEE project to provide a 927 method of carrying time sensitive information over Ethernet. It 928 introduces the concept of packet scheduling where a packet stream may 929 be given a time slot guaranteeing that it experiences no queuing 930 delay or increase in latency beyond the very small scheduling delay. 931 The mechanisms defined in TSN can be used to meet the requirements of 932 time sensitive services of an enhanced VPN. 934 Ethernet can be emulated over a Layer 3 network using an IP or MPLS 935 pseudowire. However, a TSN Ethernet payload would be opaque to the 936 underlay and thus not treated specifically as time sensitive data. 937 The preferred method of carrying TSN over a Layer 3 network is 938 through the use of deterministic networking as explained in 939 Section 5.2.1. 941 5.2. Layer Three Data Plane 943 This section considers the problem of enhanced VPN differentiation 944 and resource representation in the network layer. 946 5.2.1. Deterministic Networking 948 Deterministic Networking (DetNet) [RFC8655] is a technique being 949 developed in the IETF to enhance the ability of Layer-3 networks to 950 deliver packets more reliably and with greater control over the 951 delay. The design cannot use re-transmission techniques such as TCP 952 since that can exceed the delay tolerated by the applications. Even 953 the delay improvements that are achieved with Stream Control 954 Transmission Protocol Partial Reliability Extension (SCTP-PR) 955 [RFC3758] may not meet the bounds set by application demands. DetNet 956 pre-emptively sends copies of the packet over various paths to 957 minimize the chance of all copies of a packet being lost. It also 958 seeks to set an upper bound on latency, but the goal is not to 959 minimize latency. 961 5.2.2. MPLS Traffic Engineering (MPLS-TE) 963 MPLS-TE [RFC2702][RFC3209] introduces the concept of reserving end- 964 to-end bandwidth for a TE-LSP, which can be used to provide a point- 965 to-point Virtual Transport Path (VTP) across the underlay network to 966 support VPNs. VPN traffic can be carried over dedicated TE-LSPs to 967 provide reserved bandwidth for each specific connection in a VPN, and 968 VPNs with similar behavior requirements may be multiplexed onto the 969 same TE-LSPs. Some network operators have concerns about the 970 scalability and management overhead of MPLS-TE system especially with 971 regard to those systems that use an active control plane, and this 972 has lead them to consider other solutions for their networks. 974 5.2.3. Segment Routing 976 Segment Routing (SR) [RFC8402] is a method that prepends instructions 977 to packets at the head-end of a path. These instructions are used to 978 specify the nodes and links to be traversed, and allow the packets to 979 be routed on paths other than the shortest path. By encoding the 980 state in the packet, per-path state is transitioned out of the 981 network. 983 An SR traffic engineered path operates with a granularity of a link. 984 Hints about priority are provided using the Traffic Class (TC) or 985 Differentiated Services Code Point (DSCP) field in the header. 986 However, to achieve the latency and isolation characteristics that 987 are sought by enhanced VPN customers, it will be necessary to steer 988 packets through specific virtual links and/or queues on the same link 989 and direct them to use specific resources. With SR, it is possible 990 to introduce such fine-grained packet steering by specifying the 991 queues and resources through an SR instruction list. 993 Note that the concept of a queue is a useful abstraction for 994 different types of underlay mechanism that may be used to provide 995 enhanced isolation and latency support. How the queue satisfies the 996 requirement is implementation specific and is transparent to the 997 layer-3 data plane and control plane mechanisms used. 999 With Segment Routing, the SR instruction list could be used to build 1000 a P2P path, and a group of SR SIDs could also be used to represent an 1001 MP2MP network. Thus, the SR based mechanism could be used to provide 1002 both a Virtual Transport Path (VTP) and a Virtual Transport Network 1003 (VTN) for enhanced VPN services. 1005 5.3. Non-Packet Data Plane 1007 Non-packet underlay data plane technologies often have TE properties 1008 and behaviors, and meet many of the key requirements in particular 1009 for bandwidth guarantees, traffic isolation (with physical isolation 1010 often being an integral part of the technology), highly predictable 1011 latency and jitter characteristics, measurable loss characteristics, 1012 and ease of identification of flows. The cost is that the resources 1013 are allocated on a long-term and end-to-end basis. Such an 1014 arrangement means that the full cost of the resources has to be borne 1015 by the service that is allocated with the resources. 1017 5.4. Control Plane 1019 An enhanced VPN would likely be based on a hybrid control mechanism 1020 that takes advantage of a logically centralized controller for on- 1021 demand provisioning and global optimization, whilst still relying on 1022 a distributed control plane to provide scalability, high reliability, 1023 fast reaction, automatic failure recovery, etc. Extension to and 1024 optimization of the centralized and distributed control plane is 1025 needed to support the enhanced properties of VPN+. 1027 RSVP-TE [RFC3209] provides the signaling mechanism for establishing a 1028 TE-LSP in an MPLS network with end-to-end resource reservation. This 1029 can be seen as an approach of providing a Virtual Transport Path 1030 (VTP) which could be used to bind the VPN to specific network 1031 resources allocated within the underlay, but there remain scalability 1032 concerns as mentioned in Section 5.2.2. 1034 The control plane of SR [RFC8665] [RFC8667] [RFC9085] does not have 1035 the capability of signaling resource reservations along the path. On 1036 the other hand, the SR approach provides a potential way of binding 1037 the underlay network resource and the enhanced VPN service without 1038 requiring per-path state to be maintained in the network. A 1039 centralized controller can perform resource planning and reservation 1040 for VTNs, while it needs to ensure that resources are correctly 1041 allocated in network nodes for the enhanced VPN service. The 1042 controller could also compute the SR paths based on the planned or 1043 collected network resource and other attributes, and provision the SR 1044 paths based on the mechanism in 1045 [I-D.ietf-spring-segment-routing-policy] to the ingress nodes of the 1046 enhanced VPN services. The distributed control plane may be used to 1047 advertise the network topology and resource attributes associated 1048 with the VTNs, and compute the SR paths with VTN specific constraints 1049 for the enhanced VPN services. 1051 5.5. Management Plane 1053 The management plane provides the interface between the enhanced VPN 1054 provider and the customers for life-cycle management of the service 1055 (i.e., creation, modification, assurance/monitoring, and 1056 decommissioning). It relies on a set of service data models for the 1057 description of the information and operations needed on the 1058 interface. 1060 As an example, in the context of 5G end-to-end network slicing 1061 [TS28530], the management of enhanced VPNs is considered as the 1062 management of the transport network part of the 5G end-to-end network 1063 slice. The 3GPP management system may provide the connectivity and 1064 performance related parameters as requirements to the management 1065 plane of the transport network. It may also require the transport 1066 network to expose the capabilities and status of the network slice. 1067 Thus, an interface between the enhanced VPN management plane and the 1068 5G network slice management system, and relevant service data models 1069 are needed for the coordination of 5G end-to-end network slice 1070 management. 1072 The management plane interface and data models for enhanced VPN can 1073 be based on the service models described in Section 5.6. 1075 It is important that the management life-cycle supports in-place 1076 modification of enhanced VPNs. That is, it should be possible to add 1077 and remove end points, as well as to change the requested 1078 characteristics of the service that is delivered. The management 1079 system needs to be able to assess the revised VPN+ requests and 1080 determine whether they can be provided by the existing VTN or whether 1081 changes must be made, and it will additionally need to determine 1082 whether those changes to the VTN are possible. If not, then the 1083 customer's modification request may be rejected. 1085 When the modification of an enhanced VPN is possible, the management 1086 system should make every effort to make the changes in a non- 1087 disruptive way. That is, the modification of the enhanced VPN or the 1088 underlying VTN should not perturbate traffic on the enhanced VPN in a 1089 way that causes the service level to drop below the agreed levels. 1090 Furthermore, in the spirit of isolation, changes to one enhanced VPN 1091 should not cause disruption to other enhanced VPNs. 1093 The network operator for the underlay network (i.e., the provider of 1094 the enhanced VPN) may delegate some operational aspects of the 1095 enhanced VPN to the customer. In this way, the VPN+ is presented to 1096 the customer as a virtual network, and the customer can choose how to 1097 use that network. The customer cannot exceed the capabilities of 1098 virtual links and nodes, but can decide how to load traffic onto the 1099 network, for example, by assigning different metrics to the virtual 1100 links so that the customer can control how traffic is routed through 1101 the overlay. This approach requires a management system for the 1102 overlay network, but does not necessarily require any coordination 1103 between the underlay and overlay management systems, except that the 1104 overlay management system might notice when the enhanced VPN network 1105 is close to capacity or considerably under-used and automatically 1106 request changes in the service provided by the underlay. 1108 5.6. Applicability of Service Data Models to Enhanced VPN 1110 This section describes the applicability of the existing and in- 1111 progress service data models to enhanced VPN. New service models may 1112 also be introduced for some of the required management functions. 1114 The ACTN framework[RFC8453] supports operators in viewing and 1115 controlling different domains and presenting virtualized networks to 1116 their customers. It highlights how: 1118 * Abstraction of the underlying network resources is provided to 1119 higher-layer applications and customers. 1121 * Underlying resources are virtualized and allocated for the 1122 customer, application, or service. 1124 * A virtualized environment is created allowing operators to view 1125 and control multi-domain networks as a single virtualized network. 1127 * Networks can be presented to customers as a virtual network via 1128 open and programmable interfaces. 1130 The type of network virtualization enabled by ACTN managed 1131 infrastructure provides customers with the capability to utilize and 1132 independently control allocated virtual network resources as if they 1133 were physically their own resources. Service Data models are used to 1134 represent, monitor, and manage the virtual networks and services 1135 enabled by ACTN. The VPN customer service models (e.g., the layer 3 1136 VPN service model (L3SM) [RFC8299], the layer 2 VPN service model 1137 (L2SM) [RFC8466]), or the ACTN Virtual Network (VN) model 1138 [I-D.ietf-teas-actn-vn-yang]) are a customer view of the ACTN managed 1139 infrastructure, and is presented by the ACTN provider as a set of 1140 abstracted services or resources. The layer 3 VPN network model 1141 (L3NM) [I-D.ietf-opsawg-l3sm-l3nm] and layer 2 VPN network model 1142 (L2NM) [I-D.ietf-opsawg-l2nm] provide network views of the ACTN 1143 managed infrastructure presented by the ACTN provider as a set of 1144 virtual networks and the associated resources. The VTN network model 1145 [I-D.wd-teas-vtn-network-yang] further provides the management of the 1146 virtual underlay network topology and resources for the mapping of 1147 the VPN network models. 1149 [I-D.ietf-teas-applicability-actn-slicing] discusses the 1150 applicability of the ACTN approach in the context of network slicing. 1151 Since there is a strong correlation between network slices and 1152 enhanced VPNs, that document can also give guidance on how ACTN can 1153 be applied to enhanced VPNs. 1155 6. Applicability to Network Slice Realization 1157 One of the typical use cases of enhanced VPN is to deliver IETF 1158 network slice service. This section describes the applicability of 1159 enhanced VPN to network slice realization. 1161 In order to provide network slices to customers, a technology- 1162 agnostic network slice service Northbound Interface (NBI) data model 1163 [I-D.ietf-teas-ietf-network-slice-nbi-yang] is needed for the 1164 customers to communicate the requirements of IETF network slices (end 1165 points, connectivity, SLOs, and SLEs). These requirements may be 1166 realized using technology specified in this document to instruct the 1167 network to instantiate an enhanced VPN to meet the requirements of 1168 the IETF network slice customers. 1170 6.1. VTN Planning and Instantiation 1172 According to the network operators' network resource planning policy, 1173 or based on the requirement of one or a group of customers or 1174 services, a VTN may need to be created. One of the basic 1175 requirements for a VTN is to provide a set of dedicated network 1176 resources to avoid unexpected interference from other services in the 1177 same network. Other possible requirements may include the required 1178 topology and connectivity, bandwidth, latency, reliability, etc. 1180 A centralized network controller can be responsible for calculating a 1181 subset of the underlay network topology (which is called a logical 1182 topology) to support the VTN requirement. And on the network nodes 1183 and links within the logical topology, the set of network resources 1184 to be allocated to the VTN can also be determined by the controller. 1186 Normally such calculation needs to take the underlay network 1187 connectivity information and the available network resource 1188 information of the underlay network into consideration. The network 1189 controller may also take the status of the existing VTNs into 1190 consideration in the planning and calculation of a new VTN. 1192 According to the result of the VTN planning, the network nodes and 1193 links involved in the logical topology of the VTN are instructed to 1194 allocated the required set of network resources for the VTN. One or 1195 multiple mechanisms as specified in section 5.1 can be used to 1196 partition the forwarding plane network resources for different VTNs. 1197 In addition, the data plane VTN identifiers which are used to 1198 identify the set of network resources allocated to the VTN are also 1199 provisioned on the network nodes. Depends on the data plane 1200 technologies used, the set of network resources of a VTN can be 1201 identified using either resource aware SR segments as specified in 1202 [I-D.ietf-spring-resource-aware-segments], or a dedicated VTN 1203 resource ID as specified in [I-D.dong-6man-enhanced-vpn-vtn-id] can 1204 be introduced. The network nodes involved in a VTN may distribute 1205 the logical topology information, the VTN specific network resource 1206 information and the VTN resource identifiers using the control plane, 1207 which could be used by the controller and the network nodes to 1208 compute the TE paths within the VTN, and install the VTN specific 1209 forwarding entries. 1211 6.2. Enhanced VPN Provisioning 1213 According to the connectivity requirements of an IETF network slice 1214 service, an overlay VPN can be created using the existing or future 1215 multi-tenancy overlay technologies as described in Section 3.6. 1217 Then according to the SLOs and SLEs requirements of the network 1218 slice, the overlay VPN is mapped to an appropriate VTN as the virtual 1219 underlay. The integration of the overlay VPN and the underlay VTN 1220 together provide an enhanced VPN service which can meet the network 1221 slice service requirements. 1223 6.3. Network Slice Traffic Steering and Forwarding 1225 At the edge of the operator's network, traffic of IETF network slices 1226 can be classified based on the matching rules defined by operator's 1227 policy, so that the traffic is mapped to a specific enhanced VPN, 1228 which is further mapped to a underlay VTN. Packets belonging to the 1229 enhanced VPN will be processed and forwarded by network nodes using 1230 the set of network resources allocated to the corresponding VTN. 1232 7. Scalability Considerations 1234 An enhanced VPN provides performance guaranteed services in packet 1235 networks, but with the potential cost of introducing additional state 1236 into the network. There are at least three ways that this additional 1237 state might be brought into the network: 1239 * Introduce the complete state into the packet, as is done in SR. 1240 This allows the controller to specify the detailed series of 1241 forwarding and processing instructions for the packet as it 1242 transits the network. The cost of this is an increase in the 1243 packet header size. The cost is also that systems will have 1244 capabilities enabled in case they are called upon by a service. 1245 This is a type of latent state, and increases as the path and 1246 resources that need to be exclusively available to a VPN are 1247 specified more precisely. 1249 * Introduce the state to the network. This is normally done by 1250 creating a path using signaling such as RSVP-TE. This could be 1251 extended to include any element that needs to be specified along 1252 the path, for example explicitly specifying queuing policy. It is 1253 also possible to use other methods to introduce path state, such 1254 as via an SDN controller, or possibly by modifying a routing 1255 protocol. With this approach there is state per path: per-path 1256 characteristic that needs to be maintained over the life of the 1257 path. This is more network state than is needed using SR, but the 1258 packets are usually shorter. 1260 * Provide a hybrid approach. One example is based on using binding 1261 SIDs [RFC8402] to create path fragments, and bind them together 1262 with SR. Dynamic creation of a VPN service path using SR requires 1263 less state maintenance in the network core at the expense of 1264 larger packet headers. The packet size can be lower if a form of 1265 loose source routing is used (using a few nodal SIDs), and it will 1266 be lower if no specific functions or resources on the routers are 1267 specified. 1269 Reducing the state in the network is important to enhanced VPN, as it 1270 requires the overlay to be more closely integrated with the underlay 1271 than with traditional VPNs. This tighter coupling would normally 1272 mean that more state needs to be created and maintained in the 1273 network, as the state about fine granularity processing would need to 1274 be loaded and maintained in the routers. However, an SR approach 1275 allows much of this state to be spread amongst the network ingress 1276 nodes, and transiently carried in the packets as SIDs. 1278 Further discussion of the scalability considerations of enhanced VPNs 1279 can be found in [I-D.dong-teas-enhanced-vpn-vtn-scalability]. 1281 7.1. Maximum Stack Depth of SR 1283 One of the challenges with SR is the stack depth that nodes are able 1284 to impose on packets [RFC8491]. This leads to a difficult balance 1285 between adding state to the network and minimizing stack depth, or 1286 minimizing state and increasing the stack depth. 1288 7.2. RSVP-TE Scalability 1290 The traditional method of creating a resource allocated path through 1291 an MPLS network is to use the RSVP-TE protocol. However, there have 1292 been concerns that this requires significant continuous state 1293 maintenance in the network. Work to improve the scalability of RSVP- 1294 TE LSPs in the control plane can be found in [RFC8370]. 1296 There is also concern at the scalability of the forwarder footprint 1297 of RSVP-TE as the number of paths through a label switching router 1298 (LSR) grows. [RFC8577] addresses this by employing SR within a 1299 tunnel established by RSVP-TE. 1301 7.3. SDN Scaling 1303 The centralized approach of SDN requires state to be stored in the 1304 network, but does not have the overhead of also requiring control 1305 plane state to be maintained. Each individual network node may need 1306 to maintain a communication channel with the SDN controller, but that 1307 compares favorably with the need for a control plane to maintain 1308 communication with all neighbors. 1310 However, SDN may transfer some of the scalability concerns from the 1311 network to the centralized controller. In particular, there may be a 1312 heavy processing burden at the controller, and a heavy load in the 1313 network surrounding the controller. A centralized controller also 1314 presents a single point of failure within the network. 1316 8. OAM Considerations 1318 The design of OAM for enhanced VPNs needs to consider the following 1319 requirements: 1321 * Instrumentation of the underlay so that the network operator can 1322 be sure that the resources committed to a tenant are operating 1323 correctly and delivering the required performance. 1325 * Instrumentation of the overlay by the tenant. This is likely to 1326 be transparent to the network operator and to use existing 1327 methods. Particular consideration needs to be given to the need 1328 to verify the isolation and the various committed performance 1329 characteristics. 1331 * Instrumentation of the overlay by the network provider to 1332 proactively demonstrate that the committed performance is being 1333 delivered. This needs to be done in a non-intrusive manner, 1334 particularly when the tenant is deploying a performance sensitive 1335 application. 1337 * Verification of the conformity of the path to the service 1338 requirement. This may need to be done as part of a commissioning 1339 test. 1341 A study of OAM in SR networks has been documented in [RFC8403]. 1343 9. Telemetry Considerations 1345 Network visibility is essential for network operation. Network 1346 telemetry has been considered as an ideal means to gain sufficient 1347 network visibility with better flexibility, scalability, accuracy, 1348 coverage, and performance than conventional OAM technologies. 1350 As defined in [I-D.ietf-opsawg-ntf], the objective of Network 1351 Telemetry is to acquire network data remotely for network monitoring 1352 and operation. It is a general term for a large set of network 1353 visibility techniques and protocols. Network telemetry addresses the 1354 current network operation issues and enables smooth evolution toward 1355 intent-driven autonomous networks. Telemetry can be applied on the 1356 forwarding plane, the control plane, and the management plane in a 1357 network. 1359 How the telemetry mechanisms could be used or extended for the 1360 enhanced VPN service is out of the scope of this document. 1362 10. Enhanced Resiliency 1364 Each enhanced VPN has a life cycle, and may need modification during 1365 deployment as the needs of its tenant change. This is discussed in 1366 Section 5.5. Additionally, as the network evolves, there may need to 1367 be garbage collection performed to consolidate resources into usable 1368 quanta. 1370 Systems in which the path is imposed, such as SR or some form of 1371 explicit routing, tend to do well in these applications, because it 1372 is possible to perform an atomic transition from one path to another. 1374 That is, a single action by the head-end that changes the path 1375 without the need for coordinated action by the routers along the 1376 path. However, implementations and the monitoring protocols need to 1377 make sure that the new path is operational and meets the required SLA 1378 before traffic is transitioned to it. It is possible for deadlocks 1379 to arise as a result of the network becoming fragmented over time, 1380 such that it is impossible to create a new path or to modify an 1381 existing path without impacting the SLA of other paths. Resolution 1382 of this situation is as much a commercial issue as it is a technical 1383 issue and is outside the scope of this document. 1385 There are, however, two manifestations of the latency problem that 1386 are for further study in any of these approaches: 1388 * The problem of packets overtaking one another if a path latency 1389 reduces during a transition. 1391 * The problem of transient variation in latency in either direction 1392 as a path migrates. 1394 There is also the matter of what happens during failure in the 1395 underlay infrastructure. Fast reroute is one approach, but that 1396 still produces a transient loss with a normal goal of rectifying this 1397 within 50ms [RFC5654]. An alternative is some form of N+1 delivery 1398 such as has been used for many years to support protection from 1399 service disruption. This may be taken to a different level using the 1400 techniques of DetNet with multiple in-network replication and the 1401 culling of later packets [RFC8655]. 1403 In addition to the approach used to protect high priority packets, 1404 consideration should be given to the impact of best effort traffic on 1405 the high priority packets during a transition. Specifically, if a 1406 conventional re-convergence process is used there will inevitably be 1407 micro-loops and whilst some form of explicit routing will protect the 1408 high priority traffic, lower priority traffic on best effort shortest 1409 paths will micro-loop without the use of a loop prevention 1410 technology. To provide the highest quality of service to high 1411 priority traffic, either this traffic must be shielded from the 1412 micro-loops, or micro-loops must be prevented completely. 1414 11. Operational Considerations 1416 It is likely that enhanced VPN services will be introduced in 1417 networks which already have traditional VPN services deployed. 1418 Depending on service requirements, the tenants or the operator may 1419 choose to use a traditional VPN or an enhanced VPN to fulfill a 1420 service requirement. The information and parameters to assist such a 1421 decision needs to be reflected on the management interface between 1422 the tenant and the operator. 1424 12. Security Considerations 1426 All types of virtual network require special consideration to be 1427 given to the isolation of traffic belonging to different tenants. 1428 That is, traffic belonging to one VPN must not be delivered to end 1429 points outside that VPN. In this regard enhanced VPNs neither 1430 introduce, nor experience a greater security risks than other VPNs. 1432 However, in an enhanced Virtual Private Network service the 1433 additional service requirements need to be considered. For example, 1434 if a service requires a specific upper bound to latency then it can 1435 be damaged by simply delaying the packets through the activities of 1436 another tenant, i.e., by introducing bursts of traffic for other 1437 services. In some respects this makes the enhanced VPN more 1438 susceptible to attacks since the SLA may be broken. But another view 1439 is that the operator must, in any case, preform monitoring of the 1440 enhanced VPN to ensure that the SLA is met, and this means that the 1441 operator may be more likely to spot the early onset of a security 1442 attack and be able to take pre-emptive protective action. 1444 The measures to address these dynamic security risks must be 1445 specified as part to the specific solution are form part of the 1446 isolation requirements of a service. 1448 While an enhanced VPN service may be sold as offering encryption and 1449 other security features as part of the service, customers would be 1450 well advised to take responsibility for their own security 1451 requirements themselves possibly by encrypting traffic before handing 1452 it off to the service provider. 1454 The privacy of enhanced VPN service customers must be preserved. It 1455 should not be possible for one customer to discover the existence of 1456 another customer, nor should the sites that are members of an 1457 enhanced VPN be externally visible. 1459 13. IANA Considerations 1461 There are no requested IANA actions. 1463 14. Contributors 1465 Daniel King 1466 Email: daniel@olddog.co.uk 1468 Adrian Farrel 1469 Email: adrian@olddog.co.uk 1471 Jeff Tansura 1472 Email: jefftant.ietf@gmail.com 1474 Zhenbin Li 1475 Email: lizhenbin@huawei.com 1477 Qin Wu 1478 Email: bill.wu@huawei.com 1480 Bo Wu 1481 Email: lana.wubo@huawei.com 1483 Daniele Ceccarelli 1484 Email: daniele.ceccarelli@ericsson.com 1486 Mohamed Boucadair 1487 Email: mohamed.boucadair@orange.com 1489 Sergio Belotti 1490 Email: sergio.belotti@nokia.com 1492 Haomian Zheng 1493 Email: zhenghaomian@huawei.com 1495 15. Acknowledgements 1497 The authors would like to thank Charlie Perkins, James N Guichard, 1498 John E Drake, Shunsuke Homma, and Luis M. Contreras for their review 1499 and valuable comments. 1501 This work was supported in part by the European Commission funded 1502 H2020-ICT-2016-2 METRO-HAUL project (G.A. 761727). 1504 16. References 1506 16.1. Normative References 1508 [RFC2764] Gleeson, B., Lin, A., Heinanen, J., Armitage, G., and A. 1509 Malis, "A Framework for IP Based Virtual Private 1510 Networks", RFC 2764, DOI 10.17487/RFC2764, February 2000, 1511 . 1513 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 1514 Edge-to-Edge (PWE3) Architecture", RFC 3985, 1515 DOI 10.17487/RFC3985, March 2005, 1516 . 1518 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 1519 2 Virtual Private Networks (L2VPNs)", RFC 4664, 1520 DOI 10.17487/RFC4664, September 2006, 1521 . 1523 16.2. Informative References 1525 [BBF-SD406] 1526 "BBF SD-406: End-to-End Network Slicing", 2016, 1527 . 1530 [DETNET] "Deterministic Networking", March , 1531 . 1533 [FLEXE] "Flex Ethernet Implementation Agreement", March 2016, 1534 . 1537 [I-D.dong-6man-enhanced-vpn-vtn-id] 1538 Dong, J., Li, Z., Xie, C., Ma, C., and G. Mishra, 1539 "Carrying Virtual Transport Network Identifier in IPv6 1540 Extension Header", Work in Progress, Internet-Draft, 1541 draft-dong-6man-enhanced-vpn-vtn-id-05, 8 September 2021, 1542 . 1545 [I-D.dong-teas-enhanced-vpn-vtn-scalability] 1546 Dong, J., Li, Z., Gong, L., Yang, G., Guichard, J. N., 1547 Mishra, G., and F. Qin, "Scalability Considerations for 1548 Enhanced VPN (VPN+)", Work in Progress, Internet-Draft, 1549 draft-dong-teas-enhanced-vpn-vtn-scalability-03, 11 July 1550 2021, . 1553 [I-D.ietf-opsawg-l2nm] 1554 Barguil, S., Dios, O. G. D., Boucadair, M., and L. A. 1555 Munoz, "A Layer 2 VPN Network YANG Model", Work in 1556 Progress, Internet-Draft, draft-ietf-opsawg-l2nm-09, 20 1557 October 2021, . 1560 [I-D.ietf-opsawg-l3sm-l3nm] 1561 Barguil, S., Dios, O. G. D., Boucadair, M., Munoz, L. A., 1562 and A. Aguado, "A Layer 3 VPN Network YANG Model", Work in 1563 Progress, Internet-Draft, draft-ietf-opsawg-l3sm-l3nm-18, 1564 8 October 2021, . 1567 [I-D.ietf-opsawg-ntf] 1568 Song, H., Qin, F., Martinez-Julia, P., Ciavaglia, L., and 1569 A. Wang, "Network Telemetry Framework", Work in Progress, 1570 Internet-Draft, draft-ietf-opsawg-ntf-09, 13 October 2021, 1571 . 1574 [I-D.ietf-spring-resource-aware-segments] 1575 Dong, J., Bryant, S., Miyasaka, T., Zhu, Y., Qin, F., Li, 1576 Z., and F. Clad, "Introducing Resource Awareness to SR 1577 Segments", Work in Progress, Internet-Draft, draft-ietf- 1578 spring-resource-aware-segments-03, 12 July 2021, 1579 . 1582 [I-D.ietf-spring-segment-routing-policy] 1583 Filsfils, C., Talaulikar, K., Voyer, D., Bogdanov, A., and 1584 P. Mattes, "Segment Routing Policy Architecture", Work in 1585 Progress, Internet-Draft, draft-ietf-spring-segment- 1586 routing-policy-13, 28 May 2021, 1587 . 1590 [I-D.ietf-teas-actn-vn-yang] 1591 Lee, Y., Dhody, D., Ceccarelli, D., Bryskin, I., and B. Y. 1592 Yoon, "A YANG Data Model for VN Operation", Work in 1593 Progress, Internet-Draft, draft-ietf-teas-actn-vn-yang-13, 1594 23 October 2021, . 1597 [I-D.ietf-teas-actn-yang] 1598 Lee, Y., Zheng, H., Ceccarelli, D., Yoon, B. Y., and S. 1599 Belotti, "Applicability of YANG models for Abstraction and 1600 Control of Traffic Engineered Networks", Work in Progress, 1601 Internet-Draft, draft-ietf-teas-actn-yang-08, 8 September 1602 2021, . 1605 [I-D.ietf-teas-applicability-actn-slicing] 1606 King, D., Drake, J., Zheng, H., and A. Farrel, 1607 "Applicability of Abstraction and Control of Traffic 1608 Engineered Networks (ACTN) to Network Slicing", Work in 1609 Progress, Internet-Draft, draft-ietf-teas-applicability- 1610 actn-slicing-00, 21 September 2021, 1611 . 1614 [I-D.ietf-teas-ietf-network-slice-nbi-yang] 1615 Wu, B., Dhody, D., Rokui, R., Saad, T., and L. Han, "IETF 1616 Network Slice Service YANG Model", Work in Progress, 1617 Internet-Draft, draft-ietf-teas-ietf-network-slice-nbi- 1618 yang-00, 29 September 2021, 1619 . 1622 [I-D.ietf-teas-ietf-network-slices] 1623 Farrel, A., Gray, E., Drake, J., Rokui, R., Homma, S., 1624 Makhijani, K., Contreras, L. M., and J. Tantsura, 1625 "Framework for IETF Network Slices", Work in Progress, 1626 Internet-Draft, draft-ietf-teas-ietf-network-slices-04, 23 1627 August 2021, . 1630 [I-D.wd-teas-vtn-network-yang] 1631 Wu, B. and D. Dhody, "A VTN Network YANG Module", Work in 1632 Progress, Internet-Draft, draft-wd-teas-vtn-network-yang- 1633 00, 6 June 2021, . 1636 [NGMN-NS-Concept] 1637 "NGMN NS Concept", 2016, . 1641 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 1642 and W. Weiss, "An Architecture for Differentiated 1643 Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, 1644 . 1646 [RFC2702] Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and J. 1647 McManus, "Requirements for Traffic Engineering Over MPLS", 1648 RFC 2702, DOI 10.17487/RFC2702, September 1999, 1649 . 1651 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 1652 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 1653 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 1654 . 1656 [RFC3758] Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1657 Conrad, "Stream Control Transmission Protocol (SCTP) 1658 Partial Reliability Extension", RFC 3758, 1659 DOI 10.17487/RFC3758, May 2004, 1660 . 1662 [RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., 1663 "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", 1664 RFC 3931, DOI 10.17487/RFC3931, March 2005, 1665 . 1667 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 1668 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 1669 2006, . 1671 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 1672 "Encapsulation Methods for Transport of Ethernet over MPLS 1673 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 1674 . 1676 [RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration 1677 Guidelines for DiffServ Service Classes", RFC 4594, 1678 DOI 10.17487/RFC4594, August 2006, 1679 . 1681 [RFC4719] Aggarwal, R., Ed., Townsley, M., Ed., and M. Dos Santos, 1682 Ed., "Transport of Ethernet Frames over Layer 2 Tunneling 1683 Protocol Version 3 (L2TPv3)", RFC 4719, 1684 DOI 10.17487/RFC4719, November 2006, 1685 . 1687 [RFC5151] Farrel, A., Ed., Ayyangar, A., and JP. Vasseur, "Inter- 1688 Domain MPLS and GMPLS Traffic Engineering -- Resource 1689 Reservation Protocol-Traffic Engineering (RSVP-TE) 1690 Extensions", RFC 5151, DOI 10.17487/RFC5151, February 1691 2008, . 1693 [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., 1694 Sprecher, N., and S. Ueno, "Requirements of an MPLS 1695 Transport Profile", RFC 5654, DOI 10.17487/RFC5654, 1696 September 2009, . 1698 [RFC7149] Boucadair, M. and C. Jacquenet, "Software-Defined 1699 Networking: A Perspective from within a Service Provider 1700 Environment", RFC 7149, DOI 10.17487/RFC7149, March 2014, 1701 . 1703 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 1704 Henderickx, W., and A. Isaac, "Requirements for Ethernet 1705 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 1706 . 1708 [RFC7926] Farrel, A., Ed., Drake, J., Bitar, N., Swallow, G., 1709 Ceccarelli, D., and X. Zhang, "Problem Statement and 1710 Architecture for Information Exchange between 1711 Interconnected Traffic-Engineered Networks", BCP 206, 1712 RFC 7926, DOI 10.17487/RFC7926, July 2016, 1713 . 1715 [RFC8172] Morton, A., "Considerations for Benchmarking Virtual 1716 Network Functions and Their Infrastructure", RFC 8172, 1717 DOI 10.17487/RFC8172, July 2017, 1718 . 1720 [RFC8299] Wu, Q., Ed., Litkowski, S., Tomotaki, L., and K. Ogaki, 1721 "YANG Data Model for L3VPN Service Delivery", RFC 8299, 1722 DOI 10.17487/RFC8299, January 2018, 1723 . 1725 [RFC8370] Beeram, V., Ed., Minei, I., Shakir, R., Pacella, D., and 1726 T. Saad, "Techniques to Improve the Scalability of RSVP-TE 1727 Deployments", RFC 8370, DOI 10.17487/RFC8370, May 2018, 1728 . 1730 [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., 1731 Decraene, B., Litkowski, S., and R. Shakir, "Segment 1732 Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, 1733 July 2018, . 1735 [RFC8403] Geib, R., Ed., Filsfils, C., Pignataro, C., Ed., and N. 1736 Kumar, "A Scalable and Topology-Aware MPLS Data-Plane 1737 Monitoring System", RFC 8403, DOI 10.17487/RFC8403, July 1738 2018, . 1740 [RFC8453] Ceccarelli, D., Ed. and Y. Lee, Ed., "Framework for 1741 Abstraction and Control of TE Networks (ACTN)", RFC 8453, 1742 DOI 10.17487/RFC8453, August 2018, 1743 . 1745 [RFC8466] Wen, B., Fioccola, G., Ed., Xie, C., and L. Jalil, "A YANG 1746 Data Model for Layer 2 Virtual Private Network (L2VPN) 1747 Service Delivery", RFC 8466, DOI 10.17487/RFC8466, October 1748 2018, . 1750 [RFC8491] Tantsura, J., Chunduri, U., Aldrin, S., and L. Ginsberg, 1751 "Signaling Maximum SID Depth (MSD) Using IS-IS", RFC 8491, 1752 DOI 10.17487/RFC8491, November 2018, 1753 . 1755 [RFC8568] Bernardos, CJ., Rahman, A., Zuniga, JC., Contreras, LM., 1756 Aranda, P., and P. Lynch, "Network Virtualization Research 1757 Challenges", RFC 8568, DOI 10.17487/RFC8568, April 2019, 1758 . 1760 [RFC8577] Sitaraman, H., Beeram, V., Parikh, T., and T. Saad, 1761 "Signaling RSVP-TE Tunnels on a Shared MPLS Forwarding 1762 Plane", RFC 8577, DOI 10.17487/RFC8577, April 2019, 1763 . 1765 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1766 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1767 . 1769 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1770 "Deterministic Networking Architecture", RFC 8655, 1771 DOI 10.17487/RFC8655, October 2019, 1772 . 1774 [RFC8665] Psenak, P., Ed., Previdi, S., Ed., Filsfils, C., Gredler, 1775 H., Shakir, R., Henderickx, W., and J. Tantsura, "OSPF 1776 Extensions for Segment Routing", RFC 8665, 1777 DOI 10.17487/RFC8665, December 2019, 1778 . 1780 [RFC8667] Previdi, S., Ed., Ginsberg, L., Ed., Filsfils, C., 1781 Bashandy, A., Gredler, H., and B. Decraene, "IS-IS 1782 Extensions for Segment Routing", RFC 8667, 1783 DOI 10.17487/RFC8667, December 2019, 1784 . 1786 [RFC9085] Previdi, S., Talaulikar, K., Ed., Filsfils, C., Gredler, 1787 H., and M. Chen, "Border Gateway Protocol - Link State 1788 (BGP-LS) Extensions for Segment Routing", RFC 9085, 1789 DOI 10.17487/RFC9085, August 2021, 1790 . 1792 [SFC] "Service Function Chaining", March , 1793 . 1795 [TS23501] "3GPP TS23.501", 2016, 1796 . 1799 [TS28530] "3GPP TS28.530", 2016, 1800 . 1803 [TSN] "Time-Sensitive Networking", March , 1804 . 1806 Authors' Addresses 1808 Jie Dong 1809 Huawei 1811 Email: jie.dong@huawei.com 1813 Stewart Bryant 1814 University of Surrey 1816 Email: stewart.bryant@gmail.com 1818 Zhenqiang Li 1819 China Mobile 1821 Email: lizhenqiang@chinamobile.com 1823 Takuya Miyasaka 1824 KDDI Corporation 1826 Email: ta-miyasaka@kddi.com 1828 Young Lee 1829 Samsung 1831 Email: younglee.tx@gmail.com