idnits 2.17.00 (12 Aug 2021) /tmp/idnits16637/draft-sr-bess-evpn-vpws-gateway-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document date (7 March 2022) is 68 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-04) exists of draft-ietf-bess-rfc7432bis-03 == Outdated reference: A later version (-15) exists of draft-ietf-bess-srv6-services-12 == Outdated reference: A later version (-08) exists of draft-agrawal-spring-srv6-mpls-interworking-07 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS Workgroup J. Rabadan, Ed. 3 Internet-Draft S. Sathappan 4 Intended status: Standards Track V. Prabhu 5 Expires: 8 September 2022 Nokia 6 W. Lin 7 Juniper 8 P. Brissette 9 Cisco Systems 10 7 March 2022 12 Ethernet VPN Virtual Private Wire Services Gateway Solution 13 draft-sr-bess-evpn-vpws-gateway-00 15 Abstract 17 Ethernet VPN Virtual Private Wire Services (EVPN VPWS) need to be 18 deployed in high scale multi-domain networks, where each domain can 19 use a different transport technology, such as MPLS, VXLAN or Segment 20 Routing with MPLS or IPv6 Segment Identifiers (SIDs). While the 21 transport interworking solutions on border routers spare the border 22 routers from having to process service routes, they do not always 23 meet the multi-homing, redundancy, and operational requirements, or 24 provide the isolation that each domain requires. This document 25 analyzes the scenarios in which an interconnect solution for EVPN 26 VPWS using EVPN Domain Gateways is needed, and adds the required 27 extensions to support it. 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Internet-Drafts are working documents of the Internet Engineering 35 Task Force (IETF). Note that other groups may also distribute 36 working documents as Internet-Drafts. The list of current Internet- 37 Drafts is at https://datatracker.ietf.org/drafts/current/. 39 Internet-Drafts are draft documents valid for a maximum of six months 40 and may be updated, replaced, or obsoleted by other documents at any 41 time. It is inappropriate to use Internet-Drafts as reference 42 material or to cite them other than as "work in progress." 44 This Internet-Draft will expire on 8 September 2022. 46 Copyright Notice 48 Copyright (c) 2022 IETF Trust and the persons identified as the 49 document authors. All rights reserved. 51 This document is subject to BCP 78 and the IETF Trust's Legal 52 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 53 license-info) in effect on the date of publication of this document. 54 Please review these documents carefully, as they describe your rights 55 and restrictions with respect to this document. Code Components 56 extracted from this document must include Revised BSD License text as 57 described in Section 4.e of the Trust Legal Provisions and are 58 provided without warranty as described in the Revised BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 63 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3 64 1.2. EVPN Interconnect Options . . . . . . . . . . . . . . . . 3 65 1.3. When is the Service Interworking Solution Required for EVPN 66 VPWS . . . . . . . . . . . . . . . . . . . . . . . . . . 6 67 1.4. Service Gateway Extensions for EVPN VPWS . . . . . . . . 9 68 2. Conventions used in this document . . . . . . . . . . . . . . 10 69 3. Service Interworking procedures for EVPN VPWS . . . . . . . . 10 70 3.1. Redistribution of EVPN Routes Across Domains . . . . . . 10 71 3.2. EVPN Domain Anycast Gateways for redundancy . . . . . . . 12 72 3.3. EVPN Multi-Homing for Domain Gateway Redundancy (I-ES) . 14 73 4. Security Considerations . . . . . . . . . . . . . . . . . . . 16 74 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 75 6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 16 76 7. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 16 77 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 16 78 8.1. Normative References . . . . . . . . . . . . . . . . . . 16 79 8.2. Informative References . . . . . . . . . . . . . . . . . 17 80 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18 82 1. Introduction 84 Ethernet VPN Virtual Private Wire Services (EVPN VPWS) [RFC8214] need 85 to be deployed in high scale multi-domain networks, where each domain 86 can use a different transport technology, such as MPLS, VXLAN or 87 Segment Routing with MPLS or IPv6 Segment Identifiers (SIDs). While 88 the transport interworking solutions on border routers spare the 89 border routers from having to process service routes, they do not 90 always meet the multi-homing, redundancy, and operational 91 requirements, or provide the isolation that each domain requires. 92 This document analyzes the scenarios in which an interconnect 93 solution for EVPN VPWS using EVPN Domain Gateways is needed, and adds 94 the required extensions to support it. 96 1.1. Terminology 98 This section summarizes the terminology that is used throughout the 99 rest of the document. 101 * BR: Border Router, router that provides connectivity between 102 domains, typically an Area Border Router (ABR) or Autonomous 103 System Border Router (ASBR). 105 * I-PE: Ingress Provider Edge router 107 * E-PE: Egress Provider Edge router 109 * ES and ESI: Ethernet Segment and Ethernet Segment Identifier, as 110 defined in [I-D.ietf-bess-rfc7432bis]. 112 * I-ES and I-ESI: Interconnect Ethernet Segment and Interconnect 113 Ethernet Segment Identifier. An I-ES is defined for multihoming 114 to the domains to which a Service Gateway is attached [RFC9014]. 116 * NVO: Network Virtualization Overlay. 118 * EVPN Domain and EVPN Domain Gateway: two PEs are in the same EVPN 119 Domain if they are attached to the same service and the packets 120 between them do not require a data path lookup of the inner frame 121 in any intermediate router. An EVPN Domain is typically a group 122 of PE, P and Border Routers that belong to the same IGP instance 123 or BGP domain. EVPN services are instantiated on the PEs and 124 Border Routers, which are referred to as EVPN Domain Gateways in 125 this document. An EVPN Domain Gateway connects two or more EVPN 126 Domains and is configured with multiple Domain identifiers (EVPN 127 Domain-ID) in the VPWS that connects those EVPN Domains. Each 128 EVPN Domain-ID representing an EVPN Domain. Another definition of 129 EVPN Domain Gateway is a Border Router that implements the Service 130 Interworking procedures described in this document. 132 * Domain: in this document Domain and EVPN Domain are used 133 interchangeably. 135 1.2. EVPN Interconnect Options 137 This section describes the EVPN [I-D.ietf-bess-rfc7432bis] high level 138 interconnect options and discusses their applicability to EVPN VPWS. 140 1. Service Interworking solution: 142 A-D per EVI A-D per EVI A-D per EVI 143 RD2 tag2 L22 RD3 tag3 SID33 RD4 tag4 vni44 144 <------------+ <--------------+ <-------------+ 145 +------------+ +--------------+ +-------------+ 146 | | | | | | 147 I-PE BR-1 BR-2 E-PE 148 +-------+ +-------+ +-------+ +-------+ 149 |+-----+| |+-----+| |+-----+| |+-----+| 150 CE1--||VPWS1|| ||VPWS1|| ||VPWS1|| ||VPWS1||-->CE2 151 |+-----+| |+-----+| |+-----+| |+-----+| 152 +-------+ +-------+ +-------+ +-------+ 153 | SR-MPLS | | SRv6 | | VXLAN | 154 +------------+ +--------------+ +--------------+ 155 <------------> <--------------> <--------------> 156 Domain-1 Domain-2 Domain-3 158 Figure 1: Service Interworking Interconnect 160 [RFC9014] section 4 describes an end-to-end EVPN interconnect 161 solution using EVPN Domain Gateways, or simply Gateways. The 162 Gateways provide connectivity across EVPN Domains, where those 163 Domains can use MPLS tunnels, overlay tunnels (e.g., VXLAN) or 164 Segment Routing tunnels. Procedures are extrapolated to SRv6 165 domains too. The Gateways provide independence in terms of the 166 Route Targets and Route Distinguishers used in each Domain, or 167 the type of multicast tree used for BUM traffic in each domain, 168 while keeping the key EVPN properties end-to-end, such as MAC 169 mobility, MAC protection or ARP suppression. The Gateways also 170 provide all-active and single-active multi-homing redundancy by 171 extending the concept of the multi-homing Ethernet Segment for 172 interconnect domains (I-ES). In this document, we refer to this 173 solution as the Service Interworking option, and the Border 174 Routers play the role of EVPN Domain Gateways. Since [RFC9014] 175 section 4 only describes the solution for EVPN multi-point 176 services, this document extends the procedures to support EVPN 177 VPWS services with the required extensions. Figure 1 illustrates 178 the Service Interworking solution across domains of different 179 transport encapsulations when applied to EVPN VPWS services. 181 2. Inter-domain Option-B solution: 183 NHSelf NHSelf A-D per EVI 184 L22<-L33 L33<-L44 RD4 tag4 L44 185 <------------+ <--------------+ <-------------+ 186 +------------+ +--------------+ +-------------+ 187 | | | | | | 188 I-PE BR-1 BR-2 E-PE 189 +-------+ +-------+ +-------+ +-------+ 190 |+-----+| | | | | |+-----+| 191 CE1--||VPWS1|| | | | | ||VPWS1||-->CE2 192 |+-----+| | | | | |+-----+| 193 +-------+ +-------+ +-------+ +-------+ 194 | SR-MPLS | | SR-MPLS | | SR-MPLS | 195 +------------+ +--------------+ +--------------+ 196 <------------> <--------------> <--------------> 197 Domain-1 Domain-2 Domain-3 199 Figure 2: Inter-domain Option-B 201 [RFC8365] section 10 provides an alternative interconnect 202 solution for EVPN services by using Border Routers that re-write 203 the EVPN BGP next-hops and program a swap operation of the VNIs 204 or MPLS labels (depending on whether the encapsulation is NVO- 205 based or MPLS-based). This solution does not require the 206 instantiation of Services on the Border Routers that perform a 207 lookup on the inner destination MAC (as in the case of 208 [RFC9014]), however the solution is limited to the interconnect 209 of domains of the same encapsulation. In addition, the solution 210 does not support per-ES mass withdraw of the EVPN MAC/IP 211 Advertisement routes, as described in [RFC8365]. In this 212 document we refer to this solution as Inter-domain Option-B. 213 Figure 2 illustrates this model when applied to EVPN VPWS, where 214 the three domains are all now of the same encapsulation, and 215 there is no service instantiation on the Border Routers. 217 3. Transport Interworking solution: 219 A-D per EVI 220 RD4 tag4 L44 221 <---------------------------------------------+ 222 +------------+ +--------------+ +-------------+ 223 | | | | | | 224 I-PE BR-1 BR-2 E-PE 225 +-------+ +-------+ +-------+ +-------+ 226 |+-----+| |Transp | |Transp | |+-----+| 227 CE1--||VPWS1|| |IW | |IW | ||VPWS1||-->CE2 228 |+-----+| | | | | |+-----+| 229 +-------+ +-------+ +-------+ +-------+ 230 | SR-MPLS | | SRv6 | | SR-MPLS | 231 +------------+ +--------------+ +--------------+ 232 <------------> <--------------> <--------------> 233 Domain-1 Domain-2 Domain-3 235 Figure 3: Transport Interworking option 237 Other proposals are currently being investigated, in the context 238 of SRv6 to MPLS interworking, e.g., 239 [I-D.agrawal-spring-srv6-mpls-interworking]. In these solutions, 240 the Border Routers do not change the EVPN BGP next-hops, or 241 process EVPN routes for that matter. The Border Routers provide 242 stitching between MPLS and SRv6 tunnels. In this case, the 243 solution allows the interconnect of domains of different 244 encapsulation, as long as the ingress and egress PEs support the 245 same encapsulation. A variation of this solution is the Inter- 246 domain Option-C solution, where a BGP LU (Label Unicast) tunnel 247 provides the stitching across the domains, as long as all the 248 domains use the same encapsulation. In this document, we refer 249 to this solution as Transport Interworking option. Figure 3 250 illustrates this model when applied to EVPN VPWS, where I-PE and 251 E-PE are attached to domains of the same encapsulation. 252 Intermediate domains, e.g., Domain-2, can be of encapsulations 253 different from the ones used at the ingress and egress Domains. 254 The EVPN route is not processed or changed by the Border Routers. 256 1.3. When is the Service Interworking Solution Required for EVPN VPWS 258 The three interconnect solutions described in Section 1.2 are valid, 259 however, this section describes the requirements that make the 260 Service Interworking solution needed. Those requirements are: 262 a. Per-domain EVPN Multi-Homing 264 The Service Interworking solution allows the use of different 265 Ethernet Segment Identifiers (ESI) per domain, as well as the 266 implementation of the aliasing and backup procedures on a per- 267 domain basis. The use of different ESIs per domain may help 268 guarantee the uniqueness of the ESI when different domains 269 independently managed and operated are interconnected. The 270 implementation of independent aliasing and backup procedures per 271 domain, spares the need for propagation of the EVPN A-D per ES 272 routes by the Border Routers (which are EVPN Domain Gateways in 273 the Service Interworking solution). These A-D per ES routes are 274 consumed within the domain, which results in a significant 275 reduction of the number of routes that the ingress PEs need to 276 process. Another consequence of the processing of A-D per ES 277 routes per domain, is a faster convergence in case of ES PE or 278 link failure, since A-D per ES routes are no longer propagated by 279 all the Border Routers along the domains, but processed by the 280 Border Routers of the originating domain. Per-domain EVPN Multi- 281 Homing procedures are not possible in the Inter-domain Option-B 282 or Transport Interworking solutions. 284 b. Per-ES Mass Withdrawal 286 In order to benefit from the per-ES mass withdrawal property of 287 EVPN Multi-Homing, the received BGP next-hops of the selected 288 EVPN A-D per EVI and A-D per ES routes need to match on a PE. 289 This cannot be guaranteed in an Inter-domain Option-B solution, 290 as described in [RFC8365] section 10.2.2., however it is always 291 the case in the Service Interworking or Transport Interworking 292 solution. 294 c. Per-domain Route Distinguishers (RDs) and Route Targets (RTs) 296 In case of merge of domains coming from different administrative 297 entities, the uniqueness of RDs and RTs across domains for the 298 same service is not guaranteed. Hence the re-write of RD/RTs at 299 the Border Routers may be required. If that is the case, the 300 Service Interworking solution provides the support for re-writing 301 RD/RTs. The Inter-domain Option-B may allow re-writing RD/RTs, 302 however, it is not considered a common practice. The Transport 303 Interworking solution does not support the translation of RD/RTs. 305 d. Ethernet Tag IDs per domain 307 Similar to per-domain RDs and RTs, re-writing of Ethernet Tag IDs 308 used in the A-D per EVI routes may be needed in case of 309 interconnect of domains that belong to different administrative 310 entities. This can be only supported by a Service Interworking 311 solution. 313 e. Control Word, Flow Label and MTU (Maximum Transfer Unit) 314 signaling per domain 315 As described in [I-D.ietf-bess-rfc7432bis], the use of Control 316 Word and Flow Label, as well as the MTU are signaled in the EVPN 317 Layer 2 Attributes extended community along with the A-D per EVI 318 routes. The signaling and use of Control Word is recommended in 319 those domains where P routers can get confused when hashing based 320 on the tunneled EVPN packet payload, but the Control Word may not 321 be needed in some domains. Similarly, the Flow Label introduces 322 an additional level of entropy in EVPN encapsulated packets, that 323 may be needed in some domains but adding unnecessary extra 324 overhead in other domains. Different MTUs may be supported in 325 different domains, due to the domains running on different 326 physical media. A Service Interworking model allows the 327 signaling and use of Control Word, Flow Label, and Layer-2 MTU on 328 a per domain basis. This is not the case in the other two models 329 analyzed in this document. 331 f. Heterogeneous Encapsulations 333 Interconnecting domains that use different encapsulations (e.g., 334 VXLAN, SRv6, MPLS, SR-MPLS, etc.) is a common requirement. This 335 becomes important in case the domains have different platform 336 features, or migrations to new encapsulations or transport types 337 are needed. In the Service Interworking model the EVPN routes 338 are generated and consumed at every Border Router (which is an 339 EVPN Domain Gateway), hence the encapsulation indicated along 340 with the route can be advertised independently at each Border 341 Router. That is not the case in the models 2 and 3 in 342 Section 1.2. The Inter-domain Option-B model requires the same 343 encapsulation in each of the domains the Border Router connects, 344 whereas the Transport Interworking model requires that at least 345 the ingress and egress domains have the same encapsulation. 347 g. Per-domain EVPN Service OAM 349 [RFC9062] defines the Service OAM requirements for EVPN services. 350 When applied to the Interconnect solutions, the three solutions 351 in Section 1.2 allow for the use of MEPs and MIPs on the ingress 352 and egress PEs, but only the Service Interworking solution 353 supports MEPs and MIPs on the Border Routers. In other words, 354 per-domain EVPN Service OAM is only supported in the Service 355 Interworking option. 357 The above requirements and their support across the Interconnect 358 solutions are summarized in Table 1. 360 +======================+==============+==============+==============+ 361 | Requirement | Service | Inter-domain | Transport | 362 | | Interworking | Option-B | Interworking | 363 +======================+==============+==============+==============+ 364 | Per-domain | Yes | No | No | 365 | EVPN Multi- | | | | 366 | Homing | | | | 367 +----------------------+--------------+--------------+--------------+ 368 | Per-ES Mass | Yes | No | Yes | 369 | Withdrawal | | | | 370 +----------------------+--------------+--------------+--------------+ 371 | Per-domain RD/ | Yes | Yes* | No | 372 | RTs | | | | 373 +----------------------+--------------+--------------+--------------+ 374 | Ethernet Tag | Yes | No | No | 375 | IDs per domain | | | | 376 +----------------------+--------------+--------------+--------------+ 377 | Control Word, | Yes | No | No | 378 | Flow Label and | | | | 379 | MTU signaling | | | | 380 | per domain | | | | 381 +----------------------+--------------+--------------+--------------+ 382 | Heterogeneous | Yes | No | Yes** | 383 | encapsulations | | | | 384 +----------------------+--------------+--------------+--------------+ 385 | Per-domain | Yes | No | No | 386 | EVPN Service | | | | 387 | OAM | | | | 388 +----------------------+--------------+--------------+--------------+ 390 Table 1: EVPN VPWS Interconnect Options Comparison 392 * Although possible, it is unusual to re-write RD/RTs in the Inter- 393 domain Option-B solution 395 ** Supported only when the ingress and egress domains are of the same 396 encapsulation 398 1.4. Service Gateway Extensions for EVPN VPWS 400 The rest of the document specifies the extensions required for the 401 EVPN Domain Gateways to implement the Service Interworking solution 402 to deploy end-to-end EVPN VPWS services. In a nutshell, the AD per 403 EVI routes advertised by I-PE and E-PE are redistributed across 404 domains, while ES and A-D per ES routes advertised by these PEs are 405 not redistributed by the EVPN Domain Gateways. In addition, this 406 document defines how Gateway redundancy works using either an Anycast 407 Gateway solution, or by extending the I-ES concept already defined 408 for multi-point EVPN services in [RFC9014]. 410 2. Conventions used in this document 412 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 413 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 414 "OPTIONAL" in this document are to be interpreted as described in BCP 415 14 [RFC2119] [RFC8174] when, and only when, they appear in all 416 capitals, as shown here. 418 3. Service Interworking procedures for EVPN VPWS 420 This section describes the EVPN VPWS extensions on the EVPN Domain 421 Gateways (or simply Gateways) to support the Service Interworking 422 model. An EVPN Domain Gateway in this context is a Border Router 423 that connects EVPN Domains and implements the Service Interworking 424 model of Section 1.2. Section 3.1 specifies the Gateway rules to 425 redistribute EVPN routes. When redundant Gateways attached to two or 426 more EVPN Domains are deployed, there are two redundancy mechanisms 427 that can be used. Section 3.2 describes a redundancy method that we 428 refer to as "Anycast" and is based on the redundant Gateways behaving 429 as a single system for the remote PEs. Section 3.3 describes the 430 redundancy based on I-ES, as an extension of the I-ES procedures 431 specified in [RFC9014], only for EVPN VPWS services. The Anycast 432 redundancy does not require the use of I-ES and supports single- 433 active multi-homing connectivity, but it will not support all-active, 434 aliasing, backup, or mass withdraw features that are supported along 435 with the use of I-ES and EVPN Multi-Homing. 437 3.1. Redistribution of EVPN Routes Across Domains 439 The EVPN Domain Gateways MUST establish separate BGP sessions for 440 sending/receiving EVPN routes to/from each different Domain to which 441 they are attached. We refer to redistribution of an EVPN route to 442 all the procedures in the Gateway that involve the reception and 443 process of the source domain EVPN route, the programming of the 444 forwarding path for the route, and the readvertisement of the route 445 to a different domain (the next destination domain). 447 The reception and processing of EVPN routes for an EVPN VPWS service 448 follows [RFC8214]. If the D-PATH attribute is contained in the EVPN 449 A-D per EVI route, loop detection and best path selection follows 450 [I-D.sr-bess-evpn-dpath]. The Gateway imports the valid best EVPN 451 A-D per EVI route required for an Ethernet Tag ID based on the import 452 Route Target. If a non-zero ESI is included in the route, the 453 [RFC8214] procedures for aliasing, backup, and mass withdraw are 454 followed on the Gateway. 456 If an A-D per EVI route for a service is successfully imported and 457 processed, forwarding state is programmed in the data path using the 458 MPLS label, VNI or SRv6 SID that was received in the EVPN A-D per EVI 459 route. In addition, depending on the encapsulation of the route's 460 next destination domain, the router allocates a new MPLS label, VNI 461 or SRv6 SID and programs a data path switching operation between the 462 identifiers of the source and next destination domains. Immediately 463 after, the Gateway re-advertises the route to the BGP speaker in the 464 next domain. We refer to the source domain as the domain from which 465 the Gateway receives the route, and the next domain as the EVPN 466 Domain in which the Gateway redistributes the route. The following 467 considerations apply to the redistributed EVPN A-D per EVI routes: 469 a. The redistributed A-D per EVI route MUST carry a different RD 470 than the source A-D per EVI route did. This ensures that, in 471 case of redundant Gateways, there is full path visibility in the 472 next domain where the route is advertised. 474 b. The redistributed route MAY carry the same set of Route Targets 475 as the source route did, if the source and next destination 476 domains use different encapsulations, however translation or re- 477 write of Route Targets SHOULD be supported in this case. In case 478 the source and next destination domains use the same 479 encapsulation, the Gateway MUST use either different import Route 480 Targets in the two domains, or use different Ethernet Tag IDs to 481 create forwarding state in the two domains. This ensures the 482 Gateway does not loop packets back to the source domain and the 483 redistributed routes are not leaked back to the source domain. 485 c. The ESI of the redistributed route MUST be set to zero or the 486 value of the I-ESI defined in the Gateway (if any). 488 d. The Ethernet Tag ID of the redistributed route MAY have the same 489 value as the source route. Translation of the Ethernet Tag IDs 490 SHOULD be supported though. 492 e. The EVPN Layer 2 Attributes extended community is regenerated for 493 the redistributed route. The value of the P and B flags are set 494 based on the Gateway's I-ES and MUST NOT be propagated from the 495 source route. The Control Word, Flow Label flags, as well as the 496 MTU, MAY be set to different values from the source A-D route. 498 f. The encapsulation specific attributes of the redistributed route 499 are regenerated based on the encapsulation of the next domain. 500 That includes the encoding of the A-D per EVI route NLRI as 501 specified in [RFC8214] or [RFC8365], or the addition of the SRv6 502 Services TLV as in [I-D.ietf-bess-srv6-services]. 504 g. The redistributed route should carry the Communities, Extended 505 Communities and Large Communities of the source route, except for 506 Route Targets (which are reoriginated), EVPN Extended Communities 507 and BGP Encapsulation Extended Communities [RFC9012]. EVPN 508 Extended Communities and BGP Encapsulation Extended Communities 509 MUST NOT be propagated across domains. 511 h. The redistributed A-D per EVI route MUST update the D-PATH 512 attribute of the received route, or add the D-PATH attribute if 513 the received route did not contain a D-PATH 514 [I-D.sr-bess-evpn-dpath]. 516 EVPN VPWS services also make use of multi-homing routes, that is, 517 EVPN A-D per ES routes and Ethernet Segment routes. These multi- 518 homing routes are processed in the Gateway as in [RFC8214]. The A-D 519 per ES and Ethernet Segment routes are only processed in the context 520 of the domain they are received, and they MUST NOT be redistributed 521 to any other domain. A-D per ES and Ethernet Segment routes may be 522 originated at the Gateway though, if the Gateway is attached to an 523 I-ES, as described in Section 3.3. 525 3.2. EVPN Domain Anycast Gateways for redundancy 527 The Anycast Service Gateway redundancy is specified as follows: 529 a. All the Anycast Gateways attached to the same two domains MUST 530 redistribute the EVPN A-D per EVI routes between domains as per 531 Section 3.1 with the following considerations: 533 * No I-ES is used on the Gateways, therefore the ESI value MUST 534 be set to zero when redistributing EVPN A-D per EVI routes. 536 * All the redundant Gateways may set the same (or different) 537 Ethernet Tag ID in the redistributed A-D per EVI route. 539 b. All Anycast Gateways MUST process the received D-PATH attribute 540 and update the D-PATH (with the source domain-id) when 541 redistributing the A-D per EVI route to the next domain. The 542 D-PATH attribute will avoid control plane loops. 544 As an illustration of this redundancy method, suppose all four 545 Service Gateways in Figure 4 are configured as Anycast Service 546 Gateways, and local and remote Ethernet Tag IDs are configured as 1, 547 2 and 3 on all routers in the domains 1, 2 and 3 respectively. 549 A-D per EVI A-D per EVI 550 RD11 tag1 L111 RD21 tag2 SID21 551 <------------+ <--------------+ 552 +------------+ +--------------+ +-------------+ 553 | Domain-1 | | Domain-2 | | Domain-3 | 554 | BR-11 BR-21 A-D per EVI 555 | +-------+ +-------+ RD4 tag3 vni33 556 | |+-----+| |+-----+| <---------+ 557 I-PE +--> ||VPWS1||-----> ||VPWS1||--+ E-PE 558 +-------+ | |+-----+| |+-----+| | +-------+ 559 |+-----+|--+ +-------+ +-------+ | |+-----+| 560 CE1--||VPWS1|| | | | | +--> ||VPWS1||-->CE2 561 |+-----+| BR-12 BR-22 |+-----+| 562 +-------+ +-------+ +-------+ +-------+ 563 | |+-----+| |+-----+| | 564 | ||VPWS1|| ||VPWS1|| | 565 | |+-----+| |+-----+| | 566 | +-------+ +-------+ | 567 | SR-MPLS | | SRv6 | | VXLAN | 568 +------------+ +--------------+ +-------------+ 569 A-D per EVI A-D per EVI 570 RD12 tag1 L121 RD22 tag2 SID22 571 <------------+ <--------------+ 573 Figure 4: Anycast Redundancy 575 In the example in Figure 4 E-PE advertises an EVPN A-D per EVI route 576 for Ethernet Tag ID 3. Both BR-21 and BR-22 import the route and 577 redistribute it with Ethernet Tag ID 2 and new RD and encapsulation 578 into domain-2. When redistributing, both BR-21 and BR-22 update (if 579 it existed before) or insert a D-PATH attribute with the domain-id of 580 domain-3. That prevents BR-21 and BR-22 from redistributing back 581 into domain-3 each other's route [I-D.sr-bess-evpn-dpath]. BR-11 and 582 BR-12 import the routes after best path selection and perform the 583 same process and redistribution into domain-1. I-PE will receive two 584 routes for Ethernet Tag ID 1, from BR-11 and BR-12, and will perform 585 best path selection for Ethernet Tag ID 1. Based on the best path 586 selection carried out by I-PE and the BRs along the way, all flows 587 from CE1 to CE2 will follow, e.g., I-PE, BR-11, BR-21 and E-PE. In 588 case of failure on any of the BRs in the data path, the routers will 589 select the alternate route for the Ethernet Tag ID. The same control 590 plane exchange and traffic flow happen in the reverse direction, 591 where I-PE becomes the egress PE and E-PE the ingress PE. 593 As illustrated in Figure 4, this model does not support per-flow load 594 balancing (all-active multi-homing) to all the BR nodes along the way 595 from CE to CE. 597 3.3. EVPN Multi-Homing for Domain Gateway Redundancy (I-ES) 599 EVPN Multi-Homing procedures can be used on the EVPN Domain Gateways. 600 For that, an I-ES and its assigned I-ESI will be configured on the 601 Gateways for multihoming. The I-ES concept is introduced in 602 [RFC9014], and it is used in this document for EVPN VPWS services. 603 This I-ES represents a domain to the next domain, in both directions. 604 Therefore two or more Gateways attached to the same two domains will 605 use the same I-ESI when advertising routes to the two domains. 607 The Gateways attached to the same I-ES: 609 a. Advertise EVPN Ethernet Segment routes and A-D per ES routes for 610 the I-ES. Those routes are not redistributed beyond the Domain 611 into which they are originated. 613 b. Receive Ethernet Segment and A-D per ES routes from the I-ES 614 peer(s), and use them for I-ES Designated Forwarding (DF) 615 Election and mass withdraw respectively, as described in 616 [RFC8214] and [I-D.ietf-bess-rfc7432bis]. 618 c. Set the I-ESI into the EVPN A-D per EVI routes that are 619 redistributed across domains. P and B flags are set based on the 620 result of the DF Election [RFC8214]. 622 d. Identify loops if the received EVPN A-D per EVI routes include a 623 local domain-id in the D-PATH attribute. Also EVPN A-D per EVI 624 routes that include a local ESI MUST NOT be redistributed to 625 another domain, irrespective of the presence of the D-PATH 626 attribute. 628 Figure 5 illustrates the use of I-ES or EVPN Multi-Homing procedures 629 in EVPN Domain Gateways. In the example, BR-11 and BR-12 are 630 attached to I-ES-1 (with ESI-1 as identifier), whereas BR-21 and 631 BR-22 are attached to I-ES-2 (using ESI-2). 633 A-D per EVI A-D per EVI 634 RD11 tag1 ESI-1 L111 RD21 tag2 ESI-2 SID21 635 <------------+ <--------------+ 636 +------------+ +--------------+ +-------------+ 637 | Domain-1 | | Domain-2 | | Domain-3 | 638 | BR-11 BR-21 A-D per EVI 639 | +-------+ +-------+ RD4 tag3 vni33 640 | |+-----+| |+-----+| <---------+ 641 I-PE +--> ||VPWS1||-+---> ||VPWS1||--+ E-PE 642 +-------+ | |+-----+| | +-> |+-----+| | +-------+ 643 |+-----+|--+ +-------+ | | +-------+ +--> |+-----+| 644 CE1--||VPWS1|| I-ES1 | | | | | | I-ES2 ||VPWS1||-->CE2 645 |+-----+|--+ BR-12 | | BR-22 +--> |+-----+| 646 +-------+ | +-------+ +-|-> +-------+ | +-------+ 647 | | |+-----+| | |+-----+| | | 648 | +--> ||VPWS1||---+-> ||VPWS1||--+ | 649 | |+-----+| |+-----+| | 650 | +-------+ +-------+ | 651 | SR-MPLS | | SRv6 | | VXLAN | 652 +------------+ +--------------+ +-------------+ 653 A-D per EVI A-D per EVI 654 RD12 tag1 ESI-1 L121 RD22 tag2 ESI-2 SID22 655 <------------+ <--------------+ 657 Figure 5: EVPN Multi-Homing 659 E-PE advertises an A-D per EVI route for tag3, that gets 660 redistributed by BR-21/BR-22 first, and BR-11/BR-12 later, 661 translating the Ethernet Tag ID and encapsulation in each 662 redistribution. The BR nodes implement the EVPN Multi-Homing 663 procedures for their own Ethernet Segment as in [RFC8214], and set 664 the P and B flags accordingly when redistributing the A-D per EVI 665 routes, to indicate the forwarding mode to the receiving nodes. If 666 I-ES-1 and I-ES-2 are defined as all-active multi-homing Ethernet 667 Segments, per-flow load balancing will be performed not only by the 668 I-PE to the Gateways in domain-1, but also by the Gateways at each 669 domain of the EVPN VPWS service, as depicted in Figure 5. The same 670 control plane exchange and traffic flow happen in the reverse 671 direction, where I-PE becomes the egress PE and E-PE the ingress PE. 673 I-ES-1 and I-ES-2 are independent of each other, e.g., I-ES-1 can 674 work in single-active mode, whereas I-ES-2 uses all-active mode. If 675 that is the case, BR-11 and BR-12 run Designated Forwarded (DF) 676 Election and BR-11 signals P=1 and B=0 (in the EVPN Layer 2 677 Attributes extended community) if it is elected as DF, whereas BR-12 678 signals P=0 and B=1 if elected as Backup DF router. I-PE then sends 679 all traffic to BR-11, and BR-21/BR-22 send all traffic to BR-11 in 680 the reverse direction. Since BR-21/BR-22 work in all-active mode, 681 they both signal P=1/B=0 to both, E-PE and BR-11/BR-12. Therefore 682 traffic from BR-11/BR-12 is sprayed to both BR-21/BR-22, and so is 683 traffic from E-PE. 685 The Anycast Gateway and the EVPN Multi-Homing redundancy solutions 686 can coexist. The Gateways of the same redundancy group MUST 687 implement the same redundancy method, but different redundancy 688 Gateway groups MAY implement different methods. In the example, BR- 689 11/BR-12 constitutes a redundancy group and BR-21/BR-22 constitutes a 690 different redundancy group. 692 4. Security Considerations 694 To be added in a future version. 696 5. IANA Considerations 698 None. 700 6. Acknowledgments 702 7. Contributors 704 8. References 706 8.1. Normative References 708 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 709 Requirement Levels", BCP 14, RFC 2119, 710 DOI 10.17487/RFC2119, March 1997, 711 . 713 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 714 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 715 May 2017, . 717 [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., 718 Uttaro, J., and W. Henderickx, "A Network Virtualization 719 Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, 720 DOI 10.17487/RFC8365, March 2018, 721 . 723 [I-D.sr-bess-evpn-dpath] 724 Rabadan, J., Sathappan, S., Gautam, M., Brissette, P., and 725 W. Lin, "Domain Path (D-PATH) for Ethernet VPN (EVPN) 726 Interconnect Networks", Work in Progress, Internet-Draft, 727 draft-sr-bess-evpn-dpath-01, 7 March 2022, 728 . 731 [I-D.ietf-bess-rfc7432bis] 732 Sajassi, A., Burdet, L. A., Drake, J., and J. Rabadan, 733 "BGP MPLS-Based Ethernet VPN", Work in Progress, Internet- 734 Draft, draft-ietf-bess-rfc7432bis-03, 28 February 2022, 735 . 738 [RFC9014] Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi, 739 A., and J. Drake, "Interconnect Solution for Ethernet VPN 740 (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014, 741 May 2021, . 743 [RFC8214] Boutros, S., Sajassi, A., Salam, S., Drake, J., and J. 744 Rabadan, "Virtual Private Wire Service Support in Ethernet 745 VPN", RFC 8214, DOI 10.17487/RFC8214, August 2017, 746 . 748 [I-D.ietf-bess-srv6-services] 749 Dawra, G., Filsfils, C., Talaulikar, K., Raszuk, R., 750 Decraene, B., Zhuang, S., and J. Rabadan, "SRv6 BGP based 751 Overlay Services", Work in Progress, Internet-Draft, 752 draft-ietf-bess-srv6-services-12, 5 March 2022, 753 . 756 [RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder, 757 "The BGP Tunnel Encapsulation Attribute", RFC 9012, 758 DOI 10.17487/RFC9012, April 2021, 759 . 761 8.2. Informative References 763 [RFC9062] Salam, S., Sajassi, A., Aldrin, S., Drake, J., and D. 764 Eastlake 3rd, "Framework and Requirements for Ethernet VPN 765 (EVPN) Operations, Administration, and Maintenance (OAM)", 766 RFC 9062, DOI 10.17487/RFC9062, June 2021, 767 . 769 [I-D.agrawal-spring-srv6-mpls-interworking] 770 Agrawal, S., ALI, Z., Filsfils, C., Voyer, D., and Z. Li, 771 "SRv6 and MPLS interworking", Work in Progress, Internet- 772 Draft, draft-agrawal-spring-srv6-mpls-interworking-07, 21 773 February 2022, . 776 Authors' Addresses 778 J. Rabadan (editor) 779 Nokia 780 520 Almanor Avenue 781 Sunnyvale, CA 94085 782 United States of America 783 Email: jorge.rabadan@nokia.com 785 S. Sathappan 786 Nokia 787 520 Almanor Avenue 788 Sunnyvale, CA 94085 789 United States of America 790 Email: senthil.sathappan@nokia.com 792 V. Prabhu 793 Nokia 794 600 March Rd 795 Kanata ON K2K 2T6 796 Canada 797 Email: vinod.prabhu@nokia.com 799 W. Lin 800 Juniper 801 United States of America 802 Email: wlin@juniper.net 804 P. Brissette 805 Cisco Systems 806 Canada 807 Email: pbrisset@cisco.com