idnits 2.17.00 (12 Aug 2021) /tmp/idnits23345/draft-ietf-bess-evpn-ipvpn-interworking-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 25, 2020) is 726 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: draft-ietf-bess-evpn-prefix-advertisement has been published as RFC 9136 == Outdated reference: draft-ietf-bess-evpn-inter-subnet-forwarding has been published as RFC 9135 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS Workgroup J. Rabadan, Ed. 3 Internet-Draft Nokia 4 Intended status: Standards Track A. Sajassi, Ed. 5 Expires: November 26, 2020 Cisco 6 E. Rosen 7 Individual 8 J. Drake 9 W. Lin 10 Juniper 11 J. Uttaro 12 AT&T 13 A. Simpson 14 Nokia 15 May 25, 2020 17 EVPN Interworking with IPVPN 18 draft-ietf-bess-evpn-ipvpn-interworking-03 20 Abstract 22 EVPN is used as a unified control plane for tenant network intra and 23 inter-subnet forwarding. When a tenant network spans not only EVPN 24 domains but also domains where IPVPN provides inter-subnet 25 forwarding, there is a need to specify the interworking aspects 26 between both EVPN and IPVPN domains, so that the end to end tenant 27 connectivity can be accomplished. This document specifies how EVPN 28 should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families 29 for inter-subnet forwarding. 31 Status of This Memo 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. 36 Internet-Drafts are working documents of the Internet Engineering 37 Task Force (IETF). Note that other groups may also distribute 38 working documents as Internet-Drafts. The list of current Internet- 39 Drafts is at https://datatracker.ietf.org/drafts/current/. 41 Internet-Drafts are draft documents valid for a maximum of six months 42 and may be updated, replaced, or obsoleted by other documents at any 43 time. It is inappropriate to use Internet-Drafts as reference 44 material or to cite them other than as "work in progress." 46 This Internet-Draft will expire on November 26, 2020. 48 Copyright Notice 50 Copyright (c) 2020 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (https://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the Simplified BSD License. 63 Table of Contents 65 1. Introduction and Problem Statement . . . . . . . . . . . . . 2 66 2. Terminology and Interworking PE Components . . . . . . . . . 3 67 3. Domain Path Attribute (D-PATH) . . . . . . . . . . . . . . . 9 68 3.1. D-PATH and Loop Prevention . . . . . . . . . . . . . . . 11 69 4. BGP Path Attribute Propagation across ISF SAFIs . . . . . . . 12 70 4.1. No-Propagation-Mode . . . . . . . . . . . . . . . . . . . 12 71 4.2. Uniform-Propagation-Mode . . . . . . . . . . . . . . . . 12 72 4.3. Aggregation of Routes and Path Attribute Propagation . . 13 73 5. Route Selection Process between EVPN and other ISF SAFIs . . 14 74 6. Composite PE Procedures . . . . . . . . . . . . . . . . . . . 15 75 7. Gateway PE Procedures . . . . . . . . . . . . . . . . . . . . 17 76 8. Interworking Use-Cases . . . . . . . . . . . . . . . . . . . 19 77 9. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 21 78 10. Conventions used in this document . . . . . . . . . . . . . . 21 79 11. Security Considerations . . . . . . . . . . . . . . . . . . . 21 80 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 81 13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 22 82 14. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 22 83 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 84 15.1. Normative References . . . . . . . . . . . . . . . . . . 22 85 15.2. Informative References . . . . . . . . . . . . . . . . . 22 86 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 23 88 1. Introduction and Problem Statement 90 EVPN is used as a unified control plane for tenant network intra and 91 inter-subnet forwarding. When a tenant network spans not only EVPN 92 domains but also domains where IPVPN provides inter-subnet 93 forwarding, there is a need to specify the interworking aspects 94 between both EVPN and IPVPN domains, so that the end to end tenant 95 connectivity can be accomplished. This document specifies how EVPN 96 should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families 97 for inter-subnet forwarding. 99 EVPN supports the advertisement of IPv4 or IPv6 prefixes in two 100 different route types: 102 o Route Type 2 - MAC/IP route (only for /32 and /128 host routes), 103 as described by [I-D.ietf-bess-evpn-inter-subnet-forwarding]. 105 o Route Type 5 - IP Prefix route, as described by 106 [I-D.ietf-bess-evpn-prefix-advertisement]. 108 When interworking with other BGP address families (AFIs/SAFIs) for 109 inter-subnet forwarding, the IP prefixes in those two EVPN route 110 types must be propagated to other domains using different SAFIs. 111 Some aspects of that propagation must be clarified. Examples of 112 these aspects or procedures across BGP families are: route selection, 113 loop prevention or BGP Path attribute propagation. The Interworking 114 PE concepts are defined in section 2, and the rest of the document 115 describes the interaction between Interworking PEs and other PEs for 116 end-to-end inter-subnet forwarding. 118 2. Terminology and Interworking PE Components 120 This section summarizes the terminology related to the "Interworking 121 PE" concept that will be used throughout the rest of the document. 123 +-------------------------------------------------------------+ 124 | | 125 | +------------------+ Interworking PE | 126 | Attachment | +------------------+ | 127 | Circuit(AC1) | | +----------+ | MPLS/NVO tnl 128 ----------------------*Bridge | | +------ 129 | | | |Table(BT1)| | +-----------+ / \ \ 130 MPLS/NVO tnl +-------->| *---------* |<--> | Eth | 131 -------+ | | | |Eth-Tag x + |IRB1| | \ / / 132 / Eth / \<-+ | | +----------+ | | | +------ 133 | | | | | ... | | IP-VRF1 | | 134 \ \ /<-+ | | +----------+ | | RD2/RT2 |MPLS/NVO tnl 135 -------+ | | | |Bridge | | | | +------ 136 | +-------->|Table(BT2)| |IRB2| | / \ \ 137 | | | | *---------* |<--> | IP | 138 ----------------------*Eth-Tag y | | +-----*-----+ \ / / 139 | AC2 | | +----------+ | AC3| +------ 140 | | | MAC-VRF1 | | | 141 | +-+ RD1/RT1 | | | 142 | +------------------+ | SAFIs | 143 | | 1 +---+ | 144 -------------------------------------------------+ 128 |BGP| | 145 | EVPN +---+ | 146 | | 147 +-------------------------------------------------------------+ 149 Figure 1: EVPN-IPVPN Interworking PE 151 o ISF SAFI: Inter-Subnet Forwarding (ISF) SAFI is a MP-BGP Sub- 152 Address Family that advertises reachability for IP prefixes and 153 can be used for inter-subnet forwarding within a given tenant 154 network. The ISF SAFIs are 1 (including IPv4 and IPv6 AFIs), 128 155 (including IPv4 and IPv6 AFIs) and 70 (EVPN, including only AFI 156 25). 158 o ISF route: a route for a given prefix whose ISF SAFI may change as 159 it transits different domains. 161 o IP-VRF: an IP Virtual Routing and Forwarding table, as defined in 162 [RFC4364]. It is also the instantiation of an IPVPN in a PE. 163 Route Distinguisher and Route Target(s) are required properties of 164 an IP- VRF. 166 o MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in 167 [RFC7432]. It is also the instantiation of an EVI (EVPN Instance) 168 in a PE. Route Distinguisher and Route Target(s) are required 169 properties and they are normally different than the ones defined 170 in the associated IP-VRF. 172 o BT: a Bridge Table, as defined in [RFC7432]. A BT is the 173 instantiation of a Broadcast Domain in a PE. When there is a 174 single Broadcast Domain in a given EVI, the MAC-VRF in each PE 175 will contain a single BT. When there are multiple BTs within the 176 same MAC-VRF, each BT is associated to a different Ethernet Tag. 177 The EVPN routes specific to a BT, will indicate which Ethernet Tag 178 the route corresponds to. 180 Example: In Figure 1, MAC-VRF1 has two BTs: BT1 and BT2. Ethernet 181 Tag x is defined in BT1 and Ethernet Tag y in BT2. 183 o AC: Attachment Circuit or logical interface associated to a given 184 BT or IP-VRF. To determine the AC on which a packet arrived, the 185 PE will examine the combination of a physical port and VLAN tags 186 (where the VLAN tags can be individual c-tags, s-tags or ranges of 187 both). 189 Example: In Figure 1, AC1 is associated to BT1, AC2 to BT2 and AC3 190 to IP-VRF1. 192 o IRB: Integrated Routing and Bridging interface. It refers to the 193 logical interface that connects a BT to an IP-VRF and allows to 194 forward packets with destination in a different subnet. 196 o MPLS/NVO tnl: It refers to a tunnel that can be MPLS or NVO-based 197 (Network Virtualization Overlays) and it is used by MAC-VRFs and 198 IP-VRFs. Irrespective of the type, the tunnel may carry an 199 Ethernet or an IP payload. MAC-VRFs can only use tunnels with 200 Ethernet payloads (setup by EVPN), whereas IP-VRFs can use tunnels 201 with Ethernet (setup by EVPN) or IP payloads (setup by EVPN or 202 IPVPN). IPVPN-only PEs have IP-VRFs but they cannot send or 203 receive traffic on tunnels with Ethernet payloads. 205 Example: Figure 1 shows an MPLS/NVO tunnel that is used to 206 transport Ethernet frames to/from MAC-VRF1. The PE determines the 207 MAC-VRF and BT the packets belong to based on the EVPN label (MPLS 208 or VNI). Figure 1 also shows two MPLS/NVO tunnels being used by 209 IP- VRF1, one carrying Ethernet frames and the other one carrying 210 IP packets. 212 o RT-2: Route Type 2 or MAC/IP route, as per [RFC7432]. 214 o RT-5: Route Type 5 or IP Prefix route, as per 215 [I-D.ietf-bess-evpn-prefix-advertisement]. 217 o Domain: Two PEs are in the same domain if they are attached to the 218 same tenant and the packets between them do not require a data 219 path IP lookup (in the tenant space) in any intermediate router. 220 A gateway PE is always configured with multiple Domain-IDs. 222 Example 1: Figure 2 depicts an example where TS1 and TS2 belong to 223 the same tenant, and they are located in different Data Centers 224 that are connected by gateway PEs (see the gateway PE definition 225 later). These gateway PEs use IPVPN in the WAN. When TS1 sends 226 traffic to TS2, the intermediate routers between PE1 and PE2 227 require a tenant IP lookup in their IP-VRFs so that the packets 228 can be forwarded. In this example there are three different 229 domains. The gateway PEs connect the EVPN domains to the IPVPN 230 domain. 232 GW1------------GW3 233 +------+ +------+ 234 +-------------|IP-VRF| |IP-VRF|-------------+ 235 PE1 +------+ +------+ PE2 236 +------+ DC1 | WAN | DC2 +------+ 237 TS1-|IP-VRF| EVPN | IPVPN | EVPN |IP-VRF|-TS2 238 +------+ GW2 GW4 +---+--+ 239 | +------+ +------+ | 240 +-------------|IP-VRF| |IP-VRF|-------------+ 241 +------+ +------+ 242 +--------------+ 243 DOMAIN 1 DOMAIN 2 DOMAIN 3 244 <---------------> <------------> <----------------> 246 Figure 2: Multiple domain DCI example 248 Example 2: Figure 3 illustrates a similar example, but PE1 and PE2 249 are now connected by a BGP-LU (BGP Labeled Unicast) tunnel, and 250 they have a BGP peer relationship for EVPN. Contrary to Example 1, 251 there is no need for tenant IP lookups on the intermediate routers 252 in order to forward packets between PE1 and PE2. Therefore, there 253 is only one domain in the network and PE1/PE2 belong to it. 255 EVPN 256 <-------------------------------------------------> 257 BGP-LU 258 <-------------------------------------------------> 260 ASBR------------ASBR 261 +------+ +------+ 262 +-------------| | | |-------------+ 263 PE1 +------+ +--+---+ PE2 264 +------+ DC1 | WAN | DC2 +------+ 265 TS1-|IP-VRF| EVPN | | EVPN |IP-VRF|-TS2 266 +------+ ASBR ASBR +---+--+ 267 | +------+ +------+ | 268 +-------------| | | |-------------+ 269 +------+ +------+ 270 +--------------+ 272 <--------------------DOMAIN-1---------------------> 274 Figure 3: Single domain DCI example 276 o Regular Domain: a domain in which a single control plane, IPVPN or 277 EVPN, is used and which is composed of regular PEs, see below. In 278 Figures 2 and 3, above, all domains are regular domains. 280 o Composite Domain: a domain in which multiple control planes, IPVPN 281 and EVPN, are used and which is composed of regular PEs, see 282 below, and composite PEs, see below. 284 o Regular PE: a PE that is attached to a domain, either regular or 285 composite, and which uses one of the control plane protocols 286 (IPVPN or EVPN) operating in the domain. 288 o Interworking PE: a PE that may advertise a given prefix with an 289 EVPN ISF route (RT-2 or RT-5) and/or an IPVPN ISF route. An 290 interworking PE has one IP-VRF per tenant, and one or multiple 291 MAC- VRFs per tenant. Each MAC-VRF may contain one or more BTs, 292 where each BT may be attached to that IP-VRF via IRB. There are 293 two types of Interworking PEs: composite PEs and gateway PEs. 294 Both PE functions can be independently implemented per tenant and 295 they may both be implemented for the same tenant. 297 Example: Figure 1 shows an interworking PE of type gateway, where 298 ISF SAFIs 1, 128 and 70 are enabled. IP-VRF1 and MAC-VRF1 are 299 instantiated on the PE, and together provide inter-subnet 300 forwarding for the tenant. 302 o Composite PE: an interworking PE that is attached to a composite 303 domain and which advertises a given prefix to an IPVPN peer with 304 an IPVPN ISF route, to an EVPN peer with an EVPN ISF route, and to 305 a route reflector with both an IPVPN and EVPN ISF route. A 306 composite PE performs the procedures of Sections 5 and 6. 308 Example: Figure 4 shows an example where PE1 is a composite PE 309 since PE1 has EVPN and another ISF SAFI enabled to the same route- 310 reflector, and PE1 advertises a given IP prefix IPn/x twice, one 311 using EVPN and another one using ISF SAFI 128. PE2 and PE3 are 312 not composite PEs. 314 +---+ 315 |PE2| 316 +---+ 317 ^ 318 |EVPN 319 IW EVPN v 320 +---+ IPVPN ++-+ +---+ 321 |PE1| <----> |RR| <---> |PE3| 322 +---+ +--+ IPVPN +---+ 323 Composite 325 Figure 4: Interworking composite PE example 327 o Gateway PE: an interworking PE that is attached to two domains, 328 each either regular or composite, and which, based on 329 configuration, does one of the following: 331 - Propagates the same control plane protocol, either IPVPN or 332 EVPN, between the two domains. 334 - Propagates an ISF route with different ISF SAFIs between the 335 two domains. E.g., propagate an EVPN ISF route in one domain 336 as an IPVPN ISF route in the other domain and vice versa. A 337 gateway PE performs the procedures of Sections 3, 4, 5 and 7. 339 A gateway PE is always configured with multiple Domain-IDs. 340 The Domain-ID is encoded in the Domain Path Attribute (D-PATH), 341 and advertised along with EVPN and other ISF SAFI routes. 342 Section 3 describes the D-PATH attribute. 344 Example: Figure 5 illustrates an example where PE1 is a gateway 345 PE since the EVPN and IPVPN SAFIs are enabled on different BGP 346 peers, and a given local IP prefix IPn/x is sent to both BGP 347 peers for the same tenant. PE2 and PE1 are in one domain and 348 PE3 and PE1 are in another domain. 350 IW 351 +---+ EVPN +---+ IPVPN +---+ 352 |PE2| <----> |PE1| <----> |PE3| 353 +---+ +---+ +---+ 354 Gateway 356 Figure 5: Interworking gateway PE example 358 o Composite/Gateway PE: an interworking PE that is both a composite 359 PE and a gateway PE that is attached to two domains, one regular 360 and one composite, and which does the following: 362 - Propagates an ISF route, either IPVPN or EVPN, from the regular 363 domain into the composite domain. Within the composite domain 364 it acts as a composite PE. 366 - Propagates an ISF route, either IPVPN or EVPN, from the 367 composite domain into the regular domain. Within the regular 368 domain it is propagated as an ISF route using the ISF SAFI for 369 that domain. 371 This is particularly useful when a tenant network is attached 372 to both IPVPN and EVPN domains, any-to-any connectivity is 373 required, and end-to-end control plane consistency, when 374 possible, is desired. 376 It would be instantiated by attaching the disparate, regular 377 IPVPN and EVPN domains via these PEs to a central composite 378 domain. 380 3. Domain Path Attribute (D-PATH) 382 The BGP Domain Path (D-PATH) attribute is an optional and transitive 383 BGP path attribute. 385 Similar to AS_PATH, D-PATH is composed of a sequence of Domain 386 segments. Each Domain segment is comprised of , where the domain segment value is a 388 sequence of one or more Domains. Each domain is represented by 389 . 391 o The domain segment length field is a 1-octet field, containing the 392 number of domains in the segment. 394 o DOMAIN-ID is a 6-octet field that represents a domain. It is 395 composed of a 4-octet Global Administrator sub-field and a 2-octet 396 Local Administrator sub-field. The Global Administrator sub-field 397 MAY be filled with an Autonomous System Number (ASN), an IPv4 398 address, or any value that guarantees the uniqueness of the 399 DOMAIN- ID when the tenant network is connected to multiple 400 Operators. 402 o ISF_SAFI_TYPE is a 1-octet field that indicates the Inter-Subnet 403 Forwarding SAFI type in which a route was advertised in the 404 DOMAIN. The following types are valid in this document: 406 Value Type 408 1 SAFI 1 409 70 EVPN 410 128 SAFI 128 412 About the BGP D-PATH attribute: 414 a) Identifies the sequence of domains, each identified by a 415 through which a given ISF route has 416 passed. 418 - This attribute list may contain zero, one or more segments. 420 - The first entry in the list (leftmost) is the from which a gateway PE is propagating an ISF 422 route. The last entry in the list (rightmost) is the from which a gateway PE received an ISF route 424 without a D-PATH attribute. Intermediate entries in the list 425 are domains that the ISF route has transited. 427 - As an example, an ISF route received with a D-PATH attribute 428 containing a domain segment of {length=2, 429 <6500:2:IPVPN>,<6500:1:EVPN>} indicates that the ISF route was 430 originated in EVPN domain 6500:1, and propagated into IPVPN 431 domain 6500:2. 433 b) It is added/modified by a gateway PE when propagating 434 an update to a different domain: 436 - A gateway PE's IP-VRF, that connects two domains, belongs to 437 two DOMAIN-IDs, e.g. 6500:1 for EVPN and 6500:2 for IPVPN. 439 - Whenever a prefix arrives at a gateway PE in a particular ISF 440 SAFI route, if the gateway PE needs to export that prefix to a 441 BGP peer, the gateway PE will prepend a to the list of domains in the received 443 D-PATH. 445 - For instance, in an IP-VRF configured with DOMAIN-IDs 6500:1 446 for EVPN and 6500:2 for IPVPN, if an EVPN route for prefix P is 447 received and P installed in the IP-VRF, the IPVPN route for P 448 that is exported to an IPVPN peer will prepend the domain 449 <6500:1:EVPN> to the previously received D-PATH attribute. 450 Likewise, IP-VRF prefixes that are received from IP-VPN, will 451 be exported to EVPN peers with the domain <6500:2:IPVPN> added 452 to the segment. 454 - In the above example, if the EVPN route is received without D- 455 PATH, the gateway PE will add the D-PATH attribute with one 456 segment {length=1, <6500:1:EVPN>} when re-advertising to domain 457 6500:2. 459 - Within the originating domain, the update does not contain a D- 460 PATH attribute because the update has not passed through a 461 gateway PE yet. 463 c) The gateway PE MUST NOT add the D-PATH attribute to ISF routes 464 generated for IP-VRF prefixes that are not learned via any ISF 465 SAFI, for instance, local prefixes. 467 d) An ISF route received by a gateway PE with a D-PATH 468 attribute that contains one or more of its locally configured 469 domains for the IP-VRF is considered to be a looped ISF route and 470 MUST be dropped. 472 e) The number of domains in the D-PATH attribute 473 indicates the number of gateway PEs that the ISF route update has 474 transited. 476 3.1. D-PATH and Loop Prevention 478 The D-PATH attribute is used to prevent loops in interworking PE 479 networks. For instance, in the example of Figure 4, gateway GW1 480 receives TS1 prefix in two different ISF routes: 482 o In an EVPN RT-5 with next-hop PE1 and no D-PATH attribute. 484 o In a SAFI 128 route with next-hop GW2 and D-PATH = {length=1, 485 <6500:1:EVPN>}, assuming that DOMAIN-ID for domain 1 is 6500:1. 487 Gateway GW1 flags the SAFI 128 route as a loop, and does not re- 488 advertise it to the EVPN neighbors since the route includes the GW1's 489 local domain. 491 In general, any interworking PE that imports an ISF route MUST flag 492 the route as "looped" if its D-PATH contains a segment, where DOMAIN-ID matches a local DOMAIN-ID 494 in the tenant IP-VRF. 496 4. BGP Path Attribute Propagation across ISF SAFIs 498 Based on configurations a gateway PE is required to propagate an ISF 499 route with different ISF SAFIs between two domains. This requires a 500 definition of what a gateway PE has to do with Path attributes 501 attached to the ISF route that it is propagating. 503 4.1. No-Propagation-Mode 505 This is the default mode of operation. In this mode, the gateway PE 506 will simply re-initialize the Path Attributes when propagating an ISF 507 route, as though it would for direct or local IP prefixes. This 508 model may be enough in those use-cases where the EVPN domain is 509 considered an "abstracted" CE and remote IPVPN/IP PEs don't need to 510 consider the original EVPN Attributes for path calculations. 512 Since this mode of operation does not propagate the D-PATH attribute 513 either, redundant gateway PEs are exposed to routing loops. Those 514 loops may be resolved by policies and the use of other attributes, 515 such as the Route Origin extended community [RFC4360], however not 516 all the loop situations may be solved. 518 4.2. Uniform-Propagation-Mode 520 In this mode, the gateway PE simply keeps accumulating or mapping 521 certain key commonly used Path Attributes when propagating an ISF 522 route. This mode is typically used in networks where EVPN and IPVPN 523 SAFIs are used seamlessly to distribute IP prefixes. 525 The following rules MUST be observed by the gateway PE when 526 propagating Path Attributes: 528 o The gateway PE imports an ISF route in the IP-VRF and stores the 529 original Path Attributes. The following set of Path Attributes 530 SHOULD be propagated by the gateway PE to other ISF SAFIs (other 531 Path Attributes SHOULD NOT be propagated): 533 - AS_PATH 534 - D-PATH 535 - IBGP-only Path Attributes: LOCAL_PREF, ORIGINATOR_ID, CLUSTER_ID 536 - MED 537 - AIGP 539 o Communities, (non-EVPN) Extended Communities and Large Communities 540 - When propagating an ISF route to a different ISF SAFI and IBGP 541 peer, the gateway PE SHOULD copy the AS_PATH of the originating 542 family and add it to the destination family without any 543 modification. When re-advertising to a different ISF SAFI and 544 EBGP peer, the gateway PE SHOULD copy the AS_PATH of the 545 originating family and prepend the IP-VRF's AS before sending 546 the route. 548 - When propagating an ISF route to IBGP peers, the gateway PE 549 SHOULD copy the IBGP-only Path Attributes from the originating 550 SAFI to the re-advertised route. 552 - Communities, non-EVPN Extended Communities and Large 553 Communities SHOULD be copied by the gateway PE from the 554 originating SAFI route. 556 4.3. Aggregation of Routes and Path Attribute Propagation 558 Instead of propagating a high number of (host) ISF routes between ISF 559 SAFIs, a gateway PE that receives multiple ISF routes of one ISF SAFI 560 MAY choose to propagate a single ISF aggregate route with a different 561 ISF SAFI. In this document, aggregation is used to combine the 562 characteristics of multiple ISF routes of the same ISF SAFI in such 563 way that a single aggregate ISF route of a different ISF SAFI can be 564 propagated. Aggregation of multiple ISF routes of one ISF SAFI into 565 an aggregate ISF route of a different ISF SAFI is only done by a 566 gateway PE. 568 Aggregation on gateway PEs may use either the No-Propagation-Mode or 569 the Uniform-Propagation-Mode explained in Sections 4.1. and 4.2, 570 respectively. 572 When using Uniform-Propagation-Mode, Path Attributes of the same type 573 code MAY be aggregated according to the following rules: 575 o AS_PATH is aggregated based on the rules in [RFC4271]. The 576 gateway PEs SHOULD NOT receive AS_PATH attributes with path 577 segments of type AS_SET [RFC6472]. Routes received with AS_PATH 578 attributes including AS_SET path segments MUST NOT be aggregated. 580 o ISF routes that have different attributes of the following type 581 codes MUST NOT be aggregated: D-PATH, LOCAL_PREF, ORIGINATOR_ID, 582 CLUSTER_ID, MED or AIGP. 584 o The Community, Extended Community and Large Community attributes 585 of the aggregate ISF route MUST contain all the Communities/ 586 Extended Communities/Large Communities from all of the aggregated 587 ISF routes. 589 Assuming the aggregation can be performed (the above rules are 590 applied), the operator should consider aggregation to deal with 591 scaled tenant networks where a significant number of host routes 592 exists. For a example, large Data Centers. 594 5. Route Selection Process between EVPN and other ISF SAFIs 596 A PE may receive an IP prefix in ISF routes with different ISF SAFIs, 597 from the same or different BGP peer. It may also receive the same IP 598 prefix (host route) in an EVPN RT-2 and RT-5. A route selection 599 algorithm across all ISF SAFIs is needed so that: 601 o Different gateway and composite PEs have a consistent and 602 deterministic view on how to reach a given prefix. 604 o Prefixes advertised in EVPN and other ISF SAFIs can be compared 605 based on path attributes commonly used by operators across 606 networks. 608 o Equal Cost Multi-Path (ECMP) is allowed across EVPN and other ISF 609 SAFI routes. 611 For a given prefix advertised in one or more non-EVPN ISF routes, the 612 BGP best path selection procedure will produce a set of "non-EVPN 613 best paths". For a given prefix advertised in one or more EVPN ISF 614 routes, the BGP best path selection procedure will produce a set of 615 "EVPN best paths". To support IP/EVPN interworking, it is then 616 necessary to run a tie-breaking selection algorithm on the union of 617 these two sets. This tie-breaking algorithm begins by considering 618 all EVPN and other ISF SAFI routes, equally preferable routes to the 619 same destination, and then selects routes to be removed from 620 consideration. The process terminates as soon as only one route 621 remains in consideration. 623 The route selection algorithm must remove from consideration the 624 routes following the rules and the order defined in [RFC4271], with 625 the following exceptions and in the following order: 627 1- Immediately after removing from consideration all routes that are 629 not tied for having the highest Local Preference, any routes that 630 do not have the shortest D-PATH are also removed from 631 consideration. Routes with no D-PATH are considered to have a 632 zero-length D-PATH. 634 2- Then regular [RFC4271] selection criteria is followed. 636 3- At the end of the selection algorithm, if at least one route still 638 under consideration is an RT-2 route, remove from consideration 639 any RT-5 routes. 641 4- Steps 1-3 could possibly leave Equal Cost Multi-Path (ECMP) 642 between IP and EVPN paths. By default, the EVPN path is 643 considered (and the IP path removed from consideration). However, 644 if ECMP across ISF SAFIs is enabled by policy, and an "IP path" 645 and an "EVPN path" remain at the end of step 3, both path types 646 will be used. 648 Example 1 - PE1 receives the following routes for IP1/32, that are 649 candidate to be imported in IP-VRF-1: 651 {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=(100,200)} 652 {SAFI=EVPN, RT-5, Local-Pref=100, AS-Path=(100,200)} 653 {SAFI=128, Local-Pref=100, AS-Path=(100,200)} 655 Selected route: {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=100,200] 656 (due to step 3, and no ECMP) 658 Example 2 - PE1 receives the following routes for IP2/24, that are 659 candidate to be imported in IP-VRF-1: 661 {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-Path=(100,200), 662 MED=10} 663 {SAFI=128, D-PATH=(6500:1:EVPN,6500:2:IPVPN), AS-Path=(200), 664 MED=200} 666 Selected route: {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS- 667 Path=(100,200), MED=10} (due to step 1) 669 6. Composite PE Procedures 671 As described in Section 2, composite PEs are typically used in tenant 672 networks where EVPN and IPVPN are both used to provide inter-subnet 673 forwarding within the same composite domain. 675 Figure 6 depicts an example of a composite domain, where PE1/PE2/PE4 676 are composite PEs (they support EVPN and IPVPN ISF SAFIs on their 677 peering to the Route Reflector), and PE3 is a regular IPVPN PE. 679 +-----------------------------------+ 680 | | 681 | MPLS IPVPN PE3 682 | Network +----------+ IP3/24 683 | IPVPN |+------+ | +---+ 684 | +----->||IP-VRF|------|CE3| 685 Composite PE1 | |+------+ | +---+ 686 +---------------+ | +----------+ 687 | +------+ | EVPN v | 688 | |IP-VRF| | IPVPN +--+ | 689 | +----| | | <------> |RR| | 690 +---+ | | +------+ | +--+ Composite PE4 691 |CE2|----|MAC-VRF| | ^ ^ +---------+ IP4/24 692 +---+ | +-------+ | EVPN | | EVPN |+------+ | +---+ 693 +---|-----------+ IPVPN | | IPVPN ||IP-VRF|-----|CE4| 694 | | +----+ +-------->|+------+ | +---+ 695 IP1/24 | | v +---------+ 696 +---+ | | +---------------+ | 697 |CE1|--+ +----| +------+ +--------------+ 698 +---+ | |IP-VRF| | 699 | | +----| | | 700 | | | +------+ | 701 +--------------|MAC-VRF| | 702 | +-------+ | 703 +---------------+ 704 Composite PE2 706 Figure 6: Composite PE example 708 In a composite domain with composite and regular PEs: 710 o The composite PEs advertise the same IP prefixes in each ISF SAFI 711 to the RR. For example, in Figure 6, the prefix IP1/24 is 712 advertised by PE1 and PE2 to the RR in two separate NLRIs, one for 713 AFI/SAFI 1/128 and another one for EVPN. 715 o The RR does not forward EVPN routes to PE3 (since the RR does not 716 have the EVPN SAFI enabled on its BGP session to PE3), whereas the 717 IPVPN routes are forwarded to all the PEs. 719 o PE3 receives only the IPVPN route for IP1/24 and resolves the BGP 720 next-hop to an MPLS tunnel (with IP payload) to PE1 and/or PE2. 722 o Composite PE4 receives IP1/24 encoded in EVPN and another ISF SAFI 723 route (EVPN RT-5 and IPVPN). The route selection follows the 724 procedures in Section 5. Assuming an EVPN route is selected, PE4 725 resolves the BGP next-hop to an MPLS tunnel (with Ethernet or IP 726 payload) to PE1 and/or PE2. As described in Section 2, two EVPN 727 PEs may use tunnels with Ethernet or IP payloads to connect their 728 IP- VRFs, depending on the 729 [I-D.ietf-bess-evpn-prefix-advertisement] model implemented. If 730 some attributes are modified so that the route selection process 731 (Section 5) results in PE4 selecting the IPVPN path instead of the 732 EVPN path, the operator should be aware that the EVPN advanced 733 forwarding features, e.g. recursive resolution to overlay indexes, 734 will be lost for PE4. 736 o The other composite PEs (PE1 and PE2) receive also the same IP 737 prefix via EVPN and IPVPN SAFIs and they also follow the route 738 selection in Section 5. 740 o When a given route has been selected as the route for a particular 741 packet, the transmission of the packet is done according to the 742 rules for that route's AFI/SAFI. 744 o It is important to note that in composite domains, such as the one 745 in Figure 6, the EVPN advanced forwarding features will only be 746 available to composite and EVPN PEs (assuming they select an RT-5 747 to forward packets for a given IP prefix), and not to IPVPN PEs. 748 For example, assuming PE1 sends IP1/24 in an EVPN and an IPVPN 749 route and the EVPN route is the best one in the selection, the 750 recursive resolution of the EVPN RT-5s can only be used in PE2 and 751 PE4 (composite PEs), and not in PE3 (IPVPN PE). As a consequence 752 of this, the indirection provided by the RT5's recursive 753 resolution and its benefits in a scaled network, will not be 754 available in all the PEs in the network. 756 7. Gateway PE Procedures 758 Section 2 defines a gateway PE as an Interworking PE that advertises 759 IP prefixes to different BGP peers, using EVPN to one BGP peer and 760 another ISF SAFI to another BGP peer. Examples of gateway PEs are 761 Data Center gateways connecting domains that make use of EVPN and 762 other ISF SAFIs for a given tenant. Figure 7 illustrates this use- 763 case, in which PE1 and PE2 (and PE3/PE4) are gateway PEs 764 interconnecting domains for the same tenant. 766 <----EVPN----> <----------IPVPN---------> <----EVPN----> 767 6500:1:EVPN 6500:2:IPVPN 6500:3:EVPN 768 769 +-----------------------+ 770 Gateway PE1 Gateway PE3 771 +----------+ +----------+ 772 +-----------|+------+ | MPLS tnls |+------+ |-------------+ 773 | ||IP-VRF| | ||IP-VRF| | | 774 PE5 |+------+ | |+------+ | PE6 775 +------+ +----------+ +----------+ +------+ 776 |IP-VRF| NVO tnls | | | | NVO tnls |IP-VRF| 777 | | | | | | | | 778 +------+ +----------+ +----------+ +------+ 779 IP1/24--> |+------+ | |+------+ | | 780 | ||IP-VRF| | ||IP-VRF| | | 781 +-----------|+------+ | |+------+ |-------------+ 782 +----------+ +----------+ 783 Gateway PE2 +------+ Gateway PE4 784 +-------|IP-VRF|---------+ 785 | | 786 +------+ 787 PE7 789 Figure 7: Gateway PE example 791 The gateway PE procedures are described as follows: 793 o A gateway PE that imports an ISF SAFI-x route to prefix P in an 794 IP-VRF, MUST export P in ISF SAFI-y if: 796 1. P is installed in the IP-VRF (hence the SAFI-x route is the 797 best one for P) and 799 2. PE has a BGP peer for SAFI-y (enabled for the same IP-VRF) and 801 3. Either x or y is EVPN 803 In the example of Figure 7, gateway PE1 and PE2 receive an EVPN 804 RT-5 with IP1/24, install the prefix in the IP-VRF and re- 805 advertise it using SAFI 128. 807 o ISF SAFI routes advertised by a gateway PE MUST include a D-PATH 808 attribute, so that loops can be detected in remote gateway PEs. 809 When a gateway PE propagates an IP prefix between EVPN and another 810 ISF SAFI, it MUST prepend a to the 811 received D-PATH attribute. The DOMAIN-ID and ISF_SAFI_TYPE fields 812 refer to the domain over which the gateway PE received the IP 813 prefix and the ISF SAFI of the route, respectively. If the 814 received IP prefix route did not include any D-PATH attribute, the 815 gateway IP MUST add the D-PATH when readvertising. The D-PATH in 816 this case will have only one segment on the list, the of the received route. 819 In the example of Figure 7, gateway PE1/PE2 receive the EVPN RT-5 820 with no D-PATH attribute since the route is originated at PE5. 821 Therefore PE1 and PE2 will add the D-PATH attribute including 822 = <6500:1:EVPN>. Gateways PE3/PE4 will 823 propagate the route again, now prepending their = <6500:2:IPVPN>. PE6 receives the EVPN RT-5 825 routes with D-PATH = {<6500:2:IPVPN>,<6500:1:EVPN>} and can use 826 that information to make BGP path decisions. 828 o The gateway PE MAY use the Route Distinguisher of the IP-VRF to 829 readvertise IP prefixes in EVPN or the other ISF SAFI. 831 o The label allocation used by each gateway PE is a local 832 implementation matter. The IP-VRF advertising IP prefixes for 833 EVPN and another ISF SAFI may use a label per-VRF, per-prefix, 834 etc. 836 o The gateway PE MUST be able to use the same or different set of 837 Route Targets per ISF SAFI on the same IP-VRF. In particular, if 838 different domains use different set of Route Targets for the same 839 tenant, the gateway PE MUST be able to import and export routes 840 with the different sets. 842 o Even though Figure 7 only shows two domains per gateway PE, the 843 gateway PEs may be connected to more than two domains. 845 o There is no limitation of gateway PEs that a given IP prefix can 846 pass through until it reaches a given PE. 848 o It is worth noting that an IP prefix that was originated in an 849 EVPN domain but traversed a different ISF SAFI domain, will lose 850 EVPN- specific attributes that are used in advanced EVPN 851 procedures. For example, even if PE1 advertises IP1/24 along with 852 a given non-zero ESI (for recursive resolution to that ESI), when 853 PE6 receives the IP prefix in an EVPN route, the ESI value will be 854 zero. This is because the route traverses an ISF SAFI domain that 855 is different than EVPN. 857 8. Interworking Use-Cases 859 While Interworking PE networks may well be similar to the examples 860 described in Sections 6 and 7, in some cases a combination of both 861 functions may be required. Figure 8 illustrates an example where the 862 gateway PEs are also composite PEs, since not only they need to re- 863 advertise IP prefixes from EVPN routes to another ISF SAFI routes, 864 but they also need to interwork with IPVPN-only PEs in a domain with 865 a mix of composite and IPVPN-only PEs. 867 +-----------------------------------+ 868 | | 869 | MPLS IPVPN PE3 870 | Network +---------+ 871 | IPVPN |+------+ | 872 | +----->||IP-VRF|---TS3 873 (GW+composite) PE1 | |+------+ | 874 +---------------+ | +---------+ 875 | +------+ | EVPN v | 876 | |IP+VRF| | IPVPN +-++ | 877 | +----| | | <------> |RR| | 878 +--------| | +------+ | +--+ Composite PE4 879 | | |MAC+VRF| | ^ ^ +---------+ 880 | | +-------+ | EVPN | | EVPN |+------+ | 881 +----+ +---------------+ IPVPN | | IPVPN ||IP-VRF|---TS4 882 TS1-|NVE1| | +----+ +-------->|+------+ | 883 +----+ | v +---------+ 884 | EVPN DC | +---------------+ | 885 | NVO tnls +----| +------+ |-------------+ 886 | | |IP+VRF| | 887 | | +----| | | 888 | | | +------+ | 889 | +----+ | |MAC+VRF| | 890 +-----|NVE2|---------| +-------+ | 891 +----+ +---------------+ 892 | (GW+composite) PE2 893 TS2 895 Figure 8: Gateway and composite combined functions - example 897 In the example above, PE1 and PE2 MUST follow the procedures 898 described in Sections 6 and 7. Compared to section 7, PE1 and PE2 899 now need to also propagate prefixes from EVPN to EVPN, in addition to 900 propagating prefixes from EVPN to IPVPN. 902 It is worth noting that PE1 and PE2 will receive TS4's IP prefix via 903 IPVPN and RT-5 routes. When readvertising to NVE1 and NVE2, PE1 and 904 PE2 will consider the D-PATH rules and attributes of the selected 905 route for TS4 (Section 5 describes the Route Selection Process). 907 9. Conclusion 909 This document describes the procedures required in PEs that use EVPN 910 and another Inter-Subnet Forwarding SAFI to import and export IP 911 prefixes for a given tenant. In particular, this document defines: 913 o A route selection algorithm so that a PE can determine what path 914 to choose between EVPN paths and other ISF SAFI paths. 916 o A new BGP Path attribute called D-PATH that provides loop 917 protection and visibility on the domains a particular route has 918 traversed. 920 o The way Path attributes should be propagated between EVPN and 921 another ISF SAFI. 923 o The procedures that must be followed on Interworking PEs that 924 behave as composite PEs, gateway PEs or a combination of both. 926 The above procedures provide an operator with the required tools to 927 build large tenant networks that may span multiple domains, use 928 different ISF SAFIs to handle IP prefixes, in a deterministic way and 929 with routing loop protection. 931 10. Conventions used in this document 933 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 934 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 935 "OPTIONAL" in this document are to be interpreted as described in BCP 936 14 [RFC2119] [RFC8174] when, and only when, they appear in all 937 capitals, as shown here. 939 11. Security Considerations 941 This section will be added in future versions. 943 12. IANA Considerations 945 This document defines a new BGP path attribute known as the BGP 946 Domain Path (D-PATH) attribute. 948 IANA has assigned a new attribute code type from the "BGP Path 949 Attributes" subregistry under the "Border Gateway Protocol (BGP) 950 Parameters" registry: 952 Path Attribute Value Code Reference 953 -------------------- ------------------------ --------------- 954 36 BGP Domain Path (D-PATH) [This document] 956 13. Acknowledgments 958 The authors want to thank Russell Kelly for his review of the D-PATH 959 section and suggestions. 961 14. Contributors 963 15. References 965 15.1. Normative References 967 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 968 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 969 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 970 2015, . 972 [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A 973 Border Gateway Protocol 4 (BGP-4)", RFC 4271, 974 DOI 10.17487/RFC4271, January 2006, 975 . 977 [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private 978 Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 979 2006, . 981 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 982 Requirement Levels", BCP 14, RFC 2119, 983 DOI 10.17487/RFC2119, March 1997, 984 . 986 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 987 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 988 May 2017, . 990 15.2. Informative References 992 [RFC4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended 993 Communities Attribute", RFC 4360, DOI 10.17487/RFC4360, 994 February 2006, . 996 [I-D.ietf-bess-evpn-prefix-advertisement] 997 Rabadan, J., Henderickx, W., Drake, J., Lin, W., and A. 998 Sajassi, "IP Prefix Advertisement in EVPN", draft-ietf- 999 bess-evpn-prefix-advertisement-11 (work in progress), May 1000 2018. 1002 [I-D.ietf-bess-evpn-inter-subnet-forwarding] 1003 Sajassi, A., Salam, S., Thoria, S., Drake, J., and J. 1004 Rabadan, "Integrated Routing and Bridging in EVPN", draft- 1005 ietf-bess-evpn-inter-subnet-forwarding-08 (work in 1006 progress), March 2019. 1008 [RFC6472] Kumari, W. and K. Sriram, "Recommendation for Not Using 1009 AS_SET and AS_CONFED_SET in BGP", BCP 172, RFC 6472, 1010 DOI 10.17487/RFC6472, December 2011, 1011 . 1013 Authors' Addresses 1015 J. Rabadan (editor) 1016 Nokia 1017 777 Middlefield Road 1018 Mountain View, CA 94043 1019 USA 1021 Email: jorge.rabadan@nokia.com 1023 A. Sajassi (editor) 1024 Cisco 1025 225 West Tasman Drive 1026 San Jose, CA 95134 1027 USA 1029 Email: sajassi@cisco.com 1031 E. Rosen 1032 Individual 1034 Email: erosen52@gmail.com 1036 J. Drake 1037 Juniper 1039 Email: jdrake@juniper.net 1041 W. Lin 1042 Juniper 1044 Email: wlin@juniper.net 1045 J. Uttaro 1046 AT&T 1048 Email: ju1738@att.com 1050 A. Simpson 1051 Nokia 1053 Email: adam.1.simpson@nokia.com