idnits 2.17.00 (12 Aug 2021) /tmp/idnits13385/draft-ietf-bess-evpn-optimized-ir-11.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (November 17, 2021) is 185 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-14) exists of draft-ietf-bess-evpn-bum-procedure-updates-13 == Outdated reference: draft-ietf-bess-evpn-proxy-arp-nd has been published as RFC 9161 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 BESS Workgroup J. Rabadan, Ed. 3 Internet-Draft S. Sathappan 4 Intended status: Standards Track Nokia 5 Expires: May 21, 2022 W. Lin 6 Juniper Networks 7 M. Katiyar 8 Versa Networks 9 A. Sajassi 10 Cisco Systems 11 November 17, 2021 13 Optimized Ingress Replication Solution for Ethernet VPN (EVPN) 14 draft-ietf-bess-evpn-optimized-ir-11 16 Abstract 18 Network Virtualization Overlay networks using Ethernet VPN (EVPN) as 19 control plane may use Ingress Replication or PIM (Protocol 20 Independent Multicast)-based trees to convey the overlay Broadcast, 21 Unknown unicast and Multicast (BUM) traffic. PIM provides an 22 efficient solution to avoid sending multiple copies of the same 23 packet over the same physical link, however it may not always be 24 deployed in the Network Virtualization Overlay core network. Ingress 25 Replication avoids the dependency on PIM in the Network 26 Virtualization Overlay network core. While Ingress Replication 27 provides a simple multicast transport, some Network Virtualization 28 Overlay networks with demanding multicast applications require a more 29 efficient solution without PIM in the core. This document describes 30 a solution to optimize the efficiency of Ingress Replication trees. 32 Status of This Memo 34 This Internet-Draft is submitted in full conformance with the 35 provisions of BCP 78 and BCP 79. 37 Internet-Drafts are working documents of the Internet Engineering 38 Task Force (IETF). Note that other groups may also distribute 39 working documents as Internet-Drafts. The list of current Internet- 40 Drafts is at https://datatracker.ietf.org/drafts/current/. 42 Internet-Drafts are draft documents valid for a maximum of six months 43 and may be updated, replaced, or obsoleted by other documents at any 44 time. It is inappropriate to use Internet-Drafts as reference 45 material or to cite them other than as "work in progress." 47 This Internet-Draft will expire on May 21, 2022. 49 Copyright Notice 51 Copyright (c) 2021 IETF Trust and the persons identified as the 52 document authors. All rights reserved. 54 This document is subject to BCP 78 and the IETF Trust's Legal 55 Provisions Relating to IETF Documents 56 (https://trustee.ietf.org/license-info) in effect on the date of 57 publication of this document. Please review these documents 58 carefully, as they describe your rights and restrictions with respect 59 to this document. Code Components extracted from this document must 60 include Simplified BSD License text as described in Section 4.e of 61 the Trust Legal Provisions and are provided without warranty as 62 described in the Simplified BSD License. 64 Table of Contents 66 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 67 2. Terminology and Conventions . . . . . . . . . . . . . . . . . 6 68 3. Solution Requirements . . . . . . . . . . . . . . . . . . . . 8 69 4. EVPN BGP Attributes for Optimized Ingress Replication . . . . 9 70 5. Non-Selective Assisted-Replication (AR) Solution Description 13 71 5.1. Non-selective AR-REPLICATOR Procedures . . . . . . . . . 14 72 5.2. Non-Selective AR-LEAF Procedures . . . . . . . . . . . . 17 73 5.3. RNVE Procedures . . . . . . . . . . . . . . . . . . . . . 19 74 6. Selective Assisted-Replication (AR) Solution Description . . 20 75 6.1. Selective AR-REPLICATOR Procedures . . . . . . . . . . . 21 76 6.2. Selective AR-LEAF Procedures . . . . . . . . . . . . . . 23 77 7. Pruned-Flood-Lists (PFL) . . . . . . . . . . . . . . . . . . 26 78 7.1. A Pruned-Flood-List Example . . . . . . . . . . . . . . . 26 79 8. AR Procedures for Single-IP AR-REPLICATORS . . . . . . . . . 28 80 9. AR Procedures and EVPN All-Active Multi-homing Split-Horizon 28 81 9.1. Ethernet Segments on AR-LEAF Nodes . . . . . . . . . . . 29 82 9.2. Ethernet Segments on AR-REPLICATOR nodes . . . . . . . . 29 83 10. Security Considerations . . . . . . . . . . . . . . . . . . . 30 84 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 85 12. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 32 86 13. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 87 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 88 14.1. Normative References . . . . . . . . . . . . . . . . . . 32 89 14.2. Informative References . . . . . . . . . . . . . . . . . 33 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 34 92 1. Introduction 94 Ethernet Virtual Private Networks (EVPN) may be used as the control 95 plane for a Network Virtualization Overlay network [RFC8365]. 96 Network Virtualization Edge (NVE) and Provider Edges (PE) devices 97 that are part of the same EVPN Broadcast Domain (BD) use Ingress 98 Replication or PIM-based trees to transport the tenant's Broadcast, 99 Unknown unicast and Multicast (BUM) traffic. 101 In the Ingress Replication approach, the ingress NVE receving a BUM 102 frame from the Tenant System will create as many copies of the frame 103 as remote NVEs/PEs are attached to the BD. Each of those copies will 104 be encapsulated into an IP packet where the outer IP Destination 105 Address (IP DA) identifies the loopback of the egress NVE/PE. The IP 106 fabric core nodes (also known as Spines) will simply route the IP 107 encapsulated BUM frames based on the outer IP DA. If PIM-based trees 108 are used instead of Ingress Replication, the NVEs/PEs attached to the 109 same BD will join a PIM-based tree. The ingress NVE receiving a BUM 110 frame will send a single copy of the frame, encapsulated into an IP 111 packet where the outer IP DA is the multicast address that represents 112 the PIM-based tree. The IP fabric core nodes are part of the PIM 113 tree and keep multicast state for the multicast group, so that IP 114 encapsulated BUM frames can be routed to all the NVEs/PEs that joined 115 the tree. 117 The two approaches are illustrated in Figure 1. On the left-hand 118 side, NVE1 uses Ingress Replication to send a BUM frame (originated 119 from Tenant System TS1) to the remote nodes attached to the BD, i.e., 120 NVE2, NV3, PE1. On the right-hand side of the diagram, the same 121 example is depicted but using a PIM-based tree, i.e., (S1,G1), 122 instead of Ingress Replication. While a single copy of the tunneled 123 BUM frame is generated in the latter approach, all the routers in the 124 fabric need to keep muticast state, e.g., the Spine keeps a PIM 125 multicast routing entry for (S1,G1) with an Incoming Interface (IIF) 126 and three Outgoing Interfaces (OIFs). 128 To-WAN To-WAN 129 ^ ^ 130 | | 131 +-----+ +-----+ 132 +----------| PE1 |-----------+ +----------| PE1 |-----------+ 133 | +--^--+ | | +--^--+ | 134 | | IP Fabric | | | IP Fabric | 135 | PE | | (S1,G1) |OIF to-G | 136 | +----PE->+-----+ No State | | IIF +-----+ OIF to-G | 137 | | +---2->|Spine|------+ | | +------>Spine|------+ | 138 | | | +-3->+-----+ | | | | +-----+ | | 139 | | | | 2 3 | | |PIM |OIF to-G | | 140 | | | |IR | | | | |tree | | | 141 |+-----+ +--v--+ +--v--+ | |+-----+ +--v--+ +--v--+ | 142 +| NVE1|---| NVE2|---| NVE3|-+ +| NVE1|---| NVE2|---| NVE3|-+ 143 +--^--+ +-----+ +-----+ +--^--+ +-----+ +-----+ 144 | | | | | | 145 | v v | v v 146 TS1 TS2 TS3 TS1 TS2 TS3 148 Figure 1: Ingress Replication vs PIM-based trees in NVO networks 150 In Network Virtualization Overlay networks where PIM-based trees 151 cannot be used, Ingress Replication is the only option. Examples of 152 these situations are Network Virtualization Overlay networks where 153 the core nodes do not support PIM or the network operator does not 154 want to run PIM in the core. 156 In some use-cases, the amount of replication for BUM traffic is kept 157 under control on the NVEs due to the following fairly common 158 assumptions: 160 a. Broadcast is greatly reduced due to the proxy ARP (Address 161 Resolution Protocol) and proxy ND (Neighbor Discovery) 162 capabilities supported by EVPN on the NVEs 163 [I-D.ietf-bess-evpn-proxy-arp-nd]. Some NVEs can even provide 164 Dynamic Host Configuration Protocol (DHCP) server functions for 165 the attached Tenant Systems, reducing the broadcast even further. 167 b. Unknown unicast traffic is greatly reduced in Network 168 Virtualization Overlay networks where all the MAC and IP 169 addresses from the Tenant Systems are learned in the control 170 plane. 172 c. Multicast applications are not used. 174 If the above assumptions are true for a given Network Virtualization 175 Overlay network, then Ingress Replication provides a simple solution 176 for multi-destination traffic. However, the statement c) above is 177 not always true and multicast applications are required in many use- 178 cases. 180 When the multicast sources are attached to NVEs residing in 181 hypervisors or low-performance-replication TORs (Top Of Rack 182 switches), the ingress replication of a large amount of multicast 183 traffic to a significant number of remote NVEs/PEs can seriously 184 degrade the performance of the NVE and impact the application. 186 This document describes a solution that makes use of two Ingress 187 Replication optimizations: 189 1. Assisted-Replication (AR) 191 2. Pruned-Flood-Lists (PFL) 193 Assisted-Replication consists of a set of procedures that allows the 194 ingress NVE/PE to send a single copy of a Broadcast or Multicast 195 frame received from a Tenant System to the Broadcast Domain, without 196 the need for PIM in the underlay. Assisted Replication defines the 197 roles of AR-REPLICATOR and AR-LEAF routers. The AR-LEAF is the 198 ingress NVE/PE attached to the Tenant System. The AR-LEAF sends a 199 single copy of a Broadcast or Multicast packet to a selected AR- 200 REPLICATOR that replicates the packet mutiple times to remote AR-LEAF 201 or AR-REPLICATOR routers, and therefore "assisting" the ingress AR- 202 LEAF in delivering the Broadcast or Multicast traffic to the remote 203 NVEs/PEs attached to the same Broadcast Domain. Assisted-Replication 204 can use a single AR-REPLICATOR or two AR-REPLICATOR routers in the 205 path between the ingress AR-LEAF and the remote destination NVE/PEs. 206 The procedures that use a single AR-REPLICATOR (Non-Selective 207 Assisted-Replication Solution) are specified in Section 5, whereas 208 Section 6 describes how multi-staged replication, i.e., two AR- 209 REPLICATOR routers in the path between the ingress AR-LEAF and 210 destination NVEs/PEs, is accomplished (Selective Assisted-Replication 211 Solution). The Assisted-Replication procedures do not impact unknown 212 unicast traffic, which follows the same forwarding procedures as 213 known unicast traffic so that packet re-ordering does not occur. 215 Pruned-Flood-Lists is a method for the ingress NVE/PE to prune or 216 remove certain destination NVEs/PEs from a flood-list, depending on 217 the interest of those NVEs/PEs in receiving Broadcast, Multicast or 218 Unknown unicast. As specfied in [RFC8365], an NVE/PE builds a flood- 219 list for BUM traffic based on the Next-Hops of the received EVPN 220 Inclusive Multicast Ethernet Tag routes for the Broadcast Domain. 221 While [RFC8365] states that the flood-list is used for all BUM 222 traffic, this document allows pruning certain Next-Hops from the 223 list. As an example, suppose an ingress NVE creates a flood-list 224 with Next-Hops PE1, PE2 and PE3. If PE2 and PE3 signaled no-interest 225 in receiving Unknown Unicast in their Inclusive Multicast Ethernet 226 Tag routes, when the ingress NVE receives an Unknown Unicast frame 227 from a Tenant System it will replicate it only to PE1. That is, PE2 228 and PE3 are "pruned" from the NVE's flood-list for Unknown Unicast 229 traffic. Pruned-Flood-Lists can be used with Ingress Replication or 230 Assisted-Replication, and it is described in Section 7. 232 Both optimizations, Assisted-Replication and Pruned-Flood-Lists, may 233 be used together or independently so that the performance and 234 efficiency of the network to transport multicast can be improved. 235 Both solutions require some extensions to the BGP attributes used in 236 [RFC7432], and they are described in Section 4. 238 The Assisted-Replication solution described in this document is 239 focused on Network Virtualization Overlay networks (hence it uses IP 240 tunnels) and MPLS transport networks are out of scope. The Pruned- 241 Flood-Lists solution MAY be used in Network Virtualization Overlay 242 and MPLS transport networks. 244 Section 3 lists the requirements of the combined optimized Ingress 245 Replication solution, whereas Section 5 and Section 6 describe the 246 Assisted-Replication solution (for Non-Selective and Selective 247 procedures, respectively), and Section 7 the Pruned-Flood-Lists 248 solution. 250 2. Terminology and Conventions 252 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 253 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 254 "OPTIONAL" in this document are to be interpreted as described in BCP 255 14 [RFC2119] [RFC8174] when, and only when, they appear in all 256 capitals, as shown here. 258 The following terminology is used throughout the document: 260 - Asisted Replication forwarding mode: for an AR-LEAF, it means 261 sending an Attachment Circuit BM packet to a single AR-REPLICATOR 262 with tunnel destination IP AR-IP. For an AR-REPLICATOR, it means 263 sending a BM packet to a selected number or all the overlay 264 tunnels when the packet was previously received from an overlay 265 tunnel. 267 - AR-LEAF: Assisted Replication - LEAF, refers to an NVE/PE that 268 sends all the Broadcast and Multicast traffic to an AR-REPLICATOR 269 that can replicate the traffic further on its behalf. An AR-LEAF 270 is typically an NVE/PE with poor replication performance 271 capabilities. 273 - AR-REPLICATOR: Assisted Replication - REPLICATOR, refers to an 274 NVE/PE that can replicate Broadcast or Multicast traffic received 275 on overlay tunnels to other overlay tunnels and local Attachment 276 Circuits. This document defines the control and data plane 277 procedures that an AR-REPLICATOR needs to follow. 279 - AR-IP: IP address owned by the AR-REPLICATOR and used to 280 differentiate the incoming traffic that must follow the AR 281 procedures. The AR-IP is also used in the Tunnel Identifier and 282 Next-Hop fields of the Replicator-AR route. 284 - AR-VNI: VNI advertised by the AR-REPLICATOR along with the 285 Replicator-AR route. It is used to identify the incoming packets 286 that must follow AR procedures ONLY in the Single-IP AR-REPLICATOR 287 case Section 8. 289 - BM traffic: Refers to Broadcast and Multicast frames (excluding 290 unknown unicast frames). 292 - BD: Broadcast Domain, as defined in [RFC7432]. 294 - DF and NDF: Designated Forwarder and Non-Designated Forwarder, are 295 roles defined in NVE/PEs attached to Multi-Homed Tenant Systems, 296 as per [RFC7432] and [RFC8365]. 298 - ES and ESI: Ethernet Segment and Ethernet Segment Identifier, as 299 EVPN Multi-Homing concepts specified in [RFC7432]. 301 - EVI: EVPN Instance. A group of Provider Edge (PE) devices 302 participating in the same EVPN service, as specified in [RFC7432]. 304 - GRE: Generic Routing Encapsulation [RFC4023]. 306 - Ingress Replication forwarding mode: it refers to the Ingress 307 Replication behavior explained in [RFC7432]. It means sending an 308 Attachment Circuit BM packet copy to each remote PE/NVE in the BD 309 and sending an overlay BM packet only to the Attachment Circuits 310 and not other overlay tunnels. 312 - IR-IP: local IP address of an NVE/PE that is used for the Ingress 313 Replication signaling and procedures in [RFC7432]. Encapsulated 314 incoming traffic with outer destination IP matching the IR-IP will 315 follow the Ingress Replication procedures and not the Assisted- 316 Replication procedures. The IR-IP is also used in the Tunnel 317 Identifier and Next-hop fields of the Regular-IR route. 319 - IR-VNI: VNI advertised along with the Inclusive Multicast Ethernet 320 Tag route for Ingress Replication Tunnel Type. 322 - MPLS: Multi-Protocol Label Switching. 324 - NVE: Network Virtualization Edge router, used in this document as 325 in [RFC8365]. 327 - NVGRE: Network Virtualization using Generic Routing Encapsulation, 328 as in [RFC7637]. 330 - PE: Provider Edge router. 332 - PMSI: P-Multicast Service Interface - a conceptual interface for a 333 PE to send customer multicast traffic to all or some PEs in the 334 same VPN [RFC6513]. 336 - RD: Route Distinguisher. 338 - Regular-IR route: an EVPN Inclusive Multicast Ethernet Tag route 339 [RFC7432] that uses Ingress Replication Tunnel Type. 341 - RNVE: Regular NVE, refers to an NVE that supports the procedures 342 of [RFC8365] and does not support the procedures in this document. 343 However, this document defines procedures to interoperate with 344 RNVEs. 346 - Replicator-AR route: an EVPN Inclusive Multicast Ethernet Tag 347 route that is advertised by an AR-REPLICATOR to signal its 348 capabilities. 350 - TOR: Top Of Rack switch. 352 - TS and VM: Tenant System and Virtual Machine. In this document 353 Tenant Systems and Virtual Machiness are the devices connected to 354 the Attachment Circuits of the PEs and NVEs. 356 - VNI: VXLAN Network Identifier, used in VXLAN tunnels. 358 - VSID: Virtual Segment Identifier, used in NVGRE tunnels. 360 - VXLAN: Virtual Extensible LAN [RFC7348]. 362 3. Solution Requirements 364 The Ingress Replication optimization solution specified in this 365 document meets the following requirements: 367 a. It provides an Ingress Replication optimization for Broadcast and 368 Multicast traffic without the need for PIM, while preserving the 369 packet order for unicast applications, i.e., unknown unicast 370 traffic should follow the same path as known unicast traffic. 371 This optimization is required in low-performance NVEs. 373 b. It reduces the flooded traffic in Network Virtualization Overlay 374 networks where some NVEs do not need broadcast/multicast and/or 375 unknown unicast traffic. 377 c. The solution is compatible with [RFC7432] and [RFC8365] and has 378 no impact on the CE procedures for BM traffic. In particular, 379 the solution supports the following EVPN functions: 381 o All-active multi-homing, including the split-horizon and 382 Designated Forwarder (DF) functions. 384 o Single-active multi-homing, including the DF function. 386 o Handling of multi-destination traffic and processing of 387 broadcast and multicast as per [RFC7432]. 389 d. The solution is backwards compatible with existing NVEs using a 390 non-optimized version of Ingress Replication. A given BD can 391 have NVEs/PEs supporting regular Ingress Replication and 392 optimized Ingress Replication. 394 e. The solution is independent of the Network Virtualization Overlay 395 specific data plane encapsulation and the virtual identifiers 396 being used, e.g.: VXLAN VNIs, NVGRE VSIDs or MPLS labels, as long 397 as the tunnel is IP-based. 399 4. EVPN BGP Attributes for Optimized Ingress Replication 401 This solution extends the [RFC7432] Inclusive Multicast Ethernet Tag 402 routes and attributes so that an NVE/PE can signal its optimized 403 Ingress Replication capabilities. 405 The NLRI of the Inclusive Multicast Ethernet Tag route as in 406 [RFC7432] is shown in Figure 2 and it is used in this document 407 without any modifications to its format. The PMSI Tunnel Attribute's 408 general format as in [RFC7432] (which takes it from [RFC6514]) is 409 used in this document, only a new Tunnel Type and new flags are 410 specified, as shown in Figure 3: 412 +---------------------------------+ 413 | RD (8 octets) | 414 +---------------------------------+ 415 | Ethernet Tag ID (4 octets) | 416 +---------------------------------+ 417 | IP Address Length (1 octet) | 418 +---------------------------------+ 419 | Originating Router's IP Addr | 420 | (4 or 16 octets) | 421 +---------------------------------+ 423 Figure 2: EVPN Inclusive Multicast Tag route's NLRI 425 0 1 2 3 4 5 6 7 426 +---------------------------------+ +--+--+--+--+--+--+--+--+ 427 | Flags (1 octet) | -> |x |E |x | T |BM|U |L | 428 +---------------------------------+ +--+--+--+--+--+--+--+--+ 429 | Tunnel Type (1 octets) | T = Assisted-Replication Type 430 +---------------------------------+ BM = Broadcast and Multicast 431 | MPLS Label (3 octets) | U = Unknown unicast 432 +---------------------------------+ x = unassigned 433 | Tunnel Identifier (variable) | 434 +---------------------------------+ 436 Figure 3: PMSI Tunnel Attribute 438 The Flags field in Figure 3 is 8 bits long as per [RFC7902], where 439 the Extension flag (E) and the Leaf Information Required (L) Flag are 440 already allocated. This document defines the use of 4 bits of this 441 Flags field, and suggests the following allocation to IANA: 443 - bits 3 and 4, forming together the Assisted-Replication Type (T) 444 field 446 - bit 5, called the Broadcast and Multicast (BM) flag 448 - bit 6, called the Unknown (U) flag 450 Bits 5 and 6 are collectively referred to as the Pruned-Flood Lists 451 (PFL) flags. 453 The T field and Pruned-Flood-Lists flags are defined as follows: 455 - T is the Assisted-Replication Type field (2 bits) that defines the 456 AR role of the advertising router: 458 o 00 (decimal 0) = RNVE (non-AR support) 459 o 01 (decimal 1) = AR-REPLICATOR 461 o 10 (decimal 2) = AR-LEAF 463 o 11 (decimal 3) = RESERVED 465 - The Pruned-Flood-Lists flags define the desired behavior of the 466 advertising router for the different types of traffic: 468 o Broadcast and Multicast (BM) flag. BM=1 means "prune-me" from 469 the BM flooding list. BM=0 means regular behavior. 471 o Unknown (U) flag. U=1 means "prune-me" from the Unknown 472 flooding list. U=0 means regular behavior. 474 - Flag L is an existing flag defined in [RFC6514] (L=Leaf 475 Information Required, bit 7) and it will be used only in the 476 Selective AR Solution. 478 Please refer to Section 11 for the IANA considerations related to the 479 PMSI Tunnel Attribute flags. 481 In this document, the above Inclusive Multicast Ethernet Tag route 482 Figure 2 and PMSI Tunnel Attribute Figure 3 can be used in two 483 different modes for the same BD: 485 - Regular-IR route: in this route, Originating Router's IP Address, 486 Tunnel Type (0x06), MPLS Label and Tunnel Identifier MUST be used 487 as described in [RFC7432] when Ingress Replication is in use. The 488 NVE/PE that advertises the route will set the Next-Hop to an IP 489 address that we denominate IR-IP in this document. When 490 advertised by an AR-LEAF node, the Regular-IR route MUST be 491 advertised with type T set to 10 (AR-LEAF). 493 - Replicator-AR route: this route is used by the AR-REPLICATOR to 494 advertise its AR capabilities, with the fields set as follows: 496 o Originating Router's IP Address MUST be set to an IP address of 497 the advertising router that is common to all the EVIs on the PE 498 (usually this is a loopback address of the PE). 500 + The Tunnel Identifier and Next-Hop SHOULD be set to the same 501 IP address as the Originating Router's IP address when the 502 NVE/PE originates the route, that is, when the NVE/PE is not 503 an ASBR as in section 10.2 of [RFC8365]. Irrespective of 504 the values in the Tunnel Identifier and Originating Router's 505 IP Address fields, the ingress NVE/PE will process the 506 received Replicator-AR route and will use the IP Address in 507 the Next-Hop field to create IP tunnels to the AR- 508 REPLICATOR. 510 + The Next-Hop address is referred to as the AR-IP and MUST be 511 different from the IR-IP for a given PE/NVE, unless the 512 procedures in Section 8 are followed. 514 o Tunnel Type MUST be set to Assisted-Replication Tunnel. 515 Section 11 provides the allocated type value. 517 o T (AR role type) MUST be set to 01 (AR-REPLICATOR). 519 o L (Leaf Information Required) MUST be set to 0 (for non- 520 selective AR), and MUST be set to 1 (for selective AR). 522 An NVE/PE configured as AR-REPLICATOR for a BD MUST advertise a 523 Replicator-AR route for the BD and MAY advertise a Regular-IR route. 524 The advertisement of the Replicator-AR route will indicate the AR- 525 LEAFs what outer IP DA, i.e., the AR-IP, they need to use for IP 526 encapsulated BM frames that use Assisted Replication forwarding mode. 527 The AR-REPLICATOR will forward an IP encapsulated BM frame in 528 Assisted Replication forwarding mode if the outer IP DA matches its 529 AR-IP, but will forward in Ingress Replication forwarding mode if the 530 outer IP DA matches its IR-IP. 532 In addition, this document also uses the Leaf Auto-Discovery route 533 defined in [I-D.ietf-bess-evpn-bum-procedure-updates] in case the 534 selective AR mode is used. An AR-LEAF MAY send a Leaf A-D route in 535 response to reception of a Replicator-AR route whose L flag is set. 536 The Leaf Auto-Discovery route is only used for selective AR and the 537 fields of such route are set as follows: 539 o Originating Router's IP Address is set to the advertising 540 router's IP address (same IP used by the AR-LEAF in regular-IR 541 routes). The Next-Hop address is set to the IR-IP, which 542 SHOULD be the same IP address as the advertising router's IP 543 address, when the NVE/PE originates the route, i.e., when the 544 NVE/PE is not an ASBR as in section 10.2 of [RFC8365]. 546 o Route Key is the "Route Type Specific" NLRI of the Replicator- 547 AR route for which this Leaf Auto-Discovery route is generated. 549 o The AR-LEAF constructs an IP-address-specific route-target, 550 analogously to [I-D.ietf-bess-evpn-bum-procedure-updates], by 551 placing the IP address carried in the Next-Hop field of the 552 received Replicator-AR route in the Global Administrator field 553 of the Community, with the Local Administrator field of this 554 Community set to 0, and setting the Extended Communities 555 attribute of the Leaf Auto-Discovery route to that Community. 556 The same IP-address-specific import route-target is auto- 557 configured by the AR-REPLICATOR that sent the Replicator-AR 558 route, in order to control the acceptance of the Leaf Auto- 559 Discovery routes. 561 o The Leaf Auto-Discovery route MUST include the PMSI Tunnel 562 attribute with the Tunnel Type set to AR (Section 11), T (AR 563 role type) set to AR-LEAF and the Tunnel Identifier set to the 564 IP address of the advertising AR-LEAF. The PMSI Tunnel 565 attribute MUST carry a downstream-assigned MPLS label or VNI 566 that is used by the AR-REPLICATOR to send traffic to the AR- 567 LEAF. 569 Each AR-enabled node understands and process the T (Assisted- 570 Replication type) field in the PMSI Tunnel Attribute (Flags field) of 571 the routes, and MUST signal the corresponding type (AR-REPLICATOR or 572 AR-LEAF type) according to its administrative choice. An NVE/PE 573 following this specification is not expected to set the AR type field 574 to decimal 3 (which is a RESERVED value). If a route with the AR 575 type field set to decimal 3 is received by an AR-REPLICATOR or AR- 576 LEAF, the router will process the route as a Regular-IR route 577 advertised by an RNVE. 579 Each node attached to the BD may understand and process the BM/U 580 flags (Pruned-Flood-Lists flags). Note that these BM/U flags may be 581 used to optimize the delivery of multi-destination traffic and their 582 use SHOULD be an administrative choice, and independent of the AR 583 role. When the Pruned-Flood-List capability is enabled, the BM/U 584 flags can be used with the Regular-IR, Replicator-AR and Leaf Auto- 585 Discovery routes. 587 Non-optimized Ingress Replication NVEs/PEs will be unaware of the new 588 PMSI Tunnel Attribute flag definition as well as the new Tunnel Type 589 (AR), i.e., non-upgraded NVEs/PEs will ignore the information 590 contained in the flags field or an unknown Tunnel Type (type AR in 591 this case) for any Inclusive Multicast Ethernet Tag route. 593 5. Non-Selective Assisted-Replication (AR) Solution Description 595 Figure 4 illustrates an example Network Virtualization Overlay 596 network where the non-selective AR function is enabled. Three 597 different roles are defined for a given BD: AR-REPLICATOR, AR-LEAF 598 and RNVE (Regular NVE). The solution is called "non-selective" 599 because the chosen AR-REPLICATOR for a given flow MUST replicate the 600 BM traffic to all the NVE/PEs in the BD except for the source NVE/PE. 602 Network Virtualization Overlay tunnels, i.e., IP tunnels, exist among 603 all the PEs and NVEs in the diagram. The PEs and NVEs in the diagram 604 have Tenant Systems or Virtual Machines connected to their Attachment 605 Circuits. 607 ( ) 608 (_ WAN _) 609 +---(_ _)----+ 610 | (_ _) | 611 PE1 | PE2 | 612 +------+----+ +----+------+ 613 TS1--+ (BD-1) | | (BD-1) +--TS2 614 |REPLICATOR | |REPLICATOR | 615 +--------+--+ +--+--------+ 616 | | 617 +--+----------------+--+ 618 | | 619 | | 620 +----+ VXLAN/nvGRE/MPLSoGRE +----+ 621 | | IP Fabric | | 622 | | | | 623 NVE1 | +-----------+----------+ | NVE3 624 Hypervisor| TOR | NVE2 |Hypervisor 625 +---------+-+ +-----+-----+ +-+---------+ 626 | (BD-1) | | (BD-1) | | (BD-1) | 627 | LEAF | | RNVE | | LEAF | 628 +--+-----+--+ +--+-----+--+ +--+-----+--+ 629 | | | | | | 630 VM11 VM12 TS3 TS4 VM31 VM32 632 Figure 4: Non-Selective AR scenario 634 In AR BDs such as BD-1 in the example, BM (Broadcast and Multicast) 635 traffic between two NVEs may follow a different path than unicast 636 traffic. This solution recommends the replication of BM through the 637 AR-REPLICATOR node, whereas unknown/known unicast will be delivered 638 directly from the source node to the destination node without being 639 replicated by any intermediate node. 641 Note that known unicast forwarding is not impacted by this solution, 642 i.e., unknown unicast SHALL follow the same path as known unicast 643 traffic. 645 5.1. Non-selective AR-REPLICATOR Procedures 647 An AR-REPLICATOR is defined as an NVE/PE capable of replicating 648 incoming BM traffic received on an overlay tunnel to other overlay 649 tunnels and local Attachment Circuits. The AR-REPLICATOR signals its 650 role in the control plane and understands where the other roles (AR- 651 LEAF nodes, RNVEs and other AR-REPLICATORs) are located. A given AR- 652 enabled BD service may have zero, one or more AR-REPLICATORs. In our 653 example in Figure 4, PE1 and PE2 are defined as AR-REPLICATORs. The 654 following considerations apply to the AR-REPLICATOR role: 656 a. The AR-REPLICATOR role SHOULD be an administrative choice in any 657 NVE/PE that is part of an AR-enabled BD. This administrative 658 option to enable AR-REPLICATOR capabilities MAY be implemented as 659 a system level option as opposed to as a per-BD option. 661 b. An AR-REPLICATOR MUST advertise a Replicator-AR route and MAY 662 advertise a Regular-IR route. The AR-REPLICATOR MUST NOT 663 generate a Regular-IR route if it does not have local attachment 664 circuits (AC). If the Regular-IR route is advertised, the 665 Assisted-Replication Type field of the Regular-IR route MUST be 666 set to zero. 668 c. The Replicator-AR and Regular-IR routes are generated according 669 to Section 4. The AR-IP and IR-IP are different IP addresses 670 owned by the AR-REPLICATOR. 672 d. When a node defined as AR-REPLICATOR receives a BM packet on an 673 overlay tunnel, it will do a tunnel destination IP address lookup 674 and apply the following procedures: 676 o If the destination IP address is the AR-REPLICATOR IR-IP 677 Address the node will process the packet normally as in 678 [RFC7432]. 680 o If the destination IP address is the AR-REPLICATOR AR-IP 681 Address the node MUST replicate the packet to local Attachment 682 Circuits and overlay tunnels (excluding the overlay tunnel to 683 the source of the packet). When replicating to remote AR- 684 REPLICATORs the tunnel destination IP address will be an IR- 685 IP. That will be an indication for the remote AR-REPLICATOR 686 that it MUST NOT replicate to overlay tunnels. The tunnel 687 source IP address used by the AR-REPLICATOR MUST be its IR-IP 688 when replicating to AR-REPLICATOR or AR-LEAF nodes. 690 An AR-REPLICATOR MUST follow a data path implementation compatible 691 with the following rules: 693 - The AR-REPLICATORs will build a flooding list composed of 694 Attachment Circuits and overlay tunnels to remote nodes in the BD. 695 Some of those overlay tunnels MAY be flagged as non-BM receivers 696 based on the BM flag received from the remote nodes in the BD. 698 - When an AR-REPLICATOR receives a BM packet on an Attachment 699 Circuit, it will forward the BM packet to its flooding list 700 (including local Attachment Circuits and remote NVE/PEs), skipping 701 the non-BM overlay tunnels. 703 - When an AR-REPLICATOR receives a BM packet on an overlay tunnel, 704 it will check the destination IP address of the underlay IP header 705 and: 707 o If the destination IP address matches its IR-IP, the AR- 708 REPLICATOR will skip all the overlay tunnels from the flooding 709 list, i.e. it will only replicate to local Attachment Circuits. 710 This is the regular Ingress Replication behavior described in 711 [RFC7432]. 713 o If the destination IP address matches its AR-IP, the AR- 714 REPLICATOR MUST forward the BM packet to its flooding list (ACs 715 and overlay tunnels) excluding the non-BM overlay tunnels. The 716 AR-REPLICATOR will ensure the traffic is not sent back to the 717 originating AR-LEAF. 719 o If the encapsulation is MPLSoGRE or MPLSoUDP and the received 720 BD label (or label that the AR-REPLICATOR advertised in the 721 Replicator-AR route) is not the bottom of the stack, the AR- 722 REPLICATOR MUST copy the all the labels below the BD label and 723 propagate them when forwarding the packet to the egress overlay 724 tunnels. 726 - The AR-REPLICATOR/LEAF nodes will build an Unknown unicast flood- 727 list composed of Attachment Circuits and overlay tunnels to the 728 IR-IP Addresses of the remote nodes in the BD. Some of those 729 overlay tunnels MAY be flagged as non-U (Unknown unicast) 730 receivers based on the U flag received from the remote nodes in 731 the BD. 733 o When an AR-REPLICATOR/LEAF receives an unknown unicast packet 734 on an Attachment Circuit, it will forward the unknown unicast 735 packet to its flood-list, skipping the non-U overlay tunnels. 737 o When an AR-REPLICATOR/LEAF receives an unknown unicast packet 738 on an overlay tunnel, it will forward the unknown unicast 739 packet to its local Attachment Circuits and never to an overlay 740 tunnel. This is the regular Ingress Replication behavior 741 described in [RFC7432]. 743 5.2. Non-Selective AR-LEAF Procedures 745 AR-LEAF is defined as an NVE/PE that - given its poor replication 746 performance - sends all the BM traffic to an AR-REPLICATOR that can 747 replicate the traffic further on its behalf. It MAY signal its AR- 748 LEAF capability in the control plane and understands where the other 749 roles are located (AR-REPLICATOR and RNVEs). A given service can 750 have zero, one or more AR-LEAF nodes. Figure 4 shows NVE1 and NVE3 751 (both residing in hypervisors) acting as AR-LEAF. The following 752 considerations apply to the AR-LEAF role: 754 a. The AR-LEAF role SHOULD be an administrative choice in any NVE/PE 755 that is part of an AR-enabled BD. This administrative option to 756 enable AR-LEAF capabilities MAY be implemented as a system level 757 option as opposed to as per-BD option. 759 b. In this non-selective AR solution, the AR-LEAF MUST advertise a 760 single Regular-IR inclusive multicast route as in [RFC7432]. The 761 AR-LEAF SHOULD set the Assisted-Replication Type field to AR- 762 LEAF. Note that although this field does not make any difference 763 for the remote nodes when creating an EVPN destination to the AR- 764 LEAF, this field is useful for an easy operation and 765 troubleshooting of the BD. 767 c. In a BD where there are no AR-REPLICATORs due to the AR- 768 REPLICATORs being down or reconfigured, the AR-LEAF MUST use 769 regular Ingress Replication, based on the remote Regular-IR 770 Inclusive Multicast Routes as described in [RFC7432]. This may 771 happen in the following cases: 773 o The AR-LEAF has a list of AR-REPLICATORs for the BD, but it 774 detects that all the AR-REPLICATORs for the BD are down (via 775 next-hop tracking in the IGP or any other detection 776 mechanism). 778 o The AR-LEAF receives updates from all the former AR- 779 REPLICATORs containing a non-REPLICATOR AR type in the 780 Inclusive Multicast Etherner Tag routes. 782 o The AR-LEAF never discovered an AR-REPLICATOR for the BD. 784 d. In a service where there is one or more AR-REPLICATORs (based on 785 the received Replicator-AR routes for the BD), the AR-LEAF can 786 locally select which AR-REPLICATOR it sends the BM traffic to: 788 o A single AR-REPLICATOR MAY be selected for all the BM packets 789 received on the AR-LEAF attachment circuits (ACs) for a given 790 BD. This selection is a local decision and it does not have 791 to match other AR-LEAF's selections within the same BD. 793 o An AR-LEAF MAY select more than one AR-REPLICATOR and do 794 either per-flow or per-BD load balancing. 796 o In case of a failure of the selected AR-REPLICATOR, another 797 AR-REPLICATOR SHOULD be selected by the AR-LEAF. 799 o When an AR-REPLICATOR is selected for a given flow or BD, the 800 AR-LEAF MUST send all the BM packets targeted to that AR- 801 REPLICATOR using the forwarding information given by the 802 Replicator-AR route for the chosen AR-REPLICATOR, with tunnel 803 type = 0x0A (AR tunnel). The underlay destination IP address 804 MUST be the AR-IP advertised by the AR-REPLICATOR in the 805 Replicator-AR route. 807 o An AR-LEAF MAY change the AR-REPLICATOR(s) selection 808 dynamically, due to an administrative or policy configuration 809 change. 811 o AR-LEAF nodes SHALL send service-level BM control plane 812 packets following regular Ingress Replication procedures. An 813 example would be IGMP, MLD or PIM multicast packets, and in 814 general any packets using link-local scope multicast IPv4 or 815 IPv6 packets. The AR-REPLICATORs MUST NOT replicate these 816 control plane packets to other overlay tunnels since they will 817 use the regular IR-IP Address. 819 e. The use of an AR-REPLICATOR-activation-timer (in seconds, default 820 value is 3) on the AR-LEAF nodes is RECOMMENDED. Upon receiving 821 a new Replicator-AR route where the AR-REPLICATOR is selected, 822 the AR-LEAF will run a timer before programming the new AR- 823 REPLICATOR. In case of a new added AR-REPLICATOR, or in case the 824 AR-REPLICATOR reboots, this timer will give the AR-REPLICATOR 825 some time to program the AR-LEAF nodes before the AR-LEAF sends 826 BM traffic. The AR-REPLICATOR-activation-timer SHOULD be 827 configurable in seconds, and its value account for the time it 828 takes for the AR-LEAF Regular-IR inclusive multicast route to get 829 to the AR-REPLICATOR and be programmed. While the AR-REPLICATOR- 830 activation-time is running, the AR-LEAF node will use regular 831 ingress replication. 833 f. If the AR-LEAF has selected an AR-REPLICATOR, it is a matter of 834 local policy to change to a new preferred AR-REPLICATOR for the 835 existing BM traffic flows. 837 An AR-LEAF MUST follow a data path implementation compatible with the 838 following rules: 840 - The AR-LEAF nodes will build two flood-lists: 842 1. Flood-list #1 - composed of Attachment Circuits and an AR- 843 REPLICATOR-set of overlay tunnels. The AR-REPLICATOR-set is 844 defined as one or more overlay tunnels to the AR-IP Addresses 845 of the remote AR-REPLICATOR(s) in the BD. The selection of 846 more than one AR-REPLICATOR is described in point d) above and 847 it is a local AR-LEAF decision. 849 2. Flood-list #2 - composed of Attachment Circuits and overlay 850 tunnels to the remote IR-IP Addresses. 852 - When an AR-LEAF receives a BM packet on an Attachment Circuit, it 853 will check the AR-REPLICATOR-set: 855 o If the AR-REPLICATOR-set is empty, the AR-LEAF MUST send the 856 packet to flood-list #2. 858 o If the AR-REPLICATOR-set is NOT empty, the AR-LEAF MUST send 859 the packet to flood-list #1, where only one of the overlay 860 tunnels of the AR-REPLICATOR-set is used. 862 - When an AR-LEAF receives a BM packet on an overlay tunnel, it will 863 forward the BM packet to its local Attachment Circuits and never 864 to an overlay tunnel. This is the regular Ingress Replication 865 behavior described in [RFC7432]. 867 - AR-LEAF nodes process Unknown unicast traffic in the same way AR- 868 REPLICATORS do, as described in Section 5.1. 870 5.3. RNVE Procedures 872 RNVE (Regular Network Virtualization Edge node) is defined as an NVE/ 873 PE without AR-REPLICATOR or AR-LEAF capabilities that does Ingress 874 Replication as described in [RFC7432]. The RNVE does not signal any 875 AR role and is unaware of the AR-REPLICATOR/LEAF roles in the BD. 876 The RNVE will ignore the Flags in the Regular-IR routes and will 877 ignore the Replicator-AR routes (due to an unknown tunnel type in the 878 PMSI Tunnel Attribute) and the Leaf Auto-Discovery routes (due to the 879 IP-address-specific route-target). 881 This role provides EVPN with the backwards compatibility required in 882 optimized Ingress Replication BDs. Figure 4 shows NVE2 as RNVE. 884 6. Selective Assisted-Replication (AR) Solution Description 886 Figure 5 is used to describe the selective AR solution. 888 ( ) 889 (_ WAN _) 890 +---(_ _)----+ 891 | (_ _) | 892 PE1 | PE2 | 893 +------+----+ +----+------+ 894 TS1--+ (BD-1) | | (BD-1) +--TS2 895 |REPLICATOR | |REPLICATOR | 896 +--------+--+ +--+--------+ 897 | | 898 +--+----------------+--+ 899 | | 900 | | 901 +----+ VXLAN/nvGRE/MPLSoGRE +----+ 902 | | IP Fabric | | 903 | | | | 904 NVE1 | +-----------+----------+ | NVE3 905 Hypervisor| TOR | NVE2 |Hypervisor 906 +---------+-+ +-----+-----+ +-+---------+ 907 | (BD-1) | | (BD-1) | | (BD-1) | 908 | LEAF-set1 | |LEAF-set-1 | |LEAF-set-2 | 909 +--+-----+--+ +--+-----+--+ +--+-----+--+ 910 | | | | | | 911 VM11 VM12 TS3 TS4 VM31 VM32 913 Figure 5: Selective AR scenario 915 The solution is called "selective" because a given AR-REPLICATOR MUST 916 replicate the BM traffic to only the AR-LEAFs that requested the 917 replication (as opposed to all the AR-LEAF nodes) and MUST replicate 918 the BM traffic to the RNVEs (if there are any). The same AR roles 919 defined in Section 4 are used here, however the procedures are 920 different. 922 The Selective AR procedures create multiple AR-LEAF-sets in the EVPN 923 BD, and builds single-hop trees among AR-LEAFs of the same set (AR- 924 LEAF->AR-REPLICATOR->AR-LEAF), and two-hop trees among AR-LEAFs of 925 different sets (AR-LEAF->AR-REPLICATOR->AR-REPLICATOR->AR-LEAF). 926 Compared to the Selective solution, the Non-Selective AR method 927 assumes that all the AR-LEAFs of the BD are in the same set and 928 always create two-hop trees among AR-LEAFs. While the Selective 929 solution is more efficient than the Non-Selective solution in multi- 930 stage IP fabrics, the trade-off is additional signaling and an 931 additional outer source IP address lookup. 933 The following sub-sections describe the differences in the procedures 934 of AR-REPLICATOR/LEAFs compared to the non-selective AR solution. 935 There is no change on the RNVEs. 937 6.1. Selective AR-REPLICATOR Procedures 939 In our example in Figure 5, PE1 and PE2 are defined as Selective AR- 940 REPLICATORs. The following considerations apply to the Selective AR- 941 REPLICATOR role: 943 a. The Selective AR-REPLICATOR capability SHOULD be an 944 administrative choice in any NVE/PE that is part of an Assisted- 945 Replication-enabled BD, as the AR role itself. This 946 administrative option MAY be implemented as a system level option 947 as opposed to as a per-BD option. 949 b. Each AR-REPLICATOR will build a list of AR-REPLICATOR, AR-LEAF 950 and RNVE nodes. In spite of the 'Selective' administrative 951 option, an AR-REPLICATOR MUST NOT behave as a Selective AR- 952 REPLICATOR if at least one of the AR-REPLICATORs has the L flag 953 NOT set. If at least one AR-REPLICATOR sends a Replicator-AR 954 route with L=0 (in the BD context), the rest of the AR- 955 REPLICATORs will fall back to non-selective AR mode. 957 c. The Selective AR-REPLICATOR MUST follow the procedures described 958 in Section 5.1, except for the following differences: 960 o The Replicator-AR route MUST include L=1 (Leaf Information 961 Required) in the Replicator-AR route. This flag is used by 962 the AR-REPLICATORs to advertise their 'selective' AR- 963 REPLICATOR capabilities. In addition, the AR-REPLICATOR auto- 964 configures its IP-address-specific import route-target as 965 described in the third bullet of the procedures for Leaf Auto- 966 Discovery route in Section 4. 968 o The AR-REPLICATOR will build a 'selective' AR-LEAF-set with 969 the list of nodes that requested replication to its own AR-IP. 970 For instance, assuming NVE1 and NVE2 advertise a Leaf Auto- 971 Discovery route with PE1's IP-address-specific route-target 972 and NVE3 advertises a Leaf Auto-Discovery route with PE2's IP- 973 address-specific route-target, PE1 will only add NVE1/NVE2 to 974 its selective AR-LEAF-set for BD-1, and exclude NVE3. 975 Likewise, PE2 will only add NVE3 to its selective AR-LEAF-set 976 for BD-1, and exclude NVE1/NVE2. 978 o When a node defined and operating as a Selective AR-REPLICATOR 979 receives a packet on an overlay tunnel, it will do a tunnel 980 destination IP lookup and if the destination IP address is the 981 AR-REPLICATOR AR-IP Address, the node MUST replicate the 982 packet to: 984 + local Attachment Circuits 986 + overlay tunnels in the Selective AR-LEAF-set, excluding the 987 overlay tunnel to the source AR-LEAF. 989 + overlay tunnels to the RNVEs if the tunnel source IP 990 address is the IR-IP of an AR-LEAF. In any other case, the 991 AR-REPLICATOR MUST NOT replicate the BM traffic to remote 992 RNVEs. In other words, only the first-hop selective AR- 993 REPLICATOR will replicate to all the RNVEs. 995 + overlay tunnels to the remote Selective AR-REPLICATORs if 996 the tunnel source IP address (of the encapsulated packet 997 that arrived on the overlay tunnel) is an IR-IP of its own 998 AR-LEAF-set. In any other case, the AR-REPLICATOR MUST NOT 999 replicate the BM traffic to remote AR-REPLICATORs. When 1000 doing this replication, the tunnel destination IP address 1001 is the AR-IP of the remote Selective AR-REPLICATOR. The 1002 tunnel destination IP AR-IP will be an indication for the 1003 remote Selective AR-REPLICATOR that the packet needs 1004 further replication to its AR-LEAFs. 1006 A Selective AR-REPLICATOR data path implementation MUST be compatible 1007 with the following rules: 1009 - The Selective AR-REPLICATORs will build two flood-lists: 1011 1. Flood-list #1 - composed of Attachment Circuits and overlay 1012 tunnels to the remote nodes in the BD, always using the IR-IPs 1013 in the tunnel destination IP addresses. 1015 2. Flood-list #2 - composed of Attachment Circuits, a Selective 1016 AR-LEAF-set and a Selective AR-REPLICATOR-set, where: 1018 + The Selective AR-LEAF-set is composed of the overlay 1019 tunnels to the AR-LEAFs that advertise a Leaf Auto- 1020 Discovery route for the local AR-REPLICATOR. This set is 1021 updated with every Leaf Auto-Discovery route received/ 1022 withdrawn from a new AR-LEAF. 1024 + The Selective AR-REPLICATOR-set is composed of the overlay 1025 tunnels to all the AR-REPLICATORs that send a Replicator-AR 1026 route with L=1. The AR-IP addresses are used as tunnel 1027 destination IP. 1029 - Some of the overlay tunnels in the flood-lists MAY be flagged as 1030 non-BM receivers based on the BM flag received from the remote 1031 nodes in the routes. 1033 - When a Selective AR-REPLICATOR receives a BM packet on an 1034 Attachment Circuit, it MUST forward the BM packet to its flood- 1035 list #1, skipping the non-BM overlay tunnels. 1037 - When a Selective AR-REPLICATOR receives a BM packet on an overlay 1038 tunnel, it will check the destination and source IPs of the 1039 underlay IP header and: 1041 o If the destination IP address matches its AR-IP and the source 1042 IP address matches an IP of its own Selective AR-LEAF-set, the 1043 AR-REPLICATOR MUST forward the BM packet to its flood-list #2, 1044 unless some AR-REPLICATOR within the BD has advertised L=0. In 1045 the latter case, the node reverts back to non-selective mode 1046 and flood-list #1 MUST be used. Non-BM overlay tunnels are 1047 skipped when sending BM packets. 1049 o If the destination IP address matches its AR-IP and the source 1050 IP address does not match any IP address of its Selective AR- 1051 LEAF-set, the AR-REPLICATOR MUST forward the BM packet to 1052 flood-list #2 but skipping the AR-REPLICATOR-set. Non-BM 1053 overlay tunnels are skipped when sending BM packets. 1055 o If the destination IP address matches its IR-IP, the AR- 1056 REPLICATOR MUST use flood-list #1 but MUST skip all the overlay 1057 tunnels from the flooding list, i.e. it will only replicate to 1058 local Attachment Circuits. This is the regular-IR behavior 1059 described in [RFC7432]. Non-BM overlay tunnels are skipped 1060 when sending BM packets. 1062 - In any case, the AR-REPLICATOR ensures the traffic is not sent 1063 back to the originating source. If the encapsulation is MPLSoGRE 1064 or MPLSoUDP and the received BD label (or label that the AR- 1065 REPLICATOR advertised in the Replicator-AR route) is not the 1066 bottom of the stack, the AR-REPLICATOR MUST copy the rest of the 1067 labels when forwarding them to the egress overlay tunnels. 1069 6.2. Selective AR-LEAF Procedures 1071 A Selective AR-LEAF chooses a single Selective AR-REPLICATOR per BD 1072 and: 1074 - Sends all the BD's BM traffic to that AR-REPLICATOR and 1075 - Expects to receive all the BM traffic for a given BD from the same 1076 AR-REPLICATOR (except for the BM traffic from the RNVEs, which 1077 comes directly from the RNVEs) 1079 In the example of Figure 5, we consider NVE1/NVE2/NVE3 as Selective 1080 AR-LEAFs. NVE1 selects PE1 as its Selective AR-REPLICATOR. If that 1081 is so, NVE1 will send all its BM traffic for BD-1 to PE1. If other 1082 AR-LEAF/REPLICATORs send BM traffic, NVE1 will receive that traffic 1083 from PE1. These are the differences in the behavior of a Selective 1084 AR-LEAF compared to a non-selective AR-LEAF: 1086 a. The AR-LEAF role selective capability SHOULD be an administrative 1087 choice in any NVE/PE that is part of an Assisted-Replication- 1088 enabled BD. This administrative option to enable AR-LEAF 1089 capabilities MAY be implemented as a system level option as 1090 opposed to as per-BD option. 1092 b. The AR-LEAF MAY advertise a Regular-IR route if there are RNVEs 1093 in the BD. The Selective AR-LEAF MUST advertise a Leaf Auto- 1094 Discovery route after receiving a Replicator-AR route with L=1. 1095 It is RECOMMENDED that the Selective AR-LEAF waits for an AR- 1096 LEAF-join-wait-timer (in seconds, default value is 3) before 1097 sending the Leaf Auto-Discovery route, so that the AR-LEAF can 1098 collect all the Replicator-AR routes for the BD before 1099 advertising the Leaf Auto-Discovery route. If the Replicator-AR 1100 route with L=1 is withdrawn, the corresponding Leaf Auto- 1101 Discovery route is withdrawn too. 1103 c. In a service where there is more than one Selective AR-REPLICATOR 1104 the Selective AR-LEAF MUST locally select a single Selective AR- 1105 REPLICATOR for the BD. Once selected: 1107 o The Selective AR-LEAF MUST send a Leaf Auto-Discovery route 1108 including the Route-key and IP-address-specific route-target 1109 of the selected AR-REPLICATOR. 1111 o The Selective AR-LEAF MUST send all the BM packets received on 1112 the attachment circuits (ACs) for a given BD to that AR- 1113 REPLICATOR. 1115 o In case of a failure on the selected AR-REPLICATOR (detected 1116 when the Replicator-AR route becomes infeasible as the result 1117 of any of the underlying BGP mechanisms), another AR- 1118 REPLICATOR will be selected and a new Leaf Auto-Discovery 1119 update will be issued for the new AR-REPLICATOR. This new 1120 route will update the selective list in the new Selective AR- 1121 REPLICATOR. In case of failure of the active Selective AR- 1122 REPLICATOR, it is RECOMMENDED for the Selective AR-LEAF to 1123 revert to Ingress Replication behavior for a timer AR- 1124 REPLICATOR-activation-timer (in seconds, default value is 3) 1125 to mitigate the traffic impact. When the timer expires, the 1126 Selective AR-LEAF will resume its AR mode with the new 1127 Selective AR-REPLICATOR. The AR-REPLICATOR-activation-timer 1128 MAY be the same configurable parameter as in Section 5.2. 1130 o A Selective AR-LEAF MAY change the AR-REPLICATOR(s) selection 1131 dynamically, due to an administrative or policy configuration 1132 change. 1134 All the AR-LEAFs in a BD are expected to be configured as either 1135 selective or non-selective. A mix of selective and non-selective AR- 1136 LEAFs SHOULD NOT coexist in the same BD. In case there is a non- 1137 selective AR-LEAF, its BM traffic sent to a selective AR-REPLICATOR 1138 will not be replicated to other AR-LEAFs that are not in its 1139 Selective AR-LEAF-set. 1141 A Selective AR-LEAF MUST follow a data path implementation compatible 1142 with the following rules: 1144 - The Selective AR-LEAF nodes will build two flood-lists: 1146 1. Flood-list #1 - composed of Attachment Circuits and the 1147 overlay tunnel to the selected AR-REPLICATOR (using the AR-IP 1148 as the tunnel destination IP address). 1150 2. Flood-list #2 - composed of Attachment Circuits and overlay 1151 tunnels to the remote IR-IP addresses. 1153 - Some of the overlay tunnels in the flood-lists MAY be flagged as 1154 non-BM receivers based on the BM flag received from the remote 1155 nodes in the routes. 1157 - When an AR-LEAF receives a BM packet on an Attachment Circuit, it 1158 will check if there is any selected AR-REPLICATOR. If there is, 1159 flood-list #1 MUST be used. Otherwise, flood-list #2 MUST be 1160 used. Non-BM overlay tunnels are skipped when sending BM packets. 1162 - When an AR-LEAF receives a BM packet on an overlay tunnel, it MUST 1163 forward the BM packet to its local Attachment Circuits and never 1164 to an overlay tunnel. This is the regular Ingress Replication 1165 behavior described in [RFC7432]. 1167 7. Pruned-Flood-Lists (PFL) 1169 In addition to AR, the second optimization supported by this solution 1170 is the ability for the all the BD nodes to signal Pruned-Flood-Lists 1171 (PFL). As described in Section 4, an EVPN node can signal a given 1172 value for the BM and U Pruned-Food-Lists flags in the Regular-IR, 1173 Replicator-AR or Leaf Auto-Discovery routes, where: 1175 - BM is the Broadcast and Multicast flag. BM=1 means "prune-me" 1176 from the BM flood-list. BM=0 means regular behavior. 1178 - U is the Unknown flag. U=1 means "prune-me" from the Unknown 1179 flood-list. U=0 means regular behavior. 1181 The ability to signal and process these Pruned-Flood-Lists flags 1182 SHOULD be an administrative choice. If a node is configured to 1183 process the Pruned-Flood-Lists flags, upon receiving a non-zero 1184 Pruned-Flood-Lists flag for a route, the NVE/PE will add the 1185 corresponding flag to the created overlay tunnel in the flood-list. 1186 When replicating a BM packet in the context of a flood-list, the NVE/ 1187 PE will skip the overlay tunnels marked with the flag BM=1, since the 1188 NVE/PE at the end of those tunnels are not expecting BM packets. 1189 Similarly, when replicating Unknown unicast packets, the NVE/PE will 1190 skip the overlay tunnels marked with U=1. 1192 An NVE/PE not following this document or not configured for this 1193 optimization will ignore any of the received Pruned-Flood-Lists 1194 flags. An AR-LEAF or RNVE receiving BUM traffic on an overlay tunnel 1195 MUST replicate the traffic to its local Attachment Circuits, 1196 regardless of the BM/U flags on the overlay tunnels. 1198 This optimization MAY be used along with the Assisted-Replication 1199 solution. 1201 7.1. A Pruned-Flood-List Example 1203 In order to illustrate the use of the solution described in this 1204 document, we will assume that BD-1 in Figure 4 is optimized Ingress 1205 Replication enabled and: 1207 - PE1 and PE2 are administratively configured as AR-REPLICATORs, due 1208 to their high-performance replication capabilities. PE1 and PE2 1209 will send a Replicator-AR route with BM/U flags = 00. 1211 - NVE1 and NVE3 are administratively configured as AR-LEAF nodes, 1212 due to their low-performance software-based replication 1213 capabilities. They will advertise a Regular-IR route with type 1214 AR-LEAF. Assuming both NVEs advertise all the attached Virtual 1215 Machines MAC and IP addresses in EVPN as soon as they come up, and 1216 these NVEs do not have any Virtual Machines interested in 1217 multicast applications, they will be configured to signal BM/U 1218 flags = 11 for BD-1. That is, neither NVE1 nor NVE3 are 1219 interested in receiving BM or Unknown Unicast traffic since: 1221 o Their attached VMs (VM11, VM12, VM31, VM32) do not support 1222 multicast applications. 1224 o Their attached VMs will not receive ARP Requests. Proxy-ARP 1225 [I-D.ietf-bess-evpn-proxy-arp-nd] on the remote NVE/PEs will 1226 reply ARP Requests locally, and no other Broadcast is expected. 1228 o Their attached VMs will not receive unknown unicast traffic, 1229 since the VMs' MAC and IP addresses are always advertised by 1230 EVPN as long as the VMs are active. 1232 - NVE2 is optimized Ingress Replication unaware; therefore it takes 1233 on the RNVE role in BD-1. 1235 Based on the above assumptions the following forwarding behavior will 1236 take place: 1238 1. Any BM packets sent from VM11 will be sent to VM12 and PE1. PE1 1239 will forward further the BM packets to TS1, WAN link, PE2 and 1240 NVE2, but not to NVE3. PE2 and NVE2 will replicate the BM 1241 packets to their local Attachment Circuits but we will avoid NVE3 1242 having to replicate unnecessarily those BM packets to VM31 and 1243 VM32. 1245 2. Any BM packets received on PE2 from the WAN will be sent to PE1 1246 and NVE2, but not to NVE1 and NVE3, sparing the two hypervisors 1247 from replicating unnecessarily to their local Virtual Machines. 1248 PE1 and NVE2 will replicate to their local Attachment Circuits 1249 only. 1251 3. Any Unknown unicast packet sent from VM31 will be forwarded by 1252 NVE3 to NVE2, PE1 and PE2 but not NVE1. The solution avoids the 1253 unnecessary replication to NVE1, since the destination of the 1254 unknown traffic cannot be at NVE1. 1256 4. Any Unknown unicast packet sent from TS1 will be forwarded by PE1 1257 to the WAN link, PE2 and NVE2 but not to NVE1 and NVE3, since the 1258 target of the unknown traffic cannot be at those NVEs. 1260 8. AR Procedures for Single-IP AR-REPLICATORS 1262 The procedures explained in sections Section 5 and Section 6 assume 1263 that the AR-REPLICATOR can use two local routable IP addresses to 1264 terminate and originate Network Virtualization Overlay tunnels, i.e. 1265 IR-IP and AR-IP addresses. This is usually the case for PE-based AR- 1266 REPLICATOR nodes. 1268 In some cases, the AR-REPLICATOR node does not support more than one 1269 IP address to terminate and originate Network Virtualization Overlay 1270 tunnels, i.e. the IR-IP and AR-IP are the same IP addresses. This 1271 may be the case in some software-based or low-end AR-REPLICATOR 1272 nodes. If this is the case, the procedures in sections Section 5 and 1273 Section 6 MUST be modified in the following way: 1275 - The Replicator-AR routes generated by the AR-REPLICATOR use an AR- 1276 IP that will match its IR-IP. In order to differentiate the data 1277 plane packets that need to use Ingress Replication from the 1278 packets that must use Assisted Replication forwarding mode, the 1279 Replicator-AR route MUST advertise a different VNI/VSID than the 1280 one used by the Regular-IR route. For instance, the AR-REPLICATOR 1281 will advertise AR-VNI along with the Replicator-AR route and IR- 1282 VNI along with the Regular-IR route. Since both routes have the 1283 same key, different Route Distinguishers are needed in each route. 1285 - An AR-REPLICATOR will perform Ingress Replication or Assisted 1286 Replication forwarding mode for the incoming Overlay packets based 1287 on an ingress VNI lookup, as opposed to the tunnel IP DA lookup. 1288 Note that, when replicating to remote AR-REPLICATOR nodes, the use 1289 of the IR-VNI or AR-VNI advertised by the egress node will 1290 determine the Ingress Replication or Assisted Replication 1291 forwarding mode at the subsequent AR-REPLICATOR. 1293 The rest of the procedures will follow what is described in sections 1294 Section 5 and Section 6. 1296 9. AR Procedures and EVPN All-Active Multi-homing Split-Horizon 1298 This section extends the procedures for the cases where two or more 1299 AR-LEAF nodes are attached to the same Ethernet Segment, and two or 1300 more AR-REPLICATOR nodes are attached to the same Ethernet Segment in 1301 the BD. The mixed case, that is, an AR-LEAF node and an AR- 1302 REPLICATOR node are attached to the same Ethernet Segment, would 1303 require extended procedures and it is out of scope. 1305 9.1. Ethernet Segments on AR-LEAF Nodes 1307 If VXLAN or NVGRE are used, and if the Split-horizon is based on the 1308 tunnel IP Source Address and "Local-Bias" as described in [RFC8365], 1309 the Split-horizon check will not work if there is an Ethernet-Segment 1310 shared between two AR-LEAF nodes, and the AR-REPLICATOR replaces the 1311 tunnel IP Source Address of the packets with its own AR-IP. 1313 In order to be compatible with the IP Source Address split-horizon 1314 check, the AR-REPLICATOR MAY keep the original received tunnel IP 1315 Source Address when replicating packets to a remote AR-LEAF or RNVE. 1316 This will allow AR-LEAF nodes to apply Split-horizon check procedures 1317 for BM packets, before sending them to the local Ethernet-Segment. 1318 Even if the AR-LEAF's IP Source Address is preserved when replicating 1319 to AR-LEAFs or RNVEs, the AR-REPLICATOR MUST always use its IR-IP as 1320 the IP Source Address when replicating to other AR-REPLICATORs. 1322 When EVPN is used for MPLS over GRE (or UDP), the ESI-label based 1323 split-horizon procedure as in [RFC7432] will not work for multi-homed 1324 Ethernet-Segments defined on AR-LEAF nodes. "Local-Bias" is 1325 recommended in this case, as in the case of VXLAN or NVGRE explained 1326 above. The "Local-Bias" and tunnel IP Source Address preservation 1327 mechanisms provide the required split-horizon behavior in non- 1328 selective or selective AR. 1330 Note that if the AR-REPLICATOR implementation keeps the received 1331 tunnel IP Source Address, the use of uRPF (unicast Reverse Path 1332 Forwarding) checks in the IP fabric based on the tunnel IP Source 1333 Address MUST be disabled. 1335 9.2. Ethernet Segments on AR-REPLICATOR nodes 1337 AR-REPLICATOR nodes attached to the same all-active Ethernet Segment 1338 will follow "Local-Bias" procedures [RFC8365], as follows: 1340 a. For BUM traffic received on a local AR-REPLICATOR's Attachment 1341 Circuit, "Local-Bias" procedures as in [RFC8365] MUST be 1342 followed. 1344 b. For BUM traffic received on an AR-REPLICATOR overlay tunnel with 1345 AR-IP as the IP Destination Address, "Local-Bias" MUST also be 1346 followed. That is, traffic received with AR-IP as IP Destination 1347 Address will be treated as though it had been received on a local 1348 Attachment Circuit that is part of the Ethernet Segment and will 1349 be forwarded to all local Ethernet Segments, irrespective of 1350 their DF or NDF state. 1352 c. BUM traffic received on an AR-REPLICATOR overlay tunnel with IR- 1353 IP as the IP Destination Address, will follow regular [RFC8365] 1354 "Local-Bias" rules and will not be forwarded to local Ethernet 1355 Segments that are shared with the AR-LEAF or AR-REPLICATOR 1356 originating the traffic. 1358 d. In cases where the AR-REPLICATOR supports a single IP address, 1359 the IR-IP and the AR-IP are the same IP address, as discussed in 1360 Section 8. The received BUM traffic will be treated as in 'b' 1361 above if the received VNI is the AR-VNI, and as in 'c' if the VNI 1362 is the IR-VNI. 1364 10. Security Considerations 1366 The Security Considerations in [RFC7432] and [RFC8365] apply to this 1367 document. The Security Considerations related to the Leaf Auto- 1368 Discovery route in [I-D.ietf-bess-evpn-bum-procedure-updates] apply 1369 too. 1371 In addition, the Assisted-Replication method introduced by this 1372 document may bring some new risks for the successful delivery of BM 1373 traffic. Unicast traffic is not affected by Assisted-Replication 1374 (although Unknown unicast traffic is affected by the Pruned-Flood- 1375 Lists procedures). The forwarding of Broadcast and Multicast (BM) 1376 traffic is modified; and BM traffic from the AR-LEAF nodes will be 1377 attracted by the existence of AR-REPLICATORs in the BD. An AR-LEAF 1378 will forward BM traffic to its selected AR-REPLICATOR, therefore an 1379 attack on the AR-REPLICATOR could impact the delivery of the BM 1380 traffic using that node. Also, an attack on the AR-REPLICATOR and 1381 change of the advertised AR type will modify the selection on the AR- 1382 LEAF nodes. If no other AR-REPLICATOR is selected, the AR-LEAF nodes 1383 will be forced to use Ingress Replication forwarding mode, which will 1384 impact on their performance, since the AR-LEAF nodes are usually 1385 NVEs/PEs with poor replication performance. 1387 This document introduces the ability for the AR-REPLICATOR to forward 1388 traffic received on an overlay tunnel to another overlay tunnel. The 1389 reader may interpret that this introduces the risk of BM loops. That 1390 is, an AR-LEAF receiving a BM encapsulated packet that the AR-LEAF 1391 originated in the first place, due to one or two AR-REPLICATORs 1392 "looping" the BM traffic back to the AR-LEAF. The procedures in this 1393 document prevent these BM loops, since the AR-REPLICATOR will always 1394 forward the BM traffic using the correct tunnel IP Destination 1395 Address (or correct VNI in case of single-IP AR-REPLICATORs) that 1396 instructs the remote nodes how to forward the traffic. This is true 1397 in both the Non-Selective and Selective modes defined in this 1398 document. However, a wrong implementation of the procedures in this 1399 document may lead to those unexpected BM loops. 1401 The Selective mode provides a multi-staged replication solution, 1402 where a proper configuration of all the AR-REPLICATORs will avoid any 1403 issues. A mix of mistakenly configured Selective and Non-Selective 1404 AR-REPLICATORs in the same BD could theoretically create packet 1405 duplication in some AR-LEAFs, however this document specifies a fall 1406 back solution to Non-Selective mode in case the AR-REPLICATORs 1407 advertised an inconsistent AR Replication mode. 1409 This document allows the AR-REPLICATOR to preserve the tunnel IP 1410 Source Address of the AR-LEAF (as an option) when forwarding BM 1411 packets from an overlay tunnel to another overlay tunnel. Preserving 1412 the AR-LEAF IP Source Address makes the "Local Bias" filtering 1413 procedures possible for AR-LEAF nodes that are attached to the same 1414 Ethernet Segment. If the AR-REPLICATOR does not preserve the AR-LEAF 1415 IP Source Address, AR-LEAF nodes attached to all-active Ethernet 1416 Segments will cause packet duplication on the multi-homed CE. 1418 The AR-REPLICATOR nodes are, by design, using more bandwidth than 1419 [RFC7432] PEs or [RFC8365] NVEs would use. Certain network events or 1420 unexpected low performance may exceed the AR-REPLICATOR local 1421 bandwidth and cause service disruption. 1423 Finally, the use of PFL as in Section 7, should be handled with care. 1424 An intentional or unintentional misconfiguration of the BDs on a 1425 given leaf node may result in the leaf not receiving the required BM 1426 or Unknown unicast traffic. 1428 11. IANA Considerations 1430 IANA has allocated the following Border Gateway Protocol (BGP) 1431 Parameters: 1433 - Allocation in the P-Multicast Service Interface Tunnel (PMSI 1434 Tunnel) Tunnel Types registry: 1436 Value Meaning Reference 1437 0x0A Assisted-Replication Tunnel [This document] 1439 - Allocations in the P-Multicast Service Interface (PMSI) Tunnel 1440 Attribute Flags registry: 1442 Value Name Reference 1443 3-4 Assisted-Replication Type (T) [This document] 1444 5 Broadcast and Multicast (BM) [This document] 1445 6 Unknown (U) [This document] 1447 12. Contributors 1449 In addition to the names in the front page, the following co-authors 1450 also contributed to this document: 1452 Wim Henderickx 1453 Nokia 1455 Kiran Nagaraj 1456 Nokia 1458 Ravi Shekhar 1459 Juniper Networks 1461 Nischal Sheth 1462 Juniper Networks 1464 Aldrin Isaac 1465 Juniper 1467 Mudassir Tufail 1468 Citibank 1470 13. Acknowledgments 1472 The authors would like to thank Neil Hart, David Motz, Dai Truong, 1473 Thomas Morin, Jeffrey Zhang, Shankar Murthy and Krzysztof Szarkowicz 1474 for their valuable feedback and contributions. 1476 14. References 1478 14.1. Normative References 1480 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1481 Requirement Levels", BCP 14, RFC 2119, 1482 DOI 10.17487/RFC2119, March 1997, 1483 . 1485 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1486 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1487 May 2017, . 1489 [RFC6514] Aggarwal, R., Rosen, E., Morin, T., and Y. Rekhter, "BGP 1490 Encodings and Procedures for Multicast in MPLS/BGP IP 1491 VPNs", RFC 6514, DOI 10.17487/RFC6514, February 2012, 1492 . 1494 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 1495 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 1496 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 1497 2015, . 1499 [I-D.ietf-bess-evpn-bum-procedure-updates] 1500 Zhang, Z., Lin, W., Rabadan, J., Patel, K., and A. 1501 Sajassi, "Updates on EVPN BUM Procedures", draft-ietf- 1502 bess-evpn-bum-procedure-updates-13 (work in progress), 1503 November 2021. 1505 [RFC7902] Rosen, E. and T. Morin, "Registry and Extensions for 1506 P-Multicast Service Interface Tunnel Attribute Flags", 1507 RFC 7902, DOI 10.17487/RFC7902, June 2016, 1508 . 1510 [RFC6513] Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/ 1511 BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February 1512 2012, . 1514 [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R., 1515 Uttaro, J., and W. Henderickx, "A Network Virtualization 1516 Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, 1517 DOI 10.17487/RFC8365, March 2018, 1518 . 1520 14.2. Informative References 1522 [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, 1523 L., Sridhar, T., Bursell, M., and C. Wright, "Virtual 1524 eXtensible Local Area Network (VXLAN): A Framework for 1525 Overlaying Virtualized Layer 2 Networks over Layer 3 1526 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, 1527 . 1529 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., 1530 "Encapsulating MPLS in IP or Generic Routing Encapsulation 1531 (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, 1532 . 1534 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 1535 Virtualization Using Generic Routing Encapsulation", 1536 RFC 7637, DOI 10.17487/RFC7637, September 2015, 1537 . 1539 [I-D.ietf-bess-evpn-proxy-arp-nd] 1540 Rabadan, J., Sathappan, S., Nagaraj, K., Hankins, G., and 1541 T. King, "Operational Aspects of Proxy-ARP/ND in Ethernet 1542 Virtual Private Networks", draft-ietf-bess-evpn-proxy-arp- 1543 nd-16 (work in progress), October 2021. 1545 Authors' Addresses 1547 J. Rabadan (editor) 1548 Nokia 1549 777 Middlefield Road 1550 Mountain View, CA 94043 1551 USA 1553 Email: jorge.rabadan@nokia.com 1555 S. Sathappan 1556 Nokia 1558 Email: senthil.sathappan@nokia.com 1560 W. Lin 1561 Juniper Networks 1563 Email: wlin@juniper.net 1565 M. Katiyar 1566 Versa Networks 1568 Email: mukul@versa-networks.com 1570 A. Sajassi 1571 Cisco Systems 1573 Email: sajassi@cisco.com