idnits 2.17.00 (12 Aug 2021) /tmp/idnits24553/draft-dunbar-idr-5g-edge-compute-app-meta-data-06.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** There is 1 instance of too long lines in the document, the longest one being 17 characters in excess of 72. ** The abstract seems to contain references ([RFC9012]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (February 23, 2022) is 80 days in the past. Is this intentional? Checking references for intended status: Full Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Missing Reference: 'RFC9012' is mentioned on line 614, but not defined == Missing Reference: '5GC' is mentioned on line 135, but not defined == Missing Reference: 'EC' is mentioned on line 167, but not defined == Missing Reference: 'ToR' is mentioned on line 170, but not defined == Missing Reference: 'A-ER' is mentioned on line 444, but not defined == Missing Reference: '5G-Sticky-Service' is mentioned on line 570, but not defined == Missing Reference: 'Section 5' is mentioned on line 692, but not defined == Unused Reference: 'RFC4364' is defined on line 850, but no explicit reference was found in the text == Unused Reference: 'RFC8200' is defined on line 858, but no explicit reference was found in the text == Unused Reference: '3GPP-EdgeComputing' is defined on line 863, but no explicit reference was found in the text == Unused Reference: 'RFC5521' is defined on line 881, but no explicit reference was found in the text == Unused Reference: 'BGP-SDWAN-Port' is defined on line 885, but no explicit reference was found in the text == Unused Reference: 'SDWAN-EDGE-Discovery' is defined on line 889, but no explicit reference was found in the text == Unused Reference: 'Tunnel-Encap' is defined on line 894, but no explicit reference was found in the text ** Downref: Normative reference to an Proposed Standard RFC: RFC 4364 == Outdated reference: A later version (-02) exists of draft-dunbar-ippm-5g-edge-compute-ip-layer-metrics-00 -- No information found for draft-dunbar-6man-5g-ec-sticky-service - is the name correct? == Outdated reference: A later version (-04) exists of draft-dunbar-idr-sdwan-edge-discovery-00 == Outdated reference: draft-ietf-idr-tunnel-encaps has been published as RFC 9012 Summary: 3 errors (**), 0 flaws (~~), 18 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network Working Group L. Dunbar 2 Internet Draft Futurewei 3 Intended status: Standard K. Majumdar 4 Expires: August 23, 2022 CommScope 5 H. Wang 6 Huawei 7 G. Mishra 8 Verizon 9 February 23, 2022 11 BGP Update for 5G Edge Computing Service Metadata 12 draft-dunbar-idr-5g-edge-compute-app-meta-data-06 14 Abstract 15 This draft describes a new AppMetaData subTLV carried by 16 Tunnel Encap[RFC9012] Path Attribute for egress router to 17 advertise the running status and environment for the directly 18 attached 5G Edge Computing (EC) servers. The AppMetaData can 19 be used by the ingress routers in the 5G Local Data Network to 20 make path selection not only based on the routing distance but 21 also the running environment of the destinations. The goal is 22 to improve latency and performance for 5G EC services. 24 The extension enables an EC server at one specific location to 25 be more preferred than the others with the same IP address to 26 receive data flows from a specific source (UE). 28 Status of this Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 This Internet-Draft is submitted in full conformance with the 34 provisions of BCP 78 and BCP 79. This document may not be 35 modified, and derivative works of it may not be created, 36 except to publish it as an RFC and to translate it into 37 languages other than English. 39 Internet-Drafts are working documents of the Internet 40 Engineering Task Force (IETF), its areas, and its working 41 groups. Note that other groups may also distribute working 42 documents as Internet-Drafts. 44 Internet-Draft AppMetaData for 5G EC Service 46 Internet-Drafts are draft documents valid for a maximum of six 47 months and may be updated, replaced, or obsoleted by other 48 documents at any time. It is inappropriate to use Internet- 49 Drafts as reference material or to cite them other than as 50 "work in progress." 52 The list of current Internet-Drafts can be accessed at 53 http://www.ietf.org/ietf/1id-abstracts.txt 55 The list of Internet-Draft Shadow Directories can be accessed 56 at http://www.ietf.org/shadow.html 58 This Internet-Draft will expire on April 7, 2021. 60 Copyright Notice 62 Copyright (c) 2022 IETF Trust and the persons identified as 63 the document authors. All rights reserved. 65 This document is subject to BCP 78 and the IETF Trust's Legal 66 Provisions Relating to IETF Documents 67 (http://trustee.ietf.org/license-info) in effect on the date 68 of publication of this document. Please review these documents 69 carefully, as they describe your rights and restrictions with 70 respect to this document. Code Components extracted from this 71 document must include Simplified BSD License text as described 72 in Section 4.e of the Trust Legal Provisions and are provided 73 without warranty as described in the Simplified BSD License. 75 Table of Contents 77 1. Introduction.............................................. 3 78 1.1. 5G Edge Computing Background......................... 3 79 1.2. 5G Edge Computing Network Properties................. 4 80 1.3. Problem#1: ANYCAST in 5G EC Environment.............. 5 81 1.4. Problem #2: Unbalanced Anycast Distribution due to UE 82 Mobility.................................................. 7 83 1.5. Problem 3: Application Server Relocation............. 8 84 2. Conventions used in this document......................... 8 85 3. Usage of AppMetaData for 5G Edge Computing................ 9 86 3.1. Assumptions.......................................... 9 87 3.2. IP Layer Metrics to Gauge Application Behavior...... 10 88 3.3. AppMetaData Constrained Optimal Path Selection...... 11 90 Internet-Draft AppMetaData for 5G EC Service 92 4. BGP Protocol Extension to advertise Load & Capacity...... 12 93 4.1. Ingress Node BGP Path Selection Behavior............ 12 94 4.1.1. AppMetaData Influenced BGP Path Selection...... 12 95 4.1.2. Ingress Router Forwarding Behavior............. 12 96 4.1.3. Forwarding Behavior when UEs moving to new 5G 97 Sites................................................. 14 98 5. The Sub-TLVs for AppMetaData............................. 14 99 5.1. Load Measurement sub-TLV format..................... 15 100 5.2. Capacity Index sub-TLV format....................... 16 101 5.3. The Site Preference Index sub-TLV format............ 16 102 6. AppMetaData Propagation Scope............................ 17 103 7. Minimum Interval for Metrics Change Advertisement........ 17 104 8. Soft Anchoring of an ANYCAST Flow........................ 17 105 9. Manageability Considerations............................. 19 106 10. Security Considerations................................. 19 107 11. IANA Considerations..................................... 19 108 12. References.............................................. 20 109 12.1. Normative References............................... 20 110 12.2. Informative References............................. 20 111 13. Acknowledgments......................................... 21 113 1. Introduction 115 This document describes a new subTLV, AppMetaData, for egress 116 routers to advertise the running status and environment for 117 the directly attached Edge Computing (EC) servers. The 118 AppMetaData can be used by the ingress routers in the 5G Local 119 Data Network to make path selection not only based on the 120 routing distance but also the running environment of the 121 destinations. The goal is to improve latency and performance 122 for 5G Edge Computing services. 124 1.1. 5G Edge Computing Background 126 In 5G Edge Computing (EC), one Application can be hosted on 127 multiple Servers in different EC data centers that are close 128 in proximity. The 5G Local Data Networks (LDN)that connect the 129 EC data centers with the 5G Base stations consist of a small 130 number of dedicated routers. 132 When a User Equipment (UE) initiates application packets using 133 the destination address from a DNS reply or its cache, the 134 packets from the UE are carried in a PDU session through 5G 135 Core [5GC] to the 5G UPF-PSA (User Plan Function - PDU Session 136 Anchor). The UPF-PSA decapsulates the 5G GTP outer header and 138 Internet-Draft AppMetaData for 5G EC Service 140 forwards the packets from the UEs to its directly connected 141 Ingress router of the 5G LDN. The LDN for 5G EC is responsible 142 for forwarding the packets to the intended destinations. 144 When the UE moves out of coverage of its current gNB (next- 145 generation Node B) and anchors to a new gNB, the 5G SMF 146 (Session Management Function) could select the same UPF or a 147 new UPF for the UE per standard handover procedures described 148 in 3GPP TS 23.501 and TS 23.502. If the UE is anchored to a 149 new UPF-PSA when the handover process is complete, the packets 150 to/from the UE is carried by a GTP tunnel to the new UPF-PSA. 151 Per TS 23.501-h20 Section 5.8.2, the UE may maintain its IP 152 address when anchored to a new UPF-PSA unless the new UFP-PSA 153 belongs to different mobile operators. 5GC may maintain a path 154 from the old UPF to the new UPF for a short time for the SSC 155 [Session and Service Continuity] mode 3 to make the handover 156 process more seamless. 158 1.2. 5G Edge Computing Network Properties 160 In this document, 5G Edge Computing Network refers to multiple 161 Local IP Data Networks (LDN) in one region that interconnect 162 the Edge Computing data centers. Those IP LDN networks are the 163 N6 interfaces from 3GPP 5G perspective. 165 The ingress routers to the 5G Edge Computing Network are the 166 routers directly connected to 5G UPFs. The egress routers to 167 the 5G Edge Computing [EC] Network are the routers that have a 168 direct link to the EC servers. The EC servers and the egress 169 routers are co-located. Some of those Edge Computing Data 170 centers may have virtual switches or Top of Rack [ToR] 171 switches between the egress routers and the servers. But 172 transmission delay between the egress routers and the EC 173 servers is negligible, which is too small to be considered in 174 this document. 176 When multiple EC servers are attached to one App Layer Load 177 Balancer, only the IP addresses of the App Layer Load Balancer 178 are visible to the 5G LDNs. How an App Layer Load balancer 179 manages the individual servers is out of the scope of the 180 network layer. 182 Internet-Draft AppMetaData for 5G EC Service 184 The 5G EC Services are registered premium services that 185 require super-low latency and very high SLA. Most services by 186 the UEs are not part of the registered 5G EC Services. 188 +--+ 189 |UE|---\+---------+ +------------------+ 190 +--+ | 5G | +--------+ | S1: aa08::4450 | 191 +--+ | Site +--+-+---+ +----+ | 192 |UE|----| A |PSA1| Ra| | R1 | S2: aa08::4460 | 193 +--+ | +----+---+ +----+ | 194 +---+ | | | | | S3: aa08::4470 | 195 |UE1|---/+---------+ | | +------------------+ 196 +---+ |IP Network | L-DN1 197 |(3GPP N6) | 198 | | | +------------------+ 199 | UE1 | | | S1: aa08::4450 | 200 | moves to | +----+ | 201 | Site B | | R3 | S2: aa08::4460 | 202 v | +----+ | 203 | | | S3: aa08::4470 | 204 | | +------------------+ 205 | | L-DN3 206 +--+ | | 207 |UE|---\+---------+ | | +------------------+ 208 +--+ | 5G | | | | S1: aa08::4450 | 209 +--+ | Site +--+--+---+ +----+ | 210 |UE|----| B |PSA2| Rb | | R2 | S2: aa08::4460 | 211 +--+ | +--+-+----+ +----+ | 212 +--+ | | +-----------+ | S3: aa08::4470 | 213 |UE|---/+---------+ +------------------+ 214 +--+ L-DN2 215 Figure 1: App Servers in different edge DCs 217 1.3. Problem#1: ANYCAST in 5G EC Environment 219 Increasingly, Anycast is used by various application providers 220 and CDNs because Anycast provides better and faster resiliency 221 to failover events than GEO database DNS-based load balancing, 222 which relies on DNS to provide a different IP based on source 223 address. 225 Internet-Draft AppMetaData for 5G EC Service 227 Anycast address leverages the proximity information present in 228 the network (routing) layer. It eliminates the single point of 229 failure and bottleneck at the DNS resolvers. Anycast address 230 can be assigned to multiple app layer load balancers to 231 leverage network condition for balanced forwarding. Another 232 benefit of using the ANYCAST address is removing the 233 dependency on UEs. Some UEs (or clients) might use their 234 cached IP addresses for an extended period instead of querying 235 DNS. 237 Client using Virtual IP address is a common practice in Cloud 238 Native networking, e.g., Kubernetes, to scale dynamic changes 239 of app servers' instantiations. Virtual IP requires the 240 destination gateway node to perform address translation for 241 return traffic, which is unsuitable for underlay network nodes 242 with millions of flows passing by. The Cloud Native network 243 can also leverage network condition to balance forwarding 244 among multiple Cloud Gateway nodes by assigning the same 245 virtual IP address (ANYCAST). 247 Having multiple locations of the same IP address in the 5G EC 248 LDN can be problematic if path selection is solely based on 249 routing cost as the routing cost differences to reach 250 different egress routers can be very small. This list 251 elaborates the issues in detail: 253 a) Path Selection: When a new flow comes to an ingress node 254 (Ra), how to avoid instability with Anycast flipping 255 between paths to the same address. The problem also 256 exists in the BGP multipath environment, with the optimal 257 path selected based on routing cost metrics. 259 b) Ingress node forwards the packets from one flow to the 260 same ANYCAST server. 262 a.k.a. Flow Affinity, or Flow-based load balancing. 263 Almost all vendors have supported flow or session based 264 ECMP load balancing and not per packet to avoid out of 265 order packets for decades. When a flow is hashed to an 266 ECMP path, the flow remains on that path for the life of 267 the flow until the flow ends. 269 The ingress node, (Ra/Rb), can use Flow ID (in IPv6 270 header) or UDP/TCP port number combined with the source 271 address to enforce packets in one flow being placed in 273 Internet-Draft AppMetaData for 5G EC Service 275 one tunnel to one Egress router. No new features are 276 needed. 278 c) When a UE moves to a new 5G site in the middle of a 279 communication session with an EC server, a method is 280 needed to stick the flow to the same EC server, which is 281 required by 5G Edge Computing: 3GPP TR 23.748. [5g-edge- 282 compute-sticky-service] describes several approaches to 283 achieve stickiness in the IPv6 domain. 285 Note: most EC services have shorter sessions, e.g., 286 shorter TCP sessions. Most likely, when a UE is moving to 287 a new 5G site, the TCP session via the old UPF to an EC 288 server is already finished. Only a very small percentage 289 of registered EC services need to stick to the original 290 server when handover to a new cell tower. 292 From BGP perspective, the multiple servers with the same IP 293 address (ANYCAST)attached to different egress routers is the 294 same as multiple next hops for the IP address. 296 This draft describes the BGP UPDATE to enable ingress routers 297 to take the App Server load, the capacity index, and the 298 location preference into consideration when computing the 299 optimal path to the egress routers. 301 1.4. Problem #2: Unbalanced Anycast Distribution due to UE 302 Mobility 304 Usually, higher capacity EC servers are placed in a metro data 305 center to accommodate more UEs in the proximity needing the 306 services, and fewer are placed in remote sites. When there is 307 a special event occurring at a remote site for a short period, 308 e.g., 1~2 days, the EC servers in the remote site might be 309 heavily utilized. In contrast, the EC servers of the same app 310 in the metro DC can be very underutilized. Since the condition 311 can be short-lived, it might not make business sense to adjust 312 EC capacity among DCs. Sometimes, UEs swarming to a specific 313 site are not anticipated. 315 Internet-Draft AppMetaData for 5G EC Service 317 1.5. Problem 3: Application Server Relocation 319 When an EC server is added to, moved, or deleted from a 5G EC 320 Data Center, the routing protocol needs to propagate the 321 changes to 5G PSA or the PSA adjacent routers. After the 322 change, the cost associated with the site might change as 323 well. 325 Note: for ease of description, the Edge Application Server and 326 Application Server are used interchangeably throughout this 327 document. 329 2. Conventions used in this document 331 A-ER: Egress Router to an Application Server, [A-ER] is 332 used to describe the last router that the 333 Application Server is attached. For a 5G EC 334 environment, the A-ER can be the gateway router to 335 a (mini) Edge Computing Data Center. 337 Application Server: An application server is a physical or 338 virtual server that hosts the software system for 339 the application. 341 Application Server Location: Represent a cluster of servers at 342 one location serving the same Application. One 343 application may have a Layer 7 Load balancer, 344 whose address(es) are reachable from an external 345 IP network, in front of a set of application 346 servers. From an IP network perspective, this 347 whole group of servers is considered as the 348 Application server at the location. 350 Edge Application Server: used interchangeably with Application 351 Server throughout this document. 353 EC: Edge Computing 355 Edge Hosting Environment: An environment providing the support 356 required for Edge Application Server's execution. 358 Internet-Draft AppMetaData for 5G EC Service 360 NOTE: The above terminologies are the same as 361 those used in 3GPP TR 23.758 363 Edge DC: Edge Data Center, which provides the Edge 364 Computing Hosting Environment. An Edge DC might 365 host 5G core functions in addition to the 366 frequently used application servers. 368 gNB next generation Node B 370 L-DN: Local Data Network 372 PSA: PDU Session Anchor (UPF) 374 SSC: Session and Service Continuity 376 UE: User Equipment 378 UPF: User Plane Function 380 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL 381 NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT 382 RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be 383 interpreted as described in BCP 14 [RFC2119] [RFC8174] when, 384 and only when, they appear in all capitals, as shown here. 386 3. Usage of AppMetaData for 5G Edge Computing 388 AppMetaData consists of metrics about the running environment 389 at the egress routers to which EC servers are directly 390 attached. 392 3.1. Assumptions 394 From the IP Layer, the EC servers or their respective load 395 balancers are identified by their IP addresses. Those IP 396 addresses are the identifiers to the EC servers throughout 397 this document. Here are some assumptions about the 5G EC 398 services: 399 - Only the registered EC services, which are only a small 400 portion of the services, need to incorporate the 401 destination capacity metrics for optimal forwarding. 403 Internet-Draft AppMetaData for 5G EC Service 405 - The 5G EC controller or management system can send those 406 EC service identifiers to relevant routers. 407 - The ingress routers' local BGP path compute algorithm 408 includes a special plugin that can compute the path to 409 the optimal Next Hop (egress router) based on the BGP 410 AppMetaData TLV received for the registered EC services. 412 The proposed solution is for the egress routers, a.k.a. A-ERs 413 in this document, that have direct links to the EC Servers to 414 collect various measurements about the Servers' running status 415 [5G-EC-Metrics] and advertise the metrics to other routers in 416 5G EC LDN (Local Data Network). 418 3.2. IP Layer Metrics to Gauge Application Behavior 420 [5G-EC-Metrics] describes the IP Layer Metrics that can gauge 421 the application servers running status and environment: 423 - IP-Layer Metric for App Server Load Measurement: 424 The Load Measurement to an App Server is a weighted 425 combination of the number of packets/bytes to the App Server 426 and the number of packets/bytes from the App Server which 427 are collected by the A-ER to which the App Server is 428 directly attached. 429 The A-ER is configured with an ACL that can filter out the 430 packets for the Application Server. 431 - Capacity Index 432 a numeric number, configured on all A-ERs in the domain 433 consistently, is used to represent the capacity of the 434 application server attached to an A-ER. At some sites, the 435 IP address exposed to the A-ER is the App Layer Load 436 balancer that have many instances attached. At other sites, 437 the IP address exposed is the server instance itself. 438 - Site preference index: 439 is used to describe some sites are more preferred than 440 others. For example, a site with higher bandwidth has a 441 higher preference number than other. 443 In this document, the term "Application Server Egress Router" 444 [A-ER] is used to describe the last router that an Application 445 Server is attached. For the 5G EC environment, the A-ER can be 447 Internet-Draft AppMetaData for 5G EC Service 449 the gateway router to the EC DC where multiple Application 450 servers are hosted. 452 3.3. AppMetaData Constrained Optimal Path Selection 454 The main benefit of using ANYCAST is to leverage the network 455 layer conditions to select an optimal path to the application 456 instantiated in multiple locations. 458 When the ingress routers to the 5G LDN are informed of the 459 Load and Capacity Index of the App Servers at different EC 460 data centers, they can incorporate those metrics with the 461 network path conditions for path selection. 463 Here is an algorithm that computes the cost to reach the App 464 Servers attached to Site-i relative to another site, say Site- 465 b. When the reference site, Site-b, is plugged in the formula, 466 the cost is 1. So, if the formula returns a value less than 1, 467 the cost to reach Site-i is less than reaching Site-b. 469 CP-b * Load-i Pref-b * Network-Delay-i 470 Cost-i= (w *(----------------) + (1-w) *(-------------------------)) 471 CP-i * Load-b Pref-i * Network-Delay-b 473 Load-i: Load Index at Site-i, it is the weighted 474 combination of the total packets or/and bytes sent to and 475 received from the Application Server at Site-i during a 476 fixed time period. 478 CP-i: capacity index at Site-i, a higher value means higher 479 capacity. 481 Delay-i: Network latency measurement (RTT) to the A-ER that 482 has the Application Server attached at the site-i. 484 Pref-i: Preference index for the Site-i, a higher value 485 means higher preference. 487 w: Weight for load and site information, which is a value 488 between 0 and 1. If smaller than 0.5, Network latency and 489 the site Preference have more influence; otherwise, Server 490 load and its capacity have more influence. 492 Internet-Draft AppMetaData for 5G EC Service 494 4. BGP Protocol Extension to advertise Load & Capacity 496 The goal of the BGP extension is for egress routers to 497 propagate the metrics about their running environment to 498 ingress routers. Here are some examples of the metrics 499 propagated by the egress routers: 500 - the Load Measurement Index for the attached EC Servers, 502 - the Capacity Index, and 504 - Site Preference Index. 506 This section specifies the Load Index Sub-TLV, Capacity Sub- 507 TLV, and the Site Preference Sub-TLV that can be carried by 508 the Tunnel Encap Path Attribute associated with the routes. 510 4.1. Ingress Node BGP Path Selection Behavior 512 4.1.1. AppMetaData Influenced BGP Path Selection 514 When an ingress router receives BGP updates for the same IP 515 address from multiple egress routers, all those egress routers 516 are considered the next hops for the IP address. For the 517 selected EC services, the ingress router's BGP engine would 518 call a Plugin function that can select paths based on the 519 AppMetaData received. The Plugin function is called Load 520 Compute Engine throughout this document. 522 Assume that both Ra and Rb in Figure-1 have BGP Multipath 523 enabled. As a result, Dst Address: S1:aa08::4450 is resolved 524 via multiple NextHop: R1, R2, R3. 526 Suppose the local BGP's Load Compute Engine identifies R1 as 527 the optimal NextHop for the flow towards S1:aa08::4450. Then 528 the Load Compute Engine can insert a higher weight for the 529 path R1 so that BGP Best Path is locally influenced by the 530 weight parameter based on the local decision. 532 4.1.2. Ingress Router Forwarding Behavior 534 When the ingress router receives a packet and lookup the FIB, 535 it gets the destination prefix's whole path. It encapsulates 536 the packet destined towards the optimal egress node. 538 For subsequent packets belonging to the same flow, the ingress 539 router needs to forward them to the same egress router unless 541 Internet-Draft AppMetaData for 5G EC Service 543 the selected egress router is no longer reachable. Keeping 544 packets from one flow to the same egress router, a.k.a. Flow 545 Affinity, is supported by many commercial routers. Most 546 registered EC services have relatively short flows. 548 How Flow Affinity is implemented is out of the scope for this 549 document. Here is one example to illustrate how Flow Affinity 550 can be achieved. This illustration is not to be standardized. 552 For the registered EC services, the ingress node keeps a 553 table of 554 - Service ID (i.e., IP address) 555 - Flow-ID 556 - Sticky Egress ID (egress router loopback address) 557 - A timer 559 The Flow-ID in this table is to identify a flow, initialized 560 to NULL. How Flow-ID is constructed is out of the scope for 561 this document. Here is one example of constructing the Flow- 562 ID: 563 - For IPv6, the Flow-ID can be the Flow-ID extracted from 564 the IPv6 packet header with or without the source 565 address. 566 - For IPv4, the Flow-ID can be the combination of the 567 Source Address with or without the TCP/UDP Port number. 569 The Sticky Egress ID is the egress node address for the same 570 flow. [5G-Sticky-Service] describes several methods to 571 derive the Sticky Egress ID. 573 The Timer is always refreshed when a packet with the 574 matching EC Service ID (IP address) is received by the node. 576 If there is no Stick Egress ID present in the table for the 577 EC Service ID, the forwarding plane can select a NextHop 578 influenced by the Load Compute Engine. The forwarding plane 579 encapsulates the packet with a tunnel to the chosen 580 NextHop. The chosen NextHop and the Flow ID are recorded in 581 the EC Service table entry. 583 When the selected optimal NextHop (egress router) is no longer 584 reachable, refer to Section 6 Soft Anchoring on how another 585 path is selected. 587 Internet-Draft AppMetaData for 5G EC Service 589 4.1.3. Forwarding Behavior when UEs moving to new 5G Sites 591 When a UE moves to a new 5G eNB which is anchored to the same 592 UPF, the packets from the UE traverse to the same ingress 593 router. Path selection and forwarding behavior are same as 594 before. 596 When the new eNB is anchored to a different UPF, the packets 597 from the UE traverse a different ingress router. If the UE 598 source IP address has been changed, indicating the new UPF 599 might belong to a different administrative domain, the new 600 ingress router treats the packets from the UE as a new flow 601 and select the optimal path based on the configured policies. 602 If the UE maintains the same IP address when anchored to a new 603 UPF, the directly connected ingress router might use the pre- 604 computed Egress Router, which is passed from a neighboring 605 router. [5G-Edge-Sticky] describes methods for the ingress 606 router connected to the UPF in the new site to consider the 607 information passed from other ingress routers in selecting the 608 optimal paths. The detailed algorithm is out of the scope of 609 this document. 611 5. The Sub-TLVs for AppMetaData 613 The AppMetaData attribute is encoded in an optional subTLV 614 within the Tunnel Encap [RFC9012] Path Attribute. 616 All values in the Sub-TLVs are unsigned 32 bits integers. 618 Internet-Draft AppMetaData for 5G EC Service 620 5.1. Load Measurement sub-TLV format 622 Two types of Load Measurement Sub-TLVs are specified. One is 623 to carry the aggregated cost Index based on a weighted 624 combination of the collected measurements; another one is to 625 carry the raw measurements of packets/bytes to/from the App 626 Server address. The raw measurement is useful when ingress 627 routers have embedded analytics relying on the raw 628 measurements. 630 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 631 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 632 | Type (TBD1) | Length | 633 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 634 | Measurement Period | 635 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 636 | Aggregated Load Index to reach the App Server | 637 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 638 Figure 2: Aggregated Load Index Sub-TLV 640 Raw Load Measurement sub-TLV has the following format: 642 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 643 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 644 | Type (TBD2) | Length | 645 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 646 | Measurement Period | 647 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 648 | total number of packets to the AppServer | 649 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 650 | total number of packets from the AppServer | 651 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 652 | total number of bytes to the AppServer | 653 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 654 | total number of bytes from the AppServer | 655 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 656 Figure 3: Raw Load Measurement Sub-TLV 658 Type =TBD1: Aggregated Load Measurement Index derived from 659 the Weighted combination of bytes/packets sent to/received 660 from the App server: 662 Index=w1*ToPackets+w2*FromPackes+w3*ToBytes+w4*FromBytes 664 Where wi is a value between 0 and 1; w1+ w2+ w3+ w4 = 1. 666 Internet-Draft AppMetaData for 5G EC Service 668 Type= TBD2: Raw measurements of packets/bytes to/from the 669 App Server address. 671 Measure Period: BGP Update period or user-specified period. 673 5.2. Capacity Index sub-TLV format 675 The Capacity Index sub-TLV has the following format: 677 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 678 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 679 | Type (TBD3) | Length | 680 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 681 | Capacity Index | 682 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 683 Figure 4: Capacity Index Sub-TLV 685 Note: "Capacity Index" can be more stable for each site. If 686 those values are configured to nodes, they might not need to 687 be included in every BGP UPDATE. 689 5.3. The Site Preference Index sub-TLV format 691 The site Preference Index is used to achieve Soft Anchoring 692 [Section 5] an application flow from a UE to a specific 693 location when the UE moves from one 5G site to another. 695 The Preference Index sub-TLV has the following format: 697 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 698 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 699 | Type (TBD4) | Length | 700 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 701 | Preference Index | 702 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 703 Figure 5: Preference Index Sub-TLV 705 Note: "Site Preference Index" can be more stable for each 706 site. If those values are configured to nodes, they might not 707 need to be included in every BGP UPDATE. 709 Internet-Draft AppMetaData for 5G EC Service 711 6. AppMetaData Propagation Scope 713 AppMetaData is only to be distributed to the relevant ingress 714 nodes of the 5G EC local data networks. Only the ingress 715 routers that are configured with the 5G EC services need to 716 receive the AppMetaData for specific Service IDs. 718 For each registered EC service, a corresponding filter group 719 can be formed on RR to represent the interested ingress 720 routers that are interested in receiving the corresponding 721 AppMetaData information. 723 7. Minimum Interval for Metrics Change Advertisement 725 As the metrics change can impact the path selection, the 726 Minimum Interval for Metrics Change Advertisement is 727 configured to control the update frequency to avoid route 728 oscillations. Default is 30s. 730 Significant load changes at EC data centers can be triggered 731 by short-term gatherings of UEs, like conventions, lasting a 732 few hours or days, which are too short to justify adjusting EC 733 server capacities among DCs. Therefore, the load metrics 734 change rate can be in the magnitude of hours or days. 736 8. Soft Anchoring of an ANYCAST Flow 737 "Sticky Service" in the 3GPP Edge Computing specification 738 (3GPP TR 23.748) is about flows from a UE sticking to a 739 specific location when the UE moves from one 5G Site to 740 another. 742 "Soft Anchoring" is a mechanism for ingress routers to apply 743 preference to the path towards the previous server location 744 when the UE is anchored to a new UPF and continue using its 745 cached IP for the EC server. 747 Let's assume one application "App.net" is instantiated on 748 four servers that are attached to four different routers R1, 749 R2, R3, and R4 respectively. It is desired for packets to the 750 "App.net" from UE-1 to stick with one server, say the App 751 Server attached to R1, even when the UE moves from one 5G 752 site to another. However, when there is a failure reaching R1 753 or the Application Server attached to R1, the packets of the 754 flow "App.net" from UE-1 need to be forwarded to the 755 Application Server attached to R2, R3, or R4. 757 Internet-Draft AppMetaData for 5G EC Service 759 We call this kind of sticky service "Soft Anchoring", meaning 760 that anchoring to the site of R1 is preferred, but other 761 sites can be chosen when the preferred site encounters a 762 failure. 764 Here is a mechanism to achieve Soft Anchoring: 766 - Assign a group of ANYCAST addresses to one application. 767 For example, "App.net" is assigned with 4 ANYCAST 768 addresses, L1, L2, L3, and L4. L1/L2/L3/L4 represents 769 the location preferred ANYCAST addresses. 770 - For the App.net Server attached to a router, the router 771 has four Stub links to the same Server, L1, L2, L3, and 772 L4 respectively. The cost to L1, L2, L3, and L4 is 773 assigned differently for different egress routers. For 774 example, 775 o When attached to R1, the L1 has the lowest cost, 776 say 10, when attached to R2, R3, and R4, the L1 can 777 have a higher cost, say 30. 778 o ANYCAST L2 has the lowest cost when attached to R2, 779 higher cost when attached to R1, R3, R4 780 respectively. 781 o ANYCAST L3 has the lowest cost when attached to R3, 782 higher cost when attached to R1, R2, R4 783 respectively, and 784 o ANYCAST L4 has the lowest cost when attached to R4, 785 higher cost when attached to R1, R2, R3 786 respectively 787 - When a UE queries for the "App.net" for the first time, 788 the DNS reply has the location preferred ANYCAST 789 address, say L1, based on where the query is initiated. 790 - When the UE moves from one 5G site-A to Site-B, UE 791 continues sending packets of the "App.net" to ANYCAST 792 address L1. The routers will continue sending packets to 793 R1 because the total cost for the App.net instance for 794 ANYCAST L1 is lowest at R1. If any failure occurs making 795 R1 not reachable, the packets of the "App.net" from UE-1 796 will be sent to R2, R3, or R4 (depending on the total 797 cost to reach L1 attached to R2/R3/R4). 799 Internet-Draft AppMetaData for 5G EC Service 801 If the Application Server supports the HTTP redirect, more 802 optimal forwarding can be achieved. 804 - When a UE queries for the "App.net" for the first time, 805 the global DNS reply has the ANYCAST address G1, which 806 has the same cost regardless of where the Application 807 servers are attached. 808 - When the UE initiates the communication to G1, the 809 packets from the UE will be sent to the Application 810 Server that has the lowest cost, say the Server attached 811 to R1. The Application server is instructed with HTTPs 812 Redirect to reply with a location-specific URL, say 813 App.net-Loc1. The client on the UE will query the DNS 814 for App.net-Loc1 and get the response of ANYCAST L1. The 815 subsequent packets from the UE-1 for App.net are sent to 816 L1. 818 9. Manageability Considerations 820 To be added. 822 10. Security Considerations 824 To be added. 826 11. IANA Considerations 828 Here are new Sub-TLV types requiring IANA registration: 830 Type = TBD1: Aggregated Load Measurement Index derived from 831 the Weighted combination of bytes/packets sent to/received 832 from the App server. 834 Type = TBD2: Raw measurements of packets/bytes to/from the 835 App Server address. 837 Type = TBD3: Capacity value sub-TLV 839 Type = TBD4: Site preference value sub-TLV 841 Internet-Draft AppMetaData for 5G EC Service 843 12. References 845 12.1. Normative References 847 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 848 Requirement Levels", BCP 14, RFC 2119, March 1997. 850 [RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private 851 networks (VPNs)", Feb 2006. 853 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in 854 RFC 2119 Key Words", BCP 14, RFC 8174, DOI 855 10.17487/RFC8174, May 2017, . 858 [RFC8200] s. Deering R. Hinden, "Internet Protocol, Version 6 859 (IPv6) Specification", July 2017 861 12.2. Informative References 863 [3GPP-EdgeComputing] 3GPP TR 23.748, "3rd Generation 864 Partnership Project; Technical Specification Group 865 Services and System Aspects; Study on enhancement of 866 support for Edge Computing in 5G Core network 867 (5GC)", Release 17 work in progress, Aug 2020. 869 [5G-EC-Metrics] L. Dunbar, H. Song, J. Kaippallimalil, "IP 870 Layer Metrics for 5G Edge Computing Service", draft- 871 dunbar-ippm-5g-edge-compute-ip-layer-metrics-00, 872 work-in-progress, Oct 2020. 874 [5G-Edge-Sticky] L. Dunbar, J. Kaippallimalil, "IPv6 Solution 875 for 5G Edge Computing Sticky Service", draft-dunbar- 876 6man-5g-ec-sticky-service-00, work-in-progress, Oct 877 2020. 879 Internet-Draft AppMetaData for 5G EC Service 881 [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation 882 Subsequent Address Family Identifier (SAFI) and the 883 BGP Tunnel Encapsulation Attribute", April 2009. 885 [BGP-SDWAN-Port] L. Dunbar, H. Wang, W. Hao, "BGP Extension 886 for SDWAN Overlay Networks", draft-dunbar-idr-bgp- 887 sdwan-overlay-ext-03, work-in-progress, Nov 2018. 889 [SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K. 890 Majumdar, "BGP UPDATE for SDWAN Edge Discovery", 891 draft-dunbar-idr-sdwan-edge-discovery-00, work-in- 892 progress, July 2020. 894 [Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation 895 Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 896 2018. 898 13. Acknowledgments 900 Acknowledgements to Donald Eastlake for their review and 901 contributions. 903 This document was prepared using 2-Word-v2.0.template.dot. 905 Internet-Draft AppMetaData for 5G EC Service 907 Authors' Addresses 909 Linda Dunbar 910 Futurewei 911 Email: ldunbar@futurewei.com 913 Kausik Majumdar 914 CommScope 915 350 W Java Drive, Sunnyvale, CA 94089 916 Email: kausik.majumdar@commscope.com 918 Haibo Wang 919 Huawei 920 Email: rainsword.wang@huawei.com 922 Gyan Mishra 923 Verizon 924 Email: gyan.s.mishra@verizon.com