idnits 2.17.00 (12 Aug 2021) /tmp/idnits11973/draft-li-dyncast-architecture-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document date (7 March 2022) is 68 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-03) exists of draft-liu-dyncast-ps-usecases-02 Summary: 0 errors (**), 0 flaws (~~), 1 warning (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 rtgwg Y. Li 3 Internet-Draft L. Iannone 4 Intended status: Informational D. Trossen 5 Expires: 8 September 2022 Huawei Technologies 6 P. Liu 7 China Mobile 8 C. Li 9 Huawei Technologies 10 7 March 2022 12 Dynamic-Anycast Architecture 13 draft-li-dyncast-architecture-03 15 Abstract 17 This document describes a proposal for an architecture for the 18 Dynamic-Anycast (Dyncast). It includes an architecture overview, 19 main components that shall exist, and the workflow. An example of 20 workflow is provided, focusing on the load-balance multi-edge based 21 service use-case, where load is distributed in terms of both 22 computing and networking resources through the dynamic anycast 23 architecture. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on 8 September 2022. 42 Copyright Notice 44 Copyright (c) 2022 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 49 license-info) in effect on the date of publication of this document. 50 Please review these documents carefully, as they describe your rights 51 and restrictions with respect to this document. Code Components 52 extracted from this document must include Revised BSD License text as 53 described in Section 4.e of the Trust Legal Provisions and are 54 provided without warranty as described in the Revised BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 59 2. Definition of Terms . . . . . . . . . . . . . . . . . . . . . 3 60 3. Architecture Main Concepts . . . . . . . . . . . . . . . . . 4 61 4. Dyncast Architecture Workflow . . . . . . . . . . . . . . . . 8 62 4.1. Service Notification/Metrics Update . . . . . . . . . . . 8 63 4.2. Service Demand Dispatch and Instance Affinity . . . . . . 10 64 5. Dyncast Control-plane vs Data-plane operations . . . . . . . 11 65 6. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 66 7. Security Considerations . . . . . . . . . . . . . . . . . . . 12 67 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12 68 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 12 69 10. Informative References . . . . . . . . . . . . . . . . . . . 13 70 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 13 71 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13 73 1. Introduction 75 Edge computing is expanding from a single edge nodes to multiple 76 networked collaborating edge nodes to solve the issues like response 77 time, resource optimization, and network efficiency. 79 The current network architecture in edge computing provides 80 relatively static service dispatching, for example, to the closest 81 edge from an IGP perspective, or to the server with the most 82 computing resources without considering the network status, and even 83 sometimes just based on static configuration. 85 Networking taking into account computing resource metrics seems to be 86 an interesting paradigm that fits numbers of use-cases that would 87 benefit from such capability [I-D.liu-dyncast-ps-usecases]. Yet, 88 more investigation is still needed in key areas for this paradigm 89 and, to this end, this document aims at providing an architectural 90 framework, which will enable service notification, status update, and 91 service dispatch in edge computing.. 93 The Dyncast architecture presents an anycast based service and access 94 model addressing the problematic aspects of existing network layer 95 edge computing service deployment, including the unawareness of 96 computing resource information of service, static edge selection, 97 isolated network and computing metrics and/or slow refresh of status. 99 Dyncast assumes that there are multiple equivalent service instances 100 running on different edge nodes, globally providing (from a logical 101 point of view) one single service. A single edge may have limited 102 computing resources available, and different edges likely have 103 different resources available, such as CPU or GPU. The main 104 principle of Dyncast is that multiple edge nodes are interconnected 105 and collaborate with each other to achieve a holistic objective, 106 namely to dispatch service demands taking into account both service 107 instances status as well as network state (e.g., paths length and 108 their congestion). For this, computing resources available to serve 109 a request is one of the top metrics to be considered. At the same 110 time, the quality of the network path to an edge node may vary over 111 time and may hence be another key attribute to be considered for said 112 dispatching of service demands. 114 2. Definition of Terms 116 Dyncast: As defined in [I-D.liu-dyncast-ps-usecases], Dynamic 117 Anycast, taking the dynamic nature of computing resource metrics 118 into account to steer an anycast routing decision. 120 Service: As defined in [I-D.liu-dyncast-ps-usecases], a monolithic 121 functionality that is provided by an endpoint according to the 122 specification for said service. A composite service can be built 123 by orchestrating monolithic services. 125 Service instance: As defined in [I-D.liu-dyncast-ps-usecases], 126 running environment (e.g., a node) that makes the functionality of 127 a service available. One service can have several instances 128 running at different network locations. 130 D-Router: A node supporting Dyncast functionalities as described in 131 this document. Namely it is able to understand both network- 132 related and service-instances-related metrics, take forwarding 133 decision based upon and manitain instance affinity, i.e., forwards 134 packets belonging to the same service demand to the same instance. 136 D-MA: Dyncast Metric Agent (D-MA): A dyncast specific agent able to 137 gather and send metric updates (from both network and instance 138 prespective) but not performing forwarding decisions. May run on a 139 D-Router, but it can be also implementated as a separate module 140 (e.g., a software library) collocated with a service instance. 142 D-SID: Dyncast Service ID, an identifier representing a service, 143 which the clients use to access said service. Such identifier 144 identifies all of the instances of the same service, no matter on 145 where they are actually running. D-SID is independent of which 146 service instance serves the service demand. Usually multiple 147 instances provide a (logically) single service, and service demands 148 are dispatched to the different instance through an anycast model, 149 i.e., choosing one instance among all available instances. 151 D-BID: Dyncast Binding ID, an address to reach a service instance 152 for a given D-SID. It is usually a unicast IP where service 153 instances are attached. Different service instances provide the 154 same service identified through D-SID but with different Dyncast 155 Binding IDs. 157 Service demand: The demand for a specific service and addressed to a 158 specific D-SID. 160 Service request: The request for a specific service and addressed to 161 a specific service instance identified with D-BID. 163 3. Architecture Main Concepts 165 Dyncast assumes that there are multiple equivalent service instances 166 running on different edge sites, globally providing one single 167 service which is represented by D-SID. The network will take 168 forwarding decision for the service demand from the client according 169 to both service instances status as well as network state. 171 The architecture of Dyncast has two typical modes, distributed or 172 centralized. 174 * Distributed mode: The resources and status of the different 175 service instances are propagated from the D-Routers connecting the 176 edge sites where the service is deployed to the D-Routers with 177 clients. In addition D-Routers have the network topology and 178 status. The ingress D-Router which receives the service demand 179 from the client decides independently which service instance to 180 access according to the service instances status and network state 181 and maintains instance affinity. 183 * Centralized mode: The resources and status of the different 184 service instances are reported to the network controller from the 185 D-Routers connecting the edge sites where the service is deployed. 186 At the same time the controller collects the network topology and 187 status. The controller makes routing decisions for each ingress 188 D-Router according to the service instances status and network 189 state and downloads the decisions to all the ingress D-Routers. 191 When the ingress D-Router receives the service demand from the 192 client, it selects which service instance to access according to 193 the decision made by the controller, and maintains the instance 194 affinity subsequently. 196 This document mainly introduces the detailed process of the 197 distributed mode, and the centralized mode will be introduced in 198 detail in the future. 200 Edge sites (edges for short) are normally the sites where edge 201 computing is performed. Service instances are initiated at different 202 edge sites. Thus, a single service can actually have a significant 203 number of instances running on different edges. A Dyncast Service ID 204 (D-SID) is used to uniquely identify a service (e.g., a matrix 205 computation for face recognition, or a game server). Service 206 instances can be hosted on servers, virtual machines, access routers 207 or gateway in edge data center. 209 Close to (one or more) Service instances is the Dyncast Metric Agent 210 (D-MA). This element has the task to gather information about 211 resources and status of the different instances as well as network- 212 related information. Such element may also run in a dyncast-enable 213 router (named D-Router), while other deployment scenarios may lead to 214 this element running separately on edge nodes. 216 A D-Router is actually the main element in a Dyncast network, 217 providing the capability to exchange the information about the 218 computing resources information of service instances which have been 219 gathered through D-MAs. A D-Router can also be a service access 220 point for clients. When a service demand arrives, it will be 221 delivered to the most appropriate service instance. A service demand 222 may be the first packet of a data flow rather than an explicit out of 223 band service request. This architectural document does not make any 224 specific assumption on this matter. This documents only assumes 225 that: 227 * D-Routers are able to identify new service demands. The Dyncast 228 architecture presented in this document allows then to deliver 229 such a packet to the most appropriate service instance according 230 to information received from D-MAs and other D-Routers. 232 * D-Router are able to identify packets belonging to an existing 233 service demand. The Dyncast architecture presented in this 234 document allows to deliver these packets always to the same 235 service instance selected at the initial service demand. We term 236 this capability as 'instance affinity'. 238 Note: As described above, D-Router can make routing decision based on 239 per-service-instance computing-aware information. Actually, the 240 D-Router can make the decison based on per-site computing-aware 241 information. In this case, the egress D-Router can send the packet 242 to the specific instance based on local policy, Load balancing, etc. 243 This will be described in the future. 245 The element introduced above are depicted in Figure 1, which shows 246 the proposed Dyncast architecture. In Figure 1, the "infrastructure" 247 indicates the general IP infrastructure that does not necessarily 248 need to support Dyncats, i.e., not all routers of the infrastructure 249 need to be D-Routers. 251 edge site 1 edge site 2 edge site 3 253 +------------+ +------------+ 254 +------------+ | +------------+ | 255 | service | | | service | | 256 | instance |-+ | instance |-+ 257 +------------+ +------------+ 258 | | 259 +----------+ | 260 | D-MA | | 261 +----------+ +----------+ 262 | +-----------------+ | D-MA | 263 +----------+ | | +----------+ 264 |D-Router 1| ----| Infrastructure |---- |D-Router 3| 265 +----------+ | | +----------+ 266 | +-----------------+ | 267 | | | 268 | | | 269 | +----------+ | 270 | |D-Router 2| | 271 | +----------+ | 272 | | | 273 | | | 274 +-----+ +------+ +------+ 275 +------+| +------+ | +------+ | 276 |client|+ |client|-+ |client|-+ 277 +------+ +------+ +------+ 279 Figure 1: Dyncast Architecture. 281 Figure 2 shows an example of Dyncast deployment, with 2 service 282 instantiated twice (2 instances) on two different edges, namely edge 283 site 2 and 3. Those service instances utilize different D-BIDs to 284 serve service demands. D-Router 1 doesn't connect the edge site 285 directly and needn't collect the metric updates by D-MA. But it has 286 client to access and need to take forwarding decision for the client. 287 D-Router 2 gets metric updates by D-MA which runs on it. Edge site 2 288 has client present, so D-Router 2 need to take forwarding decision. 289 D-Router 3 gets metric updates from D-MA which is a separate software 290 module on edge computing platform in edge site 3. No client is 291 present at edge site 3, so D-Router 3 doesn't need take forwarding 292 decision. 294 D-SID: Dyncast Service ID 295 D-BID: Dyncast Binding ID 297 Service/Metrics Information 298 (D-SID 1, D-BID 21, ) 299 (D-SID 2, D-BID 22, ) 300 <-----------------> 302 +-------+ 303 +-------+ | D-SID 1 304 |Clients|-+ +--------+ 305 +-------+ +--|D-BID 21| instance 1 306 | | +--------+ 307 +----------+----+ | Edge 2 308 |D-Router 2|D-MA|--| D-SID 2 309 +----------+----+ | +--------+ 310 | +--|D-BID 22| instance 2 311 +----------------+ +--------+ 312 | | 313 | | 314 +------+ +----------+ | | 315 |Client|--|D-Router 1|--| Infrastructure | 316 +------+ +----------+ | | 317 | | D-SID 2 318 | | +--------+ 319 +----------------+ +---|D-BID 32| instance 3 320 | | +--------+ 321 +----------+ +------+ Edge 3 322 |D-Router 3| -| D-MA | 323 +----------+ +------+ D-SID 1 324 | +--------+ 325 +---|D-BID 31| instance 4 326 +--------+ 328 <----------------------------------> 329 (D-SID 2, D-BID 32, ) 330 (D-SID 1, D-BID 31, ) 331 Service/Metrics Information 332 Figure 2: Dyncast deployment example. 334 In Figure 2, the Dyncast Service ID (D-SID) follows an anycast 335 semantic, such as provided through an IP anycast address. It is used 336 to access a specific service no matter which service instance 337 eventually handles the service demand of the client. Clients or 338 other entities which want to access a service need to know about its 339 D-SID in advance. It can be achieved in different ways, for example, 340 using a special range of addresses associated to a certain service or 341 coding of anycast IP address as D-SID, or using DNS. 343 The Dyncast Binding ID (D-BID) is a unicast IP address. It is 344 usually the interface IP address through to reach a specific service 345 instance. Mapping and binding a D-SID to a D-BID is dynamic and 346 depends on the computing and network status at the time the service 347 demand first arrives (see Section 4.1 for the reporting of such 348 status). To ensure instance affinity, D-Routers are requested to 349 remember the instance that has been selected (e.g., by storing the 350 mapping) for delivering all packets to the same instance (see 351 Section 4.2 for discussing this aspect). 353 4. Dyncast Architecture Workflow 355 The following subsections provide an overview of how the 356 architectural elements introduced in the previous section do work 357 together. 359 4.1. Service Notification/Metrics Update 361 When a service instance is instantiated/terminated the service 362 information consisting in the mapping between the D-SID and the D-BID 363 has to be updated/deleteted as well. An update can also be triggered 364 by a change in relevant metrics (e.g., an instance becomes 365 overloaded). Computing resource information of service instance is 366 key information in Dyncast. Some of them may be relatively static 367 like CPU/GPU capacity, and some may be very dynamic, for example, 368 CPU/GPU utilization, number of sessions associated, number of queuing 369 requests. Changes in service-related relevant information has to be 370 collected by D-MA associated to each service instance. Various ways 371 can be used, for example, via routing protocols like EBGP or via an 372 API of a management system. Conceptually a D-Router collects 373 information coming from D-MA and keeps track of the IDs and computing 374 metrics of all service instances. 376 Figure 2 shows an example of information shared by the Dyncast 377 elements. The D-MA which is deployed with D-Router2 shares binding 378 information concerning the two instances of the two services running 379 on edge 2 (upper right hand side of the figure). These information 380 is: 382 * (D-SID 1, D-BID 21, metrics) 384 * (D-SID 2, D-BID 22, metrics) 386 The D-MA which is deployed as a separate module on edge 3 (lower 387 right hand side of the figure) shares binding information concerning 388 the two instances of the two services running on edge 3. These 389 information is: 391 * (D-SID 1, D-BID 31, metrics) 393 * (D-SID 2, D-BID 32, metrics) 395 Dyncast nodes share among themselves the service information 396 including the associated computing metrics for the service instances 397 attached to them. As a network node, a D-Router can also monitor the 398 network cost or metrics (e.g., congestion) to reach other D-Routers. 399 This is the focus of Dyncast control plane. Different mechanisms can 400 be used to share such information, for instance BGP ([RFC4760]), an 401 IGP, or a controller based mechanism. The specific mechanism is 402 beyond the scope of this document. The architecture assumes that the 403 Dyncast elements are able to share relevant information. 405 If, for instance, the client on the left hand side of Figure 2 sends 406 a service demand for D-SID1, D-Router1 has the knowledge of the 407 status of the service instance on both edge 2 and edge 3 and can make 408 a decision toward which D-BID to forward the demand. 410 There are different ways to represent the computing metrics. A 411 single digitalized value calculated from weighted attributes like 412 CPU/GPU consumption and/or number of sessions associated may be used 413 for simplicity reasons. However, it may not accurately reflect the 414 computing resources of interest. Multi-dimensional values give finer 415 information. This architectural document does not make any specific 416 assumption about metrics and how to encode or even use them. As 417 stated in Section 3, the only assumption is that a D-Router is able 418 to use such metrics so to take a decision when a service demand 419 arrives in order to map the demand onto a suitable service request. 421 As explained in the problem statement document 422 [I-D.liu-dyncast-ps-usecases], computing metrics may change very 423 frequently, when and how frequent such information should be 424 exchanged among Dyncats elements should be determined also in 425 accordance with the distribution protocol used for such purpose. A 426 spectrum of approaches can be employed,such as interval based 427 updates, threshold triggered updates, policy based updates, etc. 429 4.2. Service Demand Dispatch and Instance Affinity 431 This is the focus of the Dyncast data plane. When a new flow 432 (representing a service demand) arrives at a Dyncast ingress, such 433 ingress node selects the most appropriate egress according to the 434 network status and the computing resource of the attached service 435 instances. 437 Instance affinity is one of the key features that Dyncast should 438 support. It means that packets from the same 'flow' for a service 439 should always be sent to the same egress to be processed by the same 440 service instance. The affinity is determined at the time of newly 441 formulated service demand. 443 It is worth noting that different services may have different notions 444 of what constitutes a 'flow' and may thus identify a flow 445 differently. Typically a flow is identified by the 5-tuple value. 446 However, for instance, an RTP video streaming may use different port 447 numbers for video and audio, and it may be identified as two flows if 448 5-tuple flow identifier is used. However they certainly should be 449 treated by the same service instance. Therefore a 3-tuple based flow 450 identifier is more suitable for this case. Hence, it is desired to 451 provide certain level of flexibility in identifying flows, or from a 452 more general perspective, in identifying the set of packets for which 453 to apply instance affinity. More importantly, the means for 454 identifying a flow for the purpose of ensuring instance affinity must 455 be application-independent to avoid the need for service-specific 456 instance affinity methods. 458 Specifically, Instance affinity information should be configurable on 459 a per-service basis. For each service, the information can include 460 the flow/packets identification type and means, affinity timeout 461 value, and etc. For instance, the affinity configuration can 462 indicate what are the values, e.g., 5-tuple or 3-tuple, to be used as 463 the flow identifier. 465 When the most appropriate egress and service instance is determined 466 when a new flow for a service demand arrives, a binding table should 467 save this association between new service demand and service instance 468 selection. The information in such binding table may include flow/ 469 packets identification, affinity timeout value, etc. The subsequent 470 packets matching the entry are forwarded based on the table. 471 Figure 3 shows a possible example of flow binding table at the 472 ingress D-Router. 474 +-----------------------------------------+----------------+--------+ 475 | Flow/Packets Identifier | | | 476 +------+--------+---------+--------+------+ D-BID egress | timeout| 477 |src_IP| dst_IP |src_port |dst_port|proto | | | 478 +------+--------+---------+--------+------+----------------+--------+ 479 | X | D-SID 2| - | 8888 | tcp | D-BID 32 | xxx | 480 +------+--------+---------+--------+------+----------------+--------+ 481 | Y | D-SID 2| - | 8888 | tcp | D-BID 12 | xxx | 482 +------+--------+---------+--------+------+----------------+--------+ 484 Figure 3: Example of what a binding table can look like. 486 5. Dyncast Control-plane vs Data-plane operations 488 In summary, Dyncast consists of the following Control-plane and Data- 489 plane operations: 491 * Dyncast Control Plane: 493 - Dyncast Service ID Notification: the D-SID, an anycast IP 494 address, should be available and known. This can be achieved 495 in different ways. For example, use a special range or coding 496 of anycast IP address as service IDs or using the DNS. 498 - Dyncast Binding ID Notification: the mapping of (D-SID, D-BID), 499 i.e., service ID and the binding address, should be notified to 500 the D-Routers when the service instance starts (or stops). 501 Various ways can be used, for example, EBGP or management 502 system notification. 504 - Metrics Notification: D-MA have to be able to share the metrics 505 for a service and its binding ID so that D-Routers can select 506 the "best" instance for each new service demand. 508 * Dyncast Data Plane: 510 - New service demand: an ingress D-Router selects the most 511 appropriate egress in terms of the network status and the 512 computing resources of the instances of the requested service. 514 - Instance Affinity: Make sure the subsequent packets of an 515 existing service demand are always delivered to the same 516 service instance so that they can be served by the same service 517 instance. 519 6. Summary 521 This draft introduces a Dyncast architecture that enables the service 522 demand to be sent to an optimal service instance. It can dynamically 523 adapt to the computing resources consumption and network status 524 change. Dyncast is a network based architecture that supports a 525 large number of edges and is independent of the applications or 526 services hosted on the edge. 528 More discussion and input on control plane and data plane approach 529 are welcome. 531 7. Security Considerations 533 The computing resource information changes over time very frequent 534 with the creation and termination of service instance. When such 535 information is carried in routing protocol, too many updates can make 536 the network fluctuate. Control plane approach should take it into 537 considerations. 539 More thorough security analysis to be provided in future revisions. 541 8. IANA Considerations 543 This document does not make any request to IANA. 545 9. Contributors 547 Huijuan Yao 549 yaohuijuan@chinamobile.com 551 China Mobile 553 Xia Chen 555 jescia.chenxia@huawei.com 557 Huawei 559 10. Informative References 561 [RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter, 562 "Multiprotocol Extensions for BGP-4", RFC 4760, 563 DOI 10.17487/RFC4760, January 2007, 564 . 566 [I-D.liu-dyncast-ps-usecases] 567 Liu, P., Willis, P., Trossen, D., and C. Li, "Dynamic- 568 Anycast (Dyncast) Use Cases & Problem Statement", Work in 569 Progress, Internet-Draft, draft-liu-dyncast-ps-usecases- 570 02, 17 January 2022, . 573 Acknowledgements 575 TBD 577 Authors' Addresses 579 Yizhou Li 580 Huawei Technologies 581 Email: liyizhou@huawei.com 583 Luigi Iannone 584 Huawei Technologies 585 Email: Luigi.iannone@huawei.com 587 Dirk Trossen 588 Huawei Technologies 589 Email: dirk.trossen@huawei.com 591 Peng Liu 592 China Mobile 593 Email: liupengyjy@chinamobile.com 595 Cheng Li 596 Huawei Technologies 597 Email: c.l@huawei.com