idnits 2.17.00 (12 Aug 2021) /tmp/idnits31361/draft-dunbar-e2e-latency-arch-view-and-gaps-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (March 27, 2017) is 1881 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'LTE-latency' is defined on line 476, but no explicit reference was found in the text Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Network working group L. Dunbar 2 Internet Draft 3 Category: Informational Huawei 4 Expires: November 2017 6 March 27, 2017 8 Architectural View of E2E Latency and Gaps 10 draft-dunbar-e2e-latency-arch-view-and-gaps-01.txt 12 Abstract 14 Ultra-Low Latency is a highly desired property for many types of 15 services, such as 5G MTC (Machine Type Communication) requiring 16 E2E connection for V2V to be less than 2ms, AR/VR requiring delay 17 less than 5ms, V2X less than 20ms, etc. 19 This draft examines the E2E latency from architectural 20 perspective, from studying how different OSI layers contribute to 21 E2E latency, how different domains, which can be different 22 operators' domains or administrative domains, contribute to E2E 23 latency, to analyzing the gaps of recent technology advancement 24 in reducing latency. 26 By studying the contributing factors to E2E latency from various 27 angles, the draft identifies some gaps of recent technology 28 advancement for E2E services traversing multiple domains and 29 involving multiple layers, which can be the basis for IAB to 30 organize more in-depth discussion, like workshop or cross Area 31 (or industry wide) BOF, as the scope of the discussion will be 32 too wide for one IETF WG. The discussion might touch upon 33 multiple IETF areas. 35 The goal of those further "deep-dive" workshop or cross area BOF 36 is to establish more comprehensive foundation to IESG and the 37 IETF community in identifying potential new work initiatives for 38 reducing E2E latency of services over the Internet. 40 Internet-Draft E2E Over Internet Latency Taxonomy 42 Status of this Memo 44 This Internet-Draft is submitted in full conformance with the 45 provisions of BCP 78 and BCP 79. This document may not be 46 modified, and derivative works of it may not be created, except 47 to publish it as an RFC and to translate it into languages other 48 than English. 50 Internet-Drafts are working documents of the Internet Engineering 51 Task Force (IETF), its areas, and its working groups. Note that 52 other groups may also distribute working documents as Internet- 53 Drafts. 55 Internet-Drafts are draft documents valid for a maximum of six 56 months and may be updated, replaced, or obsoleted by other 57 documents at any time. It is inappropriate to use Internet- 58 Drafts as reference material or to cite them other than as "work 59 in progress." 61 The list of current Internet-Drafts can be accessed at 62 http://www.ietf.org/ietf/1id-abstracts.txt 64 The list of Internet-Draft Shadow Directories can be accessed at 65 http://www.ietf.org/shadow.html 67 This Internet-Draft will expire on September 27, 2017. 69 Copyright Notice 71 Copyright (c) 2017 IETF Trust and the persons identified as the 72 document authors. All rights reserved. 74 This document is subject to BCP 78 and the IETF Trust's Legal 75 Provisions Relating to IETF Documents 76 (http://trustee.ietf.org/license-info) in effect on the date of 77 publication of this document. Please review these documents 78 carefully, as they describe your rights and restrictions with 79 respect to this document. Code Components extracted from this 80 document must include Simplified BSD License text as described in 82 Internet-Draft E2E Over Internet Latency Taxonomy 84 Section 4.e of the Trust Legal Provisions and are provided 85 without warranty as described in the Simplified BSD License. 87 Table of Contents 89 1. Introduction................................................. 4 90 2. Terminology.................................................. 5 91 3. AR/VR Use Case............................................... 5 92 4. Contributing Factors to E2E Latency.......................... 6 93 5. Application Layer Initiative in reducing E2E latency......... 6 94 5.1. Content Placement mechanisms need visibility to Network. 7 95 6. Transport Layer Initiatives in reducing Latency and gaps..... 7 96 6.1. TCP Layer Latency Improvement Alone is not enough....... 7 97 6.2. LTE Latency Impact on TCP Performance................... 8 98 6.3. Low Latency via Multipath TCP Extension................. 9 99 7. Network and Link Layer Initiatives in reducing E2E Latency... 9 100 8. Radio Channel Quality Impact to flows with High QoS......... 10 101 9. E2E Latency Contributed by multiple domains................. 10 102 10. Conclusion................................................. 11 103 11. Security Considerations.................................... 11 104 12. IANA Considerations........................................ 11 105 13. Acknowledgements........................................... 11 106 14. References................................................. 11 107 14.1. Normative References.................................. 11 108 14.2. Informative References................................ 12 109 15. Appendix:.................................................. 12 110 15.1. Example: multi-Segments Latency for services via 111 Cellular Access............................................. 12 112 15.2. Latency contributed by multiple nodes................. 14 113 15.3. Latency through the Data Center that hosts S-GW & P-GW 14 114 Authors' Addresses............................................. 15 116 Internet-Draft E2E Over Internet Latency Taxonomy 118 1. Introduction 120 Ultra-Low Latency is a highly desired property for many types of 121 services, such as 5G MTC (Machine Type Communication) requiring 122 E2E connection for V2V to be less than 2ms, AR/VR requiring delay 123 less than 5ms, V2X less than 20ms, etc. 125 This draft is to examine E2E latency from architectural 126 perspective, from studying how different OSI layers contribute to 127 E2E latency, how different domains, which can be different 128 operators' domains or administrative domains, contribute to E2E 129 latency, to analyzing the gaps of recent technology advancement 130 in reducing latency. 132 The primary purpose of studying E2E Latency from architectural 133 perspective is to help the IETF community identify potential work 134 areas for reducing E2E latency of services over the Internet. 136 In recent years, the internet industry has been exploring 137 technologies and innovations at all layers of the OSI stack to 138 reduce latency. At the upper (application) layer, more contents 139 are distributed to the edges closer to end points and more 140 progress in Mobile Edge Computing (MEC) has been made. At the 141 Transport layer, there are QUIC/L4S initiatives. At the network 142 layer, there are IP/MPLS Hardened pipe (RFC 7625), latency 143 optimized router design, and BBF's Broadband Assured Services 144 (BAS). At the link layer, there are IETF DETNET, IEEE 802.1 TSN 145 (Time Sensitive Networking), and Flex Ethernet (OIF). 147 By studying the contributing factors to E2E latency from various 148 angles, the draft identifies some gaps of recent technology 149 advancement for E2E services traversing multiple domains and 150 involving multiple layers, which can be the basis for IAB to 151 organize more in-depth discussion, like a workshop or cross Area 152 (or industry wide) BOF, as the scope of the discussion will be 153 too wide for one IETF WG. The discussion might touch upon 154 multiple IETF areas. 156 The goal of the further "deep-dive" workshop or cross area BOF is 157 to establish more comprehensive foundation to IESG and the IETF 158 community in identifying potential work initiatives for reducing 159 E2E latency of services over the Internet. 161 Internet-Draft E2E Over Internet Latency Taxonomy 163 2. Terminology 165 DA: Destination Address 167 DC: Data Center 169 E2E: End To End 171 GTP: GPRS Tunneling Protocol (GTP) is a group of IP-based 172 communications protocols used to carry general packet 173 radio service (GPRS) within GSM, UMTS and LTE networks. 174 In 3GPP architectures, GTP can be decomposed into 175 separate protocols, GTP-C, GTP-U and GTP'. GTP-C is 176 used for signaling. GTP-U is used for carrying user 177 data. 179 LTE: Long Term Evolution 181 TS: Tenant System 183 VM: Virtual Machines 185 VN: Virtual Network 187 3. AR/VR Use Case 189 The E-2-E delays of AR/VR system come from delay of multiple 190 systems: 192 - Tracking delay 193 - Application delay 194 - Rendering delay 195 - Display delay 197 For human beings not to feel dizzy viewing AR/VR images, the 198 oculus delay should be less than 19.3ms, which includes display 199 delay, computing delay, transport delay, and sensoring delay. 200 That means the "Network Delay" budget is only 5ms at the most. 202 Internet-Draft E2E Over Internet Latency Taxonomy 204 4. Contributing Factors to E2E Latency 206 Internet data is packaged and transported in small pieces of 207 data. The flow of these small pieces of data directly affects a 208 user's internet experience. When data packets arrive in a smooth 209 and timely manner, the user sees a continuous flow of data; if 210 data packets arrive with large and variable delays between 211 packets, the user's experience is degraded. 213 Key contributing factors to E2E latency: 215 - Generation: delay between physical event and availability of 216 data 217 - Transmission: signal propagation, initial signal encoding 218 - Processing: Forwarding, encap/decap, NAT, encryption, 219 authentication, compress, error coding, signal translation 220 - Multiplexing: Delays needed to support sharing; Shared channel 221 acquisition, output queuing, connection establishment 222 - Grouping: Reduces frequency of control information and 223 processing; Packetization, message aggregation 225 The 2013 ISOC Workshop [Latency-ISOC] on Internet Latency 226 concluded that: 228 o Bandwidth alone is not enough in reducing latency 229 o Bufferbloat is one of the main causes for high latency in 230 the Internet. 232 Figure 1 of the 2013 ISOC workshop report showed that the timing 233 of download of an apparently uncluttered example Web page 234 (ieeexplore.ieee.org), actually comprised of over one hundred 235 objects, transferred over 23 connections needing 10 different DNS 236 look-ups. This phenomenon just further proves that reducing E2E 237 latency will need multiple layers coordination and interaction. 239 5. Application Layer Initiative in reducing E2E latency 241 More and more End to End services over internet are from end 242 users/devices to applications hosted in data centers. 244 As most content today is distributed, E2E services usually do not 245 traverse the globe but rather more often than not, the network 247 Internet-Draft E2E Over Internet Latency Taxonomy 249 segments that the E2E service traverses are from end users to 250 regional data centers. The practice of content distribution to 251 the edge has transformed reaching low latency goals from fighting 252 against the speed of light to optimizing communication between 253 end users and their desired content. 255 However, without awareness of latency characteristics of network 256 segments, the content distribution mechanisms & algorithms might 257 not achieve their intended optimal result. 259 5.1. Content Placement mechanisms need visibility to Network 261 To be added. 263 6. Transport Layer Initiatives in reducing Latency and gaps 265 IETF QUIC, L4S are some of the initiatives in reducing E2E 266 latency at the Transport Layer. 268 IETF QUIC focus on the improvement from end points. It doesn't 269 take into consideration of the network latency that the data 270 packets traverse. 272 The IETF L4S uses AQM for network nodes to purposely drop packets 273 or send indication to end points when their queues are above 274 certain thresholds. The goal is for the end nodes to reduce 275 transmission rate when intermediate nodes buffers are almost 276 full. It has following issues: 278 As network aggregates many flows from many different end points 279 and most flows have variable data rate, an intermediate network 280 node+port's buffer being almost full at one specific time 281 doesn't mean that the same amount of traffic will traverse the 282 same port a few microseconds later. If all end (source) points 283 reduce transmission rate upon receiving the AQM indication (or 284 experiencing packets drop), traffic through the network can be 285 greatly reduced (i.e. leaving no queue in the buffer). Then all 286 end points can increase their rate, causing traffic pattern 287 oscillation and buffer congestion again. 289 6.1. TCP Layer Latency Improvement Alone is not enough 291 The following example shows why simply optimizing transport layer 292 alone is not enough. More details can be found at 293 https://www.w3.org/Protocols/HTTP/Performance/Pipeline.html. 295 Internet-Draft E2E Over Internet Latency Taxonomy 297 Typical web pages today contain a HyperText Markup Language 298 (HTML) document and many embedded images. Twenty or more 299 embedded images are quite common. Each of these images is an 300 independent object in the Web, retrieved (or validated for 301 change) separately. The common behavior for a web client, 302 therefore, is to fetch the base HTML document, and then 303 immediately fetch the embedded objects, which are typically 304 located on the same server. 306 The large number of embedded objects represents a change from 307 the environment in which the Web transfer protocol, the 308 Hypertext Transfer Protocol (HTTP), was designed. As a result, 309 HTTP/1.0 handles multiple requests from the same server 310 inefficiently, creating a separate TCP connection for each 311 object. 313 6.2. LTE Latency Impact on TCP Performance 315 HTTP/TCP is the dominating application and transport layer 316 protocol suite used on the internet today. According to HTTP 317 Archive (http://httparchive.org/trends.php), the typical size of 318 HTTP based transactions over the internet are in the range of a 319 few 10's of Kbytes up to 1 Mbyte. In this size range, the TCP 320 slow start period is a significant part of the total transport 321 period of the packet stream. 323 During TCP slow start, TCP exponentially increases its congestion 324 window, i.e. the number of segments it brings into flight, until 325 it fully utilizes the throughput that LTE (Radio + EPC) can 326 offer. The incremental increases are based on TCP ACKs which are 327 received after one round trip delay in the LTE system. Thus, as 328 it turns out, during TCP slow start the performance is latency 329 limited in Radio Network (LTE). Hence, improved latency in LTE 330 can improve the perceived data rate for TCP based data 331 transactions, which in its turn reduces the time it takes to 332 complete a data down-load or upload. 334 Despite rather small (in terms of milliseconds) improvements that 335 can be achieved over the radio round trip time, the total 336 increase in the perceived throughput and delay savings of 337 downloading an item below 1MB is significant due to the additive 338 effect of LTE latency improvements in the TCP slow start[LTE- 339 Research]. 341 Internet-Draft E2E Over Internet Latency Taxonomy 343 6.3. Low Latency via Multipath TCP Extension 345 There are some research work on how to use multi-path TCP to 346 reduce E2E latency, such as 347 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7510787. The 348 paper proposes an MPTCP extension that sends data redundantly 349 over multiple paths in the network, which basically exchanges 350 bandwidth for latency. The integration into the MPTCP protocol 351 provides benefits such as transparent end-to-end connection 352 establishment, multipath-enabled congestion control, and the 353 prevention of head of line blocking. The research paper claims 354 that their proposed Multipath TCP extension can halve the average 355 round-trip time and reduce its standard deviation by a factor of 356 19 for a real world mobile scenario in a stressed environment. 357 Those kind of researchers should be invited to the "Reducing 358 latency over Internet Deep-Dive" workshop or cross-area BOF (to 359 be organized by IAB). 361 7. Network and Link Layer Initiatives in reducing E2E Latency 363 Several industry initiatives already exist for improving latency 364 at the Link and Network layers: 366 - Link Layer: IEEE 802.1 TSN (Time Sensitive Networking), and 367 Flex Ethernet (OIF). 368 - The network layer: IETF DETNET, IP/MPLS Hardened pipe (RFC 369 7625). 371 Gaps: 373 IEEE 802.1 TSN (Time Sensitive Networking) requires stringent 374 synchronous timing among all the nodes, which is suitable for 375 small scoped network, but not suitable for the internet because 376 most routers/switches in the network don't support synchronous 377 timing. 379 IP/MPLS hardened pipe can guarantee no congestion and no 380 buffering on all nodes along the path, therefore, ensure the 381 lowest latency along the path. The hardened pipe is ideal for 382 flows with steady bandwidth requirement. 384 Internet-Draft E2E Over Internet Latency Taxonomy 386 But for applications that don't have steady flow size, the 387 hardened pipe requires reserving the peak rate dedicated 388 channels, which, like TDM, will incur bandwidth waste when 389 application traffic goes below peak rate. 391 Traffic Engineering is one of the most commonly used methods to 392 reduce congestion at the network layer. However, it doesn't 393 completely prevent transient congestion. Depending on the tunnel 394 sizing, there could be momentary traffic bursts that exceed the 395 tunnel size, thus causing congestion if there isn't adequate 396 headroom on the trunk carrying the tunnel to absorb the burst. Or 397 a link or node outage, that reroutes the tunnel onto a secondary 398 path that becomes overloaded, could cause congestion. 400 8. Radio Channel Quality Impact to flows with High QoS. 402 QoS is one of the key methods employed by fixed IP network to 403 reduce latency for some flows. However, in Radio network, if a 404 UE's channel condition is poor, the eNB may schedule more frames 405 to other UEs whose flow are marked with much lower QoS. 407 There are many studies showing how Radio quality negatively 408 impact to the TCP performance. 410 It is beneficial to the whole industry if there is a workshop to 411 get people or SDOs working on different layers of Internet 412 service together to showcase their work or their pain points. 414 IESG can make much more informed decision on creating useful 415 initiatives when the community is aware of other work and 416 obstacles. 418 9. E2E Latency Contributed by multiple domains 420 All of the latency improvement initiatives in the link layer have 421 been within a single domain, such as IETF DETNET, IEEE 802.1 TSN 422 (Time Sensitive Networking), and Flex Ethernet (OIF). The network 423 layer latency improvement, such as IP/MPLS Hardened pipe (RFC 424 7625) is also within a single domain. 426 But E2E services usually traverse more than one domain, which can 427 be administrative domains or multiple operators' networks. 429 Yet today, there is no interface between domains to: 431 Internet-Draft E2E Over Internet Latency Taxonomy 433 - Inquire about the latency characteristics or capabilities from 434 another domain 435 - Negotiate or reserve latency capabilities from another domain. 436 - Have a standardized method to characterize latency 438 IETF/IAB is an ideal organization to tackle those issues because 439 IETF has the expertise. 441 10. Conclusion 443 As end to end services traverse multiple types of network 444 segments and domains, and involve multiple layers, more informed 445 decision in each layer technological improvement is important. 447 - Need across domain coordination 448 - Need across layer coordination 450 11. Security Considerations 452 As the trend is going more encryption, it is getting more 453 difficult for various network segments to detect applications 454 sessions. Therefore, it is more important to create ways for 455 better coordination among different layers, for improved latency, 456 trouble shooting, restoration, etc. 458 12. IANA Considerations 460 This section gives IANA allocation and registry considerations. 462 13. Acknowledgements 464 Special thanks to Jari Arkko for encouraging writing this draft. 465 And many thanks to Andy Malis, Jim Guichard, Spenser Dawkins, and 466 Donald Eastlake for suggestions and comments to this draft. 468 14. References 470 14.1. Normative References 472 Internet-Draft E2E Over Internet Latency Taxonomy 474 14.2. Informative References 476 [LTE-latency] https://www.ericsson.com/research-blog/lte/lte- 477 latency-improvement-gains/ 479 [Latency-ISOC] 2013 ISOC organized Latency over Internet workshop 480 report 482 15. Appendix: 484 15.1. Example: multi-Segments Latency for services via Cellular 485 Access 487 Via Cellular network, there are User Plane Latency and Control 488 Plane Latency. Control plane deals with signaling and control 489 functions, while user plane deals with actual user data 490 transmission. 492 The User Plane latency can be measured by the time it takes for a 493 small IP packet to travel from the terminal through the network 494 to the internet server, and back. The Control Plane latency is 495 measured as the time required for the UE (User Equipment) to 496 transit from idle state to active state. 498 User Plane latency is relevant for the performance of many 499 applications. This document mainly focuses on the User Plane 500 Latency. The following diagram depicts a logical path from an end 501 user (smart phone) application to the application controller 502 hosted in a data center via 4G Mobile network, which utilize the 503 Evolved Packet Core (EPC). 505 +------+ +---------+ 506 |DC | | EPC | +----+ 507 |Apps |<----------->|P-GW/S-GW|< -------> | eNB|<---> UE 508 | | +---------+ Mobile +----+ Radio 509 +------+ Internet Backhaul Access 511 Mobility Management Entity (MME) is responsible for 512 authentication of the mobile device. MME retains location 513 information for each user and then selects the Serving Gateway 514 (S-GW) for a UE at the initial attach and at time of intra-LTE 515 handover involving Core Network (CN) node relocation. 517 Internet-Draft E2E Over Internet Latency Taxonomy 519 The Serving Gateway (S-GW) resides in the user plane where it 520 forwards and routes packets to and from the eNodeB (eNB) 521 and packet data network gateway (P-GW). The S-GW also serves as 522 the local mobility anchor for inter-eNodeB handover and mobility 523 between 3GPP networks. 525 P-GW (Packet Data Network Gateway) provides connectivity from the 526 UE to external packet data networks by being the point of exit 527 and entry of traffic for the UE. A UE may have simultaneous 528 connectivity with more than one P-GW for accessing multiple 529 Packet Data Networks. The P-GW performs policy enforcement, 530 packet filtering for each user, charging support, lawful 531 interception and packet screening. Another key role of the P-GW 532 is to act as the anchor for mobility between 3GPP and non-3GPP 533 technologies such as WiMAX and 3GPP2 (CDMA 1X and EvDO). 535 Very often P-GW and S-GW are co-located. The data traffic between 536 eNB and S-GW is encapsulated by GTP-U. 538 The figure above shows that the end to end services from/to UE 539 consists of the following network segments: 541 - Radio Access network - RAN 542 - Mobile Backhaul network that connect eNB to S-GW. 543 - Network within the DC that hosts S-GW & P-GW 544 - Packet Data Network, which can dedicated VPN, internet, or 545 other data network. 546 - Network within the DC that hosts the App. 548 The RAN (Radio Access Network) is between UE (e.g. smart phone) 549 and eNB. 3GPP has a group TSG RAN working on improving 550 performance (including latency) of the Radio Access network. 551 There are many factors impacting the latency through RAN. 553 The Mobile Backhaul Network connects eNBs to S-GW/P-GW, with data 554 traffic being encapsulated in GTP protocol. The number of UEs 555 that one eNB can handle are in 100s. The number of UEs that one 556 S-GW/P-GW can handle are in millions. Therefore, the mobile 557 backhaul network connects 10s of thousands of eNBs to S-GW/P-GW. 558 Therefore, the number of network nodes in the Mobile Backhaul 559 network can be very large. Therefore, any new protocol 560 improvement in reducing latency can play a big part in reducing 561 the overall latency for the end to end services. 563 Internet-Draft E2E Over Internet Latency Taxonomy 565 15.2. Latency contributed by multiple nodes 567 The variant of delay for data packets through network is caused 568 by network nodes along the path as the transmission delay on 569 physical link is fixed. When there is no congestion, the latency 570 across most routers and switches are very small, in the magnitude 571 of ~20us (worst case in ~40us). When congestion occurs within a 572 node, i.e. with buffer/queues being used to avoid dropping 573 packets, latency across a node can be in the magnitude of micro- 574 seconds. The recent improvements made within router architecture 575 have greatly improved latency through a node. However, there is 576 no standard methods for routers to characterize and expose 577 various latency characteristics through a network node. 579 Data packets also traverse through network functions, such as FW, 580 DPI, OPS, whose latency vary depending on the depth of the 581 processing and the equipment performance. 583 15.3. Latency through the Data Center that hosts S-GW & P-GW 585 S-GW and P-GW are hosted in Data center. There are typically 2~3 586 tiers of switches connecting the servers that hosts S-GW & P-GW 587 to the external network, as depicted in the following: 589 +---------+ 590 | Gateway | 591 +---------+ 593 \ +-------+ +------+ / 594 \ +/------+ | +/-----+ | / 595 \ | Aggr11| + ----- |AggrN1| + / 596 \ +---+---+/ +------+/ / 597 \ / \ / \ / 598 \ / \ / \ / 599 \ +---+ +---+ +---+ +---+ / 600 \- |T11|... |T1x| |T21| ... |T2y|--- 601 +---+ +---+ +---+ +---+ 602 | | | | 603 +-|-+ +-|-+ +-|-+ +-|-+ Servers 604 | |... |SGW| | S | | S |<- 605 +---+ +---+ +---+ +---+ 606 | |... |PGW| | S | ... | S | 607 +---+ +---+ +---+ +---+ 608 | |... | S | | S | ... | S | 609 +---+ +---+ +---+ +---+ 611 Internet-Draft E2E Over Internet Latency Taxonomy 613 As the distance within data center can be small, the transmission 614 delay within data center can be negligent. The majority of 615 latency within data center is caused by the switching within the 616 gateway routers, traffic traversing through middleware boxes such 617 as FW, DPI, IPS, value added services, the top of the rack 618 switches, and aggregation switches. 620 If the S-GW and P-GW are hosted in large data center, there could 621 be latency contributed by the 622 encapsulation/decapsulation such as work specified by 623 NVO3. 625 Authors' Addresses 627 Linda Dunbar 628 Huawei Technologies 629 5430 Legacy Drive, Suite #175 630 Plano, TX 75024, USA 631 Phone: (469) 277 5840 632 Email: linda.dunbar@huawei.com