idnits 2.17.00 (12 Aug 2021) /tmp/idnits21598/draft-han-6man-in-band-signaling-for-transport-qos-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 11, 2017) is 1676 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 2581 (Obsoleted by RFC 5681) == Outdated reference: draft-ietf-aqm-codel has been published as RFC 8289 == Outdated reference: draft-ietf-aqm-fq-codel has been published as RFC 8290 == Outdated reference: draft-ietf-aqm-pie has been published as RFC 8033 == Outdated reference: draft-ietf-tcpm-dctcp has been published as RFC 8257 Summary: 1 error (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group L. Han, Ed. 3 Internet-Draft G. Li 4 Intended status: Informational B. Tu 5 Expires: April 14, 2018 X. Tan 6 F. Li 7 R. Li 8 Huawei Technologies 9 J. Tantsura 11 K. Smith 12 Vodafone 13 October 11, 2017 15 IPv6 in-band signaling for the support of transport with QoS 16 draft-han-6man-in-band-signaling-for-transport-qos-00 18 Abstract 20 This document proposes a method to support the IP transport service 21 that could guarantee a certain level of service quality in bandwidth 22 and latency. The new transport service is fine-grained and could 23 apply to individual or aggregated TCP/UDP flow(s). 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on April 14, 2018. 42 Copyright Notice 44 Copyright (c) 2017 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (https://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 1.1. IP and Transport Technologies . . . . . . . . . . . . . . 4 61 1.2. TCP Solution Analysis . . . . . . . . . . . . . . . . . . 4 62 1.2.1. TCP Overview and Evolution . . . . . . . . . . . . . 4 63 1.2.2. TCP Solution Variants . . . . . . . . . . . . . . . . 5 64 1.2.3. Throughput Constraint . . . . . . . . . . . . . . . . 6 65 1.2.3.1. By Algorithm . . . . . . . . . . . . . . . . . . 6 66 1.2.3.2. By Fairness Principle . . . . . . . . . . . . . . 7 67 1.2.4. Latency Constraint . . . . . . . . . . . . . . . . . 7 68 1.2.5. Summary of TCP Solution . . . . . . . . . . . . . . . 7 69 1.3. Other Solution Analysis . . . . . . . . . . . . . . . . . 8 70 1.4. New approach . . . . . . . . . . . . . . . . . . . . . . 8 71 1.4.1. IP Transport with quality of service . . . . . . . . 8 72 1.4.2. Design targets . . . . . . . . . . . . . . . . . . . 9 73 1.4.3. Scope and assumption . . . . . . . . . . . . . . . . 9 74 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 10 75 2.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . 10 76 3. Control plane . . . . . . . . . . . . . . . . . . . . . . . . 11 77 3.1. Sub-layer in IP for transport control . . . . . . . . . . 12 78 3.2. IP In-band signaling . . . . . . . . . . . . . . . . . . 13 79 3.3. Control mechanism . . . . . . . . . . . . . . . . . . . . 14 80 3.4. IPv6 Approach . . . . . . . . . . . . . . . . . . . . . . 15 81 3.4.1. Basic Control Scenarios for TCP . . . . . . . . . . . 16 82 3.4.2. Details of In-band Signaling for TCP . . . . . . . . 17 83 3.5. Key Messages and Parameters in Control Protocol . . . . . 20 84 3.5.1. Setup and Setup State Report messages . . . . . . . . 20 85 3.5.2. OAM . . . . . . . . . . . . . . . . . . . . . . . . . 21 86 3.5.3. Forwarding State and Forwarding State Report messages 21 87 3.5.4. Flow Identifying Methods . . . . . . . . . . . . . . 21 88 3.5.5. Hop Number . . . . . . . . . . . . . . . . . . . . . 23 89 3.5.6. Mapping Index, Size and Mapping Index List . . . . . 23 90 3.5.7. QoS State and life of Time . . . . . . . . . . . . . 23 91 3.5.8. Authentication . . . . . . . . . . . . . . . . . . . 24 92 4. Data plane . . . . . . . . . . . . . . . . . . . . . . . . . 24 93 4.1. Basic Capability . . . . . . . . . . . . . . . . . . . . 24 94 4.2. Forwarding State and Forwarding State Report . . . . . . 25 95 4.3. Flow Identification in Packet Forwarding . . . . . . . . 26 96 4.4. QoS Forwarding State Detection and Failure Handling . . . 26 98 5. Other Issues . . . . . . . . . . . . . . . . . . . . . . . . 27 99 5.1. User and Application driven . . . . . . . . . . . . . . . 27 100 5.2. Traffic Management in Host . . . . . . . . . . . . . . . 28 101 5.3. Non-shortest-path . . . . . . . . . . . . . . . . . . . . 28 102 5.4. Heterogeneous Network . . . . . . . . . . . . . . . . . . 29 103 5.5. Proxy Control . . . . . . . . . . . . . . . . . . . . . . 29 104 6. Message Format . . . . . . . . . . . . . . . . . . . . . . . 29 105 6.1. Setup Msg . . . . . . . . . . . . . . . . . . . . . . . . 29 106 6.2. Bandwidth Msg . . . . . . . . . . . . . . . . . . . . . . 31 107 6.3. Burst Msg . . . . . . . . . . . . . . . . . . . . . . . . 31 108 6.4. Latency Msg . . . . . . . . . . . . . . . . . . . . . . . 31 109 6.5. Authentication Msg . . . . . . . . . . . . . . . . . . . 32 110 6.6. OAM Msg . . . . . . . . . . . . . . . . . . . . . . . . . 32 111 6.7. Forwarding State Msg . . . . . . . . . . . . . . . . . . 32 112 6.8. Setup State Report Msg . . . . . . . . . . . . . . . . . 33 113 6.9. Forward State Report Msg . . . . . . . . . . . . . . . . 34 114 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 34 115 8. Security Considerations . . . . . . . . . . . . . . . . . . . 35 116 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 35 117 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 36 118 10.1. Normative References . . . . . . . . . . . . . . . . . . 36 119 10.2. Informative References . . . . . . . . . . . . . . . . . 36 120 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 122 1. Introduction 124 Recently, more and more new applications for Internet are emerging. 125 These applications have a common part that is their required 126 bandwidth is very high and/or latency is very low compared with 127 traditional applications like most of web and video applications. 129 For example, AR or VR applications may need a couple of hundred Mbps 130 bandwidth (throughput) and a low single digit ms latency. Moreover, 131 the difference of mean bit rate and peak bit rate is huge due to the 132 compression algorithm [I-D.han-iccrg-arvr-transport-problem]. 134 Some future applications expect that network can provide a bounded 135 latency service, such as tactile network [Tactile]. 137 With the technology development in 5G and beyond, the wireless access 138 network is also rising the demand for the Ultra-Reliable and Low- 139 Latency Communications (URLLC), this also leads to the question if IP 140 transport can provide such service in Evolved Packet Core (EPC) 141 network. IP is becoming more and more important in EPC when the 142 Multi-access Edge Computing (MEC) for 5G will require the cloud and 143 data service moving closer to eNodeB. 145 Following sections will brief the current transport and QoS 146 technologies, and analyze the limitations to support above new 147 applications. 149 A new approach that could provide QoS for transport service will be 150 proposed. The scope and criteria for the new technology will also be 151 summarized. 153 1.1. IP and Transport Technologies 155 The traditional IP network can only provide the best-effort service. 156 The transport layer (TCP/UDP) on top of IP are based on this 157 fundamental architecture. The best-effort-only service has 158 influenced the transport evolution for quite long time, and results 159 in some widely accepted assumptions and solutions, such as: 161 1. The IP layer can only provide the basic P2P (point to point) or 162 P2MP (point to multi-point) end-to-end connectivity in Internet, 163 but the connectivity is not reliable and does not guarantee any 164 quality of service to end-user or application, such as bandwidth, 165 packet loss, latency etc. Due to this assumption, the transport 166 layer or application must have its own control mechanism in 167 congestion and flow to obtain the reliable and satisfactory 168 service to cooperate with the under layer network quality. 170 2. The transport layer assumes that the IP layer can only process 171 all IP flows equally in the hardware since the best effort 172 service is actually an un-differentiated service. The process 173 includes scheduling, queuing and forwarding. Thus, the transport 174 layer must behave nicely and friendly to make sure all flows will 175 only obtain its own faired share of resource, and no one could 176 consume more and no one could be starved. 178 1.2. TCP Solution Analysis 180 As a most popular and widely used transport technology, TCP traffic 181 is dominating in Internet from the born of Internet. It is important 182 to analyze the TCP. This section will brief the TCP, its variation, 183 and some key factors. 185 1.2.1. TCP Overview and Evolution 187 The major functionalities of TCP are flow control and congestion 188 control. 190 The flow control is based on the sliding window algorithm. In each 191 TCP segment, the receiver specifies in the receive window field the 192 amount of additionally received data (in bytes) that it is willing to 193 buffer for the connection. The sending host can send only up to that 194 amount of data before it must wait for an acknowledgment and window 195 update from the receiving host. 197 The congestion control is algorithms to prevent the hosts and network 198 device fall into congestion state while trying to achieve the maximum 199 throughput. There are many algorithm variations developed so far. 201 All congestion control will use some congestion detection scheme to 202 detect the congestion and adjust the rate of source to avoid the 203 congestion. 205 No matter what congestion control algorithm is used, traditionally, 206 all TCP solutions are pursuing three targets, high efficiency in 207 bandwidth utilization, high fairness in bandwidth allocation, and 208 fast convergence to the equilibrium state. [TCP_Targets] 210 Recently, with the growth of new TCP applications in data center, 211 more and more solutions were proposed to solve bufferbloat, incast 212 problems typically happened in data center. These solutions include 213 DCTCP, PIE, CoDel, FQ-CoDel, etc. In addition to the three 214 traditional targets mentioned above, these solutions have another 215 target which is to minimize the latency. 217 1.2.2. TCP Solution Variants 219 There are many TCP variants and optimization solutions since TCP was 220 introduced 40 years ago. We have collected major TCP variants 221 including typical traditional solution and some new solutions 222 proposed recently. 224 The traditional solutions: 225 These solutions are implemented on host only. They use different 226 congestion detection and inference mechanism, either based on 227 packet loss, RTT or both, to dynamically adjust the TCP window to 228 do the congestion control, such as: TCP-reno [RFC2581], TCP-vegas 229 [TCP-vegas], TCP-cubic [TCP-cubic], TCP-compound 230 [I-D.sridharan-tcpm-ctcp], TIMELY [TIMELY], etc 232 The explicit rate solutions: 233 These solutions do not use the traditional black box mechanism 234 executed at host to infer the TCP congestion status, instead, they 235 rely on the rate calculation on routers to let host adjust 236 accordingly. Both network devices and hosts must be changed. 237 Typical solutions are: XCP [I-D.falk-xcp-spec], RCP [RCP]. Note, 238 we put XCP and RCP as TCP here is referring to the scenario when 239 XCP and RCP are used with TCP 241 The AQM solutions: 242 These solutions use AQM (Active Queue Management) techniques on 243 routers to control the buffer size, thus control the congestion 244 and minimize the latency indirectly. Both network devices and 245 hosts must be changed. They include: DCTCP [I-D.ietf-tcpm-dctcp], 246 PIE [I-D.ietf-aqm-pie], CoDel [I-D.ietf-aqm-codel], FQ-CoDel 247 [I-D.ietf-aqm-fq-codel], etc. 249 The new concept solutions: 250 Unlike above categories, these solutions use completely new 251 concepts and methods to either accurately calculate, or figure out 252 the optimized rate and latency of TCP, such as: PERC [PERC], BBR 253 [BBR], PCC [PCC], Fastpass [Fastpass], etc 255 1.2.3. Throughput Constraint 257 For the traditional TCP optimization solutions, the efficiency target 258 is to obtain the high bandwidth utilization as much as possible to 259 approach the link capacity. The link utilization is defined as the 260 total throughput of all TCP flows on a network device to the network 261 bandwidth for links. 263 For individual TCP flow, its actual throughput is not guaranteed at 264 all. It depends on many factors, such as TCP algorithm used, the 265 number of TCP flows sharing the same link, host CPU power, network 266 device congestion status, delay in transmission, etc. 268 For traditional TCP, the real throughput for a flow is limited by 269 three factors: The 1st one is the available maximum throughput at the 270 physical layer, accounting for maximum theoretical bandwidth, network 271 load, buffering configuration, maximum segment size, signal strength, 272 etc; The another is related to congestion control algorithm; The 3rd 273 is related to the TCP fairness principle. Below we will analyze the 274 last two factors. 276 1.2.3.1. By Algorithm 278 No matter what algorithm is used, The TCP throughput is always 279 related to some flow and network characteristics, such as the RTT 280 (Round Trip Time) and PLR (packet loss ratio). For example, TCP-reno 281 throughput is shown in the formula (3) in [Reno_throughput]; And TCP- 282 cubic throughput is expressed in formula (21) in [Cubic_throughput]. 284 This limit will prevent the link capacity to be utilized by all TCP 285 flows. Each TCP flow may only get a few portion of the link 286 bandwidth as the real throughput for application. Even there is one 287 TCP flow in a link, the throughput for the TCP could be way below the 288 link capacity for a network which RTT and PLR are high. 290 1.2.3.2. By Fairness Principle 292 TCP fairness is a de facto principle for all TCP solutions. By this 293 rule, each router will process all TCP flows equally and fairly to 294 allocate the required resource to all TCP flows. Different Fair 295 Queuing algorithms were used, such as Packet based Round Robin, Core- 296 Stateless Fair Queuing(CSFQ), WFQ, etc. The targets of all 297 algorithms are to reach the so called max-min fairness [Fairness] of 298 TCP in terms of bandwidth. 300 TCP fairness played an important and critical role in saving internet 301 from collapse caused by congestions since TCP was introduced. 303 The analysis [RCP] on page 35 has given the formula of the fair share 304 rate at bottleneck routers, the rate or throughput is capped for 305 applications which required bandwidth are not satisfied under the 306 rule of fairness. 308 1.2.4. Latency Constraint 310 TCP fairness will not process some TCP flows differently with others, 311 or there is no TCP micro-flow handling. 313 As described above, for the traditional solutions and explicit rate 314 solution, the latency is not considered as a target, thus no latency 315 guarantee at all. 317 For AQM solutions and some new concept solutions which try to control 318 the buffer bloat or flow latency, it can only provide the statistic 319 bounded latency for all TCP flows. The latency is related to the 320 queue size and other factors. And the real latency for specific 321 flow(s) is not deterministic. It could be very small or pretty large 322 due to the long tail effect if the flow is blocked by other slower 323 TCP flows. 325 1.2.5. Summary of TCP Solution 327 The bandwidth and latency can hardly be satisfied simultaneously 328 without micro flow handling and management. While trying to get 329 higher bandwidth, it may lead to more queued packet in router and 330 result in longer latency. While approaching shorter latency, it may 331 cause the queue under run, and lead to the lower bandwidth. 333 As a summary, to support some special TCP applications that are very 334 sensitive to bandwidth and/or latency, we need to handle those TCP 335 flows differently with others, and the TCP fairness must be relaxed 336 for these scenarios. 338 It must be noted that the fairness based transport service could 339 satisfy most of the applications, and it is the most efficient and 340 economical way for hardware implementation and the network bandwidth 341 efficiency. 343 When providing some TCP flows with differentiated service, the 344 traditional transport service must be able to coexist with the new 345 service. The resource partitioning between different service is a 346 operation and management job for service provider. 348 1.3. Other Solution Analysis 350 DiffServ 351 DiffServ [DiffServ] or Differentiated services is a network 352 architecture that specifies a simple, scalable and coarse-grained 353 mechanism for classifying and managing network traffic and 354 providing QoS on modern IP networks. DiffServ is designed to 355 support the QoS of aggregated traffic and normally is deployed in 356 Service Provider networks. End user application cannot directly 357 use DiffServ. 359 IntServ 360 IntServ [IntServ] or integrated services specifies more fine- 361 grained QoS, which is often contrasted with DiffServ's coarse- 362 grained control system. IntServ definitely can support the 363 applications requiring special QoS guarantee if it is deployed in 364 a network, supported by Host OS and integrated with application. 365 However, IntServ works on a small-scale only. When you scale up 366 the network, it is difficult to keep track of all of the 367 reservations and session states. Thus, IntServ is not scalable. 368 Another problem of IntServ is it is not application driven, 369 tedious provisioning cross different network must be done earlier. 370 The provisioning is slow and hard to maintain. 372 MPLS-TE 373 MPLS-TE can provide aggregated QoS or fine-grained QoS service for 374 different class of traffic. Similar to DiffServ, MPLS-TE is 375 majorly used for service providers network. It requires extra 376 protocol sets like LDP, MPLS-TE, etc to operate. It is not 377 practical to extend MPLS-TE to end user's desktop. 379 1.4. New approach 381 1.4.1. IP Transport with quality of service 383 Semiconductor chip technology has advanced a lot for last decades, 384 the widely used network process can not only forward the packet in 385 line speed, but also support fast packet processing for other 386 features, such as Qos for DiffServ/MPLS, Access Control List (ACL), 387 fire wall, Deep Packet Inspection (DIP), etc. To treat some TCP/IP 388 flows differently with others and give them specified resource are 389 feasible now by using network processor. 391 Network processor is also able to do the general process to handle 392 the simple control message for traffic management, such as signaling 393 for hardware programming, congestion state report, OAM, etc. 395 This document proposes a mechanism to provide the capability of IP 396 network to support the transport layer with quality of service. The 397 solution is based on the QoS implemented in network processor. the 398 proposal of the document is composed of two parts: 400 1. Control plane, it explains a transport control sub-layer for IP, 401 the details of control mechanism. 403 2. Data plane, the realization of QoS in data forwarding, QoS and 404 error handling. 406 1.4.2. Design targets 408 The new transport service is expected to satisfy following criteria: 410 1. End user or application can directly use and control the new 411 service 413 2. The new service can coexist with the current transport service 414 and is backward compatible. 416 3. The service provider can manage the new service. 418 4. Performance and scalability targets of new service are practical 419 for vendors to achieve. 421 5. The new service is transport agnostic. Both TCP, UDP and other 422 transport protocols on top of IP can use it 424 1.4.3. Scope and assumption 426 The initial aim is to propose a solution for IPv6. 428 To limit the scope of the document and simplify the design and 429 solution, the following constraints are given. 431 1. The transport with QoS is aimed to be supplementary to the 432 regular transport service. At the current situation, It is 433 targeted for the applications that are bandwidth and/or latency 434 sensitive. It is not intended to replace the TCP variants that 435 have been proved to be efficient and successful for current 436 applications. 438 2. The new service is limited within one administrative domain, even 439 it does not exclude the possibilities to extend the mechanism for 440 inter-domain scenarios. Thus, the security and other inter- 441 domain requirements are not critical. The basic security is good 442 enough, the inter-domain SLA, accounting and other issues are not 443 discussed. 445 3. Due to high bandwidth requirement of new service for individual 446 flow, the total number of the flows with the new service cannot 447 be high for a port, or a system. From another point of view, the 448 new service is targeted for the application that really needs it, 449 the number of supported applications/users are under controlled 450 and cannot be unlimited. So, the scalability requirement for the 451 new service is limited. 453 4. The new service must coexist with the regular transport service 454 in the same hardware, and backward compatible. Also, a transport 455 flow can switch without the service interruption between the 456 regular transport support and new service. 458 2. Terminology 460 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 461 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 462 document are to be interpreted as described in RFC 2119 [RFC2119]. 464 2.1. Definitions 466 E2E 467 End-to-end 469 EH 470 IPv6 Extension Header or Extension Option 472 QoS 473 Quality of Service 475 OAM 476 Operation and Management 478 In-band Signaling 479 In telecommunications, in-band signaling is the sending of 480 control information within the same band or channel used for 481 voice or video. 483 Out-of-band Signaling 484 out-of-band signaling is that the control information sent over 485 a different channel, or even over a separate network. 487 IP flow 488 For non-IPSec, a IP flow is identified by the source, 489 destination IP address, the protocol number, the source and 490 destination port number. 492 IP path 493 A IP path is the route that IP flow will traverse. It could be 494 the shortest path determined by routing protocols (IGP or BPG), 495 or the explicit path decided by another management entity, such 496 as a central controller, or Path Computation Element (PCE) 497 Communication Protocol (PCEP), etc 499 QoS channel 500 A forwarding channel that the QoS is guaranteed, it provides an 501 additional QoS service to the normal IP forwarding. A QoS 502 channel can be used for one or multiple IP flows depends on the 503 granularity of in-band signaling. 505 Cir 506 Committed Information Rate, this is the guaranteed bandwidth 508 Pir 509 Peak Information Rate. this is the up limit bandwidth. Whether 510 a flow can reach the PIR depends on the implementation. To use 511 resource more efficiently, the system normally does not 512 guarantee the PIR, but allow the sharing of resource between 513 flows. 515 HbH-EH 516 IPv6 Hop-by-Hop Extension Header 518 Dst-EH 519 IPv6 Destination Extension Header 521 HbH-EH-aware node 522 Network nodes that are configured to process the IPv6 Hop-by- 523 Hop Extension Header 525 3. Control plane 526 3.1. Sub-layer in IP for transport control 528 In order to provide some new features for the upper layer above IP, 529 it is very useful to introduce an additional sub-layer, Transport 530 Control, between layer 3 (IP) and layer 4 (TCP/UDP). The new layer 531 belongs to the IP, and is present only when the system needs to 532 provide extra control for the upper layer, in addition to the normal 533 IP forwarding. Fig 1. illustrates a new stack with the sub-layer. 535 +=========================+ 536 | APP | 537 +=========================+ 538 | TCP/UDP | 539 +=========================+ 540 | Transport Ctl | 541 +-+-+-+-+-+-+-+-+-+-+-+-+-+ 542 | IP | 543 +=========================+ 544 | Network Access | 545 +=========================+ 547 Figure 1: The new stack with a sub-layer in Layer 3 549 The new sub-layer is always bound with IP layer and can provide a 550 support of the features for upper layer, such as: 552 In-band Signaling 553 The IP header with the new sub-layer can carry the signaling 554 information for the devices on the IP path. The information may 555 include all QoS related parameters used for hardware programming. 557 Congestion control 558 The congestion state in each device on the path can be detected 559 and notified to the source of flows by the sub-layer; The dynamic 560 congestion control instruction can also be carried by the sub- 561 layer and examined by network devices on the IP path. 563 IP Path OAM 564 The OAM instruction can be carried in the sub-layer, and the OAM 565 state can be notified to the source of flows by the sub-layer. 566 The OAM includes the path and device property detection, QoS 567 forwarding diagnosis and report. 569 IPv6 can realize the sub-layer easily by the IPv6 extension header 570 [RFC8200]. 572 IPv4 could use the IP option for the purpose of the sub-layer. But 573 due to the limit size of the IP option, the functionalities, 574 scalability of the layer is restricted. 576 The document will focus on the solution for IPv6 by using different 577 IPv6 extension header. 579 The control plane of the propose comprises of IP in-band signaling, 580 and the detailed control mechanisms. 582 3.2. IP In-band signaling 584 There is no definition for IP in-band signaling. From the point of 585 view of similarity to traditional telecommunication technology, the 586 In-band signaling for IP is that the IP control messages are sharing 587 some common header information as the data packet. 589 In this document, we introduce three types of "in-band signaling" for 590 different signaling granularity: 592 Flow level In-band Signaling 593 The control message and data packet share the same flow 594 identification. The flow identification could be 5 tuples for non 595 IPSec IPv6 packet: the source, destination IP address, protocol 596 number, source and destination port number, and also could be 3 597 tuples for IPSec IPv6 packet: the source, destination IP address 598 and the flow label. For the flow level in-band signaling, the 599 signaling is for the individual IP flow, and there is no 600 aggregation at all. 602 Address level In-band Signaling 603 The control message and data packet share the same source, 604 destination IP address, but with different protocol number. This 605 is the scenario that the signaling is for the aggregated flows 606 which have the same source, destination address. i.e, All TCP/UDP 607 flows between the same client and same server (only one address 608 for client and one for server) 610 Transport level In-band Signaling 611 The control message and data packet share the same source, 612 destination IP address, protocol number, but with different source 613 or destination port number (non-IPSec) or different flow label 614 (IPSec). This is the situation that the signaling is for the 615 aggregated TCP or UDP flows that started and terminated at the 616 same IP addresses. 618 Using In-band signaling, the control message can be embedded into any 619 data packet, this can bring up some advantages that other methods can 620 hardly provide: 622 Diagnosis 623 The in-band signaling message takes the same path, same hops, same 624 processing at each hop as the data packet, this will make the 625 diagnosis for both signaling and data path easier. 627 Simplicity 628 The in-band signaling message is forwarded with the normal data 629 packet, it does not need to run a separate protocol. This will 630 dramatically reduce the complexity of the control. 632 Performance and scalability 633 Due to the simplicity of in-band signaling for control, it is 634 easier to provide a better performance and scalability for a new 635 future. 637 Note, the requirement of IP in-band signaling was proposed before by 638 John Harper [I-D.harper-inband-signalling-requirements]. And the in- 639 band QoS signaling for IPv6 was simply discussed in 640 [I-D.roberts-inband-qos-ipv6]. Unfortunately, both works did not 641 continue. 643 This document not only gives detailed solution for in-band signaling, 644 but also try to address issues raised for the previous proposal, such 645 as security, scalability and performance. Finally, experiments with 646 proprietary hardware and chips are given in a presentation. 648 3.3. Control mechanism 650 The in-band signaling must be cooperated with a control method to 651 achieve the QoS control. There are two categories of control, one is 652 the closed-loop control and another is the open-loop control. 654 1. Closed-loop control is that the in-band signaling is sent in one 655 direction and the feedback will return in the reverse direction. 656 For example, the closed-loop control can be achieved by inserting 657 the signaling information into a data packet sent in one 658 direction, and the feedback information is carried in the data 659 packet in reverse direction. The transport service with bi- 660 direction data flow can use this mechanism, such as TCP and 661 point-to-point UDP. In closed-loop control, a signaling message 662 in one direction is processed at each router on the path. When 663 the signaling message reaches the destination, the signaling 664 message is processed by the protocol stack in the host, and the 665 report information is generated. The report information is then 666 embedded into the flow data packet in the reverse direction and 667 return to the host of the signaling source. 669 2. Open-loop control is that the in-band signaling is sent 670 periodically in one direction without any feedback. The 671 transport service with uni-direction data flow can use this 672 mechanism, such as multicast by UDP. The transport service with 673 bi-directional data flow can also use this mechanism when the 674 simplicity of the control is wanted, i.e. no control feedback 675 needed. 677 For both closed-loop and open-loop control, the signaling message for 678 one direction is for the QoS programming for the direction. For 679 example, the TCP-SYN or TCP data packet from client to server can 680 carry the in-band signaling message to program the QoS for the 681 direction of client to service. TCP-SYNACK or TCP data packet from 682 server to client can carry the in-band signaling message to program 683 the QoS for the server to client direction 685 Due to the nature that symmetric IP path between any source and 686 destination cannot be guaranteed, in closed-loop control, the 687 feedback information may take the different path as the in-band 688 signaling path. The in-band signaling must not depend on the 689 feedback information to accomplish the signaling work, such as the 690 programming of hardware. This is one of the difference between in- 691 band signaling and RSVP protocol. 693 For this document, we will only discuss the detailed mechanism for 694 closed-loop control for TCP. 696 3.4. IPv6 Approach 698 The IPv6 In-band signaling could be realized by using the IPv6 699 extension header. 701 There are two types of extension header used for the purpose of 702 transport QoS control, one is the hop-by-hop EH (HbH-EH) and another 703 is the destination EH (Dst-EH). 705 The HbH-EH may be examined and processed by the nodes that are 706 explicitly configured to do so [RFC8200]. We call this nodes as HbH- 707 EH-aware nodes in document below. It is used to carry the QoS 708 requirement for dedicated flow(s) and then the information is 709 intercepted by HbH-EH-aware nodes on the path to program hardware 710 accordingly. 712 The destination EH will only be examined and processed by the 713 destination device that is associated with the destination IPv6 714 address in the IPv6 header. This EH is used to send the QoS related 715 report information directly to the source of the signaling at other 716 end. 718 3.4.1. Basic Control Scenarios for TCP 720 The finest grained QoS for TCP is flow level, this document will only 721 focus on the solution of the flow level in-band signaling and its 722 data plane. Other two types, address level and transport level QoS 723 for TCP are briefly discussed in section 5.3. 725 The feature of TCP with flow level QoS comprises following control 726 scenarios: 728 1. Setup: The setup is combined with the TCP 3-hand shaking, or any 729 two directional TCP packets. When used with TCP 3-hand shaking, 730 the 1st signaling embedded into HbH-EH is sent with TCP-SYN. It 731 will be processed at HbH-EH-aware nodes on the path from source 732 to destination. The signaling message includes the QoS 733 requirements, such as max/min bandwidth, burst size, the latency, 734 and the setup state. The setup state message is updated at HbH- 735 EH-aware nodes to include the QoS programming and provisioning 736 result and the necessary hardware reference information for IP 737 forwarding with QoS. The 2nd signaling message is the TCP-SYNACK 738 from server side, it includes the setup report message encoded as 739 the Dst-EH. The setup report message is from the 1st TCP-SYN 740 which represents the setup results on all HbH-EH-aware nodes on 741 the path. The setup can even be started after TCP is established 742 whenever the QoS service is required. 744 2. Dynamic control: this scenario is for the situation that previous 745 QoS programming must be refreshed, modified or re-programmed. 746 Normally, the signaling message can be embedded into HbH-EH for 747 any TCP data packet or TCP-ACK packet. There are couple cases 748 that the dynamic control is needed. 750 HW state refreshing 751 The HW state for QoS programming is data driven (see 752 Section 4.1 for details). Its state will be refreshed if 753 there is a data packet received. If there is no data received 754 for a pre-configured time, the HW programming will be erased 755 and the resource will be released. 757 HW programming modification 758 The HW QoS parameters can be modified if a new in-band 759 signaling message is received and the embedded parameters are 760 different with the old one that was used to program the HW. 761 Section 3.4.2 will explain more about this scenario. 763 HW programming repairing 764 The IP path may be changed due to rerouting, link or node 765 failures. This may result in the HW QoS programming failure. 766 To repair any QoS programming failure, the new in-band 767 signaling message can be embedded into any data packet and 768 sent to the destination. All hops on the new path will be 769 reprogramed with the QoS parameters. Section 4.4 has more 770 detailed discussion. 772 3. Congestion Control: For TCP protocol, if IP layer can provide a 773 certain level of quality service guarantee, the congestion 774 control algorithm will be impacted a lot. As for what is the new 775 congestion control, it depends on the quality service 776 implementation in hardware and the behavior of the application. 777 This is simply discussed in section 5.2. 779 3.4.2. Details of In-band Signaling for TCP 781 This document introduces following type of message for in-band 782 signaling and associated data forwarding, the detailed format of 783 messages is expressed in Section 6, 785 o Setup: This is for the setup of QoS channel through the IP path. 787 o Bandwidth: This is the required bandwidth for the QoS channel. It 788 has minimum (CIR) and maximum bandwidth (PIR). 790 o Latency: This is the required latency for the QoS channel, it is 791 the bounded latency for each hop on the path. This is not the end 792 to end latency. 794 o Burst: This is the required burst for the QoS channel, it is the 795 maximum burst size. 797 o Authentication: This is the security message for a in-band 798 signaling. 800 o OAM: This is the Operation and Management message for the QoS 801 channel. 803 o Setup State Report: This is the state report of a setup message. 805 o Forwarding State: This is the forwarding state message used for 806 data packet. 808 o Forwarding State Report: This is the forwarding state report of a 809 QoS channel. 811 There are three scenarios of QoS signaling for TCP session setup with 812 QoS 814 1. Upstream: This is for the direction of client to server. A 815 application decides to open a TCP session with upstream QoS (for 816 uploading), it will call TCP API to open a socket and connect to 817 a server. The client host will form a TCP SYN packet with the 818 HbH-EH in the IPv6 header. The EH includes Setup message and 819 Bandwidth message, and optionally Latency, Burst, Authentication 820 and OAM messages. The packet is forwarded at each hop. Each 821 HbH-EH-aware nodes will process the signaling message to finish 822 the following tasks before forwarding the packet to next hop: 824 * Retrieve the QoS parameters to program the Hardware, it 825 includes: FL, Time, Bandwidth, Latency, Burst 827 * Update the field in the EH, it includes: Hop_number, 828 Total_latency, and possibly Mapping Index List 830 When the server receives the TCP SYN, the Host kernel will also 831 check the HbH-EH while punting the TCP packet to the TCP stack 832 for processing. If the HbH-EH is present and the Report bit is 833 set, the Host kernel must form a new Setup State Report message, 834 all fields in the message must be copied from the Setup message 835 in the HbH-EH. When the TCP stack is sending the TCP-SYNACK to 836 the client, the kernel must add the Setup State Report message as 837 a Dst-EH in the IPv6 header. After this, the IPv6 packet is 838 complete and can be sent to wire; When the client receives the 839 TCP-SYNACK, the Host kernel will check the Dst-EH while punting 840 the TCP packet to the TCP stack for processing. If the Dst-EH is 841 present and the Setup State Report message is valid, the kernel 842 must read the Setup State Report message. Depending on the setup 843 state, the client will operate according to description in 844 section 5.1 846 2. Downstream: This is for the direction of server to client. A 847 application decides to open a TCP session with downstream QoS 848 (for downloading), it will call TCP API to open a socket and 849 connect to a server. The client host will form a TCP SYN packet 850 with the Dst-EH in the IPv6 header. The EH includes Bandwidth 851 message, and optionally Latency, Burst messages. The packet is 852 forwarded at each hop. Each hop will not process the Dst-EH. 853 When the server receives the TCP SYN, the Host kernel will check 854 the Dst-EH while punting the TCP packet to the TCP stack for 855 processing. If the Dst-EH is present, the Host kernel will 856 retrieve the QoS requirement information from Bandwidth, Latency 857 and Burst message, and check the QoS policy for the user. If the 858 user is allowed to get the service with the expected QoS, the 859 server will form a Setup message similar to the case of client to 860 server, and add it as the HbH-EH in the IPv6 header, and send the 861 TCP-SYNACK to client. Each HbH-EH-aware nodes on the path from 862 server to client will process the message similar to the case of 863 client to server. After the client receives the TCP-SYNACK, The 864 client will send the Setup State Report message to server as the 865 Dst-EH in the TCP-ACK. Finally the server receives the TC-ACK 866 and Setup State Report message, it can send the data to the 867 established session according to the pre-negotiated QoS 868 requirements. 870 3. Bi-direction: This is the case that the client wants to setup a 871 session with bi-direction QoS guarantee. The detailed operations 872 are actually a combination of Upstream and Downstream described 873 above. 875 After a QoS channel is setup, the in-band signaling message can still 876 be exchanged between two hosts, there are two scenarios for this. 878 1. Modify QoS on the fly: When the pre-set QoS parameters need to be 879 adjusted, the application at source host can re-send a new in- 880 band signaling message, the message can be embedded into any TCP 881 packet as a IPv6 HbH-EH. The QoS modification should not impact 882 the established TCP session and programmed QoS service. Thus, 883 there is no service impcted during the QoS modification. 884 Depending on the hardware performance, the signaling message can 885 be sent with TCP packet with different data size. If the 886 performance is high, the signaling message can be sent with any 887 TCP packet; otherwise, the signaling message should be sent with 888 small size TCP packet or zero-size TCP packet (such as TCP ACK). 889 Modification of QoS on the fly is a very critical feature for the 890 so called "Application adaptive QoS transport service". With 891 this service, an application (or the proxy from a service 892 provider) could setup an optimized CIR for different stage of 893 application for the economical and efficient purpose. For 894 example, in the transport of compressed video, the I-frame has 895 big size and cannot be lost, but P-frame and B-frame both have 896 smaller size and can tolerate some loss. There are much more 897 P-frame and B-frame than I-frame in videos with smooth changes 898 and variations in images [I-D.han-iccrg-arvr-transport-problem]. 899 Based on this characteristics, application can request a 900 relatively small CIR for the time of P-frame and P-frame, and 901 request a big CIR for the time of I-frame. 903 2. Repairing of the QoS channel: This is the case the QoS channel 904 was broken and need to be repaired, see section 4.4. 906 3.5. Key Messages and Parameters in Control Protocol 908 The detailed message format is described in the section 6, the 909 detailed explanation of key messages and parameters are below: 911 3.5.1. Setup and Setup State Report messages 913 Setup is the message used for following purpose: 915 o Setup the QoS channel for a TCP when the TCP session is 916 establishing. 918 o Dynamic Control of the QoS channel for a established TCP session. 919 See section 3.4.1 921 Setup message is intended to program the hardware for QoS channel on 922 the IP path from the source to the destination expressed in IPv6 923 header. It is embedded as the HbH-EH in an appropriate TCP packet 924 and will be processed at each HbH-EH-aware node. For the simplicity, 925 performance and scalability purpose, we can configure some hop to do 926 the processing and some hops do not. For different QoS requirement 927 and scenarios, different criteria can be used for the configuration 928 of the hop to be HbH-EH-aware node, below are some factor to 929 consider: 931 o Reserved bandwidth is required: The throttle router is the 932 critical point to be configured to process the hop-by-hop EH for 933 the bandwidth reservation. The throttle router is the device that 934 a interested TCP session cannot get the enough bandwidth to 935 support its application. The regular throttle routers include the 936 BRAS (broadband remote access server) in broadband access network, 937 the PGW (PDN Gateway) in LTE network, the TOR (Top of Rack) in 938 data center. In more general case, any routers which aggregate 939 traffic may become as a throttle router. Moreover, the direction 940 of congestion must be considered. Normally, the congestion 941 happens on the direction that more than one flows from multiple 942 ingress links are aggregated and sent to one egress link. For 943 other devices that the interested TCP session can get the enough 944 bandwidth do not need to process the hop-by-hop EH. 946 o Bounded latency is required: In theory, each router and switch 947 could contribute some delay to the end-to-end latency, but the 948 throttle router will contribute more than non-throttle routers, 949 and slow device will contribute more than fast device. We can use 950 OAM to detect the latency contribution in a network, and configure 951 those worst-cast devices to process the HbH-EH. 953 Setup State Report message is the message sent from the destination 954 host to the source host (from the point of view of the Setup 955 message). The message is embedded into the Dst-EH in any data 956 packet. The Setup State Report in the message is just a copy from 957 the Setup message received at the destination host for a typical TCP 958 session. The message is used at the source host to forward the 959 packet later and to do the congestion control. 961 3.5.2. OAM 963 OAM is a special in-band signaling message used for detection and 964 diagnosis. It can be used before and after a QoS channel is 965 established. Before a QoS channel is established, OAM message can be 966 added as a HbH-EH to any IPv6 packet and used to detect: 968 o IP path properties: Total hop number that is HbH-EH-aware node; 969 The IP address of each HbH-EH-aware node. 971 o Static properties at each HbH-EH-aware node: Protocol version; 972 Supported Flow identifying methods; Mapping index size; Supported 973 configuration range of bandwidth, latency, forwarding QoS state 974 time. 976 o Financial properties at each HbH-EH-aware node: Unit price for 977 bandwidth; Unit price for service duration; Price for different 978 latency. 980 After a QoS channel is established, OAM message can also be added as 981 a HbH-EH to any IPv6 packet and used to detect and diagnose failures: 983 o IP path dynamic properties: Total end to end latency 985 o Dynamic properties at ach HbH-EH-aware node: Queue size; Remained 986 bandwidth; Dropped packet number by different reasons. 988 o The detailed QoS forwarding failure reason. 990 3.5.3. Forwarding State and Forwarding State Report messages 992 Forwarding State and Forwarding State Report messages are used for 993 data plane, See section 4.2. 995 3.5.4. Flow Identifying Methods 997 This is a parameter to program the HW for the flow identifying 998 method. It is used for the QoS granularity definition and flow 999 identification for QoS process. The QoS is enforced for a group of 1000 flows or a dedicated flow that can be identified by the same flow 1001 identification. The QoS granularity is determined by the flow 1002 identification method during the setup and packet forwarding process. 1003 There are three levels of QoS granularities: Flow level, Address 1004 level and transport level. Each level of QoS granularity is realized 1005 by corresponding in-band signaling. The document focus on the flow 1006 level in-band signaling, other two level in-band signaling are 1007 discussed in the section 5.3. 1009 There are two ways for the flow identifying method. One is by the 1010 tuples in IP header, another is by a local significant number ( see 1011 mapping index) generated and maintained in a router. When "Mapping 1012 Index Size" (Mis) is zero, it means the "Flow identification method" 1013 (FI) is used for both control plane and data plane. When "Mis" is 1014 not zero, it means "FI" is only used in signaling, and the data plane 1015 will only use the "Mapping Index". 1017 There are four types for "Flow identification method": 1019 1. Individual Flow: Non-IPSec case: flow is identified by source and 1020 destination address, source and destination port number, and 1021 protocol number; IPSec case: flow is identified by source and 1022 destination address, flow label. For both case, FI = 0; the 1023 associated QoS is flow level, and QoS is guaranteed for a 1024 dedicated IP flow. 1026 2. TCP flows: flow is identified by source and destination address, 1027 and TCP protocol number. The associated QoS is transport level, 1028 and QoS is guaranteed for TCP flows that have the same source and 1029 destination address. For this case, FI=1. 1031 3. UDP flows: flow is identified by source and destination address, 1032 and UDP protocol number. The associated QoS is transport level, 1033 and QoS is guaranteed for UDP flows that have the same source and 1034 destination address. For this case, FI=2. 1036 4. All flows: flow is identified by source and destination address. 1037 The associated QoS is address level, and QoS is guaranteed for 1038 all IP flows that have the same source and destination address. 1039 For this case, FI=3 1041 The use of local generated number to identify flow is to speed up the 1042 flow lookup and QoS process for data plane. The number could be the 1043 MPLS label or a local tag for a MPLS capable router. The difference 1044 between this method and the MPLS switch is that there is no MPLS LDP 1045 protocol running and the IP packet does not need to be encapsulated 1046 as MPLS packet at the source host. When the MPLS label is used, the 1047 "Mapping Index Size" is 20 bits. 1049 3.5.5. Hop Number 1051 This is a parameter for the total number of hop that is HbH-EH-aware 1052 node on the path. it is the field "Hop_num" in Setup message. It is 1053 used to locate the bit position for "Setup State" and the "Mapping 1054 Index" in "Mapping Index List". The value of "Hop_num" must be 1055 decremented at each hop. And at the receive host of the in-band 1056 signaling, the Hop_num must be zero. 1058 The source host must know the exact hop number, and setup the initial 1059 value in the Setup message. The exact hop number can be detected by 1060 the OAM message. 1062 3.5.6. Mapping Index, Size and Mapping Index List 1064 Mapping Index is the local significant number generated and 1065 maintained in a router, and The "Mapping Index List" is just a list 1066 of "Mapping Index" for all hops that are HbH-EH-aware nodes on the IP 1067 path. 1069 Mapping Index Size is the size for each mapping index in the Mapping 1070 Index List. The source host must know Mapping Index Size, and setup 1071 the initial value in the Setup message. The exact Mapping Index Size 1072 can be detected by the OAM message. 1074 When a router receives a HbH-EH, it may generate a mapping index for 1075 the flow(s) that is defined by the Flow Identifying Method in "FL". 1076 Then the router must attach the mapping index value to the end of the 1077 Mapping Index List. After the packet reaches the destination host, 1078 the Mapping Index List will be that the 1st router's mapping index as 1079 the list header, and the last router's mapping index as the list 1080 tail. 1082 3.5.7. QoS State and life of Time 1084 After the chip is programmed for a QoS, a QoS state is created. The 1085 QoS state life is determined by the "Time" in the Setup message. 1086 Whenever there is a packet processed by a QoS state, the associated 1087 timer for the QoS state is reset. If the timer of a QoS state is 1088 expired, the QoS state will be erased and the associated resource 1089 will be released. 1091 In order to keep the QoS state active, a application at source host 1092 can send some zero size of data to refresh the QoS state. 1094 When the Time is set to zero, it means the life of the QoS State will 1095 be kept until the de-programming message is received. 1097 3.5.8. Authentication 1099 The in-band signaling is designed to have a basic security mechanism 1100 to protect the integrity of a signaling message. The Authentication 1101 message is to attach to a signaling message, the source host 1102 calculates the harsh value of a key and all invariable part of a 1103 signaling message (Setup message: ver, FI, R, Mis, P, Time; Bandwidth 1104 message, Latency message, Burst message). The key is only known to 1105 the hosts and all HbH-EH-aware nodes. The securely distribution of 1106 the key is out the scope of the document 1108 4. Data plane 1110 To support the QoS feature, there are couple of important 1111 requirements and schemes for implementations. These include the 1112 basic capability for the hardware, the scheme for the data 1113 forwarding, QoS processing, state report, etc. 1115 Section 4.1 will talk about the basic capability for data plane, and 1116 section 4.2 will discuss the messages used for data plane after the 1117 QoS channel is established. 1119 4.1. Basic Capability 1121 The document only proposes the protocol used for control, and it is 1122 independent of the implementation of the system. However, to achieve 1123 the satisfactory targets for performance and scalability, the 1124 protocol must be cooperated with capable hardware to provide the 1125 desired fine-grained QoS for different transport. 1127 In our experiment to implement the feature for TCP, we used a network 1128 processor with traffic management feature. The traffic management 1129 can provide the fine-grained QoS for any configured flow(s). 1130 Following capabilities are RECOMMENDED: 1132 1. The in-banding signaling is processed in network processor 1133 without punting to controller CPU for help 1135 2. The QoS forwarding state is kept and maintained in network 1136 processor without the involvement from controller CPU. 1138 3. The QoS state has a life of a pre-configured time and will be 1139 automatically deleted if there is no data packet processed by 1140 that QoS state. The timer can be changed on the fly. 1142 4. The QoS forwarding does not need to be done at the controller 1143 CPU, or so called slow path. It is at the same hardware as the 1144 normal IP forwarding. For any IP packet, the QoS forwarding is 1145 executed first. Normal forwarding will be executed if there is 1146 no QoS state associated with the identification of the flow. 1148 5. The QoS forwarding and normal forwarding can be switched on the 1149 fly. 1151 4.2. Forwarding State and Forwarding State Report 1153 After the QoS is programmed by the in-band signaling, the specified 1154 IP flows can be processed and forwarded for the QoS requirement. 1155 There are two ways for host to use the QoS channel for associated TCP 1156 session: 1158 1. Host directly send the IP packet without any changes to the 1159 packet, this is for the following cases: 1161 * The hardware was programmed to use the tuples in IP header as 1162 identification for QoS process (Mis = 0), and 1164 * The packet does not function to collect the QoS forwarding 1165 state on the path. 1167 2. Host add the Forward State message into a data packet's IP header 1168 as HbH-EH and send the packet, this is for the cases: 1170 * The hardware was programmed to use the mapping index as 1171 identification for QoS process (Mis != 0). 1173 * The hardware was programmed to use the tuples in IP header as 1174 identification for QoS process (Mis = 0), and the data packet 1175 functions to collect the QoS forwarding state on the path. 1176 This is the situation that host wants to detect the QoS 1177 forwarding state for the purpose of failure handling (See 1178 section 4.3). 1180 Forwarding State message format is shown in the Section 6.7. It is 1181 used to notify the mapping index and also update QoS forwarding state 1182 for the hops that are HbH-EH-aware nodes. 1184 After Forwarding State message is reaching the destination host, the 1185 host is supposed to retrieve it and form a Forwarding State Report 1186 message, and carry it in any data packet as the Dst-EH, then send to 1187 the host in the reverse direction. 1189 4.3. Flow Identification in Packet Forwarding 1191 Flow identification in Packet Forwarding is same as the QoS channel 1192 establishment by Setup message. It is to forward a packet with a 1193 specified QoS process if the packet is identified to be belonging to 1194 specified flow(s). 1196 There are two method used in data forwarding to identify flows: 1198 1. Hardware was programmed to use tuples in IP header implicitly. 1199 This is indicated by that the "Mis" is zero or the Mapping index 1200 is not used. When a packet is received, its tuples are looked up 1201 according to the value of "FI". If there is a QoS table has 1202 match for the packet, the packet will be processed by the QoS 1203 state found in the QoS table. This method does not need any EH 1204 added into the data packet unless the data packet function to 1205 collect the QoS forwarding state on the path. See section 4.3 1207 2. Hardware was programmed to use mapping index to identify flows. 1208 This is indicated by that the "Mis" is not zero. When a packet 1209 is received, the mapping index associated with the hop is 1210 retrieved and looked up for the QoS table. If it has match for 1211 the packet, the packet will be processed by the QoS state entry 1212 found in the QoS table. 1214 4.4. QoS Forwarding State Detection and Failure Handling 1216 QoS forwarding may be failed due to different reasons: 1218 1. Hardware failure in HbH-EH-aware node. 1220 2. IP path change due to link failure, node failure or routing 1221 changes; And the IP path change has impact to the HbH-EH-aware 1222 node. 1224 3. Network topology change; and the change leads to the changes of 1225 HbH-EH-aware nodes. 1227 Application may need to be aware of the service status of QoS 1228 guarantee when the application is using a TCP session with QoS. In 1229 order to provide such feature, the TCP stack in the source host can 1230 detect the QoS forwarding state by sending TCP data packet with 1231 Forwarding State message coded as HbH-EH. After the TCP data packet 1232 reaches the destination host, the host will copy the forwarding state 1233 into a Forwarding State Report message, and send it with another TCP 1234 packet (for example, TCP-ACK) in reverse direction to the source 1235 host. Thereafter, the source host can obtain the QoS forwarding 1236 state on all HbH-EH-aware nodes. 1238 A host can do the QoS forwarding state detection by three ways: on 1239 demand, periodically or constantly. 1241 After a host detects that there is QoS forwarding state failure, it 1242 can repair such failure by sending another Setup message embedded 1243 into a HbH-EH of any TCP packet. This repairing can handle all 1244 failure case mentioned above. 1246 If a failure cannot be repaired, host will be notified, and 1247 appropriate action can be taken, see section 5.1 1249 5. Other Issues 1251 Above document only covers the details for the QoS support of 1252 individual TCP session by using the flow level in-band signaling. 1253 Due to the extensive scope of in-band signaling, there are many other 1254 associated issues for IP transport control. Below lists some of 1255 them, and we only brief the solution but do not go to details. 1257 The details of each topic can be expressed in other drafts. 1259 5.1. User and Application driven 1261 The QoS transport service is initiated and controlled by end user's 1262 application. Following tasks are done in host 1264 1. The detailed QoS parameters in in-band signaling is set by end 1265 user application. New socket option must be added, the option is 1266 a place holder for QoS parameters (Setup, Bandwidth, etc), Setup 1267 State Report and Forwarding State Report messages. 1269 2. The Setup State Report and Forwarding State Report message 1270 received at host are processed by transport service in kernel. 1271 The Setup State Report message processed at host can result in 1272 the notification to the application whether the setup is 1273 successful. If the setup is successful, the application can 1274 start to use the socket having the QoS support; If the setup is 1275 failed, the application may have three choices: 1277 * Lower the QoS requirement and re-setup a new QoS channel with 1278 new in-band signaling message. 1280 * Use the TCP session as traditional transport without any QoS 1281 support. 1283 * Lookup the service provider for help to locate the problem in 1284 network. 1286 5.2. Traffic Management in Host 1288 In order to accommodate in-band signaling and the QoS transport 1289 service, the OS on a host muts be changed in traffic management 1290 related areas. There are two parts for traffic management to be 1291 changed, One is to manage traffic going out a host's shared links. 1292 Another is congestion control for TCP flows: 1294 1. The current traffic management in a host manages traffic from 1295 different TCP/UDP session going out host link(s), in the way 1296 similar to routers to send traffic out. All TCP/UDP sessions 1297 will share the bandwidth for all egress links. For the purpose 1298 to work with the differentiated service provided by under layer 1299 network in bandwidth and latency, the kernel may allocate 1300 expected resource to applications that are using the QoS 1301 transport service. For example, kernel can queue different 1302 packets from different applications or users to different queue 1303 and schedule them in different priority. Only after this change, 1304 some application can use more bandwidth and get less queuing 1305 delay for a link than others. 1307 2. The congestion control in a host manages the behavior of TCP 1308 flow(s). This includes important features like slow start, AIMD, 1309 fast retransmit, selective ACK, etc. To accommodate the benefit 1310 of the QoS guaranteed transport service, the congestion control 1311 will be much simpler. The new congestion control is related to 1312 the implementation of QoS guarantee. Following is a simple 1313 congestion control algorithm assuming that the CIR is guaranteed 1314 and PIR is shared between flows: 1316 * There is no slow start, the TCP can start the traffic at the 1317 rate of CIR. 1319 * The AIMD is kept, but the range of the sawtooth pattern should 1320 be maintained between CIR and PIR. 1322 * Other congestion control features can be kept. 1324 5.3. Non-shortest-path 1326 The above method for the transport service with QoS is for the normal 1327 IP flows passing along the shortest path determined by the IGP or 1328 BGP. However, the IP shortest path may not be the best path in terms 1329 of the QoS. For example, the original IP path may not have enough 1330 bandwidth for a transport QoS service. The latency of the IP path is 1331 not the minimum in the network. There are two problems involved. 1332 One is how to find the best path for a QoS criteria, bandwidth or 1333 latency. Another is how to setup the transport QoS for a non- 1334 shortest-path. 1336 The 1st problem is out of scope of this document and many 1337 technologies have be discovered or in research. 1339 The 2nd problem can be solved by combining the segment routing and 1340 in-band signaling. The use of the HbH-EH and Dst-EH is independent 1341 of the type of IP path, thus can be used with segment routing for any 1342 path determined by source. Note, the HbH-EH-aware nodes may not be 1343 different as the explicit IPv6 address in the segment routing header. 1345 5.4. Heterogeneous Network 1347 When IP network is crossing a non-IP network, such as MPLS or 1348 Ethernet network, the in-band signaling needs to be interworking with 1349 that network. The behavior, protocol and rules in the interworking 1350 with non-IP network is not the problem this document will address. 1351 More study and research need to be done, and new draft should be 1352 written to solve the problem. 1354 5.5. Proxy Control 1356 It is expected that for a real service provider network, the in-band 1357 signaling will be checked, filtered and managed at a proxy routers. 1358 This will serve following purpose: 1360 1. Proxy can check if a in-band signaling from end user for the SLA 1361 compliance, security and DOS attack prevention. 1363 2. Proxy can collect the statistics for user's TCP flows and check 1364 the in-band signaling for accounting and charging. 1366 3. Proxy can insert and process appropriate in-band signaling for 1367 TCP flows that the host does not support the new feature, and 1368 this can provide the backward compatibility for host to use the 1369 new feature. 1371 6. Message Format 1373 6.1. Setup Msg 1374 0 1 2 3 1375 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1376 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1377 |0 0 0 0|ver| FI|R|Mis|P| Time | Hop_num |u| Total_latency | 1378 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1379 | State for each hop index | 1380 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1381 | Mapping index list for hops | 1382 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-. . . + 1384 Type = 0, Setup state; 1385 Version: The version of the protocol for the QoS 1386 FI: Flow identification method, 1387 0: 5 tuples; 1: src,dst,TCP; 2: src,dst,UDP; 3: src,dst 1388 R: If the destination host report the received Setup state to 1389 the src address by Destination EH. 0: dont report; 1: report 1390 Mis: Mapping index size; 0: 0bits, 1: 16bits, 2: 20bits, 3: 32bits 1391 P: Programming the HW for QoS; 0: program HW for the QoS from 1392 src to dst; 1: De-program HW for the QoS from src to dst 1393 Time: The life time of QoS forwarding state in second. 1394 Hop_num: The total hop number on the path set by host. It must be 1395 decremented at each hop after the processing. 1396 u: the unit of latency, 0: ms; 1: us 1397 Total_latency : Latency accumulated from each hop, each hop will 1398 add the latency in the device to this value. 1399 Setup state for each hop index: each bit is the setup state on 1400 each hop on the path, 0: failed; 1: success. The 1st hop is at the 1401 most significant bit. 1402 Mapping index list for hops: the mapping index list for all hops 1403 on the path, each index bit size is defined in Mis. The 1st 1404 mapping index is at the top of the stack. Each hop add its mapping 1405 index at the correct position indexed by the current hop number 1406 for the router. 1408 Figure 2: The Setup message 1410 The Setup message is embedded into the hop-by-hop EH to setup the QoS 1411 in the device on the IP forwarding path. At each hop, if the router 1412 is configured to process the header and to enforce the QoS, it must 1413 retrieve the hardware required information from the header, and then 1414 update some fields in the hader. 1416 To keep the whole setup message size unchanged at each hop, the total 1417 hop number must be known at the source host The total hop number can 1418 be detected by OAM. The mapping index list is empty before the 1st 1419 hop receives the in-band signaling. Each hop then fill up the 1420 associated mapping index into the correct place determined by the 1421 index of the hop. 1423 6.2. Bandwidth Msg 1425 0 1 2 3 1426 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1427 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1428 |0 0 0 1| reserved | Minimum bandwidth | 1429 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1430 | Maximum bandwidth | 1431 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1433 Type = 1, 1434 Minimum bandwidth : The minimum bandwidth required, or CIR, unit Mbps 1435 Maximum bandwidth : The maximum bandwidth required, or PIR, unit Mbps 1437 Figure 3: The Bandwidth message 1439 6.3. Burst Msg 1441 0 1 2 3 1442 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1443 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1444 |0 0 1 0| Burst size | 1445 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1447 Type = 2, 1448 Burst size : The burst size, unit M bytes 1450 Figure 4: The burst message 1452 6.4. Latency Msg 1454 0 1 2 3 1455 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1456 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1457 |0 0 1 1|u| Latency | 1458 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1460 Type = 3, 1461 u: the unit of the latency 1462 0: ms; 1: us 1463 Latency: Expected maximum latency for each hop 1465 Figure 5: The Latency message 1467 6.5. Authentication Msg 1469 0 1 2 3 1470 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1471 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1472 |0 1 0 0| MAC_ALG | res | MAC data (variable length) | 1473 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+. . .+ 1475 Type = 4, 1476 MAC_ALG: Message Authentication Algorithm 1477 0: MD5; 1:SHA-0; 2: SHA-1; 3: SHA-256; 4: SHA-512 1478 MAC data: Message Authentication Data; 1479 Res: Reserved bits 1480 Size of signaling data (opt_len): Size of MAC data + 2 1481 MD5: 18; SHA-0: 22; SHA-1: 22; SHA-256: 34; SHA-512: 66 1483 Figure 6: The Authentication message 1485 6.6. OAM Msg 1487 0 1 2 3 1488 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1489 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1490 |0 1 0 1| OAM_t | OAM_len | OAM data (variable length) | 1491 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+. . .+ 1493 Type = 5, 1494 OAM_t : OAM type 1495 OAM_len : 8-bit unsigned integer. Length of the OAM data, in octets; 1496 OAM data: OAM data, details of OAM data are TBD. 1498 Figure 7: The OAM message 1500 6.7. Forwarding State Msg 1501 0 1 2 3 1502 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1503 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1504 |0 1 1 0|ver| FI|R|Mis|P| Time | Hop_num |u| Total_latency | 1505 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1506 | Forwarding state for each hop index | 1507 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1508 | Mapping index list for hops | 1509 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-. . . + 1511 Type = 6, Forwarding state; 1512 All parameter definitions and process in the 1st row are same in 1513 the setup message. 1514 Forward state for each hop index : each bit is the fwd state on each 1515 hop on the path, 0: failed; 1: success; The 1st hop is at the 1516 most significant bit. 1517 Mapping index list for hops: the mapping index list for all hops on 1518 the path, each index bit size is defined in Mis. The list is from 1519 the setup report message. 1521 Figure 8: The Forwarding State message 1523 6.8. Setup State Report Msg 1525 0 1 2 3 1526 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1527 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1528 |0 1 1 1|ver|H|u| Total_latency | Reserved | 1529 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1530 | State for each hop index | 1531 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1532 | Mapping index list for hops | 1533 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-. . . + 1535 Type = 7, Setup state report; 1536 H: Hop number bit. When a host receives a setup message and form 1537 a setup report message, it must check if the Hop_num in setup 1538 message is zero. If it is zero, the H bit is set to one, and if 1539 it is not zero, the H bit is clear. This will notify the source 1540 of setup message that if the original Hop_num was correct. 1541 Following are directly copied from the setup message: 1542 u, Total_latency; 1543 State for each hop index 1544 Mapping index list for hops. 1546 Figure 9: The Setup State Report message 1548 6.9. Forward State Report Msg 1550 0 1 2 3 1551 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1552 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1553 |1 0 0 0|ver|H|u| Total_latency | Reserved | 1554 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1555 | Forwarding state for each hop index | 1556 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1558 Type = 8, Forwarding state report; 1559 H: Hop number bit. When a host receives a Forward State message 1560 and form a Forward State Report message, it must check if the 1561 Hop_num in Forward State message is zero. If it is zero, the H bit 1562 is set to one, and if it is not zero, the H bit is clear. 1563 This will notify the source of Forward State message that if the 1564 original Hop_num was set correct. 1565 Following are directly copied from the Forward State message: 1566 u, Total_latency; 1567 Forwarding State for each hop index 1569 Figure 10: The Fwd State Report message 1571 7. IANA Considerations 1573 This document defines a new option type for the Hop-by-Hop Options 1574 header and the Destination Options header. According to [RFC8200], 1575 the detailed value are: 1577 +-----------+----------------+---------------------+---------------+ 1578 | | Binary Value | | | 1579 | Hex Value +----+---+-------+ Description | Reference | 1580 | | act|chg| rest | | | 1581 +-----------+----+---+-------+---------------------+---------------+ 1582 | 0x0 | 00 | 0 | 10000 | In-band Signaling | Section 6 | 1583 | | | | | | in this doc | 1584 +-----------+----+---+-------+---------------------+---------------+ 1586 Figure 11: The New Option Type 1588 1. The highest-order 2 bits: 00, indicating if the processing IPv6 1589 node does not recognize the Option type, skip over this option and 1590 continue processing the header. 1592 2. The third-highest-order bit: 0, indicating the Option Data does 1593 not change en route. 1595 3. The low-order 5 bits: 10000, assigned by IANA. 1597 This document also defines a 4-bit subtype field, for which IANA will 1598 create and will maintain a new sub-registry entitled "In-band 1599 signaling Subtypes" under the "Internet Protocol Version 6 (IPv6) 1600 Parameters" [IPv6_Parameters] registry. Initial values for the 1601 subtype registry are given below 1603 +-------+------------+-----------------------------+---------------+ 1604 | Type | Mnemonic | Description | Reference | 1605 +-------+------------+-----------------------------+---------------+ 1606 | 0 | SETUP | Setup message | Section 6.1 | 1607 +-------+------------+-----------------------------+---------------+ 1608 | 1 | BANDWIDTH | Bandwidth message | Section 6.2 | 1609 +-------+------------+-----------------------------+---------------+ 1610 | 2 | BURST | Burst message | Section 6.3 | 1611 +-------+------------+-----------------------------+---------------+ 1612 | 3 | LATENCY | Latency message | Section 6.4 | 1613 +-------+------------+-----------------------------+---------------+ 1614 | 4 | AUTH | Authentication message | Section 6.5 | 1615 +-------+------------+-----------------------------+---------------+ 1616 | 5 | OAM | OAM message | Section 6.6 | 1617 +-------+------------+-----------------------------+---------------+ 1618 | 6 | FWD STATE | Forward state | Section 6.7 | 1619 +-------+------------+-----------------------------+---------------+ 1620 | 7 |SETUP REPORT| Setup state report | Section 6.8 | 1621 +-------+------------+-----------------------------+---------------+ 1622 | 8 | FWD REPORT | Forwarding state report | Section 6.9 | 1623 +-------+------------+-----------------------------+---------------+ 1625 Figure 12: The In-band Signaling Sub Type 1627 8. Security Considerations 1629 There is no security issue introduced by this document 1631 9. Acknowledgements 1633 We like to thank Huawei's Nanjing research team leaded by Feng Li to 1634 provide the Product on Concept (POC) development and test, the team 1635 member includes Fengxin Sun, Xingwang Zhou, Weiguang Wang. We also 1636 like to thank other people involved in the discussion of solution: 1637 Tao Ma from Future Network Streategy dept. 1639 10. References 1641 10.1. Normative References 1643 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1644 Requirement Levels", BCP 14, RFC 2119, 1645 DOI 10.17487/RFC2119, March 1997, 1646 . 1648 [RFC2581] Allman, M., Paxson, V., and W. Stevens, "TCP Congestion 1649 Control", RFC 2581, DOI 10.17487/RFC2581, April 1999, 1650 . 1652 [RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1653 (IPv6) Specification", STD 86, RFC 8200, 1654 DOI 10.17487/RFC8200, July 2017, 1655 . 1657 10.2. Informative References 1659 [BBR] Neal Cardwell, et al, Google, "BBR Congestion Control", 1660 2016, . 1663 [Cubic_throughput] 1664 Wei Bao, et al. The University of British Columbia, 1665 Vancouver, Canada, IEEE Globecom 2010 proceedings, "A 1666 Model for Steady State Throughput of TCP CUBIC", 2010, 1667 . 1670 [DiffServ] 1671 wiki, "Differentiated services", 2016, 1672 . 1674 [Fairness] 1675 Jain, R., et al. DEC Research Report TR-301, "A 1676 Quantitative Measure of Fairness and Discrimination for 1677 Resource Allocation in Shared Computer Systems", 1984, 1678 . 1680 [Fastpass] 1681 Jonathan Perry, et al, MIT, "Fastpass: A Centralized 1682 ?Zero-Queue? Datacenter Network", 2014, 1683 . 1685 [I-D.falk-xcp-spec] 1686 Falk, A., "Specification for the Explicit Control Protocol 1687 (XCP)", draft-falk-xcp-spec-03 (work in progress), July 1688 2007. 1690 [I-D.han-iccrg-arvr-transport-problem] 1691 Han, L. and K. Smith, "Problem Statement: Transport 1692 Support for Augmented and Virtual Reality Applications", 1693 draft-han-iccrg-arvr-transport-problem-01 (work in 1694 progress), March 2017. 1696 [I-D.harper-inband-signalling-requirements] 1697 Harper, J., "Requirements for In-Band QoS Signalling", 1698 draft-harper-inband-signalling-requirements-00 (work in 1699 progress), January 2007. 1701 [I-D.ietf-aqm-codel] 1702 Nichols, K., Jacobson, V., McGregor, A., and J. Iyengar, 1703 "Controlled Delay Active Queue Management", draft-ietf- 1704 aqm-codel-06 (work in progress), December 2016. 1706 [I-D.ietf-aqm-fq-codel] 1707 Hoeiland-Joergensen, T., McKenney, P., 1708 dave.taht@gmail.com, d., Gettys, J., and E. Dumazet, "The 1709 FlowQueue-CoDel Packet Scheduler and Active Queue 1710 Management Algorithm", draft-ietf-aqm-fq-codel-06 (work in 1711 progress), March 2016. 1713 [I-D.ietf-aqm-pie] 1714 Pan, R., Natarajan, P., Baker, F., and G. White, "PIE: A 1715 Lightweight Control Scheme To Address the Bufferbloat 1716 Problem", draft-ietf-aqm-pie-10 (work in progress), 1717 September 2016. 1719 [I-D.ietf-tcpm-dctcp] 1720 Bensley, S., Eggert, L., Thaler, D., Balasubramanian, P., 1721 and G. Judd, "Datacenter TCP (DCTCP): TCP Congestion 1722 Control for Datacenters", draft-ietf-tcpm-dctcp-03 (work 1723 in progress), November 2016. 1725 [I-D.roberts-inband-qos-ipv6] 1726 Roberts, L. and J. Harford, "In-Band QoS Signaling for 1727 IPv6", draft-roberts-inband-qos-ipv6-00 (work in 1728 progress), July 2005. 1730 [I-D.sridharan-tcpm-ctcp] 1731 Sridharan, M., Tan, K., Bansal, D., and D. Thaler, 1732 "Compound TCP: A New TCP Congestion Control for High-Speed 1733 and Long Distance Networks", draft-sridharan-tcpm-ctcp-02 1734 (work in progress), November 2008. 1736 [IntServ] wiki, "Integrated services", 2016, 1737 . 1739 [IPv6_Parameters] 1740 IANA, "Internet Protocol Version 6 (IPv6) Parameters", 1741 2015, . 1744 [PCC] Mo Dong, et al, University of Illinois at Urbana- 1745 Champaign, Hebrew University of Jerusalem, "PCC: Re- 1746 architecting Congestion Control for Consistent High 1747 Performance", 2014, . 1749 [PERC] Lavanya Jose, et al, Stanford University, MIT, Microsoft, 1750 "High Speed Networks Need Proactive Congestion Control", 1751 2016, . 1754 [RCP] Nandita Dukkipati, Ph.D. Thesis, Department of Electrical 1755 Engineering, Stanford University, "Rate Control Protocol 1756 (RCP): Congestion control to make flows complete quickly", 1757 2007, 1758 . 1760 [Reno_throughput] 1761 Matthew Mathis, et al, Pittsburgh Supercomputing Center, 1762 "The Macroscopic Behavior of the TCP Congestion Avoidance 1763 Algorithm", 1997, 1764 . 1767 [Tactile] JDavid Szabo, et al. Proceedings of European Wireless 1768 2015; 21th European Wireless Conference, "Towards the 1769 Tactile Internet: Decreasing Communication Latency with 1770 Network Coding and Software Defined Networking", 2015, 1771 . 1773 [TCP-cubic] 1774 Ha, S., Rhee, I., and L. Xu, "CUBIC: A New TCP-Friendly 1775 High-Speed TCP Variant", 2008. 1777 [TCP-vegas] 1778 Peterson, L., "TCP Vegas: New Techniques for Congestion 1779 Detection and Avoidance - CiteSeer page on the 1994 1780 SIGCOMM paper", 1994. 1782 [TCP_Targets] 1783 Andreas Benthin, Stefan Mischke, University of Paderborn, 1784 "Bandwidth Allocation of TCP", 2004. 1786 [TIMELY] Radhika Mittal, et al. Google, Inc., "TIMELY: RTT-based 1787 Congestion Control for the Datacenter", 2010, 1788 . 1791 Authors' Addresses 1793 Lin Han (editor) 1794 Huawei Technologies 1795 2330 Central Expressway 1796 Santa Clara, CA 95050 1797 USA 1799 Phone: +10 408 330 4613 1800 Email: lin.han@huawei.com 1802 Guoping Li 1803 Huawei Technologies 1804 Beijing 1805 China 1807 Email: liguoping@huawei.com 1809 Boyan Tu 1810 Huawei Technologies 1811 Beijing 1812 China 1814 Email: tuboyan@huawei.com 1816 Xuefei Tan 1817 Huawei Technologies 1818 Beijing 1819 China 1821 Email: tanxuefei@huawei.com 1822 Frank Li 1823 Huawei Technologies 1824 Nanjing 1825 China 1827 Email: frank.lifeng@huawei.com 1829 Richard Li 1830 Huawei Technologies 1831 2330 Central Expressway 1832 Santa Clara, CA 95050 1833 USA 1835 Email: renwei.li@huawei.com 1837 Jeff Tantsura 1839 Email: jefftant.ietf@gmail.com 1841 Kevin Smith 1842 Vodafone 1843 UK 1845 Email: Kevin.Smith@vodafone.com