idnits 2.17.00 (12 Aug 2021) /tmp/idnits58570/draft-ietf-sfc-nsh-ecn-support-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == Line 275 has weird spacing: '... sender doma...' == Line 276 has weird spacing: '...ingress v |...' -- The document date (April 17, 2022) is 27 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) No issues found here. Summary: 0 errors (**), 0 flaws (~~), 2 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 INTERNET-DRAFT D. Eastlake 2 Intended status: Proposed Standard Futurewei Technologies 3 B. Briscoe 4 Independent 5 Y. Li 6 Huawei Technologies 7 A. Malis 8 Malis Consulting 9 X. Wei 10 Huawei Technologies 11 Expires: October 16, 2022 April 17, 2022 13 Explicit Congestion Notification (ECN) and Congestion Feedback 14 Using the Network Service Header (NSH) and IPFIX 15 17 Abstract 19 Explicit congestion notification (ECN) allows a forwarding element to 20 notify downstream devices of the onset of congestion without having 21 to drop packets. Coupled with a means to feed information about 22 congestion back to upstream nodes, this can improve network 23 efficiency through better congestion control, frequently without 24 packet drops. This document specifies ECN and congestion feedback 25 support within a Service Function Chaining (SFC) enabled domain 26 through use of the Network Service Header (NSH, RFC 8300) and IP Flow 27 Information Export (IPFIX, RFC 7011). 29 Status of This Memo 31 This Internet-Draft is submitted in full conformance with the 32 provisions of BCP 78 and BCP 79. 34 Distribution of this document is unlimited. Comments should be sent 35 to the SFC Working Group mailing list or to the 36 authors. 38 Internet-Drafts are working documents of the Internet Engineering 39 Task Force (IETF), its areas, and its working groups. Note that 40 other groups may also distribute working documents as Internet- 41 Drafts. 43 Internet-Drafts are draft documents valid for a maximum of six months 44 and may be updated, replaced, or obsoleted by other documents at any 45 time. It is inappropriate to use Internet-Drafts as reference 46 material or to cite them other than as "work in progress." 47 The list of current Internet-Drafts can be accessed at 48 https://www.ietf.org/1id-abstracts.html. The list of Internet-Draft 49 Shadow Directories can be accessed at 50 https://www.ietf.org/shadow.html. 52 Table of Contents 54 1. Introduction............................................4 55 1.1 NSH Background.........................................4 56 1.2 ECN Background.........................................6 57 1.3 Tunnel Congestion Feedback Background..................6 58 1.4 Conventions Used in This Document......................8 60 2. The NSH ECN Field......................................10 62 3. ECN Support in the NSH.................................12 63 3.1 At The Ingress........................................13 64 3.2 At Transit Nodes......................................14 65 3.2.1 At NSH Transit Nodes................................14 66 3.2.2 At an SF/Proxy......................................15 67 3.2.3 At Other Forwarding Nodes...........................15 68 3.3 At Exit/Egress........................................16 69 3.4 Congestion Statistics and the Conservation of Packets.16 71 4. Tunnel Congestion Feedback Support.....................18 72 4.1 Congestion Level Measurements.........................18 73 4.3 Congestion Information Delivery.......................19 74 4.3 IPFIX Extensions......................................21 75 4.3.1 nshServicePathID....................................21 76 4.3.2 tunnelEcnCeCeByteTotalCount.........................22 77 4.3.3 tunnelEcnEctNectBytetTotalCount.....................22 78 4.3.4 tunnelEcnCeNectByteTotalCount.......................22 79 4.3.5 tunnelEcnCeEctByteTotalCount........................23 80 4.3.6 tunnelEcnEctEctByteTotalCount.......................23 81 4.3.7 tunnelEcnCEMarkedRatio..............................24 83 5. Example of Use.........................................25 85 6. IANA Considerations....................................28 86 6.1 SFC NSH Header ECN Bits...............................28 87 6.2 IPFIX Information Element IDs.........................28 89 7. Security Considerations................................30 90 8. Acknowledgements.......................................30 92 Normative References......................................31 93 Informative References....................................32 95 Authors' Addresses........................................33 97 1. Introduction 99 Explicit Congestion Notification (ECN [RFC3168]) allows a forwarding 100 element to notify downstream nodes of the onset of congestion without 101 having to drop packets. Coupled with a means to feed information 102 about congestion back to upstream nodes, this can improve network 103 efficiency through better congestion control, frequently without 104 packet drops. This document specifies ECN and congestion feedback 105 support within a Service Function Chaining (SFC [RFC7665]) enabled 106 domain through use of the Network Service Header (NSH [RFC8300]) and 107 IP Flow Information Export (IPFIX [RFC7011]). 109 This document requires that all ingress and egress nodes of the SFC 110 domain implement ECN. While congestion management will be the most 111 effective if all interior nodes of the SFC enabled domain implement 112 ECN, some benefit is obtained even if some interior nodes do not 113 implement ECN. Congestion at any interior bottleneck where ECN 114 marking is not implemented will be unmanaged. 116 The following subsections provide background information on NSH, ECN, 117 congestion feedback, and terminology used in this document. 119 1.1 NSH Background 121 The Service Function Chaining (SFC [RFC7665]) architecture calls for 122 the encapsulation of traffic within a service function chaining 123 domain with a Network Service Header (NSH [RFC8300]) added by the 124 "Classifier" (ingress node) on entry to the domain and the NSH being 125 removed on exit from the domain at the egress node. The NSH is used 126 to control the path of a packet in an SFC domain. The NSH is a 127 reasonable place, in a domain where traffic is NSH encapsulated, to 128 accumulate congestion information. 130 | 131 v 132 +----------+ 133 . .|Classifier|. . . . . . . . . . . . . . 134 . +----------+ . 135 . | +----+ . 136 . | --+ SF | Service . 137 . | / +----+ Function . 138 . v --- Chaining . 139 . +-----+/ +----+ domain . 140 . | SFF |--------+ SF | . 141 . +-----+\ +----+ . 142 . | --- . 143 . | \ +----+ . 144 . | --+ SF | . 145 . v +----+ . 146 . +-----+ +----+ . 147 . | SFF |-----------------+ SF | . 148 . +-----+ +----+ . 149 . | +----+ . 150 . | --+ SF | . 151 . | / +----+ . 152 . v --- . 153 . +-----+/ +----+ . 154 . | SFF |--------+ SF | . 155 . +-----+\ +----+ . 156 . | --- . 157 . | \ +----+ . 158 . | --+ SF | . 159 . v +----+ . 160 . +------+ . 161 . . .|Egress|. . . . . . . . . . . . . . . 162 +------+ 163 | 164 v 166 Figure 1. Example SFC Forwarding Nodes Path 168 Figure 1 shows an SFC enabled domain for the purpose of illustrating 169 the use of the NSH. Traffic passes through a sequence of Service 170 Function Forwarders (SFFs) each of which sends the traffic to one or 171 more Service Functions (SFs). Each SF performs some operation on the 172 traffic, for example firewalling or Network Address Translation (NAT) 173 or load balancing, and then returns the traffic to the SFF from which 174 it was received. 176 Logically, during the transit of each SFF, the outer transport header 177 that got the packet to the SFF is stripped (see Figure 3), the SFF 178 decides on the next forwarding step, either adding a new outer 179 transport header or, if the SFF is the exit/egress, removing the NSH 180 header. The outer transport headers added may be different in 181 different regions of the SFC enabled domain. For example, IP could be 182 used for some SFF-to-SFF communication and MPLS used for other SFF- 183 to-SFF communication. 185 1.2 ECN Background 187 Explicit Congestion Notification (ECN [RFC3168]) allows a forwarding 188 element (such as a router or a Service Function Forwarder (SFF) or 189 Service Function (SF)) to notify downstream nodes of the onset of 190 congestion without having to drop packets. This can be used as an 191 element in active queue management (AQM) [RFC7567] to improve network 192 efficiency through better traffic control without packet drops. The 193 forwarding element can explicitly mark some packets in an ECN field 194 instead of dropping the packet. For example, a two-bit field is 195 available for ECN marking in IP headers [RFC3168]. 197 1.3 Tunnel Congestion Feedback Background 199 Tunnels are widely deployed in various networks including data center 200 networks, enterprise network, and the public Internet. A tunnel 201 consists of ingress, egress, and a set of intermediate nodes 202 including routers. Tunnel Congestion Feedback (Section 4) is a 203 building block for congestion mitigation methods. It supports 204 feedback of congestion information from an egress node to an ingress 205 node. This document treats the SFC emabled domain as a tunnel with 206 the initial Classifier node being the ingress; however, the tunnel 207 congestion feedback facilities specified in this document MAY be used 208 in contexts other than SFC. 210 Any action by a tunnel ingress to reduce congestion needs to allow 211 sufficient time for the end-to-end congestion control loop to respond 212 first, otherwise the system could go unstable. For instance by the 213 ingress taking a smoothed average of the level of congestion signaled 214 by feedback from the tunnel egress or delaying any action for at 215 least the worst case end-to-end round-trip time (for example, 200 216 milliseconds). 218 Examples of actions that can be taken by an ingress node when it has 219 knowledge of downstream congestion include those listed below. 220 Details of implementing these traffic control methods, beyond those 221 given here, are outside the scope of this document. 223 (1) Traffic throttling (policing), where the downstream traffic 224 flowing out of the ingress node is limited to reduce or eliminate 225 congestion. 227 (2) Upstream congestion feedback, where the ingress node sends 228 messages upstream to or towards the ultimate traffic source, a 229 function that can throttle traffic generation/transmission. 231 (3) Traffic re-direction, where the ingress node configures the NSH 232 of some future traffic so that it avoids congested paths. Great 233 care must be taken with this option to avoid (a) significant re- 234 ordering of traffic in flows that it is desirable to keep in 235 order (due to end-to-end requiresment or due to a stateful SF) 236 and (b) oscillation/instability in traffic paths due to alternate 237 congestion of previously idle paths and the idling of previously 238 congested paths. For example, it is preferable to classify 239 traffic into flows of a sufficiently coarse granularity that the 240 flows are long lived and to use a stable path per flow, sending 241 only newly appearing flows on apparently uncongested paths. 243 Figure 2 shows an example path from an original sender to a final 244 receiver passing through a chain of service functions between the 245 ingress and egress of an SFC enabled domain. The path is also likely 246 to pass through other network nodes outside the SFC enabled domain 247 (not shown) before entering that domain and after leaving that 248 domain. 250 Figure 2 shows typical congestion feedback that would be expected 251 from the final receiver to the origin sender, which controls the load 252 the origin sender directs to all elements on the path. The figure 253 also shows the congestion feedback from the egress to the ingress of 254 the SFC enabled domain that is described in this document, to control 255 or balance load within that domain. 257 .:= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = :. 258 _||_ End-to-End Congestion Feedback || 259 \ / || 260 \/ || 261 __ Inner Transport Header and Payload __ 262 | | ->- - - - - - - - - - - - - - ->- - - - - -- - - - - - ->- | | 263 | | | | 264 | | .:= = = = = = = = = = = = = = = = = = = = = =:. | | 265 | | _||_ Tunnel Congestion Feedback || | | 266 | | \ / || | | 267 | | \/ || | | 268 | | __ NSH __ | | 269 | | | |-------------------------->--------------| | | | 270 | |. . . | | ___ ___ ___ | |. . .| | 271 | | | | OT1 | | OT4 | | . . . | | OTn | | | | 272 | | | |-->--|SFF|--->---|SFF| |SFF|-->--| | | | 273 |__| |__| |___| |___| |___| |__| |__| 274 origin SFC | ^ | ^ SFC final 275 sender domain OT2| |OT3 OT6| |OT7 domain rcvr 276 ingress v | v | egress 277 +---+ +---+ SFF 278 |SF | |SF | 279 +---+ +---+ 281 Figure 2. Congestion Feedback across an SFC enabled Domain 283 SFC enabled Domain congestion feedback in Figure 2 is shown within 284 the context of an end-to-end congestion feedback loop. Also shown is 285 the encapsulated layering of NSH headers within a series of outer 286 transport headers (OT1, OT2, ... OTn). 288 1.4 Conventions Used in This Document 290 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 291 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 292 "OPTIONAL" in this document are to be interpreted as described in BCP 293 14 [RFC2119] [RFC8174] when, and only when, they appear in all 294 capitals, as shown here. 296 Acronyms: 298 AQM - Active Queue Management [RFC7567] 300 CE - Congestion Experienced [RFC3168] 302 downstream - The direction from ingress to egress 303 ECN - Explicit Congestion Notification [RFC3168] 305 ECT - ECN Capable Transport [RFC3168] 307 IPFIX - IP Flow Information Export [RFC7011] 309 Not-ECT - Not ECN-Capable Transport [RFC3168] 311 NSH - Network Service Header [RFC8300] 313 SF - Service Function [RFC7665] 315 SFC - Service Function Chaining [RFC7665] 317 SFF - Service Function Forwarder [RFC7665] - A type of node that 318 forwards based on the NSH. 320 TLV - Type Length Value 322 upstream - The direction from egress to ingress 324 2. The NSH ECN Field 326 The NSH is used to encapsulate traffic and control its subsequent 327 path (see Section 2 of [RFC8300]). The NSH also provides for optional 328 metadata inclusion, as shown in Figure 3. 330 +-----------------------------------+ 331 | Outer Transport Header | 332 +-----------------------------------+ 333 | Network Service Header (NSH) | 334 | +------------------------------+ | 335 | | Base Header | | 336 | +------------------------------+ | 337 | | Service Path Header | | 338 | +------------------------------+ | 339 | | Metadata (Context Header(s)) | | 340 | +------------------------------+ | 341 +-----------------------------------+ 342 | Original Packet / Frame / Payload | 343 +-----------------------------------+ 345 Figure 3. Data Encapsulation with the NSH 347 This document assigns two currently unused bits (indicated by "U") in 348 the NSH Base Header (Section 2.2 of [RFC8300]) for the purpose of ECN 349 indication as shown in Figure 4. 351 0 1 2 3 352 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 353 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 354 |Ver|O|U| TTL | Length |U|U|U|U|MD Type| Next Protocol | 355 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 356 ^ ^ 357 | | 358 +-------+ 359 |NSH ECN| 360 | field | 361 +-------+ 363 Figure 4. Updated NSH Base Header 365 RFC Editor NOTE: The above figure should be adjusted based on the 366 bits actually assigned by IANA (see Section 5) and this note deleted. 368 Table 1 shows the meaning of the code points in the NSH ECN field. 369 These have the same meaning as the ECN field code points in the IPv4 370 or IPv6 header as defined in Section 23.1 of [RFC3168]. 372 Binary Name Meaning 373 ------ ------- -------------------------------- 374 00 Not-ECT Not ECN-Capable Transport 375 01 ECT(1) ECN-Capable Transport 376 10 ECT(0) ECN-Capable Transport 377 11 CE Congestion Experienced 379 Table 1. ECN Field Code Points 381 3. ECN Support in the NSH 383 This section describes the required behavior to support ECN using the 384 NSH. There are two aspects to ECN support: 385 1. ECN propagation during encapsulation or decapsulation; 386 2. ECN marking during congestion at bottlenecks. 388 While this section covers all combinations of ECN-aware and ECN- 389 unaware, it is expected that in most cases the NSH domain will be 390 uniform so that, if this document is applicable, all SFFs will 391 support ECN; however, some SFs might not support ECN. 393 ECN Propagation: 395 The specification of ECN tunneling [RFC6040] explains that an 396 ingress must not propagate ECN support into an encapsulating 397 header unless the egress supports correct onward propagation of 398 the ECN field during decapsulation. We define Compliant ECN 399 Decapsulation here as decapsulation compliant with either 400 [RFC6040] or an earlier compatible equivalent ([RFC4301], or the 401 full functionality mode of [RFC3168]). 403 The procedures in Section 3.2.1 ensure that each ingress of the 404 transport links within the SFC enabled domain does not propagate 405 ECN support into the encapsulating outer transport header unless 406 the corresponding egress of that link supports Compliant ECN 407 Decapsulation. 409 Section 3.3 requires that all the egress nodes of the SFC enabled 410 domain support Compliant ECN Decapsulation in conjunction with 411 tunnel congestion feedback, otherwise the scheme in this document 412 will not work. 414 ECN Marking: 416 At transit nodes the marking behavior specified in Section 3.2.1 417 is recommended and if not implemented at such transit nodes, there 418 may be unmanaged congestion. 420 Detection of congestion will be most effective if ECN marking is 421 supported by all potential bottlenecks inside the domain in which 422 NSH is being used to route traffic as well as at the ingress and 423 egress. Nodes that do not support ECN marking, or that support 424 AQM but not ECN, will naturally use drop to relieve congestion. 425 The gap in the end-to-end packet sequence will be detected as 426 congestion by the final receiving endpoint, but not by the NSH 427 egress (see Figure 2). 429 3.1 At The Ingress 431 When the ingress/Classifier encapsulates an incoming packet with an 432 NSH, it MUST set the NSH ECN field using the "Normal mode" specified 433 in [RFC6040] (e.g., copied from the incoming IP header). 435 Then, if the resulting NSH ECN field is Not-ECT, the ingress SHOULD 436 set it to ECT(0). This indicates that, even though the end-to-end 437 transport is not ECN-capable, the egress and ingress of the SFC 438 enabled domain are acting as an ECN-capable transport. This approach 439 supports all known variants of ECN, including the experimental L4S 440 capability [RFC8311] [ecnL4S]. 442 Packets arriving at the ingress might not use IP. If the protocol of 443 arriving packets supports an ECN field similar to IP, for example 444 MPLS [RFC5129], the procedures for IP packets can be used. If 445 arriving packets do not support an ECN field similar to IP, they MUST 446 be treated as if they are Not-ECT IP packets. 448 Then, as the NSH encapsulated packet is further encapsulated with a 449 transport header, if ECN marking is available for that transport (as 450 it is for IP [RFC3168] and MPLS [RFC5129]), the ECN field of the 451 transport header MUST be set using the "Normal mode" specified in 452 [RFC6040] (i.e., copied from the NSH ECN field). 454 A summary of these normative steps is given in Table 2. 456 +-----------------+---------------+ 457 | Incoming Header | Departing NSH | 458 | (also equal to | and Outer | 459 | departing Inner | Headers | 460 | Header) | | 461 +-----------------+---------------+ 462 | Not-ECT | ECT(0) | 463 | ECT(0) | ECT(0) | 464 | ECT(1) | ECT(1) | 465 | CE | CE | 466 +-----------------+---------------+ 468 Table 2. Setting of ECN fields by an Ingress/Classifier 470 The requirements in this section apply to all ingress nodes for the 471 domain in which an NSH is being used to steer traffic. 473 3.2 At Transit Nodes 475 This section describes the behavior at nodes that forward based on 476 the NSH such as SFF and other forwarding nodes such as IP routers. 477 Figure 5 shows a packet on the wire between forwarding nodes. 479 +-----------------+ 480 | Outer Header | 481 +-----------------+ 482 | NSH | 483 +-----------------+ 484 | Inner Header | 485 +-----------------+ 486 | Payload | 487 +-----------------+ 489 Figure 5. Packet in Transit 491 3.2.1 At NSH Transit Nodes 493 When a packet is received at an NSH based forwarding node such as an 494 SFF, say N1, the outer transport encapsulation is removed and its ECN 495 marking SHOULD be combined into the NSH ECN marking as specified in 496 [RFC6040]. If this is not done, any congestion encountered at non-NSH 497 transit nodes between N1 and the previous upstream NSH based 498 forwarding node will be lost and not transmitted downstream. 500 The NSH forwarding node SHOULD use a recognized AQM algorithm 501 [RFC7567] to detect congestion. If the NSH ECN field indicates ECT, 502 it will probabilistically set the NSH ECN field to the Congestion 503 Experienced (CE) value or, in cases of extreme congestion, drop the 504 packet. 506 When the NSH encapsulated packet is further encapsulated for 507 transmission to the next SFF or SF, ECN marking behavior depends on 508 whether or not the node that will decapsulate the outer header 509 supports Compliant ECN Decapsulation (see Section 3). If it does, 510 then the encapsulating node propagates the NSH ECN field to this 511 outer encapsulation using the "Normal Mode" of ECN encapsulation 512 [RFC6040] (the ECN field is copied). If it does not, then the 513 encapsulating node MUST clear ECN in the outer encapsulation to non- 514 ECT (the "Compatibility Mode" of [RFC6040]). 516 3.2.2 At an SF/Proxy 518 If the SF is NSH and ECN-aware, the processing is essentially the 519 same at the SF as at an SFF as discussed in Section 3.2.1 (except in 520 the case where the SF terminates the packets path). 522 If the SF is NSH-aware but ECN-unaware, then the SFF transmitting the 523 packet to the SF will use Compatibility Mode. Congestion encountered 524 in the SFF to SF and SF to SFF paths will be unmanaged. 526 If the SF is not NSH-aware, then an NSH proxy will be between the SFF 527 and the SF to avoid exposure of the SF that does not understand NSHs 528 to the NSH as shown in Figure 6. This is described in Section 4.6 of 529 [RFC7665]. The SF and proxy together look to the SFF like an NSH- 530 aware SF. The behavior at the proxy and SF in this case is as below: 532 If such a proxy is not ECN-aware then congestion in the entire 533 path from SFF to proxy to SF back to proxy to SFF will be 534 unmanaged. 536 | 537 v 538 +----------+ +---------+ 539 | | +-------+ | NSH | 540 | SFF +---->| NSH +---->|un-aware | 541 |(Service | | aware | | SF | 542 | Function |<----+ proxy |<----+(Service | 543 |Forwarder)| +-------+ |Function)| 544 +----------+ +---------+ 545 | 546 v 548 Figure 6. Proxy for NSH Un-aware SFF 550 If the proxy is ECN-aware, the proxy uses an AQM to indicate 551 congestion within the proxy in the NSH that it returns to the SFF. 552 The outer header used for the proxy-to-SF path uses Normal Mode. 553 The outer header used for the proxy-to-SFF path uses Normal Mode 554 based copying of the NSH ECN field to the outer header. Thus 555 congestion in the proxy will be managed. 557 Congestion in the SF will be managed only if the SF is ECN-aware 558 and implements an AQM. 560 3.2.3 At Other Forwarding Nodes 562 Other forwarding nodes, that is non-NSH forwarding nodes between NSH 563 forwarding nodes, such as IP or label switched routers, might also 564 contain potential bottlenecks. If so, they SHOULD implement an AQM 565 algorithm to update the ECN marking in the outer transport header as 566 specified in [RFC3168]. 568 3.3 At Exit/Egress 570 At the SFC enabled domain egress node, first any actions are taken 571 based on Congestion Experienced or other values of ECN marking, such 572 as accumulating statistics to send back to the ingress (see Section 573 4) or for other uses. If the packet being carried inside the NSH is 574 IP, when the NSH is removed the NSH ECN field MUST be combined with 575 the IP ECN field as specified in Table 3 that was extracted from 576 Section 3.2 of [RFC6040]. This requirement applies to all egress 577 nodes for the domain in which NSH is being used to route traffic. 579 +---------+---------------------------------------------+ 580 |Arriving | Arriving Outer Header | 581 | Inner +---------+-----------+-----------+-----------+ 582 | Header | Not-ECT | ECT(0) | ECT(1) | CE | 583 +---------+---------+-----------+-----------+-----------+ 584 | Not-ECT | Not-ECT | Not-ECT | Not-ECT | | 585 | ECT(0) | ECT(0) | ECT(0) | ECT(0) | CE | 586 | ECT(1) | ECT(1) | ECT(1) | ECT(1) | CE | 587 | CE | CE | CE | CE | CE | 588 +---------+---------+-----------+-----------+-----------+ 590 Table 3. Exit ECN Fields Merger (Source [RFC6040]) 592 All the egress nodes of the SFC enabled domain MUST support Compliant 593 ECN Decapsulation as specified in this section. If this is not the 594 case, the scheme described in this document will not work, and cannot 595 be used. 597 3.4 Congestion Statistics and the Conservation of Packets 599 The SFC specification permits an SF to absorb packets and to generate 600 new packets as well as simply processing and returning the packets it 601 receives to an SFF. Such actions might appear to be packet loss due 602 to congestion or might mask the loss of packets by generating 603 additional packets. 605 The tunnel congestion feedback approach (Section 4) can detect 606 congestions in several ways. One way detects traffic loss by counting 607 payload packets and bytes in at the ingress and counting them out at 608 the egress. This does not work unless nodes conserve the number of 609 payload packets and/or bytes. Therefore, it will not be possible to 610 accurately detect packet loss using this technique if traffic volume, 611 as measured by the metric in used (packets or bytes), is not 612 conserved by the service function chain processing that traffic. 614 Nonetheless, if a bottleneck supports ECN marking, it will be 615 possible to detect the high level of CE markings that are associated 616 with congestion at that bottleneck by looking at the ratio of CE- 617 marked to non-CE-marked packets. However, it will not be possible to 618 detect any congestion based on ECN marking, whether slight or severe, 619 if it occurs at a bottleneck that does not support ECN marking. 621 4. Tunnel Congestion Feedback Support 623 The collection and storage of congestion information at the egress 624 may be useful for later analysis and MAY be used without the feedback 625 mechanisms specified in this Section. However, if congestion 626 information is not fed back to a point which can take action to 627 reduce congestion, it will not be useful in real time. Such 628 congestion feedback to the ingress enables it to take actions such as 629 those listed in Section 1.3. 631 IP Flow Information Export (IPFIX [RFC7011]) provides a standard for 632 communicating traffic flow statistics. As extended by this document, 633 IPFIX messages from the egress to the ingress are used to communicate 634 the extent of congestion between an ingress and egress based on ECN 635 marking in the NSH. The ingress MUST be configured to know the 636 relevant egress for a flow. The egress MUST be able to identify the 637 relevant ingress for a packet based on the SPI, the Ingress Network 638 Node Information Context Header [NSHTLV], or the like. 640 4.1 Congestion Level Measurements 642 The congestion level measurements are based on ECN marking in the NSH 643 and packet drop. In particular congestion information includes at 644 least one of cumulative bytes counts of packets with each type of 645 outer/inner header ECN marking combination, the ratio of CE-marked 646 packets to all packets, and the ratio of dropped packets to all 647 packets. 649 If the congestion level is low enough, the packets are marked as CE 650 instead of being dropped, and then it is easy to calculate congestion 651 level according to the ratio of CE-marked packets. If the congestion 652 level is so high that ECT packets will be dropped, then the packet 653 loss ratio could be calculated by comparing total packets entering 654 ingress and total packets arriving at egress over the same span of 655 packets. If packet loss is detected for a flow that would preserve 656 the number of packets in the absence of congestion, then it can be 657 assumed that severe congestion has occurred in the tunnel. 659 The egress calculates the CE-marked packet ratio by counting packets 660 with different ECN markings. The CE-marked packet ratio will be used 661 as an indication of tunnel load level. It is assumed that nodes 662 between the ingress and egress will not drop packets biased towards 663 certain ECN codepoints, so calculating of CE-marked packet ratio is 664 not affected by packet drop. 666 The calculation of the fraction of packets dropped is by comparing 667 the traffic volumes between ingress and egress. 669 Faked ECN-Capable Transport (ECT) is used at the ingress to defer 670 packet loss to the egress. The basic idea of faked ECT is that, when 671 encapsulating packets, the ingress first marks the tunnel outer 672 header according to [RFC6040], and then remarks the outer header of 673 Not-ECT packets as ECT. (ECT(0) and ECT(1) are treated as the same.) 674 In this case, the NSH is treated as the tunnel outer header because 675 it will be present for the entire SFC enabled domain transit while 676 transport headers may change. Thus, as transmitted by the ingress 677 node, there will be one of three combinations of outer header ECN 678 field and inner header ECN field as follows: CE|CE, ECT|N-ECT, and 679 ECT|ECT (in the format of outer-ECN|inner-ECN); when decapsulating 680 packets at the egress, [RFC6040] defined decapsulation behavior is 681 used, and according to [RFC6040], the packets marked as CE|N-ECT will 682 be dropped. Faked-ECT is used to shift some drops to the egress in 683 order to allow the egress to calculate the CE-marked packet ratio 684 more precisely. 686 The ingress encapsulates packets and marks their outer header 687 according to faked ECT as described above. The ingress cumulatively 688 counts packet bytes for three types of ECN combination (CE|CE, ECT|N- 689 ECT, and ECT|ECT) and then the ingress regularly sends cumulative 690 bytes counts message of each type of ECN combination to the egress. 692 When each message arrives at the egress, the following two steps 693 occur: (1) the egress calculates the ratio of CE-marked packets; (2) 694 the egress cumulatively counts packet bytes coming from the ingress 695 and adds its own bytes counts of each type of ECN combination (CE|CE, 696 ECT|N-ECT, CE|N-ECT, CE|ECT, and ECT|ECT) to the message for the 697 ingress to calculate packet loss. The egress feeds back the CE-marked 698 packet ratio, packet loss ratio, bytes counts information, and the 699 like to the ingress as requested for evaluating congestion level in 700 the tunnel. 702 The statistics can be at the granularity of all traffic from the 703 ingress to the egress to learn about the overall congestion status of 704 the path between the ingress and the egress or at the granularity of 705 individual customer's traffic or a specific set of flows to learn 706 about their congestion contribution. 708 For example, the tunnelEcnCEMarkedRatio field (specified below) 709 indicates the fraction of traffic that has been marked in the ECN 710 field of the NSH as Congestion Experienced (CE). 712 4.3 Congestion Information Delivery 714 As described above, the tunnel ingress sends a message containing 715 cumulative byte counts of packets of each type of ECN marking to the 716 tunnel egress, and the tunnel egress feeds back messages to the 717 ingress with at least one of the following: cumulative byte counts of 718 packets of each type of ECN combination, the ratio of CE-marked 719 packets to all packets, and the ratio of dropped packets to all 720 packets. (It is possible for these messages to contribute to 721 congestion.) This section specifies how the messages are conveyed. 723 IPFIX recommends, but does not require, use of SCTP [RFC4960] in 724 partial reliability mode [RFC3758] for the transport of its messages. 725 This mode allows loss of some packets, which is tolerable because 726 IPFIX communicates cumulative statistics. IPFIX over SCTP over IP 727 SHOULD be used directly where there is IP connectivity between the 728 ingress and egress; however, there might be different transport 729 protocols or address spaces used in different regions of an SFC 730 enabled domain that block such direct IP connectivity. The NSH 731 provides the general method of routing traffic within an SFC enabled 732 domain so the encapsulation of the required IPFIX traffic in NSH MUST 733 be implemented and, when IP connectivity is not available, IPFIX over 734 NSH SHOULD be used along with configuration of appropriate SFC paths 735 for the IPFIX over NSH traffic. 737 IPFIX messages could travel along the same path as network data 738 traffic. In any case, an IPFIX message packet may get lost in case of 739 network congestion. Even though the missing information could be 740 recovered because of the use of cumulative counts, the message SHOULD 741 be transmitted at a higher priority than users' traffic flows to 742 improve the promptness of congestion information feedback. 744 The ingress node can do congestion management at different 745 granularity which means both the overall aggregated inner tunnel 746 congestion level and congestion level contributed by certain traffic 747 flows could be measured for different congestion management purposes. 748 For example, if the ingress only wants to limit congestion volume 749 caused by certain traffic flows, such as UDP-based traffic, then 750 congestion volume for that traffic can be fed back; or if the ingress 751 is doing overall congestion management, the aggregated congestion 752 volume can be fed back. 754 When sending IPFIX messages from ingress to egress, the ingress acts 755 as IPFIX exporter and the egress acts as IPFIX collector; When 756 feeding back congestion level information from egress to ingress, 757 then the egress acts as IPFIX exporter and ingress acts as IPFIX 758 collector. 760 The combination of congestion level measurement and congestion 761 information delivery procedures are as following: 763 o The ingress node determines the IPFIX template record to be used. 764 The template record can be pre-configured or determined at 765 runtime, the content of the template record will be determined 766 according to the granularity of congestion management; if the 767 ingress wants to limit congestion volume contributed by specific 768 traffic flows then the elements such as source IP address, 769 destination IP address, flow ID, and CE-marked packet volume of 770 the flows, etc., will be included in the template record. 772 o Metering at the ingress measures traffic volume according to the 773 template record chosen and then the measurement records are sent 774 to the egress. 776 o Metering on the egress measures congestion level information 777 according to template record which SHOULD be the same as the 778 template record sent by the ingress. 780 o The egress sends its measurement records together with the 781 measurement records of the ingress back to the ingress. 783 4.3 IPFIX Extensions 785 This section specifies the new IPFIX Information Elements needed. It 786 conforms to [RFC7013]. 788 4.3.1 nshServicePathID 790 In order to identify SFC flows, so that congestion can be measured 791 and reported at that granularity, it is necessary for IPFIX to be 792 able to classify traffic based on the Service Path Identifier field 793 of the NSH [RFC8300]. Thus an NSH Service Path Identifier 794 (nshServicePathID) IPFIX Information Element [RFC7012] is specified. 796 Name: nshServicePathID 798 Description: Network Service Header [RFC8300] Service Path 799 Identifier. This is a 24-bit value which is left justified in 800 the Information Element. The low order byte MUST be sent as 801 zero and ignored on receipt. 803 Abstract Data Type: unsigned32 805 Data Type Semantics: identifier 807 ElementId: TBD0 809 Status: current 811 4.3.2 tunnelEcnCeCeByteTotalCount 813 Description: The total number of bytes of incoming packets with 814 the CE|CE ECN marking combination at the Observation Point 815 since the Metering Process (re-)initialization for this 816 Observation Point. 818 Abstract Data Type: unsigned64 820 Data Type Semantics: totalCounter 822 ElementId: TBD1 824 Statues: current 826 Units: bytes 828 4.3.3 tunnelEcnEctNectBytetTotalCount 830 Description: The total number of bytes of incoming packets with 831 the ECT|N-ECT ECN marking combination (ECT(0) and ECT(1) are 832 treated the same as each other) at the Observation Point since 833 the Metering Process (re-)initialization for this Observation 834 Point. 836 Abstract Data Type: unsigned64 838 Data Type Semantics: totalCounter 840 ElementId: TBD2 842 Statues: current 844 Units: bytes 846 4.3.4 tunnelEcnCeNectByteTotalCount 848 Description: The total number of bytes of incoming packets with 849 the CE|N-ECT ECN marking combination at the Observation Point 850 since the Metering Process (re-)initialization for this 851 Observation Point. 853 Abstract Data Type: unsigned64 855 Data Type Semantics: totalCounter 856 ElementId: TBD3 858 Statues: current 860 Units: bytes 862 4.3.5 tunnelEcnCeEctByteTotalCount 864 Description: The total number of bytes of incoming packets with 865 the CE|ECT ECN marking combination (ECT(0) and ECT(1) are 866 treated the same as each other) at the Observation Point since 867 the Metering Process (re-)initialization for this Observation 868 Point. 870 Abstract Data Type: unsigned64 872 Data Type Semantics: totalCounter 874 ElementId: TBD4 876 Statues: current 878 Units: bytes 880 4.3.6 tunnelEcnEctEctByteTotalCount 882 Description: The total number of bytes of incoming packets with 883 the ECT|ECT ECN marking combination (ECT(0) and ECT(1) are 884 treated the same as each other) at the Observation Point since 885 the Metering Process (re-)initialization for this Observation 886 Point. 888 Abstract Data Type: unsigned64 890 Data Type Semantics: totalCounter 892 ElementId: TBD5 894 Statues: current 896 Units: bytes 898 4.3.7 tunnelEcnCEMarkedRatio 900 Description: The ratio of packets that are CE-marked to packets 901 that are not CE-marked at the Observation Point. 903 Abstract Data Type: float32 905 ElementId: TBD6 907 Statues: current 909 5. Example of Use 911 This section provides an example of the solution described in this 912 document. 914 First, IPFIX template records are exchanged between ingress and 915 egress to negotiate the format of the data records to be exchanged. 916 The example here is to measure the congestion level for the overall 917 tunnel caused by all the traffic. After the negotiation is finished, 918 the ingress sends in-band messages to the egress containing the 919 number of each kind of ECN-marked packets (i.e., CE|CE, ECT|N-ECT and 920 ECT|ECT) received before it sent the message. 922 After the egress receives the message, the egress calculates the CE- 923 marked packet ratio and counts the number of different kinds of ECN- 924 marking packets received before it received the message. Then the 925 egress sends a feedback message containing the counts together with 926 the information in the ingress's message back to the ingress. 928 Figures 7 to 10 below illustrate the example procedure between 929 ingress and egress. 931 +---------------------------------+----------------------+ 932 |Set ID=2 Length=40 | 933 |---------------------------------|----------------------| 934 |Template ID=256 Field Count=8 | 935 |---------------------------------|----------------------| 936 |tunnelEcnCeCeByteTotalCount Field Length=8 | 937 |---------------------------------|----------------------| 938 |tunnelEcnEctNectByteTotalCount Field Length=8 | 939 |---------------------------------|----------------------| 940 |tunnelEcnEctEctByteTotalCount Field Length=8 | 941 |---------------------------------|----------------------| 942 |tunnelEcnCeNectByteTotalCount Field Length=8 | 943 |---------------------------------|----------------------| 944 |tunnelEcnCeEctByteTotalCount Field Length=8 | 945 +---------------------------------|----------------------+ 946 |tunnelEcnCEMarkedRatio Field Length=4 | 947 +---------------------------------+----------------------+ 949 Figure 7. Template Record Sent From Egress to Ingress 951 +---------------------------------+----------------------+ 952 |Set ID=2 Length=28 | 953 |---------------------------------|----------------------| 954 |Template ID=257 Field Count=3 | 955 |---------------------------------|----------------------| 956 |tunnelEcnCeCeByteTotalCount Field Length=8 | 957 |---------------------------------|----------------------| 958 |tunnelEcnEctNectByteTotalCount Field Length=8 | 959 |---------------------------------|----------------------| 960 |tunnelEcnEctEctByteTotalCount Field Length=8 | 961 |---------------------------------+----------------------| 963 Figure 8. Template Record Sent From Ingress to Egress 965 +-------+ +-------+ 966 | | +-+ +-+ +-+ +-+ +-+ +-+ +-+ | | 967 | | |P| |P| |M| |P| |P| |P| |M| | | 968 | | +-+ +-+ +-+ +-+ +-+ +-+ +-+ | | 969 | |--------------------------------------->| | 970 | | | | 971 |ingress| |egress | 972 | | +-+ +-+ | | 973 | | |M| |M| | | 974 | | +-+ +-+ | | 975 | |<---------------------------------------| | 976 | | | | 977 +-------+ +-------+ 979 +-+ 980 |M| : IPFIX Message Packet 981 +-+ 983 +-+ 984 |P| : User Data Packet 985 +-+ 987 Figure 9. Traffic flow Between Ingress and Egress 988 Set ID=257, Length=28 989 +-------+ A1 +-------+ 990 | | B1 | | 991 | | C1 | | 992 | | -----------------------------> | | 993 | | | | 994 | | | | 995 | | SetID=256, Length=72 | | 996 | | A1 | | 997 | | B1 | | 998 |ingress| C1 |egress | 999 | | A2 | | 1000 | | B2 | | 1001 | | C2 | | 1002 | | D | | 1003 | | E | | 1004 | | R | | 1005 | | <---------------------------- | | 1006 | | | | 1007 +-------+ +-------+ 1009 Figure 10. Messages Between Ingress and Egress 1011 The following provides an example of how the tunnel congestion level 1012 can be calculated (see Figure 10): 1014 The congestion Level could be divided into two categories: (1) 1015 slight congestion (no packets dropped); (2) serious congestion 1016 (packets are being dropped). 1018 For slight congestion, the congestion level is indicated by the 1019 ratio of CE-marked packets: 1021 R = ce_marked_ratio = ce-marked / total_egress ; 1023 For serious congestion, the congestion level is indicated as the 1024 volume of traffic loss: 1026 total_ingress = (A1 + B1 + C1) 1028 total_egress = (A2 + B2 + C2 + D + E) 1030 volume_loss = (total_ingress - total_egress) 1032 6. IANA Considerations 1034 The following subsections provide IANA assignment considerations. 1036 6.1 SFC NSH Header ECN Bits 1038 IANA is requested to assign two contiguous bits in the NSH Base 1039 Header Bits registry for ECN (bits 16 and 17 suggested) and note this 1040 assignment as follows: 1042 Bit Description Reference 1043 ---------- ----------- ----------------- 1044 tbd(16-17) NSH ECN [this document] 1046 6.2 IPFIX Information Element IDs 1048 IANA is requested to assign seven IPFIX Information Element IDs as 1049 follows: 1051 ElementID: TBD0 1052 Name: nshServicePathID 1053 Data Type: unsigned32 1054 Data Type Semantics: identifier 1055 Status: current 1056 Description: The Network Service Header [RFC8300] Service Path 1057 Identifier. 1059 ElementID: TBD1 1060 Name: tunnelEcnCeCePacketTotalCount 1061 Data Type: unsigned64 1062 Data Type Semantics: totalCounter 1063 Status: current 1064 Description: The total number of bytes of incoming packets with 1065 the CE|CE ECN marking combination at the Observation Point 1066 since the Metering Process (re-)initialization for this 1067 Observation Point. 1068 Units: octets 1070 ElementID: TBD2 1071 Name: tunnelEcnEctNectPacketTotalCount 1072 Data Type: unsigned64 1073 Data Type Semantics: totalCounter 1074 Status: current 1075 Description: The total number of bytes of incoming packets with 1076 the ECT|N-ECT ECN marking combination at the Observation Point 1077 since the Metering Process (re-)initialization for this 1078 Observation Point. 1079 Units: octets 1081 ElementID: TBD3 1082 Name: tunnelEcnCeNectPacketTotalCount 1083 Data Type: unsigned64 1084 Data Type Semantics: totalCounter 1085 Status: current 1086 Description: The total number of bytes of incoming packets with 1087 the CE|N-ECT ECN marking combination at the Observation Point 1088 since the Metering Process (re-)initialization for this 1089 Observation Point. 1090 Units: octets 1092 ElementID: TBD4 1093 Name: tunnelEcnCeEctPacketTotalCount 1094 Data Type: unsigned64 1095 Data Type Semantics: totalCounter 1096 Status: current 1097 Description: The total number of bytes of incoming packets with 1098 the CE|ECT ECN marking combination at the Observation Point 1099 since the Metering Process (re-)initialization for this 1100 Observation Point. 1101 Units: octets 1103 ElementID: TBD5 1104 Name: tunnelEcnEctEctPacketTotalCount 1105 Data Type: unsigned64 1106 Data Type Semantics: totalCounter 1107 Status: current 1108 Description: The total number of bytes of incoming packets with 1109 the CE|ECT(0) ECN marking combination at the Observation Point 1110 since the Metering Process (re-)initialization for this 1111 Observation Point. 1112 Units: octets 1114 ElementID: TBD6 1115 Name: tunnelEcnCEMarkedRatio 1116 Data Type: float32 1117 Status: current 1118 Description: The ratio of CE-marked Packet at the Observation 1119 Point. 1121 7. Security Considerations 1123 For general NSH security considerations, see [RFC8300]. 1125 For security considerations concerning ECN signaling tampering, see 1126 [RFC3168]. For security considerations concerning ECN and 1127 encapsulation, see [RFC6040]. 1129 For general IPFIX security considerations, see [RFC7011]. If deployed 1130 in an untrusted environment, the signaling traffic between ingress 1131 and egress can be protected utilizing the security mechanisms 1132 provided by IPFIX (see Section 11 in [RFC7011]). The tunnel 1133 endpoints (the ingress and egress for an SFC enabled domain) are 1134 assumed to be in the same administrative domain, so they will trust 1135 each other. 1137 The solution in this document does not introduce any greater 1138 potential to invade privacy than would have been available without 1139 the solution. 1141 8. Acknowledgements 1143 Most of the material on Tunnel Congestion Feedback was originally in 1144 draft-ietf-tsvwg-tunnel-congestion-feedback. After discussion with 1145 the authors of that draft, the authors of this draft, and the Chairs 1146 of the TSVWG and SFC Working Groups, the Tunnel Congestion Feedback 1147 draft was merged into this draft. 1149 The authors wish to thank the following for their comments, 1150 suggestions, and reviews: 1152 David Black, Mohamed Boucadair, Sami Boutros, Anthony Chan, 1153 Lingli Deng, Liang Geng, Joel Halpern, Jake Holland, 1154 John Kaippallimalil, Tal Mizrahi, Vincent Roca, Lei Zhu 1156 Normative References 1158 [RFC2119] - Bradner, S., "Key words for use in RFCs to Indicate 1159 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, 1160 March 1997, . 1162 [RFC3168] - Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 1163 of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 1164 10.17487/RFC3168, September 2001, . 1167 [RFC3758] - Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P. 1168 Conrad, "Stream Control Transmission Protocol (SCTP) Partial 1169 Reliability Extension", RFC 3758, DOI 10.17487/RFC3758, May 1170 2004, . 1172 [RFC5129] - Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 1173 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008, 1174 . 1176 [RFC6040] - Briscoe, B., "Tunnelling of Explicit Congestion 1177 Notification", RFC 6040, DOI 10.17487/RFC6040, November 2010, 1178 . 1180 [RFC7011] - Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 1181 "Specification of the IP Flow Information Export (IPFIX) 1182 Protocol for the Exchange of Flow Information", STD 77, RFC 1183 7011, DOI 10.17487/RFC7011, September 2013, . 1186 [RFC7013] - Trammell, B. and B. Claise, "Guidelines for Authors and 1187 Reviewers of IP Flow Information Export (IPFIX) Information 1188 Elements", BCP 184, RFC 7013, DOI 10.17487/RFC7013, September 1189 2013, . 1191 [RFC7567] - Baker, F., Ed., and G. Fairhurst, Ed., "IETF 1192 Recommendations Regarding Active Queue Management", BCP 197, 1193 RFC 7567, DOI 10.17487/RFC7567, July 2015, . 1196 [RFC8174] - Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1197 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 1198 2017, 1200 [RFC8300] - Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1201 "Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300, 1202 January 2018, . 1204 Informative References 1206 [RFC4301] - Kent, S. and K. Seo, "Security Architecture for the 1207 Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December 1208 2005, . 1210 [RFC4960] - Stewart, R., Ed., "Stream Control Transmission Protocol", 1211 RFC 4960, DOI 10.17487/RFC4960, September 2007, 1212 . 1214 [RFC7012] - Claise, B., Ed., and B. Trammell, Ed., "Information Model 1215 for IP Flow Information Export (IPFIX)", RFC 7012, DOI 1216 10.17487/RFC7012, September 2013, . 1219 [RFC7665] - Halpern, J., Ed., and C. Pignataro, Ed., "Service 1220 Function Chaining (SFC) Architecture", RFC 7665, DOI 1221 10.17487/RFC7665, October 2015, . 1224 [RFC8311] - Black, D., "Relaxing Restrictions on Explicit Congestion 1225 Notification (ECN) Experimentation", RFC 8311, DOI 1226 10.17487/RFC8311, January 2018, . 1229 [ecnL4S] - De Schepper, K., and B. Briscoe, "Identifying Modified 1230 Explicit Congestion Notification (ECN) Semantics for Ultra-Low 1231 Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s-id, work in 1232 progress. 1234 [NSHTLV] - Wei, Y., U. Elzur, S. Majee, C. Pignataro, and D. 1235 Eastlake, "Network Service Header Metadata Type 2 Variable- 1236 Length Context Headers" , draft-ietf-sfc-nsh-tlv, work in 1237 progress. 1239 Authors' Addresses 1241 Donald E. Eastlake, 3rd 1242 Futurewei Technologies 1243 2386 Panoramic Circle 1244 Apopka, FL 32703 USA 1246 Tel: +1-508-333-2270 1247 Email: d3e3e3@gmail.com 1249 Bob Briscoe 1250 Independent 1251 UK 1253 Email: ietf@bobbriscoe.net 1254 URI: http://bobbriscoe.net/ 1256 Yizhou Li 1257 Huawei Technologies 1258 101 Software Avenue, 1259 Nanjing 210012, P. R China 1261 Phone: +86-25-56624584 1262 EMail: liyizhou@huawei.com 1264 Andrew G. Malis 1265 Malis Consulting 1267 Email: agmalis@gmail.com 1269 Xinpeng Wei 1270 Huawei Technologies 1271 Beiqing Rd. Z-park No.156, Haidian District, 1272 Beijing, 100095, P. R. China 1274 EMail: weixinpeng@huawei.com 1276 Copyright and IPR Provisions 1278 Copyright (c) 2022 IETF Trust and the persons identified as the 1279 document authors. All rights reserved. 1281 This document is subject to BCP 78 and the IETF Trust's Legal 1282 Provisions Relating to IETF Documents 1283 (http://trustee.ietf.org/license-info) in effect on the date of 1284 publication of this document. Please review these documents 1285 carefully, as they describe your rights and restrictions with respect 1286 to this document. Code Components extracted from this document must 1287 include Revised BSD License text as described in Section 4.e of the 1288 Trust Legal Provisions and are provided without warranty as described 1289 in the Revised BSD License.