idnits 2.17.00 (12 Aug 2021) /tmp/idnits5834/draft-ietf-mpls-forwarding-04.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (December 17, 2013) is 3077 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: draft-ietf-mpls-special-purpose-labels has been published as RFC 7274 == Outdated reference: draft-ietf-pwe3-vccv-impl-survey-results has been published as RFC 7079 == Outdated reference: A later version (-07) exists of draft-ietf-tictoc-1588overmpls-05 -- Obsolete informational reference (is this intentional?): RFC 4379 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6424 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6829 (Obsoleted by RFC 8029) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPLS C. Villamizar, Ed. 3 Internet-Draft OCCNC 4 Intended status: Informational K. Kompella 5 Expires: June 20, 2014 Juniper Networks 6 S. Amante 7 Apple Inc. 8 A. Malis 9 Consultant 10 C. Pignataro 11 Cisco 12 December 17, 2013 14 MPLS Forwarding Compliance and Performance Requirements 15 draft-ietf-mpls-forwarding-04 17 Abstract 19 This document provides guidelines for implementers regarding MPLS 20 forwarding and a basis for evaluations of forwarding implementations. 21 Guidelines cover many aspects of MPLS forwarding. Topics are 22 highlighted where implementers might otherwise overlook practical 23 requirements which are unstated or under emphasized or are optional 24 for conformance to RFCs but are often considered mandatory by 25 providers. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on June 20, 2014. 44 Copyright Notice 46 Copyright (c) 2013 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction and Document Scope . . . . . . . . . . . . . . . 3 62 1.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.2. Use of Requirements Language . . . . . . . . . . . . . . 8 64 1.3. Apparent Misconceptions . . . . . . . . . . . . . . . . . 8 65 1.4. Target Audience . . . . . . . . . . . . . . . . . . . . . 9 66 2. Forwarding Issues . . . . . . . . . . . . . . . . . . . . . . 10 67 2.1. Forwarding Basics . . . . . . . . . . . . . . . . . . . . 10 68 2.1.1. MPLS Special Purpose Labels . . . . . . . . . . . . . 11 69 2.1.2. MPLS Differentiated Services . . . . . . . . . . . . 12 70 2.1.3. Time Synchronization . . . . . . . . . . . . . . . . 13 71 2.1.4. Uses of Multiple Label Stack Entries . . . . . . . . 14 72 2.1.5. MPLS Link Bundling . . . . . . . . . . . . . . . . . 15 73 2.1.6. MPLS Hierarchy . . . . . . . . . . . . . . . . . . . 15 74 2.1.7. MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . . 15 75 2.1.8. Pseudowire Encapsulation . . . . . . . . . . . . . . 16 76 2.1.8.1. Pseudowire Sequence Number . . . . . . . . . . . 16 77 2.1.9. Layer-2 and Layer-3 VPN . . . . . . . . . . . . . . . 18 78 2.2. MPLS Multicast . . . . . . . . . . . . . . . . . . . . . 18 79 2.3. Packet Rates . . . . . . . . . . . . . . . . . . . . . . 19 80 2.4. MPLS Multipath Techniques . . . . . . . . . . . . . . . . 21 81 2.4.1. Pseudowire Control Word . . . . . . . . . . . . . . . 22 82 2.4.2. Large Microflows . . . . . . . . . . . . . . . . . . 22 83 2.4.3. Pseudowire Flow Label . . . . . . . . . . . . . . . . 23 84 2.4.4. MPLS Entropy Label . . . . . . . . . . . . . . . . . 23 85 2.4.5. Fields Used for Multipath Load Balance . . . . . . . 23 86 2.4.5.1. MPLS Fields in Multipath . . . . . . . . . . . . 24 87 2.4.5.2. IP Fields in Multipath . . . . . . . . . . . . . 25 88 2.4.5.3. Fields Used in Flow Label . . . . . . . . . . . . 27 89 2.4.5.4. Fields Used in Entropy Label . . . . . . . . . . 27 90 2.5. MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . . 28 91 2.6. Local Delivery of Packets . . . . . . . . . . . . . . . . 28 92 2.6.1. DoS Protection . . . . . . . . . . . . . . . . . . . 28 93 2.6.2. MPLS OAM . . . . . . . . . . . . . . . . . . . . . . 30 94 2.6.3. Pseudowire OAM . . . . . . . . . . . . . . . . . . . 31 95 2.6.4. MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . . 32 96 2.6.5. MPLS OAM and Layer-2 OAM Interworking . . . . . . . . 33 97 2.6.6. Extent of OAM Support by Hardware . . . . . . . . . . 33 98 2.7. Number and Size of Flows . . . . . . . . . . . . . . . . 34 99 3. Questions for Suppliers . . . . . . . . . . . . . . . . . . . 35 100 3.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . 35 101 3.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 36 102 3.3. Multipath Capabilities and Performance . . . . . . . . . 37 103 3.4. Pseudowire Capabilities and Performance . . . . . . . . . 38 104 3.5. Entropy Label Support and Performance . . . . . . . . . . 38 105 3.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . 38 106 3.7. OAM Capabilities and Performance . . . . . . . . . . . . 39 107 4. Forwarding Compliance and Performance Testing . . . . . . . . 39 108 4.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . 40 109 4.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 40 110 4.3. Multipath Capabilities and Performance . . . . . . . . . 41 111 4.4. Pseudowire Capabilities and Performance . . . . . . . . . 42 112 4.5. Entropy Label Support and Performance . . . . . . . . . . 42 113 4.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . 43 114 4.7. OAM Capabilities and Performance . . . . . . . . . . . . 43 115 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 44 116 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 45 117 7. Security Considerations . . . . . . . . . . . . . . . . . . . 45 118 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 45 119 8.1. Normative References . . . . . . . . . . . . . . . . . . 45 120 8.2. Informative References . . . . . . . . . . . . . . . . . 46 121 Appendix A. Organization of References Section . . . . . . . . . 51 122 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 51 124 1. Introduction and Document Scope 126 The initial purpose of this document was to address concerns raised 127 on the MPLS WG mailing list about shortcomings in implementations of 128 MPLS forwarding. Documenting existing misconceptions and potential 129 pitfalls might potentially avoid repeating past mistakes. The 130 document has grown to address a broad set of forwarding requirements. 132 The focus of this document is MPLS forwarding, base pseudowire 133 forwarding, and MPLS OAM. The use of pseudowire control word, and 134 sequence number are discussed. Specific pseudowire AC and NSP are 135 out of scope. Specific pseudowire applications, such as various 136 forms of VPN, are out of scope. 138 MPLS support for multipath techniques is considered essential by many 139 service providers and is useful for other high capacity networks. In 140 order to obtain sufficient entropy from MPLS traffic service 141 providers and others find it essential for the MPLS implementation to 142 interpret the MPLS payload as IPv4 or IPv6 based on the contents of 143 the first nibble of payload. The use of IP addresses, the IP 144 protocol field, and UDP and TCP port number fields in multipath load 145 balancing are considered within scope. The use of any other IP 146 protocol fields, such as tunneling protocols carried within IP, are 147 out of scope. 149 Implementation details are a local matter and are out of scope. Most 150 interfaces today operate at 1 Gb/s or greater. It is assumed that 151 all forwarding operations are implemented in specialized forwarding 152 hardware rather than on a general purpose processor. This is often 153 referred to as "fast path" and "slow path" processing. Some 154 recommendations are made regarding implementing control or management 155 plane functionality in specialized hardware or with limited 156 assistance from specialized hardware. This advise is based on 157 expected control or management protocol loads and on the need for 158 denial of service (DoS) protection. 160 1.1. Acronyms 162 The following acronyms are used. 164 AC Attachment Circuit ([RFC3985]) 166 ACH Associated Channel Header (pseudowires) 168 ACK Acknowledgement (TCP flag and type of TCP packet) 170 AIS Alarm Indication Signal (MPLS-TP OAM) 172 ATM Asynchronous Transfer Mode (legacy switched circuits) 174 BFD Bidirectional Forwarding Detection 176 BGP Border Gateway Protocol 178 CC-CV Connectivity Check and Connectivity Verification 180 CE Customer Edge (LDP) 182 CPU Central Processing Unit (computer or microprocessor) 184 CT Class Type ([RFC4124]) 186 CW Control Word ([RFC4385]) 187 DCCP Datagram Congestion Control Protocol 189 DDoS Distributed Denial of Service 191 DM Delay Measurement (MPLS-TP OAM) 193 DSCP Differentiated Services Code Point ([RFC2474]) 195 DWDM Dense Wave Division Multiplexing 197 DoS Denial of Service 199 E-LSP EXP-Inferred-PSC LSP ([RFC3270]) 201 EBGP External BGP 203 ECMP Equal Cost Multi-Path 205 ECN Explicit Congestion Notification ([RFC3168] and [RFC5129]) 207 EL Entropy Label ([RFC6790]) 209 ELI Entropy Label Indicator ([RFC6790]) 211 EXP Experimental (field in MPLS renamed to TC in [RFC5462]) 213 FEC Forwarding Equivalence Classes (LDP), also Forward Error 214 Correction in other context 216 FR Frame Relay (legacy switched circuits) 218 FRR Fast Reroute ([RFC4090]) 220 G-ACh Generic Associated Channel ([RFC5586]) 222 GAL Generic Associated Channel Label ([RFC5586]) 224 GFP Generic Framing Protocol (used in OTN) 226 GMPLS Generalized MPLS ([RFC3471]) 228 GTSM Generalized TTL Security Mechanism ([RFC5082]) 230 Gb/s Gigabits per second (billion bits per second) 232 IANA Internet Assigned Numbers Authority 234 ILM Incoming Label Map ([RFC3031]) 235 IP Internet Protocol 237 IPVPN Internet Protocol VPN 239 IPv4 Internet Protocol version 4 241 IPv6 Internet Protocol version 6 243 L-LSP Label-Only-Inferred-PSC LSP ([RFC3270]) 245 L2VPN Layer 2 VPN 247 LDP Label Distribution Protocol ([RFC5036]) 249 LER Label Edge Router ([RFC3031]) 251 LM Loss Measurement (MPLS-TP OAM) 253 LSP Label Switched Path ([RFC3031]) 255 LSR Label Switching Router ([RFC3031]) 257 MP2MP Multipoint to Point 259 MPLS MultiProtocol Label Switching ([RFC3031]) 261 MPLS-TP MPLS Transport Profile ([RFC5317]) 263 Mb/s Megabits per second (million bits per second) 265 NSP Native Service Processing ([RFC3985]) 267 NTP Network Time Protocol 269 OAM Operations, Administration, and Maintenance ([RFC6291]) 271 OOB Out-of-band (not carried within a data channel) 273 OTN Optical Transport Network 275 P Provider router (LDP) 277 P2MP Point to Multi-Point 279 PE Provider Edge router (LDP) 281 PHB Per-Hop-Behavior ([RFC2475]) 282 PHP Penultimate Hop Popping ([RFC3443]) 284 POS Packet over SONET 286 PSC This acronym has multiple interpretations. 288 1. Packet Switch Capable ([RFC3471] 290 2. PHB Scheduling Class ([RFC3270]) 292 3. Protection State Coordination (MPLS-TP linear protection) 294 PTP Precision Time Protocol 296 PW Pseudowire 298 QoS Quality of Service 300 RA Router Alert ([RFC3032]) 302 RDI Remote Defect Indication (MPLS-TP OAM) 304 RSVP-TE RSVP Traffic Engineering 306 RTP Real-Time Transport Protocol 308 SCTP Stream Control Transmission Protocol 310 SDH Synchronous Data Hierarchy (European SONET, a form of TDM) 312 SONET Synchronous Optical Network (US SDH, a form of TDM) 314 T-LDP Targeted LDP (LDP sessions over more than one hop) 316 TC Traffic Class ([RFC5462]) 318 TCP Transmission Control Protocol 320 TDM Time-Division Multiplexing (legacy encapsulations) 322 TOS Type of Service (see [RFC2474]) 324 TTL Time-to-live (a field in IP and MPLS headers) 326 UDP User Datagram Protocol 328 UHP Ultimate Hop Popping (opposite of PHP) 329 VCCV Virtual Circuit Connectivity Verification ([RFC5085]) 331 VLAN Virtual Local Area Network (Ethernet) 333 VOQ Virtual Output Queuing (switch fabric design) 335 VPN Virtual Private Network 337 WG Working Group 339 1.2. Use of Requirements Language 341 This document is informational. The upper case [RFC2119] key words 342 are not used in this document, except in the following cases. 344 1. RFC 2119 keywords are used where requirements stated in this 345 document are called for in referenced RFCs. In most cases the 346 RFC containing the requirement is cited within the statement 347 using an RFC 2119 keyword. 349 2. RFC 2119 keywords are used where explicitly noted that the 350 keywords indicate that operator experiences indicate a 351 requirement, but there are no existing RFC requirements. 353 Advice provided by this document may be ignored by implementations. 354 Similarly, implementations not claiming conformance to specific RFCs 355 may ignore the requirements of those RFCs. In both cases, 356 implementers should consider the risk of doing so. 358 1.3. Apparent Misconceptions 360 In early generations of forwarding silicon (which might now be behind 361 us), there apparently were some misconceptions about MPLS. The 362 following statements provide clarifications. 364 1. There are practical reasons to have more than one or two labels 365 in an MPLS label stack. Under some circumstances the label stack 366 can become quite deep. See Section 2.1. 368 2. The label stack MUST be considered to be arbitrarily deep. 369 Section 3.27.4. "Hierarchy: LSP Tunnels within LSPs" of RFC3031 370 states "The label stack mechanism allows LSP tunneling to nest to 371 any depth." [RFC3031] If a bottom of the label stack cannot be 372 found, but sufficient number of labels exist to forward, an LSR 373 MUST forward the packet. An LSR MUST NOT assume the packet is 374 malformed unless the end of packet is found before bottom of 375 stack. See Section 2.1. 377 3. In networks where deep label stacks are encountered, they are not 378 rare. Full packet rate performance is required regardless of 379 label stack depth, except where multiple pop operations are 380 required. See Section 2.1. 382 4. Research has shown that long bursts of short packets with 40 byte 383 or 44 byte IP payload sizes in these bursts are quite common. 384 This is due to TCP ACK compression [ACK-compression]. The 385 following two sub-bullets constitutes advice that reflects very 386 common hard requirements of providers. Implementers may ignore 387 this advice but should consider the risk of doing so. 389 a. A forwarding engine SHOULD, if practical, be able to sustain 390 an arbitrarily long sequence of small packets arriving at 391 full interface rate. 393 b. If indefinite full packet rate for small packets is not 394 practical, a forwarding engine MUST be able to buffer a long 395 sequence of small packets inbound to the on-chip decision 396 engine and sustain full interface rate for some reasonable 397 average packet rate. Absent this small on-chip buffering, 398 QoS agnostic packet drops can occur. 400 See Section 2.3. 402 5. The implementer and system designer MUST support pseudowire 403 control word (CW) if MPLS-TP is supported or if ACH [RFC5586] is 404 being used on a pseudowire. The implementer and system designer 405 SHOULD support pseudowire CW even if MPLS-TP and ACH [RFC5586] 406 are not used, using instead CW and VCCV Type 1 [RFC5085] to allow 407 the use of multipath in the underlying network topology without 408 impacting the PW traffic. 409 [I-D.ietf-pwe3-vccv-impl-survey-results] does note that there are 410 still some deployments where the CW is not always used. It also 411 notes that many service providers do enable the CW. See 412 Section 2.4.1 for more discussion on why deployments SHOULD 413 enable the pseudowire CW. 415 6. The implementer and system designer SHOULD support adding a 416 pseudowire Flow Label [RFC6391]. Deployments MAY enable this 417 feature for appropriate pseudowire types. See Section 2.4.3. 419 7. The implementer and system designer SHOULD support adding an MPLS 420 entropy label [RFC6790]. Deployments MAY enable this feature. 421 See Section 2.4.4. 423 1.4. Target Audience 424 This document is intended for multiple audiences: implementer 425 (implementing MPLS forwarding in silicon or in software); systems 426 designer (putting together a MPLS forwarding systems); deployer 427 (running an MPLS network). These guidelines are intended to serve 428 the following purposes: 430 1. Explain what to do and what not to do when a deep label stack is 431 encountered. (audience: implementer) 433 2. Highlight pitfalls to look for when implementing an MPLS 434 forwarding chip. (audience: implementer) 436 3. Provide a checklist of features and performance specifications to 437 request. (audience: systems designer, deployer) 439 4. Provide a set of tests to perform. (audience: systems designer, 440 deployer). 442 The implementer, systems designer, and deployer have a transitive 443 supplier customer relationship. It is in the best interest of the 444 supplier to review their product against their customer's checklist 445 and customer's customer's checklist if applicable. 447 2. Forwarding Issues 449 A brief review of forwarding issues is provided in the subsections 450 that follow. This section provides some background on why some of 451 these requirements exist. The questions to ask of suppliers is 452 covered in Section 3. Some guidelines for testing are provided in 453 Section 4. 455 2.1. Forwarding Basics 457 Basic MPLS architecture and MPLS encapsulation, and therefore packet 458 forwarding are defined in [RFC3031] and [RFC3032]. RFC3031 and 459 RFC3032 are somewhat LDP centric. RSVP-TE supports traffic 460 engineering (TE) and fast reroute, features that LDP lacks. The base 461 document for RSVP-TE based MPLS is [RFC3209]. 463 A few RFCs update RFC3032. Those with impact on forwarding include 464 the following. 466 1. TTL processing is clarified in [RFC3443]. 468 2. The use of MPLS Explicit NULL is modified in [RFC4182]. 470 3. Differentiated Services is supported by [RFC3270] and [RFC4124]. 471 The "EXP" field is renamed to "Traffic Class" in [RFC5462], 472 removing any misconception that it was available for 473 experimentation or could be ignored. 475 4. ECN is supported by [RFC5129]. 477 5. The MPLS G-ACh and GAL are defined in [RFC5586]. 479 6. [RFC5332] redefines the two data link layer codepoints for MPLS 480 packets. 482 Tunneling encapsulations which may carry MPLS, such as MPLS in GRE, 483 L2TP, or LDP, are out of scope. 485 Other RFCs have implications to MPLS Forwarding and do not update 486 RFC3032 or RFC3209, including: 488 1. The pseudowire (PW) Associated Channel Header (ACH), defined by 489 [RFC5085], later generalized by the MPLS G-ACh [RFC5586]. 491 2. The entropy label indicator (ELI) and entropy label (EL) are 492 defined by [RFC6790]. 494 A few RFCs update RFC3209. Those that are listed as updating RFC3209 495 generally impact only RSVP-TE signaling. Forwarding is modified by 496 major extension built upon RFC3209. 498 RFCs which impact forwarding are discussed in the following 499 subsections. 501 2.1.1. MPLS Special Purpose Labels 503 [RFC3032] specifies that label values 0-15 are special purpose labels 504 with special meanings. Three values of NULL label are defined (two 505 of which are later updated by [RFC4182]) and a router-alert label is 506 defined. The original intent was that special purpose labels, except 507 the NULL labels, could be sent to the routing engine CPU rather than 508 be processed in forwarding hardware. Hardware support is required by 509 new RFCs such as those defining entropy label and OAM processed as a 510 result of receiving a GAL. For new special purpose labels, some 511 accommodation is needed for LSR that will send the labels to a 512 general purpose CPU or other highly programmable hardware. For 513 example, ELI will only be sent to LSR which have signaled support for 514 [RFC6790] and high OAM packet rate must be negotiated among 515 endpoints. 517 [RFC3429] reserves a label for ITU-T Y.1711, however Y.1711 does not 518 work with multipath and its use is strongly discouraged. 520 The current list of special purpose labels can be found on the 521 "Multiprotocol Label Switching Architecture (MPLS) Label Values" 522 registry reachable at IANA's pages at http://www.iana.org. 524 [I-D.ietf-mpls-special-purpose-labels] introduces an IANA "Extended 525 Special Purpose MPLS Label Values" registry and makes use of the 526 "extension" label, label 15, to indicate that the next label is an 527 extended special purpose label and requires special handling. The 528 range of only 16 values for special purpose labels allows a table to 529 be used. The range of extended special purpose labels with 20 bits 530 available for use may have to be handled in some other way in the 531 unlikely event that in the future the range of currently reserved 532 values 256-1048575 are used. If only the standards action range, 533 16-239, and the experimental range, 240-255, are used, then a table 534 of 256 entries can be used. 536 Unknown special purpose labels and unknown extended special purpose 537 labels are handled the same. When an unknown special purpose label 538 is encountered or a special purpose label not directly handled in 539 forwarding hardware is encountered, the packet should be sent to a 540 general purpose CPU by default. If this capability is supported, 541 there must be an option to either drop or rate limit such packets on 542 a per special purpose label value basis. 544 2.1.2. MPLS Differentiated Services 546 [RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence 547 (Prec) fields and replaces them with the Differentiated Services 548 Field more commonly known as the Differentiated Services Code Point 549 (DSCP) field. [RFC2475] defines the Differentiated Services 550 architecture, which in other forum is often called a Quality of 551 Service (QoS) architecture. 553 MPLS uses the Traffic Class (TC) field to support Differentiated 554 Services [RFC5462]. There are two primary documents describing how 555 DSCP is mapped into TC. 557 1. [RFC3270] defines E-LSP and L-LSP. E-LSP use a static mapping of 558 DSCP into TC. L-LSP uses a per LSP mapping of DSCP into TC, with 559 one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use 560 multiple Per-Hop Behavior (PHB) values. For example, the Assured 561 Forwarding service defines three PSC, each with three PHB 562 [RFC2597]. 564 2. [RFC4124] defines assignment of a class-type (CT) to an LSP, 565 where a per CT static mapping of TC to PHB is used. [RFC4124] 566 provides a means to support up to eight E-LSP-like mappings of 567 DSCP to TC. 569 To meet Differentiated Services requirements specified in [RFC3270], 570 the following forwarding requirements must be met. An ingress LER 571 MUST be able to select an LSP and then apply a per LSP map of DSCP 572 into TC. A midpoint LSR MUST be able to apply a per LSP map of TC to 573 PHB. The number of mappings supported will be far less than the 574 number of LSP supported. 576 To meet Differentiated Services requirements specified in [RFC4124], 577 the following forwarding requirements must be met. An ingress LER 578 MUST be able to select an LSP and then apply a per LSP map of DSCP 579 into TC. A midpoint LSR MUST be able to apply a per LSP map to CT 580 map and then use Class Type (CT) to map TC to PHB. Since there are 581 only eight allowed values of CT, only eight maps of TC to PHB need to 582 be supported. The LSP label can be used directly to find the TC to 583 PHB mapping, as is needed to support [RFC3270] L-LSP. 585 While support for [RFC4124] and not [RFC3270] would allow support for 586 only eight mappings of TC to PHB, it is common to support both and 587 simply state a limit on the number of unique TC to PHB mappings which 588 can be supported. 590 2.1.3. Time Synchronization 592 PTP or NTP may be carried over MPLS [I-D.ietf-tictoc-1588overmpls]. 593 Generally NTP will be carried within IP with IP carried in MPLS 594 [RFC5905]. Both PTP and NTP benefit from accurate time stamping of 595 incoming packets and the ability to insert accurate time stamps in 596 outgoing packets. PTP correction which occurs when forwarding 597 requires updating a timestamp compensation field based on the 598 difference between packet arrival at an LSR and packet transmit time 599 at that same LSR. 601 Since the label stack depth may vary, hardware should allow a 602 timestamp to be placed in an outgoing packet at any specified byte 603 position. It may be necessary to modify layer-2 checksums or frame 604 check sequences after insertion. PTP and NTP timestamp formats 605 differ slightly. If NTP or PTP is carried over UDP/IP or UDP/IP/ 606 MPLS, the UDP checksum will also have to be updated. 608 Accurate time synchronization in addition to being generally useful 609 is required for MPLS-TP delay measurement (DM) OAM. See 610 Section 2.6.4. 612 2.1.4. Uses of Multiple Label Stack Entries 614 MPLS deployments in the early part of the prior decade (circa 2000) 615 tended to support either LDP or RSVP-TE. LDP was favored by some for 616 its ability to scale to a very large number of PE devices at the edge 617 of the network, without adding deployment complexity. RSVP-TE was 618 favored, generally in the network core, where traffic engineering and 619 /or fast reroute were considered important. 621 Both LDP and RSVP-TE are used simultaneously within major Service 622 Provider networks using a technique known as "LDP over RSVP-TE 623 Tunneling". This technique allows service providers to carry LDP 624 tunnels inside RSVP-TE tunnels. This makes it possible to take 625 advantage of the Traffic Engineering and Fast Re-Route on more 626 expensive Inter-City and Inter-Continental transport paths. The 627 ingress RSVP-TE PEs places many LDP tunnels on a single RSVP-TE LSP 628 and carries it to the egress RSVP-TE PE. The LDP PEs are situated 629 further from the core, for example within a metro network. LDP over 630 RSVP-TE tunneling requires a minimum of two MPLS labels: one each for 631 LDP and RSVP-TE. 633 The use of MPLS FRR [RFC4090] might add one more label to MPLS 634 traffic, but only when FRR protection is in use (active). If LDP 635 over RSVP-TE is in use, and FRR protection is in use, then at least 636 three MPLS labels are present on the label stack on the links through 637 which the Bypass LSP traverses. FRR is covered in Section 2.1.7. 639 LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN 640 services that are deployed by the vast majority of service providers. 641 These VPN services added yet another label, bringing the label stack 642 depth (when FRR is active) to four. 644 Pseudowires and VPN are discussed in further detail in Section 2.1.8 645 and Section 2.1.9. 647 MPLS hierarchy as described in [RFC4206] can in principle add up to 648 four additional labels. MPLS hierarchy is discussed in 649 Section 2.1.6. 651 Other features such as Entropy Label (discussed in Section 2.4.4) and 652 Flow Label (discussed in Section 2.4.3) can add additional labels to 653 the label stack. 655 Although theoretical scenarios can easily result in eight or more 656 labels, such cases are rare if they occur at all today. For the 657 purpose of forwarding, only the top label needs to be examined if PHP 658 is used, a few more if UHP is used (see Section 2.5). For deep label 659 stacks, quite a few labels may have to be examined for the purpose of 660 load balancing across parallel links (see Section 2.4), however this 661 depth can be bounded by a provider through use of Entropy Label. 663 2.1.5. MPLS Link Bundling 665 MPLS Link Bundling was the first RFC to address the need for multiple 666 parallel links between nodes [RFC4201]. MPLS Link Bundling is 667 notable in that it tried not to change MPLS forwarding, except in 668 specifying the "All-Ones" component link. MPLS Link Bundling is 669 seldom if ever deployed. Instead multipath techniques described in 670 Section 2.4 are used. 672 2.1.6. MPLS Hierarchy 674 MPLS hierarchy is defined in [RFC4206]. Although RFC4206 is 675 considered part of GMPLS, the Packet Switching Capable (PSC) portion 676 of the MPLS hierarchy are applicable to MPLS and may be supported in 677 an otherwise GMPLS free implementation. The MPLS PSC hierarchy 678 remains the most likely means of providing further scaling in an 679 RSVP-TE MPLS network, particularly where the network is designed to 680 provide RSVP-TE connectivity to the edges. This is the case for 681 envisioned MPLS-TP networks. The use of the MPLS PSC hierarchy can 682 add as many as four labels to a label stack, though it is likely that 683 only one layer of PSC will be used in the near future. 685 2.1.7. MPLS Fast Reroute (FRR) 687 Fast reroute is defined by [RFC4090]. Two significantly different 688 methods are defined in RFC4090, the "One-to-One Backup" method which 689 uses the "Detour LSP" and the " Facility Backup" which uses a "bypass 690 tunnel". These are commonly referred to as the detour and bypass 691 methods respectively. 693 The detour method makes use of a presignaled LSP. Hardware 694 assistance is needed for detour FRR only if necessary to accomplish 695 local repair of a large number of LSP within the 10s of milliseconds 696 target. For each affected LSP a swap operation must be reprogrammed 697 or otherwise switched over. The use of detour FRR doubles the number 698 of LSP terminating at any given hop and will increase the number of 699 LSP within a network by a factor dependent on the average detour path 700 length. 702 The bypass method makes use of a tunnel that is unused when no fault 703 exists but may carry many LSP when a local repair is required. There 704 is no presignaling indicating which working LSP will be diverted into 705 any specific bypass LSP. The merge LSR (egress LSR of the bypass 706 LSP) MUST use platform label space (as defined in [RFC3031]) so that 707 an LSP working path on any give interface can be backed up using a 708 bypass LSP terminating on any other interface. Hardware assistance 709 is needed if necessary to accomplish local repair of a large number 710 of LSP within the 10s of milliseconds target. For each affected LSP 711 a swap operation must be reprogrammed or otherwise switched over with 712 an additional push of the bypass LSP label. The use of platform 713 label space impacts the size of the LSR ILM for LSR with a very large 714 number of interfaces. 716 2.1.8. Pseudowire Encapsulation 718 The pseudowire (PW) architecture is defined in [RFC3985]. A 719 pseudowire, when carried over MPLS, adds one or more additional label 720 entries to the MPLS label stack. A PW Control Word is defined in 721 [RFC4385] with motivation for defining the control word in [RFC4928]. 722 The PW Associated Channel defined in [RFC4385] is used for OAM in 723 [RFC5085]. The PW Flow Label is defined in [RFC6391] and is 724 discussed further in this document in Section 2.4.3. 726 There are numerous pseudowire encapsulations, supporting emulation of 727 services such as Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over 728 packet switched networks (PSNs) using IP or MPLS. 730 The pseudowire encapsulation is out of scope for this document. 731 Pseudowire impact on MPLS forwarding at midpoint LSR is within scope. 732 The impact on ingress MPLS push and egress MPLS UHP pop are within 733 scope. While pseudowire encapsulation is out of scope, some advice 734 is given on sequence number support. 736 2.1.8.1. Pseudowire Sequence Number 738 Pseudowire (PW) sequence number support is most important for PW 739 payload types with a high expectation of lossless and/or in-order 740 delivery. Identifying lost PW packets and exact amount of lost 741 payload is critical for PW services which maintain bit timing, such 742 as Time Division Multiplexing (TDM) services since these services 743 MUST compensate lost payload on a bit-for-bit basis. 745 With PW services which maintain bit timing, packets that have been 746 received out of order also MUST be identified and may be either re- 747 ordered or dropped. Reordering requires, in addition to sequence 748 numbering, a "reorder buffer" in the egress PE, and ability to 749 reorder is limited by the depth of this buffer. The down side of 750 maintaining a large reorder buffer is added end-to-end service delay. 752 For PW services which maintain bit timing or any other service where 753 jitter must be bounded, a jitter buffer is always necessary. The 754 jitter buffer is needed regardless of whether reordering is done. In 755 order to be effective, a reorder buffer must often be larger than a 756 jitter buffer needs to be creating a tradeoff between reducing loss 757 and minimizing delay. 759 PW services which are not timing critical bit streams in nature are 760 cell oriented or frame oriented. Though resequencing support may be 761 beneficial to PW cell and frame oriented payloads such as ATM, FR and 762 Ethernet, this support is desirable but not required. Requirements 763 to hamdle out of order packets at all vary among services and 764 deployments. For example for Ethernet PW, occasional (very rare) 765 reordering is usually acceptable. If the Ethernet PW is carrying 766 MPLS-TP, then this reordering may be acceptable. 768 Reducing jitter is best done by an end-system, given that the 769 tradeoff of loss vs delay varies among services. For example with 770 interactive real time services low delay is preferred, while with 771 non-interactive (one way) real time services low loss is preferred. 772 The same end-site may be receiving both types of traffic. Regardless 773 of this, bounded jitter is sometimes a requiremnet for specific 774 deployments. 776 Packet reorder should be rare except in a small number of 777 circumstances, most of which are due to network design or equipment 778 design errors: 780 1. The most common case is where reordering occurs is rare, 781 occurring only when a network or equipment fault forces traffic 782 on a new path with different delay. The packet loss that 783 accompanies a network or equipment fault is generally more 784 disruptive than any reordering which may occur. 786 2. A path change can be caused by reasons other than a network or 787 equipment fault, such as administrative routing change. This may 788 result in packet reordering but generally without any packet 789 loss. 791 3. If the edge is not using pseudowire control word (CW) and the 792 core is using multipath, reordering will be far more common. If 793 this is occurring, the best solution is to use CW on the edge, 794 rather than try to fix the reordering using resequencing. 796 4. Another avoidable case is where some core equipment has multipath 797 and for some reason insists on periodically installing a new 798 random number as the multipath hash seed. If supporting MPLS-TP, 799 equipment MUST provide a means to disable periodic hash reseeding 800 and deployments MUST disable periodic hash reseeding. Even if 801 not supporting MPLS-TP, equipment should provide a means to 802 disable periodic hash reseeding and deployments should disable 803 periodic hash reseeding. 805 In provider networks which use multipath techniques and which may 806 occassionally rebalance traffic or which may change PW paths 807 occasionally for other reasons, reordering may be far more common 808 than loss. Where reordering is more common than loss, resequencing 809 packets is beneficial, rather than dropping packets at egress when 810 out of order arrival occus. Resequencing is most important for PW 811 payload types with a high expectation of lossless delivery since in 812 such cases out of order delivery within the network results in PW 813 loss. 815 2.1.9. Layer-2 and Layer-3 VPN 817 Layer-2 VPN [RFC4664] and Layer-3 VPN [RFC4110] add one or more label 818 entry to the MPLS label stack. VPN encapsulations are out of scope 819 for this document. Its impact on forwarding at midpoint LSR are 820 within scope. 822 Any of these services may be used on an MPLS entropy label enabled 823 ingress and egress (see Section 2.4.4 for discussion of entropy 824 label) which would add an additional two labels to the MPLS label 825 stack. The need to provide a useful entropy label value impacts the 826 requirements of the VPN ingress LER but is out of scope for this 827 document. 829 2.2. MPLS Multicast 831 MPLS Multicast encapsulation is clarified in [RFC5332]. MPLS 832 Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388]. 834 [RFC4875] defines a root initiated RSVP-TE LSP setup rather than leaf 835 initiated join used in IP multicast. [RFC6388] defines a leaf 836 initiated LDP setup. Both [RFC4875] and [RFC6388] define point to 837 multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint to 838 multipoint (MP2MP) LSP setup. 840 The P2MP LSP have a single source. An LSR may be a leaf node, an 841 intermediate node, or a "bud" node. A bud serves as both a leaf and 842 intermediate. At a leaf an MPLS pop is performed. The payload may 843 be a IP Multicast packet that requires further replication. At an 844 intermediate node a MPLS swap operation is performed. The bud 845 requires that both a pop operation and a swap operation be performed 846 for the same incoming packet. 848 One strategy to support P2MP functionality is to pop at the LSR 849 interface serving as ingress to the P2MP traffic and then optionally 850 push labels at each LSR interface serving as egress to the P2MP 851 traffic at that same LSR. A given LSR egress chip may support 852 multiple egress interfaces, each of which requires a copy, but each 853 with a different set of added labels and layer-2 encapsulation. Some 854 physical interfaces may have multiple sub-interfaces (such as 855 Ethernet VLAN or channelized interfaces) each requiring a copy. 857 If packet replication is performed at LSR ingress, then the ingress 858 interface performance may suffer. If the packet replication is 859 performed within a LSR switching fabric and at LSR egress, congestion 860 of egress interfaces cannot make use of backpressure to ingress 861 interfaces using techniques such as virtual output queuing (VOQ). If 862 buffering is primarily supported at egress, then the need for 863 backpressure is minimized. There may be no good solution for high 864 volumes of multicast traffic if VOQ is used. 866 MP2MP LSP differ in that any branch may provide an input, including a 867 leaf. Packets must be replicated onto all other branches. This 868 forwarding is often implemented as multiple P2MP forwarding trees, 869 one for each potential input interface at a given LSR. 871 2.3. Packet Rates 873 While average packet size of Internet traffic may be large, long 874 sequences of small packets have both been predicted in theory and 875 observed in practice. Traffic compression and TCP ACK compression 876 can conspire to create long sequences of packets of 40-44 bytes in 877 payload length. If carried over Ethernet, the 64 byte minimum 878 payload applies, yielding a packet rate of approximately 150 Mpps 879 (million packets per second) for the duration of the burst on a 880 nominal 100 Gb/s link. The peak rate for other encapsulations can be 881 as high as 250 Mpps (for example IP or MPLS encapsulated using GFP 882 over OTN ODU4). 884 It is possible that the packet rates achieved by a specific 885 implementation is acceptable for a minimum payload size, such as 64 886 byte (64B) payload for Ethernet, but the acheived rate declines to an 887 unacceptable level for other packet sizes, such as 65B payload. 888 There are other packet rates of interest besides TCP ACK. For 889 example, a TCP ACK carried over an Ethernet PW over MPLS over 890 Ethernet may occupy 82B or 82B plus an increment of 4B if additional 891 MPLS labels are present. 893 A graph of packet rate vs. packet size often displays a sawtooth. 894 The sawtooth is commonly due to a memory bottleneck and memory 895 widths, sometimes internal cache, but often a very wide external 896 buffer memory interface. In some cases it may be due to a fabric 897 transfer width. A fine packing, rounding up to the nearest 8B or 16B 898 will result in a fine sawtooth with small degradation for 65B, and 899 even less for 82B packets. A course packing, rounding up to 64B can 900 yield a sharper drop in performance for 65B packets, or perhaps more 901 important, a larger drop for 82B packets. 903 The loss of some TCP ACK packets are not the primary concern when 904 such a burst occurs. When a burst occurs, any other packets, 905 regardless of packet length and packet QoS are dropped once on-chip 906 input buffers prior to the decision engine are exceeded. Buffers in 907 front of the packet decision engine are often very small or non- 908 existent (less than one packet of buffer) causing significant QoS 909 agnostic packet drop. 911 Internet service providers and content providers at one time 912 specified full rate forwarding with 40 byte payload packets as a 913 requirement. Today, this requirement often can be waived if the 914 provider can be convinced that when long sequence of short packets 915 occur no packets will be dropped. 917 Many equipment suppliers have pointed out that the extra cost in 918 designing hardware capable of processing the minimum size packets at 919 full line rate is significant for very high speed interfaces. If 920 hardware is not capable of processing the minimum size packets at 921 full line rate, then that hardware MUST be capable of handling large 922 burst of small packets, a condition which is often observed. This 923 level of performance is necessary to meet Differentiated Services 924 [RFC2475] requirements for without it, packets are lost prior to 925 inspection of the IP DSCP field [RFC2474] or MPLS TC field [RFC5462]. 927 With adequate on-chip buffers before the packet decision engine, an 928 LSR can absorb a long sequence of short packets. Even if the output 929 is slowed to the point where light congestion occurs, the packets, 930 having cleared the decision process, can make use of larger VOQ or 931 output side buffers and be dealt with according to configured QoS 932 treatment, rather than dropped completely at random. 934 These on-chip buffers need not contribute significant delay since 935 they are only used when the packet decision engine is unable to keep 936 up, not in response to congestion, plus these buffers are quite 937 small. For example, an on-chip buffer capable of handling 4K packets 938 of 64 bytes in length, or 256KB, corresponds to 2 msec on a 10 Mb/s 939 link and 0.2 usec on a 100 Gb/s link. If the packet decision engine 940 is capable of handling packets at 90% of the full rate for small 941 packets, then the maximum added delay is 0.2 msec and 20 nsec 942 respectively, and this delay only applies if a 4K burst of short 943 packets occurs. When no burst of short packets was being processed, 944 no delay is added. 946 Packet rate requirements apply regardless of which network tier 947 equipment is deployed in. Whether deployed in the network core or 948 near the network edges, one of the two conditions MUST be met if 949 Differentiated Services requirements are to be met: 951 1. Packets must be processed at full line rate with minimum sized 952 packets. -OR- 954 2. Packets must be processed at a rate well under generally accepted 955 average packet sizes, with sufficient buffering prior to the 956 packet decision engine to accommodate long bursts of small 957 packets. 959 2.4. MPLS Multipath Techniques 961 In any large provider, service providers and content providers, hash 962 based multipath techniques are used in the core and in the edge. In 963 many of these providers hash based multipath is also used in the 964 larger metro networks. 966 The most common multipath techniques are ECMP applied at the IP 967 forwarding level, Ethernet LAG with inspection of the IP payload, and 968 multipath on links carrying both IP and MPLS, where the IP header is 969 inspected below the MPLS label stack. In most core networks, the 970 vast majority of traffic is MPLS encapsulated. 972 In order to support an adequately balanced load distribution across 973 multiple links, IP header information must be used. Common practice 974 today is to reinspect the IP headers at each LSR and use the label 975 stack and IP header information in a hash performed at each LSR. 976 Further details are provided in Section 2.4.5. 978 The use of this technique is so ubiquitous in provider networks that 979 lack of support for multipath makes any product unsuitable for use in 980 large core networks. This will continue to be the case in the near 981 future, even as deployment of MPLS entropy label begins to relax the 982 core LSR multipath performance requirements given the existing 983 deployed base of edge equipment without the ability to add an entropy 984 label. 986 A generation of edge equipment supporting the ability to add an MPLS 987 entropy label is needed before the performance requirements for core 988 LSR can be relaxed. However, it is likely that two generations of 989 deployment in the future will allow core LSR to support full packet 990 rate only when a relatively small number of MPLS labels need to be 991 inspected before hashing. For now, don't count on it. 993 Common practice today is to reinspect the packet at each LSR and use 994 information from the packet combined plus a hash seed that is 995 selected by each LSR. Where flow labels or entropy labels are used, 996 a hash seed must be used when creating these labels. 998 2.4.1. Pseudowire Control Word 1000 Within the core of a network some form of multipath is almost certain 1001 to be used. Multipath techniques deployed today are likely to be 1002 looking beneath the label stack for an opportunity to hash on IP 1003 addresses. 1005 A pseudowire encapsulated at a network edge must have a means to 1006 prevent reordering within the core if the pseudowire will be crossing 1007 a network core, or any part of a network topology where multipath is 1008 used (see [RFC4385] and [RFC4928]). 1010 Not supporting the ability to encapsulate a pseudowire with a control 1011 word may lock a product out from consideration. A pseudowire 1012 capability without control word support might be sufficient for 1013 applications that are strictly both intra-metro and low bandwidth. 1014 However a provider with other applications will very likely not 1015 tolerate having equipment which can only support a subset of their 1016 pseudowire needs. 1018 2.4.2. Large Microflows 1020 Where multipath makes use of a simple hash and simple load balance 1021 such as modulo or other fixed allocation (see Section 2.4) the 1022 presence of large microflows that each consumes 10% of the capacity 1023 of a component link of a potentially congested composite link, one 1024 such microflow can upset the traffic balance and more than one can in 1025 effect reduce the effective capacity of the entire composite link by 1026 more than 10%. 1028 When even a very small number of large microflows are present, there 1029 is a significant probability that more than one of these large 1030 microflows could fall on the same component link. If the traffic 1031 contribution from large microflows is small, the probability for 1032 three or more large microflows on the same component link drops 1033 significantly. Therefore in a network where a significant number of 1034 parallel 10 Gb/s links exists, even a 1 Gb/s pseudowire or other 1035 large microflow that could not otherwise be subdivided into smaller 1036 flows should carry a flow label or entropy label if possible. 1038 Active management of the hash space to better accommodate large 1039 microflows has been implemented and deployed in the past, however 1040 such techniques are out of scope for this document. 1042 2.4.3. Pseudowire Flow Label 1044 Unlike a pseudowire control word, a pseudowire flow label [RFC6391], 1045 is required only for relatively large capacity pseudowires. There 1046 are many cases where a pseudowire flow label makes sense. Any 1047 service such as a VPN which carries IP traffic within a pseudowire 1048 can make use of a pseudowire flow label. 1050 Any pseudowire carried over MPLS which makes use of the pseudowire 1051 control word and does not carry a flow label is in effect a single 1052 microflow (in [RFC2475] terms) and may result in the types of 1053 problems described in Section 2.4.2. 1055 2.4.4. MPLS Entropy Label 1057 The MPLS entropy label simplifies flow group identification [RFC6790] 1058 at midpoint LSR. Prior to the MPLS entropy label midpoint LSR needed 1059 to inspect the entire label stack and often the IP headers to provide 1060 an adequate distribution of traffic when using multipath techniques 1061 (see Section 2.4.5). With the use of MPLS entropy label, a hash can 1062 be performed closer to network edges, placed in the label stack, and 1063 used by midpoint LSR without fully reinspecting the label stack and 1064 inspecting the payload. 1066 The MPLS entropy label is capable of avoiding full label stack and 1067 payload inspection within the core where performance levels are most 1068 difficult to achieve (see Section 2.3). The label stack inspection 1069 can be terminated as soon as the first entropy label is encountered, 1070 which is generally after a small number of labels are inspected. 1072 In order to provide these benefits in the core, LSR closer to the 1073 edge must be capable of adding an entropy label. This support may 1074 not be required in the access tier, the tier closest to the customer, 1075 but is likely to be required in the edge or the border to the network 1076 core. LSR peering with external networks will also need to be able 1077 to add an entropy label on incoming traffic. 1079 2.4.5. Fields Used for Multipath Load Balance 1081 The most common multipath techniques are based on a hash over a set 1082 of fields. Regardless of whether a hash is used or some other method 1083 is used, the there is a limited set of fields which can safely be 1084 used for multipath. 1086 2.4.5.1. MPLS Fields in Multipath 1088 If the "outer" or "first" layer of encapsulation is MPLS, then label 1089 stack entries are used in the hash. Within a finite amount of time 1090 (and for small packets arriving at high speed that time can be quite 1091 limited) only a finite number of label entries can be inspected. 1092 Pipelined or parallel architectures improve this, but the limit is 1093 still finite. 1095 The following guidelines are provided for use of MPLS fields in 1096 multipath load balancing. 1098 1. Only the 20 bit label field SHOULD be used. The TTL field SHOULD 1099 NOT be used. The S bit MUST NOT be used. The TC field (formerly 1100 EXP) MUST NOT be used. See text following this list for reasons. 1102 2. If an ELI label is found, then if the LSR supports entropy label, 1103 the EL label field in the next label entry (the EL) SHOULD be 1104 used and label entries below that label SHOULD NOT be used and 1105 the MPLS payload SHOULD NOT be used. See below this list for 1106 reasons. 1108 3. Special purpose labels (label values 0-15) MUST NOT be used. 1109 Extended special purpose labels (any label following label 15) 1110 MUST NOT be used. In particular, GAL and RA MUST NOT be used so 1111 that OAM traffic follows the same path as payload packets with 1112 the same label stack. 1114 4. The most entropy is generally found in the label stack entries 1115 near the bottom of the label stack (innermost label, closest to 1116 S=1 bit). If the entire label stack cannot be used (or entire 1117 stack up to an EL), then it is better to use as many labels as 1118 possible closest to the bottom of stack. 1120 5. If no ELI is encountered, and the first nibble of payload 1121 contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support 1122 the ability to interpret the payload as IPv4 or IPv6 and extract 1123 and use appropriate fields from the IP headers. This feature is 1124 considered a hard requirement by many service providers. If 1125 supported, there MUST be a way to disable it (if, for example, PW 1126 without CW are used). This ability to disable this feature is 1127 considered a hard requirement by many service providers. 1128 Therefore an implementation has a very strong incentive to 1129 support both options. 1131 6. A label which is popped at egress (UHP pop) SHOULD NOT be used. 1132 A label which is popped at the penultimate hop (PHP pop) SHOULD 1133 be used. 1135 Apparently some chips have made use of the TC (formerly EXP) bits as 1136 a source of entropy. This is very harmful since it will reorder 1137 Assured Forwarding (AF) traffic [RFC2597] when a subset does not 1138 conform to the configured rates and is remarked but not dropped at a 1139 prior LSR. Traffic which uses MPLS ECN [RFC5129] can also be 1140 reordered if TC is used for entropy. Therefore, as stated in the 1141 guidelines above, the TC field (formerly EXP) MUST NOT be used in 1142 multipath load balancing as it violates Differentiated Services 1143 Ordered Aggregate (OA) requirements in these two instances. 1145 Use of the MPLS label entry S bit would result in putting OAM traffic 1146 on a different path if the addition of a GAL at the bottom of stack 1147 removed the S bit from the prior label. 1149 If an ELI label is found, then if the LSR supports entropy label, the 1150 EL label field in the next label entry (the EL) SHOULD be used and 1151 the search for additional entropy within the packet SHOULD be 1152 terminated. Failure to terminate the search will impact client MPLS- 1153 TP LSP carried within server MPLS LSP. A network operator has the 1154 option to use administrative attributes as a means to identify LSR 1155 which do not terminate the entropy search at the first EL. 1156 Administrative attributes are defined in [RFC3209]. Some 1157 configuration is required to support this. 1159 If the label removed by a PHP pop is not used, then for any PW for 1160 which CW is used, there is no basis for multipath load split. In 1161 some networks it is infeasible to put all PW traffic on one component 1162 link. Any PW which does not use CW will be improperly split 1163 regardless of whether the label removed by a PHP pop is used. 1164 Therefore the PHP pop label SHOULD be used as recommended above. 1166 2.4.5.2. IP Fields in Multipath 1168 Inspecting the IP payload provides the most entropy in provider 1169 networks. The practice of looking past the bottom of stack label for 1170 an IP payload is well accepted and documented in [RFC4928] and in 1171 other RFCs. 1173 Where IP is mentioned in the document, both IPv4 and IPv6 apply. All 1174 LSRs MUST fully support IPv6. 1176 When information in the IP header is used, the following guidelines 1177 apply: 1179 1. Both the IP source address and IP destination address SHOULD be 1180 used. There MAY be an option to reverse the order of these 1181 addresses, improving the ability to provide symmetric paths in 1182 some cases. Many service providers require that both addresses 1183 be used. 1185 2. Implementations SHOULD allow inspection of the IP protocol field 1186 and use of the UDP or TCP port numbers. For many service 1187 providers this feature is considered mandatory, particularly for 1188 enterprise, data center, or edge equipment. If this feature is 1189 provided, it SHOULD be possible to disable use of TCP and UDP 1190 ports. Many service providers consider it a hard requirement 1191 that use of UDP and TCP ports can be disabled. Therefore there 1192 is a stong incentive for implementations to provide both options. 1194 3. Equipment suppliers MUST NOT make assumptions that because the IP 1195 version field is equal to 4 (an IPv4 packet) that the IP protocol 1196 will either be TCP (IP protocol 6) or UDP (IP protocol 17) and 1197 blindly fetch the data at the offset where the TCP or UDP ports 1198 would be found. With IPv6, TCP and UDP port numbers are not at 1199 fixed offsets. With IPv4 packets carrying IP options, TCP and 1200 UDP port numbers are not at fixed offsets. 1202 4. The IPv6 header flow field SHOULD be used. This is the explicit 1203 purpose of the IPv6 flow field, however observed flow fields 1204 rarely contains a non-zero value. Some uses of the flow field 1205 have been defined such as [RFC6438]. In the absence of MPLS 1206 encapsulation, the IPv6 flow field can serve a role equivalent to 1207 entropy label. 1209 5. Support for other protocols that share a common Layer-4 header 1210 such as RTP, UDP-lite, SCTP and DCCP SHOULD be provided, 1211 particularly for edge or access equipment where additional 1212 entropy may be needed. Equipment SHOULD also use RTP, UDP-lite, 1213 SCTP and DCCP headers when creating an entropy label. 1215 6. The following IP header fields should not or must not be used: 1217 a. Similar to avoiding TC in MPLS, the IP DSCP, and ECN bits 1218 MUST NOT be used. 1220 b. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used. 1222 c. Note that the IP TOS field was deprecated ([RFC0791] was 1223 updated by [RFC2474]). No part of the IP DSCP field can be 1224 used (formerly IP PREC and IP TOS bits). 1226 7. Some IP encapsulations support tunneling, such as IP-in-IP, GRE, 1227 L2TPv3, and IPSEC. These provide a greater source of entropy 1228 which some provider networks carrying large amounts of tunneled 1229 traffic may need. The use of tunneling header information is out 1230 of scope for this document. 1232 This document makes the following recommendations. These 1233 recommendations are not required to claim compliance to any existing 1234 RFC therefore implementers are free to ignore them, but due to 1235 service provider requirements should consider the risk of doing so. 1236 The use of IP addresses MUST be supported and TCP and UDP ports 1237 (conditional on the protocol field and properly located) MUST be 1238 supported. The ability to disable use of UDP and TCP ports MUST be 1239 available. Though potentially very useful in some networks, it is 1240 uncommon to support using payloads of tunneling protocols carried 1241 over IP. Though the use of tunneling protocol header information is 1242 out of scope for this document, it is not discouraged. 1244 2.4.5.3. Fields Used in Flow Label 1246 The ingress to a pseudowire (PW) can extract information from the 1247 payload being encapsulated to create a flow label. [RFC6391] 1248 references IP carried in Ethernet as an example. The Native Service 1249 Processing (NSP) function defined in [RFC3985] differs with 1250 pseudowire type. It is in the NSP function where information for a 1251 specific type of PW can be extracted for use in a flow label. Which 1252 fields to use for any given PW NSP is out of scope for this document. 1254 2.4.5.4. Fields Used in Entropy Label 1256 An entropy label is added at the ingress to an LSP. The payload 1257 being encapsulated is most often MPLS, a PW, or IP. The payload type 1258 is identified by the layer-2 encapsulation (Ethernet, GFP, POS, etc). 1260 If the payload is MPLS, then the information used to create an 1261 entropy label is the same information used for local load balancing 1262 (see Section 2.4.5.1). This information MUST be extracted for use in 1263 generating an entropy label even if the LSR local egress interface is 1264 not a multipath. 1266 Of the non-MPLS payload types, only payloads that are forwarded are 1267 of interest. For example, ARP is not forwarded and CNLP (used only 1268 for ISIS) is not forwarded. 1270 The non-MPLS payload type of greatest interest are IPv4 and IPv6. 1271 The guidelines in Section 2.4.5.2 apply to fields used to create and 1272 entropy label. 1274 The IP tunneling protocols mentioned in Section 2.4.5.2 may be more 1275 applicable to generation of an entropy label at edge or access where 1276 deep packet inspection is practical due to lower interface speeds 1277 than in the core where deep packet inspection may be impractical. 1279 2.5. MPLS-TP and UHP 1281 MPLS-TP introduces forwarding demands that will be extremely 1282 difficult to meet in a core network. Most troublesome is the 1283 requirement for Ultimate Hop Popping (UHP, the opposite of 1284 Penultimate Hop Popping or PHP). Using UHP opens the possibility of 1285 one or more MPLS pop operation plus an MPLS swap operation for each 1286 packet. The potential for multiple lookups and multiple counter 1287 instances per packet exists. 1289 As networks grow and tunneling of LDP LSPs into RSVP-TE LSPs is used, 1290 and/or RSVP-TE hierarchy is used, the requirement to perform one or 1291 two or more MPLS pop operations plus a MPLS swap operation (and 1292 possibly a push or two) increases. If MPLS-TP LM (link monitoring) 1293 OAM is enabled at each layer, then a packet and byte count MUST be 1294 maintained for each pop and swap operation so as to offer OAM for 1295 each layer. 1297 2.6. Local Delivery of Packets 1299 There are a number of situations in which packets are destined to a 1300 local address or where a return packet must be generated. There is a 1301 need to mitigate the potential for outage as a result of either 1302 attacks on network infrastructure, or in some cases unintentional 1303 misconfiguration resulting in processor overload. Some hardware 1304 assistance is needed for all traffic destined to the general purpose 1305 CPU that is used in MPLS control protocol processing or network 1306 management protocol processing and in most cases to other general 1307 purpose CPUs residing on an LSR. This is due to the ease of 1308 overwhelming such a processor with traffic arriving on LSR high speed 1309 interfaces, whether the traffic is malicious or not. 1311 Denial of service (DoS) protection is an area requiring hardware 1312 support that is often overlooked or inadequately considered. 1313 Hardware assist is also needed for OAM, particularly the more 1314 demanding MPLS-TP OAM. 1316 2.6.1. DoS Protection 1318 Modern equipment supports a number of control plane and management 1319 plane protocols. Generally no single means of protecting network 1320 equipment from denial of service (DoS) attacks is sufficient, 1321 particularly for high speed interfaces. This problem is not specific 1322 to MPLS, but is a topic that cannot be ignored when implementing or 1323 evaluating MPLS implementations. 1325 Two types of protections are often cited as primary means of 1326 protecting against attacks of all kinds. 1328 Isolated Control/Management Traffic 1329 Control and Management traffic can be carried out-of-band (OOB), 1330 meaning not intermixed with payload. For MPLS, use of G-ACh and 1331 GAL to carry control and management traffic provides a means of 1332 isolation from potentially malicious payload. Used alone, the 1333 compromise of a single node, including a small computer at a 1334 network operations center, could compromise an entire network. 1335 Implementations which send all G-ACh/GAL traffic directly to a 1336 routing engine CPU are subject to DoS attack as a result of such 1337 a compromise. 1339 Cryptographic Authentication 1340 Cryptographic authentication can very effectively prevent 1341 malicious injection of control or management traffic. 1342 Cryptographic authentication can is some circumstances be subject 1343 to DoS attack by overwhelming the capacity of the decryption with 1344 a high volume of malicious traffic. For very low speed 1345 interfaces, cryptographic authentication can be performed by the 1346 general purpose CPU used as a routing engine. For all other 1347 cases, cryptographic hardware may be needed. For very high speed 1348 interfaces, even cryptographic hardware can be overwhelmed. 1350 Some control and management protocols are often carried with payload 1351 traffic. This is commonly the case with BGP, T-LDP, and SNMP. It is 1352 often the case with RSVP-TE. Even when carried over G-ACh/GAL 1353 additional measures can reduce the potential for a minor breach to be 1354 leveraged to a full network attack. 1356 Some of the additional protections are supported by hardware packet 1357 filtering. 1359 GTSM 1360 [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop 1361 Limit fields to insure control traffic that can only originate 1362 from an immediate neighbor is not forged and originating from a 1363 distant source. GTSM can be applied to many control protocols 1364 which are routable, for example LDP [RFC6720]. 1366 IP Filtering 1367 At the very minimum, packet filtering plus classification and use 1368 of multiple queues supporting rate limiting is needed for traffic 1369 that could potentially be sent to a general purpose CPU used as a 1370 routing engine. The first level of filtering only allows 1371 connections to be initiated from specific IP prefixes to specific 1372 destination ports and then preferably passes traffic directly to 1373 a cryptographic engine and/or rate limits. The second level of 1374 filtering passes connected traffic, such as TCP connections 1375 having received at least one authenticated SYN or having been 1376 locally initiated. The second level of filtering only passes 1377 traffic to specific address and port pairs to be checked for 1378 cryptographic authentication. 1380 The cryptographic authentication is generally the last resort in DoS 1381 attack mitigation. If a packet must be first sent to a general 1382 purpose CPU, then sent to a cryptographic engine, a DoS attack is 1383 possible on high speed interfaces. Only where hardware can identify 1384 a signature and the portion of packet covered by the signature is 1385 cryptographic authentication highly beneficial in protecting against 1386 DoS attacks. 1388 For chips supporting multiple 100 Gb/s interfaces, only a very large 1389 number of parallel cryptographic engines can provide the processing 1390 capacity to handle a large scale DoS or distributed DoS (DDoS) 1391 attack. For many forwarding chips this much processing power 1392 requires significant chip real estate and power, and therefore 1393 reduces system space and power density. For this reason, 1394 cryptographic authentication is not considered a viable first line of 1395 defense. 1397 For some networks the first line of defense is some means of 1398 supporting OOB control and management traffic. In the past this OOB 1399 channel might make use of overhead bits in SONET or OTN or a 1400 dedicated DWDM wavelength. G-ACh and GAL provide an alternative OOB 1401 mechanism which is independent of underlying layers. In other 1402 networks, including most IP/MPLS networks, perimeter filtering serves 1403 a similar purpose, though less effective without extreme vigilance. 1405 A second line of defense is filtering, including GTSM. For protocols 1406 such as EBGP, GTSM and other filtering is often the first line of 1407 defense. Cryptographic authentication is usually the last line of 1408 defense and insufficient by itself to mitigate DoS or DDoS attacks. 1410 2.6.2. MPLS OAM 1412 [RFC4377] defines requirements for MPLS OAM that predate MPLS-TP. 1413 [RFC4379] defines what is commonly referred to as LSP Ping and LSP 1414 Traceroute. [RFC4379] is updated by [RFC6424] supporting MPLS 1415 tunnels and stitched LSP and P2MP LSP. [RFC4379] is updated by 1416 [RFC6425] supporting P2MP LSP. [RFC4379] is updated by [RFC6426] to 1417 support MPLS-TP connectivity verification (CV) and route tracing. 1419 [RFC4950] extends the ICMP format to support TTL expiration that may 1420 occur when using IP traceroute within an MPLS tunnel. The ICMP 1421 message generation can be implemented in forwarding hardware, but if 1422 sent to a general purpose CPU must be rate limited to avoid a 1423 potential denial or service (DoS) attack. 1425 [RFC5880] defines Bidirectional Forwarding Detection (BFD), a 1426 protocol intended to detect faults in the bidirectional path between 1427 two forwarding engines. [RFC5884] and [RFC5885] define BFD for MPLS. 1428 BFD can provide failure detection on any kind of path between 1429 systems, including direct physical links, virtual circuits, tunnels, 1430 MPLS Label Switched Paths (LSPs), multihop routed paths, and 1431 unidirectional links as long as there is some return path. 1433 The processing requirements for BFD are less than for LSP Ping, 1434 making BFD somewhat better suited for relatively high rate proactive 1435 monitoring. BFD does not verify that the data plane matches the 1436 control plane, where LSP Ping does. LSP Ping is somewhat better 1437 suited for on-demand monitoring including relatively low rate 1438 periodic verification of data plane and as a diagnostic tool. 1440 Hardware assistance is often provided for BFD response where BFD 1441 setup or parameter change is not involved and may be necessary for 1442 relatively high rate proactive monitoring. If both BFD and LSP Ping 1443 are recognized in filtering prior to passing traffic to a general 1444 purpose CPU, appropriate DoS protection can be applied (see 1445 Section 2.6.1). Failure to recognize BFD and LSP Ping and at least 1446 rate limit creates the potential for misconfiguration to cause 1447 outages rather than cause errors in the misconfigured OAM. 1449 2.6.3. Pseudowire OAM 1451 Pseudowire OAM makes use of the control channel provided by Virtual 1452 Circuit Connectivity Verification (VCCV) [RFC5085]. VCCV makes use 1453 of the Pseudowire Control Word. BFD support over VCCV is defined by 1454 [RFC5885]. [RFC5885] is updated by [RFC6478] in support of static 1455 pseudowires. [RFC4379] is updated by [RFC6829] supporting LSP Ping 1456 for Pseudowire FEC advertised over IPv6. 1458 G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control 1459 channel and applies to any MPLS-TP end points, including Pseudowire. 1460 See Section 2.6.4 for an overview of MPLS-TP OAM. 1462 2.6.4. MPLS-TP OAM 1464 [RFC6669] summarizes the MPLS-TP OAM toolset, the set of protocols 1465 supporting the MPLS-TP OAM requirements specified in [RFC5860] and 1466 supported by the MPLS-TP OAM framework defined in [RFC6371]. 1468 The MPLS-TP OAM toolset includes: 1470 CC-CV 1471 [RFC6428] defines BFD extensions to support proactive 1472 Connectivity Check and Connectivity Verification (CC-CV) 1473 applications. [RFC6426] provides LSP ping extensions that are 1474 used to implement on-demand connectivity verification. 1476 RDI 1477 Remote Defect Indication (RDI) is triggered by failure of 1478 proactive CC-CV, which is BFD based. For fast RDI initiation, 1479 RDI SHOULD be initiated and handled by hardware if BFD is handled 1480 in forwarding hardware. [RFC6428] provides an extension for BFD 1481 that includes the RDI indication in the BFD format and a 1482 specification of how this indication is to be used. 1484 Route Tracing 1485 [RFC6426] specifies that the LSP ping enhancements for MPLS-TP 1486 on-demand connectivity verification include information on the 1487 use of LSP ping for route tracing of an MPLS-TP path. 1489 Alarm Reporting 1490 [RFC6427] describes the details of a new protocol supporting 1491 Alarm Indication Signal (AIS), Link Down Indication, and fault 1492 management. Failure to support this functionality in forwarding 1493 hardware can potentially result in failure to meet protection 1494 recovery time requirements and is therefore strongly recommended. 1496 Lock Instruct 1497 Lock instruct is initiated on-demand and therefore need not be 1498 implemented in forwarding hardware. [RFC6435] defines a lock 1499 instruct protocol. 1501 Lock Reporting 1502 [RFC6427] covers lock reporting. Lock reporting need not be 1503 implemented in forwarding hardware. 1505 Diagnostic 1506 [RFC6435] defines protocol support for loopback. Loopback 1507 initiation is on-demand and therefore need not be implemented in 1508 forwarding hardware. Loopback of packet traffic SHOULD be 1509 implemented in forwarding hardware on high speed interfaces. 1511 Packet Loss and Delay Measurement 1512 [RFC6374] and [RFC6375] define a protocol and profile for packet 1513 loss measurement (LM) and delay measurement (DM). LM requires a 1514 very accurate capture and insertion of packet and byte counters 1515 when a packet is transmitted and capture of packet and byte 1516 counters when a packet is received. This capture and insertion 1517 MUST be implemented in forwarding hardware for LM OAM if high 1518 accuracy is needed. DM requires very accurate capture and 1519 insertion of a timestamp on transmission and capture of timestamp 1520 when a packet is received. This timestamp capture and insertion 1521 MUST be implemented in forwarding hardware for DM OAM if high 1522 accuracy is needed. 1524 See Section 2.6.2 for discussion of hardware support necessary for 1525 BFD and LSP Ping. 1527 CC-CV and alarm reporting is tied to protection and therefore SHOULD 1528 be supported in forwarding hardware in order to provide protection 1529 for a large number of affected LSP within target response intervals. 1530 Since CC-CV is supported by BFD, for MPLS-TP providing hardware 1531 assistance for BFD processing helps insure that protection recovery 1532 time requirements can be met even for faults affecting a large number 1533 of LSP. 1535 2.6.5. MPLS OAM and Layer-2 OAM Interworking 1537 [RFC6670] provides the reasons for selecting a single MPLS-TP OAM 1538 solution and examines the consequences were ITU-T to develop a second 1539 OAM solution that is based on Ethernet encodings and mechanisms. 1541 [RFC6310] and [RFC7023] specifies the mapping of defect states 1542 between many types of hardware Attachment Circuits (ACs) and 1543 associated Pseudowires (PWs). This functionality SHOULD be supported 1544 in forwarding hardware. 1546 It is beneficial if an MPLS OAM implementation can interwork with the 1547 underlying server layer and provide a means to interwork with a 1548 client layer. For example, [RFC6427] specifies an inter-layer 1549 propagation of AIS and LDI from MPLS server layer to client MPLS 1550 layers. Where the server layer is a Layer-2, such as Ethernet, PPP 1551 over SONET/SDH, or GFP over OTN, interwork among layers is also 1552 beneficial. For high speed interfaces, supporting this interworking 1553 in forwarding hardware helps insure that protection based on this 1554 interworking can meet recovery time requirements even for faults 1555 affecting a large number of LSP. 1557 2.6.6. Extent of OAM Support by Hardware 1558 Where certain requirements must be met, such as relatively high CC-CV 1559 rates and a large number of interfaces, or strict protection recovery 1560 time requirements and a moderate number of affected LSP, some OAM 1561 functionality must be supported by forwarding hardware. In other 1562 cases, such as highly accurate LM and DM OAM or strict protection 1563 recovery time requirements with a large number of affected LSP, OAM 1564 functionality must be entirely implemented in forwarding hardware. 1566 Where possible, implementation in forwarding hardware should be in 1567 programmable hardware such that if standards are later changed or 1568 extended these changes are likely to be accommodated with hardware 1569 reprogramming rather than replacement. 1571 For some functionality there is a strong case for an implementation 1572 in dedicated forwarding hardware. Examples include packet and byte 1573 counters needed for LM OAM as well as needed for management 1574 protocols. Similarly the capture and insertion of packet and byte 1575 counts or timestamps needed for transmitted LM or DM or time 1576 synchronization packets MUST be implemented in forwarding hardware if 1577 high accuracy is required. 1579 For some functions there is a strong case to provide limited support 1580 in forwarding hardware but may make use of an external general 1581 purpose processor if performance criteria can be met. For example 1582 origination of RDI triggered by CC-CV, response to RDI, and PSC 1583 functionality may be supported by hardware, but expansion to a large 1584 number of client LSP and transmission of AIS or RDI to the client LSP 1585 may occur in a general purpose processor. Some forwarding hardware 1586 supports one or more on-chip general purpose processors which may be 1587 well suited for such a role. 1589 The customer (system supplier or provider) should not dictate design, 1590 but should independently validate target functionality and 1591 performance. However, it is not uncommon for service providers and 1592 system implementers to insist on reviewing design details (under NDA) 1593 due to past experiences with suppliers and to reject suppliers who 1594 are unwilling to provide details. 1596 2.7. Number and Size of Flows 1598 Service provider networks may carry up to hundreds of millions of 1599 flows on 10 Gb/s links. Most flows are very short lived, many under 1600 a second. A subset of the flows are low capacity and somewhat long 1601 lived. When Internet traffic dominates capacity a very small subset 1602 of flows are high capacity and/or very long lived. 1604 Two types of limitations with regard to number and size of flows have 1605 been observed. 1607 1. Some hardware cannot handle some very large flows because of 1608 internal paths which are limited, such as per packet backplane 1609 paths or paths internal or external to chips such as buffer 1610 memory paths. Such designs can handle aggregates of smaller 1611 flows. Some hardware with acknowledged limitations has been 1612 successfully deployed but may be increasingly problematic if the 1613 capacity of large microflows in deployed networks continues to 1614 grow. 1616 2. Some hardware approaches cannot handle a large number of flows, 1617 or a large number of large flows due to attempting to count per 1618 flow, rather than deal with aggregates of flows. Hash techniques 1619 scale with regard to number of flows due to a fixed hash size 1620 with many flows falling into the same hash bucket. Techniques 1621 that identify individual flows have been implemented but have 1622 never successfully deployed for Internet traffic. 1624 3. Questions for Suppliers 1626 The following questions should be asked of a supplier. These 1627 questions are grouped into broad categories. The questions 1628 themselves are intended to be an open ended question to the supplier. 1629 The tests in Section 4 are intended to verify whether the supplier 1630 disclosed any compliance or performance limitations completely and 1631 accurately. 1633 3.1. Basic Compliance 1635 Q#1 Can the implementation forward packets with an arbitrarily 1636 large stack depth? What limitations exist, and under what 1637 circumstances do further limitations come into play (such as high 1638 packet rate or specific features enabled or specific types of 1639 packet processing)? See Section 2.1. 1641 Q#2 Is the entire set of basic MPLS functionality described in 1642 Section 2.1 supported? 1644 Q#3 Are the set of MPLS special purpose labels handled correctly 1645 and with adequate performance? Are extended special purpose 1646 labels handled correctly and with adequate performance? See 1647 Section 2.1.1. 1649 Q#4 Are mappings of label value and TC to PHB handled correctly, 1650 including RFC3270 L-LSP mappings and RFC4124 CT mappings to PHB? 1651 See Section 2.1.2. 1653 Q#5 Is time synchronization adequately supported in forwarding 1654 hardware? 1655 a. Are both PTP and NTP formats supported? 1657 b. Is the accuracy of timestamp insertion and incoming stamping 1658 sufficient? 1660 See Section 2.1.3. 1662 Q#6 Is link bundling supported? 1664 a. Can LSP be pinned to specific components? 1666 b. Is the "all-ones" component link supported? 1668 See Section 2.1.5. 1670 Q#7 Is MPLS hierarchy supported? 1672 a. Are both PHP and UHP supported? What limitations exist on 1673 the number of pop operations with UHP? 1675 b. Are the pipe, short-pipe, and uniform models supported? Are 1676 TTL and TC values updated correctly at egress where 1677 applicable? 1679 See Section 2.1.6 1681 Q#8 Are pseudowire sequence numbers handled correctly? See 1682 Section 2.1.8.1. 1684 Q#9 Is VPN LER functionality handled correctly and without 1685 performance issues? See Section 2.1.9. 1687 Q#10 Is MPLS multicast (P2MP and MP2MP) handled correctly? 1689 a. Are packets dropped on uncongested outputs if some outputs 1690 are congested? 1692 b. Is performance limited in high fanout situations? 1694 See Section 2.2. 1696 3.2. Basic Performance 1698 Q#11 Can very small packets be forwarded at full line rate on all 1699 interfaces indefinitely? What limitations exist, and under what 1700 circumstances do further limitations come into play (such as 1701 specific features enabled or specific types of packet 1702 processing)? 1704 Q#12 Customers must decide whether to relax the prior requirement and 1705 to what extent. If the answer to the prior question indicates 1706 that limitations exist, then: 1708 a. What is the smallest packet size where full line rate 1709 forwarding can be supported? 1711 b. What is the longest burst of full rate small packets that can 1712 be supported? 1714 Specify circumstances (such as specific features enabled or 1715 specific types of packet processing) often impact these rates and 1716 burst sizes. 1718 Q#13 How many pop operations can be supported along with a swap 1719 operation at full line rate while maintaining per LSP packet and 1720 byte counts for each pop and swap? This requirement is 1721 particularly relevant for MPLS-TP. 1723 Q#14 How many label push operations can be supported. While this 1724 limitation is rarely an issue, it applies to both PHP and UHP, 1725 unlike the pop limit which applies to UHP. 1727 Q#15 For a worst case where all packets arrive on one LSP, what is 1728 the counter overflow time? Are any means provided to avoid 1729 polling all counters at short intervals? This applies to both 1730 MPLS and MPLS-TP. 1732 3.3. Multipath Capabilities and Performance 1734 Multipath capabilities and performance do not apply to MPLS-TP but 1735 apply to MPLS and apply if MPLS-TP is carried in MPLS. 1737 Q#16 How are large microflows accommodated? Is there active 1738 management of the hash space mapping to output ports? See 1739 Section 2.4.2. 1741 Q#17 How many MPLS labels can be included in a hash based on the MPLS 1742 label stack? 1744 Q#18 Is packet rate performance decreased beyond some number of 1745 labels? 1747 Q#19 Can the IP header and payload information below the MPLS stack 1748 be used in the hash? If so, which IP fields, payload types and 1749 payload fields are supported? 1751 Q#20 At what maximum MPLS label stack depth can Bottom of Stack and 1752 an IP header appear without impacting packet rate performance? 1754 Q#21 Are special purpose labels excluded from the label stack hash? 1755 Are extended purpose labels excluded from the label stack hash? 1756 See Section 2.4.5.1. 1758 Q#22 How is multipath performance affected by very large flows or an 1759 extremely large number of flows, or by very short lived flows? 1760 See Section 2.7. 1762 3.4. Pseudowire Capabilities and Performance 1764 Q#23 Is the pseudowire control word supported? 1766 Q#24 What is the maximum rate of pseudowire encapsulation and 1767 decapsulation? Apply the same questions as in Base Performance 1768 for any packet based pseudowire such as IP VPN or Ethernet. 1770 Q#25 Does inclusion of a pseudowire control word impact performance? 1772 Q#26 Are flow labels supported? 1774 Q#27 If so, what fields are hashed on for the flow label for 1775 different types of pseudowires? 1777 Q#28 Does inclusion of a flow label impact performance? 1779 3.5. Entropy Label Support and Performance 1781 Q#29 Can an entropy label be added when acting as in ingress LER and 1782 can it be removed when acting as an egress LER? 1784 Q#30 If so, what fields are hashed on for the entropy label? 1786 Q#31 Does adding or removing an entropy label impact packet rate 1787 performance? 1789 Q#32 Can an entropy label be detected in the label stack, used in the 1790 hash, and properly terminate the search for further information 1791 to hash on? 1793 Q#33 Does using an entropy label have any negative impact on 1794 performance? It should have no impact or a positive impact. 1796 3.6. DoS Protection 1797 Q#34 For each control and management plane protocol in use, what 1798 measures are taken to provide DoS attack hardening? 1800 Q#35 Have DoS attack tests been performed? 1802 Q#36 Can compromise of an internal computer on a management subnet be 1803 leveraged for any form of attack including DoS attack? 1805 3.7. OAM Capabilities and Performance 1807 Q#37 What OAM proactive and on-demand mechanisms are supported? 1809 Q#38 What performance limits exist under high proactive monitoring 1810 rates? 1812 Q#39 Can excessively high proactive monitoring rates impact control 1813 plane performance or cause control plane instability? 1815 Q#40 Ask the prior questions for each of the following. 1817 a. MPLS OAM 1819 b. Pseudowire OAM 1821 c. MPLS-TP OAM 1823 d. Layer-2 OAM Interworking 1825 See Section 2.6.2. 1827 4. Forwarding Compliance and Performance Testing 1829 Packet rate performance of equipment supporting a large number of 10 1830 Gb/s or 100 Gb/s links is not possible using desktop computers or 1831 workstations. The use of high end workstations as a source of test 1832 traffic was barely viable 20 years ago, but is no longer at all 1833 viable. Though custom microcode has been used on specialized router 1834 forwarding cards to serve the purpose of generating test traffic and 1835 measuring it, for the most part performance testing will require 1836 specialized test equipment. There are multiple sources of suitable 1837 equipment. 1839 The set of tests listed here do not correspond one-to-one to the set 1840 of questions in Section 3. The same categorization is used and these 1841 tests largely serve to validate answers provided to the prior 1842 questions, and can also provide answers where a supplier is unwilling 1843 to disclose compliance or performance. 1845 Performance testing is the domain of the IETF Benchmark Methodology 1846 Working Group (BMWG). Below are brief descriptions of conformance 1847 and performance tests. Some very basic tests are specified in 1848 [RFC5695] which partially cover only the basic performance test T#3. 1850 The following tests should be performed by the systems designer, or 1851 deployer, or performed by the supplier on their behalf if it is not 1852 practical for the potential customer to perform the tests directly. 1853 These tests are grouped into broad categories. 1855 The tests in Section 4.1 should be repeated under various conditions 1856 to retest basic performance when critical capabilities are enabled. 1857 Complete repetition of the performance tests enabling each capability 1858 and combinations of capabilities would be very time intensive, 1859 therefore a reduced set of performance tests can be used to gauge the 1860 impact of enabling specific capabilities. 1862 4.1. Basic Compliance 1864 T#1 Test forwarding at a high rate for packets with varying number 1865 of label entries. While packets with more than a dozen label 1866 entries are unlikely to be used in any practical scenario today, 1867 it is useful to know if limitations exists. 1869 T#2 For each of the questions listed under "Basic Compliance" in 1870 Section 3, verify the claimed compliance. For any functionality 1871 considered critical to a deployment, where applicable performance 1872 using each capability under load should be verified in addition 1873 to basic compliance. 1875 4.2. Basic Performance 1877 T#3 Test packet forwarding at full line rate with small packets. 1878 See [RFC5695]. The most likely case to fail is the smallest 1879 packet size. Also test with packet sizes in four byte increments 1880 ranging from payload sizes or 40 to 128 bytes. 1882 T#4 If the prior tests did not succeed for all packet sizes, then 1883 perform the following tests. 1885 a. Increase the packet size by 4 bytes until a size is found 1886 that can be forwarded at full rate. 1888 b. Inject bursts of consecutive small packets into a stream of 1889 larger packets. Allow some time for recovery between bursts. 1890 Increase the number of packets in the burst until packets are 1891 dropped. 1893 T#5 Send test traffic where a swap operation is required. Also set 1894 up multiple LSP carried over other LSP where the device under 1895 test (DUT) is the egress of these LSP. Create test packets such 1896 that the swap operation is performed after pop operations, 1897 increasing the number of pop operations until forwarding of small 1898 packets at full line rate can no longer be supported. Also check 1899 to see how many pop operations can be supported before the full 1900 set of counters can no longer be maintained. This requirement is 1901 particularly relevant for MPLS-TP. 1903 T#6 Send all traffic on one LSP and see if the counters become 1904 inaccurate. Often counters on silicon are much smaller than the 1905 64 bit packet and byte counters in IETF MIB. System developers 1906 should consider what counter polling rate is necessary to 1907 maintain accurate counters and whether those polling rates are 1908 practical. Relevant MIBs for MPLS are discussed in [RFC4221] and 1909 [RFC6639]. 1911 4.3. Multipath Capabilities and Performance 1913 Multipath capabilities do not apply to MPLS-TP but apply to MPLS and 1914 apply if MPLS-TP is carried in MPLS. 1916 T#7 Send traffic at a rate well exceeding the capacity of a single 1917 multipath component link, and where entropy exists only below the 1918 top of stack. If only the top label is used this test will fail 1919 immediately. 1921 T#8 Move the labels with entropy down in the stack until either the 1922 full forwarding rate can no longer be supported or most or all 1923 packets try to use the same component link. 1925 T#9 Repeat the two tests above with the entropy contained in IP 1926 headers or IP payload fields below the label stack rather than in 1927 the label stack. Test with the set of IP headers or IP payload 1928 fields considered relevant to the deployment or to the target 1929 market. 1931 T#10 Determine whether traffic that contains a pseudowire control 1932 word is interpreted as IP traffic. Information in the payload 1933 MUST NOT be used in the load balancing if the first nibble of the 1934 packet is not 4 or 6 (IPv4 or IPv6). 1936 T#11 Determine whether special purpose labels and extended special 1937 purpose labels are excluded from the label stack hash. They MUST 1938 be excluded. 1940 T#12 Perform testing in the presence of combinations of: 1942 a. Very large microflows. 1944 b. Relatively short lived high capacity flows. 1946 c. Extremely large numbers of flows. 1948 d. Very short lived small flows. 1950 4.4. Pseudowire Capabilities and Performance 1952 T#13 Ensure that pseudowire can be set up with a pseudowire label and 1953 pseudowire control word added at ingress and the pseudowire label 1954 and pseudowire control word removed at egress. 1956 T#14 For pseudowire that contains variable length payload packets, 1957 repeat performance tests listed under "Basic Performance" for 1958 pseudowire ingress and egress functions. 1960 T#15 Repeat pseudowire performance tests with and without a 1961 pseudowire control word. 1963 T#16 Determine whether pseudowire can be set up with a pseudowire 1964 label, flow label, and pseudowire control word added at ingress 1965 and the pseudowire label, flow label, and pseudowire control word 1966 removed at egress. 1968 T#17 Determine which payload fields are used to create the flow label 1969 and whether the set of fields and algorithm provide sufficient 1970 entropy for load balancing. 1972 T#18 Repeat pseudowire performance tests with flow labels included. 1974 4.5. Entropy Label Support and Performance 1976 T#19 Determine whether entropy labels can be added at ingress and 1977 removed at egress. 1979 T#20 Determine which fields are used to create an entropy label. 1980 Labels further down in the stack, including entropy labels 1981 further down and IP headers or IP payload fields where applicable 1982 should be used. Determine whether the set of fields and 1983 algorithm provide sufficient entropy for load balancing. 1985 T#21 Repeat performance tests under "Basic Performance" when entropy 1986 labels are used, where ingress or egress is the device under test 1987 (DUT). 1989 T#22 Determine whether an ELI is detected when acting as a midpoint 1990 LSR and whether the search for further information on which to 1991 base the load balancing is used. Information below the entropy 1992 label SHOULD NOT be used. 1994 T#23 Ensure that the entropy label indicator and entropy label (ELI 1995 and EL) are removed from the label stack during UHP and PHP 1996 operations. 1998 T#24 Insure that operations on the TC field when adding and removing 1999 entropy label are correctly carried out. If TC is changed during 2000 a swap operation, the ability to transfer that change MUST be 2001 provided. The ability to suppress the transfer of TC MUST also 2002 be provided. See "pipe", "short pipe", and "uniform" models in 2003 [RFC3443]. 2005 T#25 Repeat performance tests for midpoint LSR with entropy labels 2006 found at various label stack depths. 2008 4.6. DoS Protection 2010 T#26 Actively attack LSR under high protocol churn load and determine 2011 control plane performance impact or successful DoS under test 2012 conditions. Specifically test for the following. 2014 a. TCP SYN attack against control plane and management plane 2015 protocols using TCP, including CLI access (typically SSH 2016 protected login), NETCONF, etc. 2018 b. High traffic volume attack against control plane and 2019 management plane protocols not using TCP. 2021 c. Attacks which can be performed from a compromised management 2022 subnet computer, but not one with authentication keys. 2024 d. Attacks which can be performed from a compromised peer within 2025 the control plane (internal domain and external domain). 2026 Assume that per peering keys and per router ID keys rather 2027 than network wide keys are in use. 2029 See Section 2.6.1. 2031 4.7. OAM Capabilities and Performance 2033 T#27 Determine maximum sustainable rates of BFD traffic. If BFD 2034 requires CPU intervention, determine both maximum rates and CPU 2035 loading when multiple interfaces are active. 2037 T#28 Verify LSP Ping and LSP Traceroute capability. 2039 T#29 Determine maximum rates of MPLS-TP CC-CV traffic. If CC-CV 2040 requires CPU intervention, determine both maximum rates and CPU 2041 loading when multiple interfaces are active. 2043 T#30 Determine MPLS-TP DM precision. 2045 T#31 Determine MPLS-TP LM accuracy. 2047 T#32 Verify MPLS-TP AIS/RDI and PSC functionality, protection speed, 2048 and AIS/RDI notification speed when a large number of Management 2049 Entities (ME) must be notified with AIS/RDI. 2051 5. Acknowledgements 2053 Numerous very useful comments have been received in private email. 2054 Some of these contributions are acknowledged here, approximately in 2055 chronologic order. 2057 Paul Doolan provided a brief review resulting in a number of 2058 clarifications, most notably regarding on-chip vs. system buffering, 2059 100 Gb/s link speed assumptions in the 150 Mpps figure, and handling 2060 of large microflows. Pablo Frank reminded us of the sawtooth effect 2061 in PPS vs. packet size graphs, prompting the addition of a few 2062 paragraphs on this. Comments from Lou Berger at IETF-85 prompted the 2063 addition of Section 2.7. 2065 Valuable comments were received on the BMWG mailing list. Jay 2066 Karthik pointed out testing methodology hints that after discussion 2067 were deemed out of scope and were removed but may benefit later work 2068 in BMWG. 2070 Nabil Bitar pointed out the need to cover QoS (Differentiated 2071 Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM. Nabil 2072 also provided a number of clarifications to the questions and tests 2073 in Section 3 and Section 4. 2075 Gregory Mirsky and Thomas Beckhaus provided useful comments during 2076 the MPLS RT review. 2078 Tal Mizrahi provided comments that prompted clarifications regarding 2079 timestamp processing, local delivery of packets, and the need for 2080 hardware assistance in processing OAM traffic. 2082 Alexander (Sasha) Vainshtein pointed out errors in Section 2.1.8.1 2083 and suggested new text which after lengthy discussion resulted in 2084 restating the sumarization of requirements from PWE3 RFCs and more 2085 clearly stating the benefits and drawbacks of packet resequencing 2086 based on PW sequence number. 2088 6. IANA Considerations 2090 This memo includes no request to IANA. 2092 7. Security Considerations 2094 This document reviews forwarding behavior specified elsewhere and 2095 points out compliance and performance requirements. As such it 2096 introduces no new security requirements or concerns. 2098 Knowledge of potential performance shortcomings may serve to help new 2099 implementations avoid pitfalls. It is unlikely that such knowledge 2100 could be the basis of new denial of service as these pitfalls are 2101 already widely known in the service provider community and among 2102 leading equipment suppliers. In practice extreme data and packet 2103 rate are needed to affect existing equipment and to affect networks 2104 that may be still vulnerable due to failure to implement adequate 2105 protection. The extreme data and packet rates make this type of 2106 denial of service unlikely and make undetectable denial of service of 2107 this type impossible. 2109 8. References 2111 8.1. Normative References 2113 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2114 Requirement Levels", BCP 14, RFC 2119, March 1997. 2116 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2117 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2118 Encoding", RFC 3032, January 2001. 2120 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2121 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2122 Tunnels", RFC 3209, December 2001. 2124 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2125 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2126 Protocol Label Switching (MPLS) Support of Differentiated 2127 Services", RFC 3270, May 2002. 2129 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2130 in Multi-Protocol Label Switching (MPLS) Networks", RFC 2131 3443, January 2003. 2133 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 2134 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, May 2135 2005. 2137 [RFC4182] Rosen, E., "Removing a Restriction on the use of MPLS 2138 Explicit NULL", RFC 4182, September 2005. 2140 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 2141 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 2143 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2144 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2145 Use over an MPLS PSN", RFC 4385, February 2006. 2147 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2148 Marking in MPLS", RFC 5129, January 2008. 2150 [RFC5586] Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic 2151 Associated Channel", RFC 5586, June 2009. 2153 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 2154 J., and S. Amante, "Flow-Aware Transport of Pseudowires 2155 over an MPLS Packet Switched Network", RFC 6391, November 2156 2011. 2158 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2159 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2160 RFC 6790, November 2012. 2162 8.2. Informative References 2164 [ACK-compression] 2165 , , , "Observations and Dynamics of a Congestion Control 2166 Algorithm: The Effects of Two-Way Traffic", Proc. ACM 2167 SIGCOMM, ACM Computer Communications Review (CCR) Vol 21, 2168 No 4, 1991, pp.133-147., 1991. 2170 [I-D.ietf-mpls-special-purpose-labels] 2171 Kompella, K., Andersson, L., and A. Farrel, "Allocating 2172 and Retiring Special Purpose MPLS Labels", draft-ietf- 2173 mpls-special-purpose-labels-03 (work in progress), July 2174 2013. 2176 [I-D.ietf-pwe3-vccv-impl-survey-results] 2177 Malis, A., "The Pseudowire (PW) & Virtual Circuit 2178 Connectivity Verification (VCCV) Implementation Survey 2179 Results", draft-ietf-pwe3-vccv-impl-survey-results-03 2180 (work in progress), October 2013. 2182 [I-D.ietf-tictoc-1588overmpls] 2183 Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. 2184 Montini, "Transporting Timing messages over MPLS 2185 Networks", draft-ietf-tictoc-1588overmpls-05 (work in 2186 progress), June 2013. 2188 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 2189 1981. 2191 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2192 "Definition of the Differentiated Services Field (DS 2193 Field) in the IPv4 and IPv6 Headers", RFC 2474, December 2194 1998. 2196 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 2197 and W. Weiss, "An Architecture for Differentiated 2198 Services", RFC 2475, December 1998. 2200 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 2201 "Assured Forwarding PHB Group", RFC 2597, June 1999. 2203 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2204 Label Switching Architecture", RFC 3031, January 2001. 2206 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2207 of Explicit Congestion Notification (ECN) to IP", RFC 2208 3168, September 2001. 2210 [RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for 2211 Multiprotocol Label Switching Architecture (MPLS) 2212 Operation and Maintenance (OAM) Functions", RFC 3429, 2213 November 2002. 2215 [RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching 2216 (GMPLS) Signaling Functional Description", RFC 3471, 2217 January 2003. 2219 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 2220 Edge (PWE3) Architecture", RFC 3985, March 2005. 2222 [RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 2223 Provider-Provisioned Virtual Private Networks (PPVPNs)", 2224 RFC 4110, July 2005. 2226 [RFC4124] Le Faucheur, F., "Protocol Extensions for Support of 2227 Diffserv-aware MPLS Traffic Engineering", RFC 4124, June 2228 2005. 2230 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2231 Hierarchy with Generalized Multi-Protocol Label Switching 2232 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 2234 [RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol 2235 Label Switching (MPLS) Management Overview", RFC 4221, 2236 November 2005. 2238 [RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. 2239 Matsushima, "Operations and Management (OAM) Requirements 2240 for Multi-Protocol Label Switched (MPLS) Networks", RFC 2241 4377, February 2006. 2243 [RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol 2244 Label Switched (MPLS) Data Plane Failures", RFC 4379, 2245 February 2006. 2247 [RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual 2248 Private Networks (L2VPNs)", RFC 4664, September 2006. 2250 [RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, 2251 "Extensions to Resource Reservation Protocol - Traffic 2252 Engineering (RSVP-TE) for Point-to-Multipoint TE Label 2253 Switched Paths (LSPs)", RFC 4875, May 2007. 2255 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 2256 Cost Multipath Treatment in MPLS Networks", BCP 128, RFC 2257 4928, June 2007. 2259 [RFC4950] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP 2260 Extensions for Multiprotocol Label Switching", RFC 4950, 2261 August 2007. 2263 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 2264 Specification", RFC 5036, October 2007. 2266 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., and C. 2267 Pignataro, "The Generalized TTL Security Mechanism 2268 (GTSM)", RFC 5082, October 2007. 2270 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 2271 Connectivity Verification (VCCV): A Control Channel for 2272 Pseudowires", RFC 5085, December 2007. 2274 [RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) 2275 Report on MPLS Architectural Considerations for a 2276 Transport Profile", RFC 5317, February 2009. 2278 [RFC5332] Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS 2279 Multicast Encapsulations", RFC 5332, August 2008. 2281 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2282 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2283 Class" Field", RFC 5462, February 2009. 2285 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 2286 Benchmarking Methodology for IP Flows", RFC 5695, November 2287 2009. 2289 [RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for 2290 Operations, Administration, and Maintenance (OAM) in MPLS 2291 Transport Networks", RFC 5860, May 2010. 2293 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 2294 (BFD)", RFC 5880, June 2010. 2296 [RFC5884] Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow, 2297 "Bidirectional Forwarding Detection (BFD) for MPLS Label 2298 Switched Paths (LSPs)", RFC 5884, June 2010. 2300 [RFC5885] Nadeau, T. and C. Pignataro, "Bidirectional Forwarding 2301 Detection (BFD) for the Pseudowire Virtual Circuit 2302 Connectivity Verification (VCCV)", RFC 5885, June 2010. 2304 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 2305 Time Protocol Version 4: Protocol and Algorithms 2306 Specification", RFC 5905, June 2010. 2308 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 2309 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 2310 Acronym in the IETF", BCP 161, RFC 6291, June 2011. 2312 [RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., 2313 Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, 2314 Administration, and Maintenance (OAM) Message Mapping", 2315 RFC 6310, July 2011. 2317 [RFC6371] Busi, I. and D. Allan, "Operations, Administration, and 2318 Maintenance Framework for MPLS-Based Transport Networks", 2319 RFC 6371, September 2011. 2321 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 2322 Measurement for MPLS Networks", RFC 6374, September 2011. 2324 [RFC6375] Frost, D. and S. Bryant, "A Packet Loss and Delay 2325 Measurement Profile for MPLS-Based Transport Networks", 2326 RFC 6375, September 2011. 2328 [RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, 2329 "Label Distribution Protocol Extensions for Point-to- 2330 Multipoint and Multipoint-to-Multipoint Label Switched 2331 Paths", RFC 6388, November 2011. 2333 [RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for 2334 Performing Label Switched Path Ping (LSP Ping) over MPLS 2335 Tunnels", RFC 6424, November 2011. 2337 [RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, 2338 S., and T. Nadeau, "Detecting Data-Plane Failures in 2339 Point-to-Multipoint MPLS - Extensions to LSP Ping", RFC 2340 6425, November 2011. 2342 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2343 On-Demand Connectivity Verification and Route Tracing", 2344 RFC 6426, November 2011. 2346 [RFC6427] Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S., 2347 and D. Ward, "MPLS Fault Management Operations, 2348 Administration, and Maintenance (OAM)", RFC 6427, November 2349 2011. 2351 [RFC6428] Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive 2352 Connectivity Verification, Continuity Check, and Remote 2353 Defect Indication for the MPLS Transport Profile", RFC 2354 6428, November 2011. 2356 [RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., 2357 and X. Dai, "MPLS Transport Profile Lock Instruct and 2358 Loopback Functions", RFC 6435, November 2011. 2360 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 2361 for Equal Cost Multipath Routing and Link Aggregation in 2362 Tunnels", RFC 6438, November 2011. 2364 [RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, 2365 "Pseudowire Status for Static Pseudowires", RFC 6478, May 2366 2012. 2368 [RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching 2369 Transport Profile (MPLS-TP) MIB-Based Management 2370 Overview", RFC 6639, June 2012. 2372 [RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, 2373 Administration, and Maintenance (OAM) Toolset for MPLS- 2374 Based Transport Networks", RFC 6669, July 2012. 2376 [RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a 2377 Single Solution for MPLS Transport Profile (MPLS-TP) 2378 Operations, Administration, and Maintenance (OAM)", RFC 2379 6670, July 2012. 2381 [RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security 2382 Mechanism (GTSM) for the Label Distribution Protocol 2383 (LDP)", RFC 6720, August 2012. 2385 [RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label 2386 Switched Path (LSP) Ping for Pseudowire Forwarding 2387 Equivalence Classes (FECs) Advertised over IPv6", RFC 2388 6829, January 2013. 2390 [RFC7023] Mohan, D., Bitar, N., Sajassi, A., DeLord, S., Niger, P., 2391 and R. Qiu, "MPLS and Ethernet Operations, Administration, 2392 and Maintenance (OAM) Interworking", RFC 7023, October 2393 2013. 2395 Appendix A. Organization of References Section 2397 The References section is split into Normative and Informative 2398 subsections. References that directly specify forwarding 2399 encapsulations or behaviors are listed as normative. References 2400 which describe signaling only, though normative with respect to 2401 signaling, are listed as informative. They are informative with 2402 respect to MPLS forwarding. 2404 Authors' Addresses 2406 Curtis Villamizar (editor) 2407 Outer Cape Cod Network Consulting, LLC 2409 Email: curtis@occnc.com 2411 Kireeti Kompella 2412 Juniper Networks 2414 Email: kireeti@juniper.net 2415 Shane Amante 2416 Apple Inc. 2417 1 Infinite Loop 2418 Cupertino, California 95014 2420 Email: samante@apple.com 2422 Andrew Malis 2423 Consultant 2425 Email: agmalis@gmail.com 2427 Carlos Pignataro 2428 Cisco Systems 2429 7200-12 Kit Creek Road 2430 Research Triangle Park, NC 27709 2431 US 2433 Email: cpignata@cisco.com