idnits 2.17.00 (12 Aug 2021) /tmp/idnits62953/draft-ietf-mpls-forwarding-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (October 11, 2013) is 3144 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: draft-ietf-mpls-special-purpose-labels has been published as RFC 7274 == Outdated reference: draft-ietf-pwe3-vccv-impl-survey-results has been published as RFC 7079 == Outdated reference: A later version (-07) exists of draft-ietf-tictoc-1588overmpls-05 -- Obsolete informational reference (is this intentional?): RFC 4379 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6424 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6829 (Obsoleted by RFC 8029) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPLS C. Villamizar, Ed. 3 Internet-Draft OCCNC 4 Intended status: Informational K. Kompella 5 Expires: April 14, 2014 Contrail Systems 6 S. Amante 7 Level 3 Communications, Inc. 8 A. Malis 9 Verizon 10 C. Pignataro 11 Cisco 12 October 11, 2013 14 MPLS Forwarding Compliance and Performance Requirements 15 draft-ietf-mpls-forwarding-02 17 Abstract 19 This document provides guidelines for implementers regarding MPLS 20 forwarding and a basis for evaluations of forwarding implementations. 21 Guidelines cover many aspects of MPLS forwarding. Topics are 22 highlighted where implementers might otherwise overlook practical 23 requirements which are unstated or under emphasized or are optional 24 for conformance to RFCs but are often considered mandatory by 25 providers. 27 Status of this Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on April 14, 2014. 44 Copyright Notice 46 Copyright (c) 2013 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction and Document Scope . . . . . . . . . . . . . . . 4 62 1.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.2. Use of Requirements Language . . . . . . . . . . . . . . . 8 64 1.3. Apparent Misconceptions . . . . . . . . . . . . . . . . . 9 65 1.4. Target Audience . . . . . . . . . . . . . . . . . . . . . 10 66 2. Forwarding Issues . . . . . . . . . . . . . . . . . . . . . . 11 67 2.1. Forwarding Basics . . . . . . . . . . . . . . . . . . . . 11 68 2.1.1. MPLS Special Purpose Labels . . . . . . . . . . . . . 12 69 2.1.2. MPLS Differentiated Services . . . . . . . . . . . . . 13 70 2.1.3. Time Synchronization . . . . . . . . . . . . . . . . . 14 71 2.1.4. Uses of Multiple Label Stack Entries . . . . . . . . . 14 72 2.1.5. MPLS Link Bundling . . . . . . . . . . . . . . . . . . 15 73 2.1.6. MPLS Hierarchy . . . . . . . . . . . . . . . . . . . . 15 74 2.1.7. MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . . 16 75 2.1.8. Pseudowire Encapsulation . . . . . . . . . . . . . . . 16 76 2.1.8.1. Pseudowire Sequence Number . . . . . . . . . . . . 17 77 2.1.9. Layer-2 and Layer-3 VPN . . . . . . . . . . . . . . . 18 78 2.2. MPLS Multicast . . . . . . . . . . . . . . . . . . . . . . 18 79 2.3. Packet Rates . . . . . . . . . . . . . . . . . . . . . . . 19 80 2.4. MPLS Multipath Techniques . . . . . . . . . . . . . . . . 21 81 2.4.1. Pseudowire Control Word . . . . . . . . . . . . . . . 21 82 2.4.2. Large Microflows . . . . . . . . . . . . . . . . . . . 22 83 2.4.3. Pseudowire Flow Label . . . . . . . . . . . . . . . . 22 84 2.4.4. MPLS Entropy Label . . . . . . . . . . . . . . . . . . 23 85 2.4.5. Fields Used for Multipath Load Balance . . . . . . . . 23 86 2.4.5.1. MPLS Fields in Multipath . . . . . . . . . . . . . 23 87 2.4.5.2. IP Fields in Multipath . . . . . . . . . . . . . . 25 88 2.4.5.3. Fields Used in Flow Label . . . . . . . . . . . . 27 89 2.4.5.4. Fields Used in Entropy Label . . . . . . . . . . . 27 90 2.5. MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . . 27 91 2.6. Local Delivery of Packets . . . . . . . . . . . . . . . . 28 92 2.6.1. DoS Protection . . . . . . . . . . . . . . . . . . . . 28 93 2.6.2. MPLS OAM . . . . . . . . . . . . . . . . . . . . . . . 30 94 2.6.3. Pseudowire OAM . . . . . . . . . . . . . . . . . . . . 31 95 2.6.4. MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . . 31 96 2.6.5. MPLS OAM and Layer-2 OAM Interworking . . . . . . . . 33 97 2.6.6. Extent of OAM Support by Hardware . . . . . . . . . . 33 98 2.7. Number and Size of Flows . . . . . . . . . . . . . . . . . 34 99 3. Questions for Suppliers . . . . . . . . . . . . . . . . . . . 35 100 3.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . . 35 101 3.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 36 102 3.3. Multipath Capabilities and Performance . . . . . . . . . . 37 103 3.4. Pseudowire Capabilities and Performance . . . . . . . . . 38 104 3.5. Entropy Label Support and Performance . . . . . . . . . . 38 105 3.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . . 38 106 3.7. OAM Capabilities and Performance . . . . . . . . . . . . . 39 107 4. Forwarding Compliance and Performance Testing . . . . . . . . 39 108 4.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . . 40 109 4.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 40 110 4.3. Multipath Capabilities and Performance . . . . . . . . . . 41 111 4.4. Pseudowire Capabilities and Performance . . . . . . . . . 42 112 4.5. Entropy Label Support and Performance . . . . . . . . . . 42 113 4.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . . 43 114 4.7. OAM Capabilities and Performance . . . . . . . . . . . . . 43 115 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 44 116 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 44 117 7. Security Considerations . . . . . . . . . . . . . . . . . . . 45 118 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 45 119 8.1. Normative References . . . . . . . . . . . . . . . . . . . 45 120 8.2. Informative References . . . . . . . . . . . . . . . . . . 46 121 Appendix A. Organization of References Section . . . . . . . . . 51 122 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 51 124 1. Introduction and Document Scope 126 The initial purpose of this document was to address concerns raised 127 on the MPLS WG mailing list about shortcomings in implementations of 128 MPLS forwarding. Documenting existing misconceptions and potential 129 pitfalls might potentially avoid repeating past mistakes. The 130 document has grown to address a broad set of forwarding requirements. 132 The focus of this document is MPLS forwarding, base pseudowire 133 forwarding, and MPLS OAM. The use of pseudowire control word, and 134 sequence number are discussed. Specific pseudowire AC and NSP are 135 out of scope. Specific pseudowire applications, such as various 136 forms of VPN, are out of scope. 138 MPLS support for multipath techniques is considered essential by many 139 service providers and is useful for other high capacity networks. In 140 order to obtain sufficient entropy from MPLS traffic service 141 providers and others find it essential for the MPLS implementation to 142 interpret the MPLS payload as IPv4 or IPv6 based on the contents of 143 the first nibble of payload. The use of IP addresses, the IP 144 protocol field, and UDP and TCP port number fields in multipath load 145 balancing are considered within scope. The use of any other IP 146 protocol fields, such as tunneling protocols carried within IP, are 147 out of scope. 149 Implementation details are a local matter and are out of scope. Most 150 interfaces today operate at 1 Gb/s or greater. It is assumed that 151 all forwarding operations are implemented in specialized forwarding 152 hardware rather than on a general purpose processor. This is often 153 referred to as "fast path" and "slow path" processing. Some 154 recommendations are made regarding implementing control or management 155 plane functionality in specialized hardware or with limited 156 assistance from specialized hardware. This advise is based on 157 expected control or management protocol loads and on the need for 158 denial of service (DoS) protection. 160 1.1. Acronyms 162 The following acronyms are used. 164 AC Attachment Circuit ([RFC3985]) 166 ACH Associated Channel Header (pseudowires) 168 ACK Acknowledgement (TCP flag and type of TCP packet) 169 AIS Alarm Indication Signal (MPLS-TP OAM) 171 ATM Asynchronous Transfer Mode (legacy switched circuits) 173 BFD Bidirectional Forwarding Detection 175 BGP Border Gateway Protocol 177 CC-CV Connectivity Check and Connectivity Verification 179 CE Customer Edge (LDP) 181 CPU Central Processing Unit (computer or microprocessor) 183 CT Class Type ([RFC4124]) 185 CW Control Word ([RFC4385]) 187 DCCP Datagram Congestion Control Protocol 189 DDoS Distributed Denial of Service 191 DM Delay Measurement (MPLS-TP OAM) 193 DSCP Differentiated Services Code Point ([RFC2474]) 195 DWDM Dense Wave Division Multiplexing 197 DoS Denial of Service 199 E-LSP EXP-Inferred-PSC LSP ([RFC3270]) 201 EBGP External BGP 203 ECMP Equal Cost Multi-Path 205 ECN Explicit Congestion Notification ([RFC3168] and [RFC5129]) 207 EL Entropy Label ([RFC6790]) 209 ELI Entropy Label Indicator ([RFC6790]) 211 EXP Experimental (field in MPLS renamed to TC in [RFC5462]) 213 FEC Forwarding Equivalence Classes (LDP), also Forward Error 214 Correction in other context 216 FR Frame Relay (legacy switched circuits) 218 FRR Fast Reroute ([RFC4090]) 220 G-ACh Generic Associated Channel ([RFC5586]) 222 GAL Generic Associated Channel Label ([RFC5586]) 224 GFP Generic Framing Protocol (used in OTN) 226 GMPLS Generalized MPLS ([RFC3471]) 228 GTSM Generalized TTL Security Mechanism ([RFC5082]) 230 Gb/s Gigabits per second (billion bits per second) 232 IANA Internet Assigned Numbers Authority 234 ILM Incoming Label Map ([RFC3031]) 236 IP Internet Protocol 238 IPVPN Internet Protocol VPN 240 IPv4 Internet Protocol version 4 242 IPv6 Internet Protocol version 6 244 L-LSP Label-Only-Inferred-PSC LSP ([RFC3270]) 246 L2VPN Layer 2 VPN 248 LDP Label Distribution Protocol ([RFC5036]) 250 LER Label Edge Router ([RFC3031]) 252 LM Loss Measurement (MPLS-TP OAM) 254 LSP Label Switched Path ([RFC3031]) 256 LSR Label Switching Router ([RFC3031]) 258 MP2MP Multipoint to Point 260 MPLS MultiProtocol Label Switching ([RFC3031]) 261 MPLS-TP MPLS Transport Profile ([RFC5317]) 263 Mb/s Megabits per second (million bits per second) 265 NSP Native Service Processing ([RFC3985]) 267 NTP Network Time Protocol 269 OAM Operations, Administration, and Maintenance ([RFC6291]) 271 OOB Out-of-band (not carried within a data channel) 273 OTN Optical Transport Network 275 P Provider router (LDP) 277 P2MP Point to Multi-Point 279 PE Provider Edge router (LDP) 281 PHB Per-Hop-Behavior ([RFC2475]) 283 PHP Penultimate Hop Popping ([RFC3443]) 285 POS Packet over SONET 287 PSC This acronym has multiple interpretations. 289 1. Packet Switch Capable ([RFC3471] 291 2. PHB Scheduling Class ([RFC3270]) 293 3. Protection State Coordination (MPLS-TP linear protection) 295 PTP Precision Time Protocol 297 PW Pseudowire 299 QoS Quality of Service 301 RA Router Alert ([RFC3032]) 303 RDI Remote Defect Indication (MPLS-TP OAM) 305 RSVP-TE RSVP Traffic Engineering 306 RTP Real-Time Transport Protocol 308 SCTP Stream Control Transmission Protocol 310 SDH Synchronous Data Hierarchy (European SONET, a form of TDM) 312 SONET Synchronous Optical Network (US SDH, a form of TDM) 314 T-LDP Targeted LDP (LDP sessions over more than one hop) 316 TC Traffic Class ([RFC5462]) 318 TCP Transmission Control Protocol 320 TDM Time-Division Multiplexing (legacy encapsulations) 322 TOS Type of Service (see [RFC2474]) 324 TTL Time-to-live (a field in IP and MPLS headers) 326 UDP User Datagram Protocol 328 UHP Ultimate Hop Popping (opposite of PHP) 330 VCCV Virtual Circuit Connectivity Verification ([RFC5085]) 332 VLAN Virtual Local Area Network (Ethernet) 334 VOQ Virtual Output Queuing (switch fabric design) 336 VPN Virtual Private Network 338 WG Working Group 340 1.2. Use of Requirements Language 342 This document is informational. The upper case [RFC2119] key words 343 are not used in this document, except in the following cases. 345 1. RFC 2119 keywords are used where requirements stated in this 346 document are called for in referenced RFCs. In most cases the 347 RFC containing the requirement is cited within the statement 348 using an RFC 2119 keyword. 350 2. RFC 2119 keywords are used where explicitly noted that the 351 keywords indicate that operator experiences indicate a 352 requirement, but there are no existing RFC requirements. 354 Advice provided by this document may be ignored by implementations. 355 Similarly, implementations not claiming conformance to specific RFCs 356 may ignore the requirements of those RFCs. In both cases, 357 implementers should consider the risk of doing so. 359 1.3. Apparent Misconceptions 361 In early generations of forwarding silicon (which might now be behind 362 us), there apparently were some misconceptions about MPLS. The 363 following statements provide clarifications. 365 1. There are practical reasons to have more than one or two labels 366 in an MPLS label stack. Under some circumstances the label stack 367 can become quite deep. See Section 2.1. 369 2. The label stack MUST be considered to be arbitrarily deep. 370 Section 3.27.4. "Hierarchy: LSP Tunnels within LSPs" of RFC3031 371 states "The label stack mechanism allows LSP tunneling to nest to 372 any depth." [RFC3031] If a bottom of the label stack cannot be 373 found, but sufficient number of labels exist to forward, an LSR 374 MUST forward the packet. An LSR MUST NOT assume the packet is 375 malformed unless the end of packet is found before bottom of 376 stack. See Section 2.1. 378 3. In networks where deep label stacks are encountered, they are not 379 rare. Full packet rate performance is required regardless of 380 label stack depth, except where multiple pop operations are 381 required. See Section 2.1. 383 4. Research has shown that long bursts of short packets with 40 byte 384 or 44 byte IP payload sizes in these bursts are quite common. 385 This is due to TCP ACK compression [ACK-compression]. The 386 following two sub-bullets constitutes advice that reflects very 387 common hard requirements of providers. Implementers may ignore 388 this advice but should consider the risk of doing so. 390 A. A forwarding engine SHOULD, if practical, be able to sustain 391 an arbitrarily long sequence of small packets arriving at 392 full interface rate. 394 B. If indefinite full packet rate for small packets is not 395 practical, a forwarding engine MUST be able to buffer a long 396 sequence of small packets inbound to the on-chip decision 397 engine and sustain full interface rate for some reasonable 398 average packet rate. Absent this small on-chip buffering, 399 QoS agnostic packet drops can occur. 401 See Section 2.3. 403 5. The implementer and system designer MUST support pseudowire 404 control word (CW) if MPLS-TP is supported or if ACH [RFC5586] is 405 being used on a pseudowire. The implementer and system designer 406 SHOULD support pseudowire CW even if MPLS-TP and ACH [RFC5586] 407 are not used, using instead CW and VCCV Type 1 [RFC5085] to allow 408 the use of multipath in the underlying network topology without 409 impacting the PW traffic. 410 [I-D.ietf-pwe3-vccv-impl-survey-results] does note that there are 411 still some deployments where the CW is not always used. It also 412 notes that many service providers do enable the CW. See 413 Section 2.4.1 for more discussion on why deployments SHOULD 414 enable the pseudowire CW. 416 6. The implementer and system designer SHOULD support adding a 417 pseudowire Flow Label [RFC6391]. Deployments MAY enable this 418 feature for appropriate pseudowire types. See Section 2.4.3. 420 7. The implementer and system designer SHOULD support adding an MPLS 421 entropy label [RFC6790]. Deployments MAY enable this feature. 422 See Section 2.4.4. 424 1.4. Target Audience 426 This document is intended for multiple audiences: implementer 427 (implementing MPLS forwarding in silicon or in software); systems 428 designer (putting together a MPLS forwarding systems); deployer 429 (running an MPLS network). These guidelines are intended to serve 430 the following purposes: 432 1. Explain what to do and what not to do when a deep label stack is 433 encountered. (audience: implementer) 435 2. Highlight pitfalls to look for when implementing an MPLS 436 forwarding chip. (audience: implementer) 438 3. Provide a checklist of features and performance specifications to 439 request. (audience: systems designer, deployer) 441 4. Provide a set of tests to perform. (audience: systems designer, 442 deployer). 444 The implementer, systems designer, and deployer have a transitive 445 supplier customer relationship. It is in the best interest of the 446 supplier to review their product against their customer's checklist 447 and customer's customer's checklist if applicable. 449 2. Forwarding Issues 451 A brief review of forwarding issues is provided in the subsections 452 that follow. This section provides some background on why some of 453 these requirements exist. The questions to ask of suppliers is 454 covered in Section 3. Some guidelines for testing are provided in 455 Section 4. 457 2.1. Forwarding Basics 459 Basic MPLS architecture and MPLS encapsulation, and therefore packet 460 forwarding are defined in [RFC3031] and [RFC3032]. RFC3031 and 461 RFC3032 are somewhat LDP centric. RSVP-TE supports traffic 462 engineering (TE) and fast reroute, features that LDP lacks. The base 463 document for RSVP-TE based MPLS is [RFC3209]. 465 A few RFCs update RFC3032. Those with impact on forwarding include 466 the following. 468 1. TTL processing is clarified in [RFC3443]. 470 2. The use of MPLS Explicit NULL is modified in [RFC4182]. 472 3. Differentiated Services is supported by [RFC3270] and [RFC4124]. 473 The "EXP" field is renamed to "Traffic Class" in [RFC5462], 474 removing any misconception that it was available for 475 experimentation or could be ignored. 477 4. ECN is supported by [RFC5129]. 479 5. The MPLS G-ACh and GAL are defined in [RFC5586]. 481 6. [RFC5332] redefines the two data link layer codepoints for MPLS 482 packets. 484 Tunneling encapsulations which may carry MPLS, such as MPLS in GRE, 485 L2TP, or LDP, are out of scope. 487 Other RFCs have implications to MPLS Forwarding and do not update 488 RFC3032 or RFC3209, including: 490 1. The pseudowire (PW) Associated Channel Header (ACH), defined by 491 [RFC5085], later generalized by the MPLS G-ACh [RFC5586]. 493 2. The entropy label indicator (ELI) and entropy label (EL) are 494 defined by [RFC6790]. 496 A few RFCs update RFC3209. Those that are listed as updating RFC3209 497 generally impact only RSVP-TE signaling. Forwarding is modified by 498 major extension built upon RFC3209. 500 RFCs which impact forwarding are discussed in the following 501 subsections. 503 2.1.1. MPLS Special Purpose Labels 505 [RFC3032] specifies that label values 0-15 are special purpose labels 506 with special meanings. Three values of NULL label are defined (two 507 of which are later updated by [RFC4182]) and a router-alert label is 508 defined. The original intent was that special purpose labels, except 509 the NULL labels, could be sent to the routing engine CPU rather than 510 be processed in forwarding hardware. Hardware support is required by 511 new RFCs such as those defining entropy label and OAM processed as a 512 result of receiving a GAL. For new special purpose labels, some 513 accommodation is needed for LSR that will send the labels to a 514 general purpose CPU or other highly programmable hardware. For 515 example, ELI will only be sent to LSR which have signaled support for 516 [RFC6790] and high OAM packet rate must be negotiated among 517 endpoints. 519 [RFC3429] reserves a label for ITU-T Y.1711, however Y.1711 does not 520 work with multipath and its use is strongly discouraged. 522 The current list of special purpose labels can be found on the 523 "Multiprotocol Label Switching Architecture (MPLS) Label Values" 524 registry reachable at IANA's pages at . 526 [I-D.ietf-mpls-special-purpose-labels] introduces an IANA "Extended 527 Special Purpose MPLS Label Values" registry and makes use of the 528 "extension" label, label 15, to indicate that the next label is an 529 extended special purpose label and requires special handling. The 530 range of only 16 values for special purpose labels allows a table to 531 be used. The range of extended special purpose labels with 20 bits 532 available for use may have to be handled in some other way in the 533 unlikely event that in the future the range of currently reserved 534 values 256-1048575 are used. If only the standards action range, 16- 535 239, and the experimental range, 240-255, are used, then a table of 536 256 entries can be used. 538 Unknown special purpose labels and unknown extended special purpose 539 labels are handled the same. When an unknown special purpose label 540 is encountered or a special purpose label not directly handled in 541 forwarding hardware is encountered, the packet should be sent to a 542 general purpose CPU by default. If this capability is supported, 543 there must be an option to either drop or rate limit such packets on 544 a per special purpose label value basis. 546 2.1.2. MPLS Differentiated Services 548 [RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence 549 (Prec) fields and replaces them with the Differentiated Services 550 Field more commonly known as the Differentiated Services Code Point 551 (DSCP) field. [RFC2475] defines the Differentiated Services 552 architecture, which in other forum is often called a Quality of 553 Service (QoS) architecture. 555 MPLS uses the Traffic Class (TC) field to support Differentiated 556 Services [RFC5462]. There are two primary documents describing how 557 DSCP is mapped into TC. 559 1. [RFC3270] defines E-LSP and L-LSP. E-LSP use a static mapping of 560 DSCP into TC. L-LSP uses a per LSP mapping of DSCP into TC, with 561 one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use 562 multiple Per-Hop Behavior (PHB) values. For example, the Assured 563 Forwarding service defines three PSC, each with three PHB 564 [RFC2597]. 566 2. [RFC4124] defines assignment of a class-type (CT) to an LSP, 567 where a per CT static mapping of TC to PHB is used. [RFC4124] 568 provides a means to support up to eight E-LSP-like mappings of 569 DSCP to TC. 571 To meet Differentiated Services requirements specified in [RFC3270], 572 the following forwarding requirements must be met. An ingress LER 573 MUST be able to select an LSP and then apply a per LSP map of DSCP 574 into TC. A midpoint LSR MUST be able to apply a per LSP map of TC to 575 PHB. The number of mappings supported will be far less than the 576 number of LSP supported. 578 To meet Differentiated Services requirements specified in [RFC4124], 579 the following forwarding requirements must be met. An ingress LER 580 MUST be able to select an LSP and then apply a per LSP map of DSCP 581 into TC. A midpoint LSR MUST be able to apply a per LSP map to CT 582 map and then use Class Type (CT) to map TC to PHB. Since there are 583 only eight allowed values of CT, only eight maps of TC to PHB need to 584 be supported. The LSP label can be used directly to find the TC to 585 PHB mapping, as is needed to support [RFC3270] L-LSP. 587 While support for [RFC4124] and not [RFC3270] would allow support for 588 only eight mappings of TC to PHB, it is common to support both and 589 simply state a limit on the number of unique TC to PHB mappings which 590 can be supported. 592 2.1.3. Time Synchronization 594 PTP or NTP may be carried over MPLS [I-D.ietf-tictoc-1588overmpls]. 595 Generally NTP will be carried within IP with IP carried in MPLS 596 [RFC5905]. Both PTP and NTP benefit from accurate time stamping of 597 incoming packets and the ability to insert accurate time stamps in 598 outgoing packets. PTP correction which occurs when forwarding 599 requires updating a timestamp compensation field based on the 600 difference between packet arrival at an LSR and packet transmit time 601 at that same LSR. 603 Since the label stack depth may vary, hardware should allow a 604 timestamp to be placed in an outgoing packet at any specified byte 605 position. It may be necessary to modify layer-2 checksums or frame 606 check sequences after insertion. PTP and NTP timestamp formats 607 differ slightly. If NTP or PTP is carried over UDP/IP or UDP/IP/ 608 MPLS, the UDP checksum will also have to be updated. 610 Accurate time synchronization in addition to being generally useful 611 is required for MPLS-TP delay measurement (DM) OAM. See 612 Section 2.6.4. 614 2.1.4. Uses of Multiple Label Stack Entries 616 MPLS deployments in the early part of the prior decade (circa 2000) 617 tended to support either LDP or RSVP-TE. LDP was favored by some for 618 its ability to scale to a very large number of PE devices at the edge 619 of the network, without adding deployment complexity. RSVP-TE was 620 favored, generally in the network core, where traffic engineering 621 and/or fast reroute were considered important. 623 Both LDP and RSVP-TE are used simultaneously within major Service 624 Provider networks using a technique known as "LDP over RSVP-TE 625 Tunneling". This technique allows service providers to carry LDP 626 tunnels inside RSVP-TE tunnels. This makes it possible to take 627 advantage of the Traffic Engineering and Fast Re-Route on more 628 expensive Inter-City and Inter-Continental transport paths. The 629 ingress RSVP-TE PEs places many LDP tunnels on a single RSVP-TE LSP 630 and carries it to the egress RSVP-TE PE. The LDP PEs are situated 631 further from the core, for example within a metro network. LDP over 632 RSVP-TE tunneling requires a minimum of two MPLS labels: one each for 633 LDP and RSVP-TE. 635 The use of MPLS FRR [RFC4090] might add one more label to MPLS 636 traffic, but only when FRR protection is in use (active). If LDP 637 over RSVP-TE is in use, and FRR protection is in use, then at least 638 three MPLS labels are present on the label stack on the links through 639 which the Bypass LSP traverses. FRR is covered in Section 2.1.7. 641 LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN 642 services that are deployed by the vast majority of service providers. 643 These VPN services added yet another label, bringing the label stack 644 depth (when FRR is active) to four. 646 Pseudowires and VPN are discussed in further detail in Section 2.1.8 647 and Section 2.1.9. 649 MPLS hierarchy as described in [RFC4206] can in principle add up to 650 four additional labels. MPLS hierarchy is discussed in 651 Section 2.1.6. 653 Other features such as Entropy Label (discussed in Section 2.4.4) and 654 Flow Label (discussed in Section 2.4.3) can add additional labels to 655 the label stack. 657 Although theoretical scenarios can easily result in eight or more 658 labels, such cases are rare if they occur at all today. For the 659 purpose of forwarding, only the top label needs to be examined if PHP 660 is used, a few more if UHP is used (see Section 2.5). For deep label 661 stacks, quite a few labels may have to be examined for the purpose of 662 load balancing across parallel links (see Section 2.4), however this 663 depth can be bounded by a provider through use of Entropy Label. 665 2.1.5. MPLS Link Bundling 667 MPLS Link Bundling was the first RFC to address the need for multiple 668 parallel links between nodes [RFC4201]. MPLS Link Bundling is 669 notable in that it tried not to change MPLS forwarding, except in 670 specifying the "All-Ones" component link. MPLS Link Bundling is 671 seldom if ever deployed. Instead multipath techniques described in 672 Section 2.4 are used. 674 2.1.6. MPLS Hierarchy 676 MPLS hierarchy is defined in [RFC4206]. Although RFC4206 is 677 considered part of GMPLS, the Packet Switching Capable (PSC) portion 678 of the MPLS hierarchy are applicable to MPLS and may be supported in 679 an otherwise GMPLS free implementation. The MPLS PSC hierarchy 680 remains the most likely means of providing further scaling in an 681 RSVP-TE MPLS network, particularly where the network is designed to 682 provide RSVP-TE connectivity to the edges. This is the case for 683 envisioned MPLS-TP networks. The use of the MPLS PSC hierarchy can 684 add as many as four labels to a label stack, though it is likely that 685 only one layer of PSC will be used in the near future. 687 2.1.7. MPLS Fast Reroute (FRR) 689 Fast reroute is defined by [RFC4090]. Two significantly different 690 methods are defined in RFC4090, the "One-to-One Backup" method which 691 uses the "Detour LSP" and the " Facility Backup" which uses a "bypass 692 tunnel". These are commonly referred to as the detour and bypass 693 methods respectively. 695 The detour method makes use of a presignaled LSP. Hardware 696 assistance is needed for detour FRR only if necessary to accomplish 697 local repair of a large number of LSP within the 10s of milliseconds 698 target. For each affected LSP a swap operation must be reprogrammed 699 or otherwise switched over. The use of detour FRR doubles the number 700 of LSP terminating at any given hop and will increase the number of 701 LSP within a network by a factor dependent on the average detour path 702 length. 704 The bypass method makes use of a tunnel that is unused when no fault 705 exists but may carry many LSP when a local repair is required. There 706 is no presignaling indicating which working LSP will be diverted into 707 any specific bypass LSP. The merge LSR (egress LSR of the bypass 708 LSP) MUST use platform label space (as defined in [RFC3031]) so that 709 an LSP working path on any give interface can be backed up using a 710 bypass LSP terminating on any other interface. Hardware assistance 711 is needed if necessary to accomplish local repair of a large number 712 of LSP within the 10s of milliseconds target. For each affected LSP 713 a swap operation must be reprogrammed or otherwise switched over with 714 an additional push of the bypass LSP label. The use of platform 715 label space impacts the size of the LSR ILM for LSR with a very large 716 number of interfaces. 718 2.1.8. Pseudowire Encapsulation 720 The pseudowire (PW) architecture is defined in [RFC3985]. A 721 pseudowire, when carried over MPLS, adds one or more additional label 722 entries to the MPLS label stack. A PW Control Word is defined in 723 [RFC4385] with motivation for defining the control word in [RFC4928]. 724 The PW Associated Channel defined in [RFC4385] is used for OAM in 725 [RFC5085]. The PW Flow Label is defined in [RFC6391] and is 726 discussed further in this document in Section 2.4.3. 728 There are numerous pseudowire encapsulations, supporting emulation of 729 services such as Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over 730 packet switched networks (PSNs) using IP or MPLS. 732 The pseudowire encapsulation is out of scope for this document. 733 Pseudowire impact on MPLS forwarding at midpoint LSR is within scope. 734 The impact on ingress MPLS push and egress MPLS UHP pop are within 735 scope. While pseudowire encapsulation is out of scope, some advice 736 is given on sequence number support. 738 2.1.8.1. Pseudowire Sequence Number 740 Pseudowire (PW) sequence number support is most important for PW 741 payload types with a high expectation of in-order delivery. 742 Resequencing support, rather than dropping at egress on out of order 743 arrival, is most important for PW payload types with a high 744 expectation of lossless delivery. For example, TDM payloads require 745 sequence number support and require resequencing support. The same 746 is true of ATM CBR service. ATM VBR or ABR may have somewhat relaxed 747 requirements, but generally require ATM Early Packet Discard (EPD) or 748 ATM Partial Packet Discard (PPD) [ATM-EPD-and-PPD]. Though sequence 749 number support and resequencing support are beneficial to PW packet 750 oriented payloads such as FR and Ethernet, they are highly desirable 751 but not as strongly required. 753 Packet reorder should be rare except in a small number of 754 circumstances, most of which are due to network design or equipment 755 design errors: 757 1. The most common case is where reordering occurs is rare, 758 occurring only when a network or equipment fault forces traffic 759 on a new path with different delay. The packet loss that 760 accompanies a network or equipment fault is generally more 761 disruptive than any reordering which may occur. 763 2. A path change can be caused by reasons other than a network or 764 equipment fault, such as administrative routing change. This may 765 result in packet reordering but generally without any packet 766 loss. 768 3. If the edge is not using pseudowire control word (CW) and the 769 core is using multipath, reordering will be far more common. If 770 this is occurring, the best solution is to use CW on the edge, 771 rather than try to fix the reordering using resequencing. 773 4. Another avoidable case is where some core equipment has multipath 774 and for some reason insists on periodically installing a new 775 random number as the multipath hash seed. If supporting MPLS-TP, 776 equipment MUST provide a means to disable periodic hash reseeding 777 and deployments MUST disable periodic hash reseeding. Even if 778 not supporting MPLS-TP, equipment should provide a means to 779 disable periodic hash reseeding and deployments should disable 780 periodic hash reseeding. 782 2.1.9. Layer-2 and Layer-3 VPN 784 Layer-2 VPN [RFC4664] and Layer-3 VPN [RFC4110] add one or more label 785 entry to the MPLS label stack. VPN encapsulations are out of scope 786 for this document. Its impact on forwarding at midpoint LSR are 787 within scope. 789 Any of these services may be used on an MPLS entropy label enabled 790 ingress and egress (see Section 2.4.4 for discussion of entropy 791 label) which would add an additional two labels to the MPLS label 792 stack. The need to provide a useful entropy label value impacts the 793 requirements of the VPN ingress LER but is out of scope for this 794 document. 796 2.2. MPLS Multicast 798 MPLS Multicast encapsulation is clarified in [RFC5332]. MPLS 799 Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388]. 801 [RFC4875] defines a root initiated RSVP-TE LSP setup rather than leaf 802 initiated join used in IP multicast. [RFC6388] defines a leaf 803 initiated LDP setup. Both [RFC4875] and [RFC6388] define point to 804 multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint to 805 multipoint (MP2MP) LSP setup. 807 The P2MP LSP have a single source. An LSR may be a leaf node, an 808 intermediate node, or a "bud" node. A bud serves as both a leaf and 809 intermediate. At a leaf an MPLS pop is performed. The payload may 810 be a IP Multicast packet that requires further replication. At an 811 intermediate node a MPLS swap operation is performed. The bud 812 requires that both a pop operation and a swap operation be performed 813 for the same incoming packet. 815 One strategy to support P2MP functionality is to pop at the LSR 816 interface serving as ingress to the P2MP traffic and then optionally 817 push labels at each LSR interface serving as egress to the P2MP 818 traffic at that same LSR. A given LSR egress chip may support 819 multiple egress interfaces, each of which requires a copy, but each 820 with a different set of added labels and layer-2 encapsulation. Some 821 physical interfaces may have multiple sub-interfaces (such as 822 Ethernet VLAN or channelized interfaces) each requiring a copy. 824 If packet replication is performed at LSR ingress, then the ingress 825 interface performance may suffer. If the packet replication is 826 performed within a LSR switching fabric and at LSR egress, congestion 827 of egress interfaces cannot make use of backpressure to ingress 828 interfaces using techniques such as virtual output queuing (VOQ). If 829 buffering is primarily supported at egress, then the need for 830 backpressure is minimized. There may be no good solution for high 831 volumes of multicast traffic if VOQ is used. 833 MP2MP LSP differ in that any branch may provide an input, including a 834 leaf. Packets must be replicated onto all other branches. This 835 forwarding is often implemented as multiple P2MP forwarding trees, 836 one for each potential input interface at a given LSR. 838 2.3. Packet Rates 840 While average packet size of Internet traffic may be large, long 841 sequences of small packets have both been predicted in theory and 842 observed in practice. Traffic compression and TCP ACK compression 843 can conspire to create long sequences of packets of 40-44 bytes in 844 payload length. If carried over Ethernet, the 64 byte minimum 845 payload applies, yielding a packet rate of approximately 150 Mpps 846 (million packets per second) for the duration of the burst on a 847 nominal 100 Gb/s link. The peak rate for other encapsulations can be 848 as high as 250 Mpps (for example IP or MPLS encapsulated using GFP 849 over OTN ODU4). 851 It is possible that the packet rates achieved by a specific 852 implementation is acceptable for a minimum payload size, such as 64 853 byte (64B) payload for Ethernet, but the acheived rate declines to an 854 unacceptable level for other packet sizes, such as 65B payload. 855 There are other packet rates of interest besides TCP ACK. For 856 example, a TCP ACK carried over an Ethernet PW over MPLS over 857 Ethernet may occupy 82B or 82B plus an increment of 4B if additional 858 MPLS labels are present. 860 A graph of packet rate vs. packet size often displays a sawtooth. 861 The sawtooth is commonly due to a memory bottleneck and memory 862 widths, sometimes internal cache, but often a very wide external 863 buffer memory interface. In some cases it may be due to a fabric 864 transfer width. A fine packing, rounding up to the nearest 8B or 16B 865 will result in a fine sawtooth with small degradation for 65B, and 866 even less for 82B packets. A course packing, rounding up to 64B can 867 yield a sharper drop in performance for 65B packets, or perhaps more 868 important, a larger drop for 82B packets. 870 The loss of some TCP ACK packets are not the primary concern when 871 such a burst occurs. When a burst occurs, any other packets, 872 regardless of packet length and packet QoS are dropped once on-chip 873 input buffers prior to the decision engine are exceeded. Buffers in 874 front of the packet decision engine are often very small or non- 875 existent (less than one packet of buffer) causing significant QoS 876 agnostic packet drop. 878 Internet service providers and content providers at one time 879 specified full rate forwarding with 40 byte payload packets as a 880 requirement. Today, this requirement often can be waived if the 881 provider can be convinced that when long sequence of short packets 882 occur no packets will be dropped. 884 Many equipment suppliers have pointed out that the extra cost in 885 designing hardware capable of processing the minimum size packets at 886 full line rate is significant for very high speed interfaces. If 887 hardware is not capable of processing the minimum size packets at 888 full line rate, then that hardware MUST be capable of handling large 889 burst of small packets, a condition which is often observed. This 890 level of performance is necessary to meet Differentiated Services 891 [RFC2475] requirements for without it, packets are lost prior to 892 inspection of the IP DSCP field [RFC2474] or MPLS TC field [RFC5462]. 894 With adequate on-chip buffers before the packet decision engine, an 895 LSR can absorb a long sequence of short packets. Even if the output 896 is slowed to the point where light congestion occurs, the packets, 897 having cleared the decision process, can make use of larger VOQ or 898 output side buffers and be dealt with according to configured QoS 899 treatment, rather than dropped completely at random. 901 These on-chip buffers need not contribute significant delay since 902 they are only used when the packet decision engine is unable to keep 903 up, not in response to congestion, plus these buffers are quite 904 small. For example, an on-chip buffer capable of handling 4K packets 905 of 64 bytes in length, or 256KB, corresponds to 2 msec on a 10 Mb/s 906 link and 0.2 usec on a 100 Gb/s link. If the packet decision engine 907 is capable of handling packets at 90% of the full rate for small 908 packets, then the maximum added delay is 0.2 msec and 20 nsec 909 respectively, and this delay only applies if a 4K burst of short 910 packets occurs. When no burst of short packets was being processed, 911 no delay is added. 913 Packet rate requirements apply regardless of which network tier 914 equipment is deployed in. Whether deployed in the network core or 915 near the network edges, one of the two conditions MUST be met if 916 Differentiated Services requirements are to be met: 918 1. Packets must be processed at full line rate with minimum sized 919 packets. -OR- 921 2. Packets must be processed at a rate well under generally accepted 922 average packet sizes, with sufficient buffering prior to the 923 packet decision engine to accommodate long bursts of small 924 packets. 926 2.4. MPLS Multipath Techniques 928 In any large provider, service providers and content providers, hash 929 based multipath techniques are used in the core and in the edge. In 930 many of these providers hash based multipath is also used in the 931 larger metro networks. 933 The most common multipath techniques are ECMP applied at the IP 934 forwarding level, Ethernet LAG with inspection of the IP payload, and 935 multipath on links carrying both IP and MPLS, where the IP header is 936 inspected below the MPLS label stack. In most core networks, the 937 vast majority of traffic is MPLS encapsulated. 939 In order to support an adequately balanced load distribution across 940 multiple links, IP header information must be used. Common practice 941 today is to reinspect the IP headers at each LSR and use the label 942 stack and IP header information in a hash performed at each LSR. 943 Further details are provided in Section 2.4.5. 945 The use of this technique is so ubiquitous in provider networks that 946 lack of support for multipath makes any product unsuitable for use in 947 large core networks. This will continue to be the case in the near 948 future, even as deployment of MPLS entropy label begins to relax the 949 core LSR multipath performance requirements given the existing 950 deployed base of edge equipment without the ability to add an entropy 951 label. 953 A generation of edge equipment supporting the ability to add an MPLS 954 entropy label is needed before the performance requirements for core 955 LSR can be relaxed. However, it is likely that two generations of 956 deployment in the future will allow core LSR to support full packet 957 rate only when a relatively small number of MPLS labels need to be 958 inspected before hashing. For now, don't count on it. 960 Common practice today is to reinspect the packet at each LSR and use 961 information from the packet combined plus a hash seed that is 962 selected by each LSR. Where flow labels or entropy labels are used, 963 a hash seed must be used when creating these labels. 965 2.4.1. Pseudowire Control Word 967 Within the core of a network some form of multipath is almost certain 968 to be used. Multipath techniques deployed today are likely to be 969 looking beneath the label stack for an opportunity to hash on IP 970 addresses. 972 A pseudowire encapsulated at a network edge must have a means to 973 prevent reordering within the core if the pseudowire will be crossing 974 a network core, or any part of a network topology where multipath is 975 used (see [RFC4385] and [RFC4928]). 977 Not supporting the ability to encapsulate a pseudowire with a control 978 word may lock a product out from consideration. A pseudowire 979 capability without control word support might be sufficient for 980 applications that are strictly both intra-metro and low bandwidth. 981 However a provider with other applications will very likely not 982 tolerate having equipment which can only support a subset of their 983 pseudowire needs. 985 2.4.2. Large Microflows 987 Where multipath makes use of a simple hash and simple load balance 988 such as modulo or other fixed allocation (see Section 2.4) the 989 presence of large microflows that each consumes 10% of the capacity 990 of a component link of a potentially congested composite link, one 991 such microflow can upset the traffic balance and more than one can in 992 effect reduce the effective capacity of the entire composite link by 993 more than 10%. 995 When even a very small number of large microflows are present, there 996 is a significant probability that more than one of these large 997 microflows could fall on the same component link. If the traffic 998 contribution from large microflows is small, the probability for 999 three or more large microflows on the same component link drops 1000 significantly. Therefore in a network where a significant number of 1001 parallel 10 Gb/s links exists, even a 1 Gb/s pseudowire or other 1002 large microflow that could not otherwise be subdivided into smaller 1003 flows should carry a flow label or entropy label if possible. 1005 Active management of the hash space to better accommodate large 1006 microflows has been implemented and deployed in the past, however 1007 such techniques are out of scope for this document. 1009 2.4.3. Pseudowire Flow Label 1011 Unlike a pseudowire control word, a pseudowire flow label [RFC6391], 1012 is required only for relatively large capacity pseudowires. There 1013 are many cases where a pseudowire flow label makes sense. Any 1014 service such as a VPN which carries IP traffic within a pseudowire 1015 can make use of a pseudowire flow label. 1017 Any pseudowire carried over MPLS which makes use of the pseudowire 1018 control word and does not carry a flow label is in effect a single 1019 microflow (in [RFC2475] terms) and may result in the types of 1020 problems described in Section 2.4.2. 1022 2.4.4. MPLS Entropy Label 1024 The MPLS entropy label simplifies flow group identification [RFC6790] 1025 at midpoint LSR. Prior to the MPLS entropy label midpoint LSR needed 1026 to inspect the entire label stack and often the IP headers to provide 1027 an adequate distribution of traffic when using multipath techniques 1028 (see Section 2.4.5). With the use of MPLS entropy label, a hash can 1029 be performed closer to network edges, placed in the label stack, and 1030 used by midpoint LSR without fully reinspecting the label stack and 1031 inspecting the payload. 1033 The MPLS entropy label is capable of avoiding full label stack and 1034 payload inspection within the core where performance levels are most 1035 difficult to achieve (see Section 2.3). The label stack inspection 1036 can be terminated as soon as the first entropy label is encountered, 1037 which is generally after a small number of labels are inspected. 1039 In order to provide these benefits in the core, LSR closer to the 1040 edge must be capable of adding an entropy label. This support may 1041 not be required in the access tier, the tier closest to the customer, 1042 but is likely to be required in the edge or the border to the network 1043 core. LSR peering with external networks will also need to be able 1044 to add an entropy label on incoming traffic. 1046 2.4.5. Fields Used for Multipath Load Balance 1048 The most common multipath techniques are based on a hash over a set 1049 of fields. Regardless of whether a hash is used or some other method 1050 is used, the there is a limited set of fields which can safely be 1051 used for multipath. 1053 2.4.5.1. MPLS Fields in Multipath 1055 If the "outer" or "first" layer of encapsulation is MPLS, then label 1056 stack entries are used in the hash. Within a finite amount of time 1057 (and for small packets arriving at high speed that time can be quite 1058 limited) only a finite number of label entries can be inspected. 1059 Pipelined or parallel architectures improve this, but the limit is 1060 still finite. 1062 The following guidelines are provided for use of MPLS fields in 1063 multipath load balancing. 1065 1. Only the 20 bit label field SHOULD be used. The TTL field SHOULD 1066 NOT be used. The S bit MUST NOT be used. The TC field (formerly 1067 EXP) MUST NOT be used. See text following this list for reasons. 1069 2. If an ELI label is found, then if the LSR supports entropy label, 1070 the EL label field in the next label entry (the EL) SHOULD be 1071 used and label entries below that label SHOULD NOT be used and 1072 the MPLS payload SHOULD NOT be used. See below this list for 1073 reasons. 1075 3. Special purpose labels (label values 0-15) MUST NOT be used. 1076 Extended special purpose labels (any label following label 15) 1077 MUST NOT be used. In particular, GAL and RA MUST NOT be used so 1078 that OAM traffic follows the same path as payload packets with 1079 the same label stack. 1081 4. The most entropy is generally found in the label stack entries 1082 near the bottom of the label stack (innermost label, closest to 1083 S=1 bit). If the entire label stack cannot be used (or entire 1084 stack up to an EL), then it is better to use as many labels as 1085 possible closest to the bottom of stack. 1087 5. If no ELI is encountered, and the first nibble of payload 1088 contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support 1089 the ability to interpret the payload as IPv4 or IPv6 and extract 1090 and use appropriate fields from the IP headers. This feature is 1091 considered a hard requirement by many service providers. If 1092 supported, there MUST be a way to disable it (if, for example, PW 1093 without CW are used). This ability to disable this feature is 1094 considered a hard requirement by many service providers. 1095 Therefore an implementation has a very strong incentive to 1096 support both options. 1098 6. A label which is popped at egress (UHP pop) SHOULD NOT be used. 1099 A label which is popped at the penultimate hop (PHP pop) SHOULD 1100 be used. 1102 Apparently some chips have made use of the TC (formerly EXP) bits as 1103 a source of entropy. This is very harmful since it will reorder 1104 Assured Forwarding (AF) traffic [RFC2597] when a subset does not 1105 conform to the configured rates and is remarked but not dropped at a 1106 prior LSR. Traffic which uses MPLS ECN [RFC5129] can also be 1107 reordered if TC is used for entropy. Therefore, as stated in the 1108 guidelines above, the TC field (formerly EXP) MUST NOT be used in 1109 multipath load balancing as it violates Differentiated Services 1110 Ordered Aggregate (OA) requirements in these two instances. 1112 Use of the MPLS label entry S bit would result in putting OAM traffic 1113 on a different path if the addition of a GAL at the bottom of stack 1114 removed the S bit from the prior label. 1116 If an ELI label is found, then if the LSR supports entropy label, the 1117 EL label field in the next label entry (the EL) SHOULD be used and 1118 the search for additional entropy within the packet SHOULD be 1119 terminated. Failure to terminate the search will impact client 1120 MPLS-TP LSP carried within server MPLS LSP. A network operator has 1121 the option to use administrative attributes as a means to identify 1122 LSR which do not terminate the entropy search at the first EL. 1123 Administrative attributes are defined in [RFC3209]. Some 1124 configuration is required to support this. 1126 If the label removed by a PHP pop is not used, then for any PW for 1127 which CW is used, there is no basis for multipath load split. In 1128 some networks it is infeasible to put all PW traffic on one component 1129 link. Any PW which does not use CW will be improperly split 1130 regardless of whether the label removed by a PHP pop is used. 1131 Therefore the PHP pop label SHOULD be used as recommended above. 1133 2.4.5.2. IP Fields in Multipath 1135 Inspecting the IP payload provides the most entropy in provider 1136 networks. The practice of looking past the bottom of stack label for 1137 an IP payload is well accepted and documented in [RFC4928] and in 1138 other RFCs. 1140 Where IP is mentioned in the document, both IPv4 and IPv6 apply. All 1141 LSRs MUST fully support IPv6. 1143 When information in the IP header is used, the following guidelines 1144 apply: 1146 1. Both the IP source address and IP destination address SHOULD be 1147 used. There MAY be an option to reverse the order of these 1148 addresses, improving the ability to provide symmetric paths in 1149 some cases. Many service providers require that both addresses 1150 be used. 1152 2. Implementations SHOULD allow inspection of the IP protocol field 1153 and use of the UDP or TCP port numbers. For many service 1154 providers this feature is considered mandatory, particularly for 1155 enterprise, data center, or edge equipment. If this feature is 1156 provided, it SHOULD be possible to disable use of TCP and UDP 1157 ports. Many service providers consider it a hard requirement 1158 that use of UDP and TCP ports can be disabled. Therefore there 1159 is a stong incentive for implementations to provide both options. 1161 3. Equipment suppliers MUST NOT make assumptions that because the IP 1162 version field is equal to 4 (an IPv4 packet) that the IP protocol 1163 will either be TCP (IP protocol 6) or UDP (IP protocol 17) and 1164 blindly fetch the data at the offset where the TCP or UDP ports 1165 would be found. With IPv6, TCP and UDP port numbers are not at 1166 fixed offsets. With IPv4 packets carrying IP options, TCP and 1167 UDP port numbers are not at fixed offsets. 1169 4. The IPv6 header flow field SHOULD be used. This is the explicit 1170 purpose of the IPv6 flow field, however observed flow fields 1171 rarely contains a non-zero value. Some uses of the flow field 1172 have been defined such as [RFC6438]. In the absence of MPLS 1173 encapsulation, the IPv6 flow field can serve a role equivalent to 1174 entropy label. 1176 5. Support for other protocols that share a common Layer-4 header 1177 such as RTP, UDP-lite, SCTP and DCCP SHOULD be provided, 1178 particularly for edge or access equipment where additional 1179 entropy may be needed. Equipment SHOULD also use RTP, UDP-lite, 1180 SCTP and DCCP headers when creating an entropy label. 1182 6. The following IP header fields should not or must not be used: 1184 A. Similar to avoiding TC in MPLS, the IP DSCP, and ECN bits 1185 MUST NOT be used. 1187 B. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used. 1189 C. Note that the IP TOS field was deprecated ([RFC0791] was 1190 updated by [RFC2474]). No part of the IP DSCP field can be 1191 used (formerly IP PREC and IP TOS bits). 1193 7. Some IP encapsulations support tunneling, such as IP-in-IP, GRE, 1194 L2TPv3, and IPSEC. These provide a greater source of entropy 1195 which some provider networks carrying large amounts of tunneled 1196 traffic may need. The use of tunneling header information is out 1197 of scope for this document. 1199 This document makes the following recommendations. These 1200 recommendations are not required to claim compliance to any existing 1201 RFC therefore implementers are free to ignore them, but due to 1202 service provider requirements should consider the risk of doing so. 1203 The use of IP addresses MUST be supported and TCP and UDP ports 1204 (conditional on the protocol field and properly located) MUST be 1205 supported. The ability to disable use of UDP and TCP ports MUST be 1206 available. Though potentially very useful in some networks, it is 1207 uncommon to support using payloads of tunneling protocols carried 1208 over IP. Though the use of tunneling protocol header information is 1209 out of scope for this document, it is not discouraged. 1211 2.4.5.3. Fields Used in Flow Label 1213 The ingress to a pseudowire (PW) can extract information from the 1214 payload being encapsulated to create a flow label. [RFC6391] 1215 references IP carried in Ethernet as an example. The Native Service 1216 Processing (NSP) function defined in [RFC3985] differs with 1217 pseudowire type. It is in the NSP function where information for a 1218 specific type of PW can be extracted for use in a flow label. Which 1219 fields to use for any given PW NSP is out of scope for this document. 1221 2.4.5.4. Fields Used in Entropy Label 1223 An entropy label is added at the ingress to an LSP. The payload 1224 being encapsulated is most often MPLS, a PW, or IP. The payload type 1225 is identified by the layer-2 encapsulation (Ethernet, GFP, POS, etc). 1227 If the payload is MPLS, then the information used to create an 1228 entropy label is the same information used for local load balancing 1229 (see Section 2.4.5.1). This information MUST be extracted for use in 1230 generating an entropy label even if the LSR local egress interface is 1231 not a multipath. 1233 Of the non-MPLS payload types, only payloads that are forwarded are 1234 of interest. For example, ARP is not forwarded and CNLP (used only 1235 for ISIS) is not forwarded. 1237 The non-MPLS payload type of greatest interest are IPv4 and IPv6. 1238 The guidelines in Section 2.4.5.2 apply to fields used to create and 1239 entropy label. 1241 The IP tunneling protocols mentioned in Section 2.4.5.2 may be more 1242 applicable to generation of an entropy label at edge or access where 1243 deep packet inspection is practical due to lower interface speeds 1244 than in the core where deep packet inspection may be impractical. 1246 2.5. MPLS-TP and UHP 1248 MPLS-TP introduces forwarding demands that will be extremely 1249 difficult to meet in a core network. Most troublesome is the 1250 requirement for Ultimate Hop Popping (UHP, the opposite of 1251 Penultimate Hop Popping or PHP). Using UHP opens the possibility of 1252 one or more MPLS pop operation plus an MPLS swap operation for each 1253 packet. The potential for multiple lookups and multiple counter 1254 instances per packet exists. 1256 As networks grow and tunneling of LDP LSPs into RSVP-TE LSPs is used, 1257 and/or RSVP-TE hierarchy is used, the requirement to perform one or 1258 two or more MPLS pop operations plus a MPLS swap operation (and 1259 possibly a push or two) increases. If MPLS-TP LM (link monitoring) 1260 OAM is enabled at each layer, then a packet and byte count MUST be 1261 maintained for each pop and swap operation so as to offer OAM for 1262 each layer. 1264 2.6. Local Delivery of Packets 1266 There are a number of situations in which packets are destined to a 1267 local address or where a return packet must be generated. There is a 1268 need to mitigate the potential for outage as a result of either 1269 attacks on network infrastructure, or in some cases unintentional 1270 misconfiguration resulting in processor overload. Some hardware 1271 assistance is needed for all traffic destined to the general purpose 1272 CPU that is used in MPLS control protocol processing or network 1273 management protocol processing and in most cases to other general 1274 purpose CPUs residing on an LSR. This is due to the ease of 1275 overwhelming such a processor with traffic arriving on LSR high speed 1276 interfaces, whether the traffic is malicious or not. 1278 Denial of service (DoS) protection is an area requiring hardware 1279 support that is often overlooked or inadequately considered. 1280 Hardware assist is also needed for OAM, particularly the more 1281 demanding MPLS-TP OAM. 1283 2.6.1. DoS Protection 1285 Modern equipment supports a number of control plane and management 1286 plane protocols. Generally no single means of protecting network 1287 equipment from denial of service (DoS) attacks is sufficient, 1288 particularly for high speed interfaces. This problem is not specific 1289 to MPLS, but is a topic that cannot be ignored when implementing or 1290 evaluating MPLS implementations. 1292 Two types of protections are often cited as primary means of 1293 protecting against attacks of all kinds. 1295 Isolated Control/Management Traffic 1296 Control and Management traffic can be carried out-of-band (OOB), 1297 meaning not intermixed with payload. For MPLS, use of G-ACh and 1298 GAL to carry control and management traffic provides a means of 1299 isolation from potentially malicious payload. Used alone, the 1300 compromise of a single node, including a small computer at a 1301 network operations center, could compromise an entire network. 1302 Implementations which send all G-ACh/GAL traffic directly to a 1303 routing engine CPU are subject to DoS attack as a result of such 1304 a compromise. 1306 Cryptographic Authentication 1307 Cryptographic authentication can very effectively prevent 1308 malicious injection of control or management traffic. 1309 Cryptographic authentication can is some circumstances be subject 1310 to DoS attack by overwhelming the capacity of the decryption with 1311 a high volume of malicious traffic. For very low speed 1312 interfaces, cryptographic authentication can be performed by the 1313 general purpose CPU used as a routing engine. For all other 1314 cases, cryptographic hardware may be needed. For very high speed 1315 interfaces, even cryptographic hardware can be overwhelmed. 1317 Some control and management protocols are often carried with payload 1318 traffic. This is commonly the case with BGP, T-LDP, and SNMP. It is 1319 often the case with RSVP-TE. Even when carried over G-ACh/GAL 1320 additional measures can reduce the potential for a minor breach to be 1321 leveraged to a full network attack. 1323 Some of the additional protections are supported by hardware packet 1324 filtering. 1326 GTSM 1327 [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop 1328 Limit fields to insure control traffic that can only originate 1329 from an immediate neighbor is not forged and originating from a 1330 distant source. GTSM can be applied to many control protocols 1331 which are routable, for example LDP [RFC6720]. 1333 IP Filtering 1334 At the very minimum, packet filtering plus classification and use 1335 of multiple queues supporting rate limiting is needed for traffic 1336 that could potentially be sent to a general purpose CPU used as a 1337 routing engine. The first level of filtering only allows 1338 connections to be initiated from specific IP prefixes to specific 1339 destination ports and then preferably passes traffic directly to 1340 a cryptographic engine and/or rate limits. The second level of 1341 filtering passes connected traffic, such as TCP connections 1342 having received at least one authenticated SYN or having been 1343 locally initiated. The second level of filtering only passes 1344 traffic to specific address and port pairs to be checked for 1345 cryptographic authentication. 1347 The cryptographic authentication is generally the last resort in DoS 1348 attack mitigation. If a packet must be first sent to a general 1349 purpose CPU, then sent to a cryptographic engine, a DoS attack is 1350 possible on high speed interfaces. Only where hardware can identify 1351 a signature and the portion of packet covered by the signature is 1352 cryptographic authentication highly beneficial in protecting against 1353 DoS attacks. 1355 For chips supporting multiple 100 Gb/s interfaces, only a very large 1356 number of parallel cryptographic engines can provide the processing 1357 capacity to handle a large scale DoS or distributed DoS (DDoS) 1358 attack. For many forwarding chips this much processing power 1359 requires significant chip real estate and power, and therefore 1360 reduces system space and power density. For this reason, 1361 cryptographic authentication is not considered a viable first line of 1362 defense. 1364 For some networks the first line of defense is some means of 1365 supporting OOB control and management traffic. In the past this OOB 1366 channel might make use of overhead bits in SONET or OTN or a 1367 dedicated DWDM wavelength. G-ACh and GAL provide an alternative OOB 1368 mechanism which is independent of underlying layers. In other 1369 networks, including most IP/MPLS networks, perimeter filtering serves 1370 a similar purpose, though less effective without extreme vigilance. 1372 A second line of defense is filtering, including GTSM. For protocols 1373 such as EBGP, GTSM and other filtering is often the first line of 1374 defense. Cryptographic authentication is usually the last line of 1375 defense and insufficient by itself to mitigate DoS or DDoS attacks. 1377 2.6.2. MPLS OAM 1379 [RFC4377] defines requirements for MPLS OAM that predate MPLS-TP. 1380 [RFC4379] defines what is commonly referred to as LSP Ping and LSP 1381 Traceroute. [RFC4379] is updated by [RFC6424] supporting MPLS 1382 tunnels and stitched LSP and P2MP LSP. [RFC4379] is updated by 1383 [RFC6425] supporting P2MP LSP. [RFC4379] is updated by [RFC6426] to 1384 support MPLS-TP connectivity verification (CV) and route tracing. 1386 [RFC4950] extends the ICMP format to support TTL expiration that may 1387 occur when using IP traceroute within an MPLS tunnel. The ICMP 1388 message generation can be implemented in forwarding hardware, but if 1389 sent to a general purpose CPU must be rate limited to avoid a 1390 potential denial or service (DoS) attack. 1392 [RFC5880] defines Bidirectional Forwarding Detection (BFD), a 1393 protocol intended to detect faults in the bidirectional path between 1394 two forwarding engines. [RFC5884] and [RFC5885] define BFD for MPLS. 1395 BFD can provide failure detection on any kind of path between 1396 systems, including direct physical links, virtual circuits, tunnels, 1397 MPLS Label Switched Paths (LSPs), multihop routed paths, and 1398 unidirectional links as long as there is some return path. 1400 The processing requirements for BFD are less than for LSP Ping, 1401 making BFD somewhat better suited for relatively high rate proactive 1402 monitoring. BFD does not verify that the data plane matches the 1403 control plane, where LSP Ping does. LSP Ping is somewhat better 1404 suited for on-demand monitoring including relatively low rate 1405 periodic verification of data plane and as a diagnostic tool. 1407 Hardware assistance is often provided for BFD response where BFD 1408 setup or parameter change is not involved and may be necessary for 1409 relatively high rate proactive monitoring. If both BFD and LSP Ping 1410 are recognized in filtering prior to passing traffic to a general 1411 purpose CPU, appropriate DoS protection can be applied (see 1412 Section 2.6.1). Failure to recognize BFD and LSP Ping and at least 1413 rate limit creates the potential for misconfiguration to cause 1414 outages rather than cause errors in the misconfigured OAM. 1416 2.6.3. Pseudowire OAM 1418 Pseudowire OAM makes use of the control channel provided by Virtual 1419 Circuit Connectivity Verification (VCCV) [RFC5085]. VCCV makes use 1420 of the Pseudowire Control Word. BFD support over VCCV is defined by 1421 [RFC5885]. [RFC5885] is updated by [RFC6478] in support of static 1422 pseudowires. [RFC4379] is updated by [RFC6829] supporting LSP Ping 1423 for Pseudowire FEC advertised over IPv6. 1425 G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control 1426 channel and applies to any MPLS-TP end points, including Pseudowire. 1427 See Section 2.6.4 for an overview of MPLS-TP OAM. 1429 2.6.4. MPLS-TP OAM 1431 [RFC6669] summarizes the MPLS-TP OAM toolset, the set of protocols 1432 supporting the MPLS-TP OAM requirements specified in [RFC5860] and 1433 supported by the MPLS-TP OAM framework defined in [RFC6371]. 1435 The MPLS-TP OAM toolset includes: 1437 CC-CV 1438 [RFC6428] defines BFD extensions to support proactive 1439 Connectivity Check and Connectivity Verification (CC-CV) 1440 applications. [RFC6426] provides LSP ping extensions that are 1441 used to implement on-demand connectivity verification. 1443 RDI 1444 Remote Defect Indication (RDI) is triggered by failure of 1445 proactive CC-CV, which is BFD based. For fast RDI initiation, 1446 RDI SHOULD be initiated and handled by hardware if BFD is handled 1447 in forwarding hardware. [RFC6428] provides an extension for BFD 1448 that includes the RDI indication in the BFD format and a 1449 specification of how this indication is to be used. 1451 Route Tracing 1452 [RFC6426] specifies that the LSP ping enhancements for MPLS-TP 1453 on-demand connectivity verification include information on the 1454 use of LSP ping for route tracing of an MPLS-TP path. 1456 Alarm Reporting 1457 [RFC6427] describes the details of a new protocol supporting 1458 Alarm Indication Signal (AIS), Link Down Indication, and fault 1459 management. Failure to support this functionality in forwarding 1460 hardware can potentially result in failure to meet protection 1461 recovery time requirements and is therefore strongly recommended. 1463 Lock Instruct 1464 Lock instruct is initiated on-demand and therefore need not be 1465 implemented in forwarding hardware. [RFC6435] defines a lock 1466 instruct protocol. 1468 Lock Reporting 1469 [RFC6427] covers lock reporting. Lock reporting need not be 1470 implemented in forwarding hardware. 1472 Diagnostic 1473 [RFC6435] defines protocol support for loopback. Loopback 1474 initiation is on-demand and therefore need not be implemented in 1475 forwarding hardware. Loopback of packet traffic SHOULD be 1476 implemented in forwarding hardware on high speed interfaces. 1478 Packet Loss and Delay Measurement 1479 [RFC6374] and [RFC6375] define a protocol and profile for packet 1480 loss measurement (LM) and delay measurement (DM). LM requires a 1481 very accurate capture and insertion of packet and byte counters 1482 when a packet is transmitted and capture of packet and byte 1483 counters when a packet is received. This capture and insertion 1484 MUST be implemented in forwarding hardware for LM OAM if high 1485 accuracy is needed. DM requires very accurate capture and 1486 insertion of a timestamp on transmission and capture of timestamp 1487 when a packet is received. This timestamp capture and insertion 1488 MUST be implemented in forwarding hardware for DM OAM if high 1489 accuracy is needed. 1491 See Section 2.6.2 for discussion of hardware support necessary for 1492 BFD and LSP Ping. 1494 CC-CV and alarm reporting is tied to protection and therefore SHOULD 1495 be supported in forwarding hardware in order to provide protection 1496 for a large number of affected LSP within target response intervals. 1497 Since CC-CV is supported by BFD, for MPLS-TP providing hardware 1498 assistance for BFD processing helps insure that protection recovery 1499 time requirements can be met even for faults affecting a large number 1500 of LSP. 1502 2.6.5. MPLS OAM and Layer-2 OAM Interworking 1504 [RFC6670] provides the reasons for selecting a single MPLS-TP OAM 1505 solution and examines the consequences were ITU-T to develop a second 1506 OAM solution that is based on Ethernet encodings and mechanisms. 1508 [RFC6310] and [RFC7023] specifies the mapping of defect states 1509 between many types of hardware Attachment Circuits (ACs) and 1510 associated Pseudowires (PWs). This functionality SHOULD be supported 1511 in forwarding hardware. 1513 It is beneficial if an MPLS OAM implementation can interwork with the 1514 underlying server layer and provide a means to interwork with a 1515 client layer. For example, [RFC6427] specifies an inter-layer 1516 propagation of AIS and LDI from MPLS server layer to client MPLS 1517 layers. Where the server layer is a Layer-2, such as Ethernet, PPP 1518 over SONET/SDH, or GFP over OTN, interwork among layers is also 1519 beneficial. For high speed interfaces, supporting this interworking 1520 in forwarding hardware helps insure that protection based on this 1521 interworking can meet recovery time requirements even for faults 1522 affecting a large number of LSP. 1524 2.6.6. Extent of OAM Support by Hardware 1526 Where certain requirements must be met, such as relatively high CC-CV 1527 rates and a large number of interfaces, or strict protection recovery 1528 time requirements and a moderate number of affected LSP, some OAM 1529 functionality must be supported by forwarding hardware. In other 1530 cases, such as highly accurate LM and DM OAM or strict protection 1531 recovery time requirements with a large number of affected LSP, OAM 1532 functionality must be entirely implemented in forwarding hardware. 1534 Where possible, implementation in forwarding hardware should be in 1535 programmable hardware such that if standards are later changed or 1536 extended these changes are likely to be accommodated with hardware 1537 reprogramming rather than replacement. 1539 For some functionality there is a strong case for an implementation 1540 in dedicated forwarding hardware. Examples include packet and byte 1541 counters needed for LM OAM as well as needed for management 1542 protocols. Similarly the capture and insertion of packet and byte 1543 counts or timestamps needed for transmitted LM or DM or time 1544 synchronization packets MUST be implemented in forwarding hardware if 1545 high accuracy is required. 1547 For some functions there is a strong case to provide limited support 1548 in forwarding hardware but may make use of an external general 1549 purpose processor if performance criteria can be met. For example 1550 origination of RDI triggered by CC-CV, response to RDI, and PSC 1551 functionality may be supported by hardware, but expansion to a large 1552 number of client LSP and transmission of AIS or RDI to the client LSP 1553 may occur in a general purpose processor. Some forwarding hardware 1554 supports one or more on-chip general purpose processors which may be 1555 well suited for such a role. 1557 The customer (system supplier or provider) should not dictate design, 1558 but should independently validate target functionality and 1559 performance. However, it is not uncommon for service providers and 1560 system implementers to insist on reviewing design details (under NDA) 1561 due to past experiences with suppliers and to reject suppliers who 1562 are unwilling to provide details. 1564 2.7. Number and Size of Flows 1566 Service provider networks may carry up to hundreds of millions of 1567 flows on 10 Gb/s links. Most flows are very short lived, many under 1568 a second. A subset of the flows are low capacity and somewhat long 1569 lived. When Internet traffic dominates capacity a very small subset 1570 of flows are high capacity and/or very long lived. 1572 Two types of limitations with regard to number and size of flows have 1573 been observed. 1575 1. Some hardware cannot handle some very large flows because of 1576 internal paths which are limited, such as per packet backplane 1577 paths or paths internal or external to chips such as buffer 1578 memory paths. Such designs can handle aggregates of smaller 1579 flows. Some hardware with acknowledged limitations has been 1580 successfully deployed but may be increasingly problematic if the 1581 capacity of large microflows in deployed networks continues to 1582 grow. 1584 2. Some hardware approaches cannot handle a large number of flows, 1585 or a large number of large flows due to attempting to count per 1586 flow, rather than deal with aggregates of flows. Hash techniques 1587 scale with regard to number of flows due to a fixed hash size 1588 with many flows falling into the same hash bucket. Techniques 1589 that identify individual flows have been implemented but have 1590 never successfully deployed for Internet traffic. 1592 3. Questions for Suppliers 1594 The following questions should be asked of a supplier. These 1595 questions are grouped into broad categories. The questions 1596 themselves are intended to be an open ended question to the supplier. 1597 The tests in Section 4 are intended to verify whether the supplier 1598 disclosed any compliance or performance limitations completely and 1599 accurately. 1601 3.1. Basic Compliance 1603 Q#1 Can the implementation forward packets with an arbitrarily 1604 large stack depth? What limitations exist, and under what 1605 circumstances do further limitations come into play (such as 1606 high packet rate or specific features enabled or specific types 1607 of packet processing)? See Section 2.1. 1609 Q#2 Is the entire set of basic MPLS functionality described in 1610 Section 2.1 supported? 1612 Q#3 Are the set of MPLS special purpose labels handled correctly 1613 and with adequate performance? Are extended special purpose 1614 labels handled correctly and with adequate performance? See 1615 Section 2.1.1. 1617 Q#4 Are mappings of label value and TC to PHB handled correctly, 1618 including RFC3270 L-LSP mappings and RFC4124 CT mappings to 1619 PHB? See Section 2.1.2. 1621 Q#5 Is time synchronization adequately supported in forwarding 1622 hardware? 1624 A. Are both PTP and NTP formats supported? 1626 B. Is the accuracy of timestamp insertion and incoming 1627 stamping sufficient? 1629 See Section 2.1.3. 1631 Q#6 Is link bundling supported? 1633 A. Can LSP be pinned to specific components? 1635 B. Is the "all-ones" component link supported? 1637 See Section 2.1.5. 1639 Q#7 Is MPLS hierarchy supported? 1641 A. Are both PHP and UHP supported? What limitations exist on 1642 the number of pop operations with UHP? 1644 B. Are the pipe, short-pipe, and uniform models supported? 1645 Are TTL and TC values updated correctly at egress where 1646 applicable? 1648 See Section 2.1.6 1650 Q#8 Are pseudowire sequence numbers handled correctly? See 1651 Section 2.1.8.1. 1653 Q#9 Is VPN LER functionality handled correctly and without 1654 performance issues? See Section 2.1.9. 1656 Q#10 Is MPLS multicast (P2MP and MP2MP) handled correctly? 1658 A. Are packets dropped on uncongested outputs if some outputs 1659 are congested? 1661 B. Is performance limited in high fanout situations? 1663 See Section 2.2. 1665 3.2. Basic Performance 1667 Q#11 Can very small packets be forwarded at full line rate on all 1668 interfaces indefinitely? What limitations exist, and under 1669 what circumstances do further limitations come into play (such 1670 as specific features enabled or specific types of packet 1671 processing)? 1673 Q#12 Customers must decide whether to relax the prior requirement 1674 and to what extent. If the answer to the prior question 1675 indicates that limitations exist, then: 1677 A. What is the smallest packet size where full line rate 1678 forwarding can be supported? 1680 B. What is the longest burst of full rate small packets that 1681 can be supported? 1683 Specify circumstances (such as specific features enabled or 1684 specific types of packet processing) often impact these rates 1685 and burst sizes. 1687 Q#13 How many pop operations can be supported along with a swap 1688 operation at full line rate while maintaining per LSP packet 1689 and byte counts for each pop and swap? This requirement is 1690 particularly relevant for MPLS-TP. 1692 Q#14 How many label push operations can be supported. While this 1693 limitation is rarely an issue, it applies to both PHP and UHP, 1694 unlike the pop limit which applies to UHP. 1696 Q#15 For a worst case where all packets arrive on one LSP, what is 1697 the counter overflow time? Are any means provided to avoid 1698 polling all counters at short intervals? This applies to both 1699 MPLS and MPLS-TP. 1701 3.3. Multipath Capabilities and Performance 1703 Multipath capabilities and performance do not apply to MPLS-TP but 1704 apply to MPLS and apply if MPLS-TP is carried in MPLS. 1706 Q#16 How are large microflows accommodated? Is there active 1707 management of the hash space mapping to output ports? See 1708 Section 2.4.2. 1710 Q#17 How many MPLS labels can be included in a hash based on the 1711 MPLS label stack? 1713 Q#18 Is packet rate performance decreased beyond some number of 1714 labels? 1716 Q#19 Can the IP header and payload information below the MPLS stack 1717 be used in the hash? If so, which IP fields, payload types and 1718 payload fields are supported? 1720 Q#20 At what maximum MPLS label stack depth can Bottom of Stack and 1721 an IP header appear without impacting packet rate performance? 1723 Q#21 Are special purpose labels excluded from the label stack hash? 1724 Are extended purpose labels excluded from the label stack hash? 1725 See Section 2.4.5.1. 1727 Q#22 How is multipath performance affected by very large flows or an 1728 extremely large number of flows, or by very short lived flows? 1729 See Section 2.7. 1731 3.4. Pseudowire Capabilities and Performance 1733 Q#23 Is the pseudowire control word supported? 1735 Q#24 What is the maximum rate of pseudowire encapsulation and 1736 decapsulation? Apply the same questions as in Base Performance 1737 for any packet based pseudowire such as IP VPN or Ethernet. 1739 Q#25 Does inclusion of a pseudowire control word impact performance? 1741 Q#26 Are flow labels supported? 1743 Q#27 If so, what fields are hashed on for the flow label for 1744 different types of pseudowires? 1746 Q#28 Does inclusion of a flow label impact performance? 1748 3.5. Entropy Label Support and Performance 1750 Q#29 Can an entropy label be added when acting as in ingress LER and 1751 can it be removed when acting as an egress LER? 1753 Q#30 If so, what fields are hashed on for the entropy label? 1755 Q#31 Does adding or removing an entropy label impact packet rate 1756 performance? 1758 Q#32 Can an entropy label be detected in the label stack, used in 1759 the hash, and properly terminate the search for further 1760 information to hash on? 1762 Q#33 Does using an entropy label have any negative impact on 1763 performance? It should have no impact or a positive impact. 1765 3.6. DoS Protection 1767 Q#34 For each control and management plane protocol in use, what 1768 measures are taken to provide DoS attack hardening? 1770 Q#35 Have DoS attack tests been performed? 1772 Q#36 Can compromise of an internal computer on a management subnet 1773 be leveraged for any form of attack including DoS attack? 1775 3.7. OAM Capabilities and Performance 1777 Q#37 What OAM proactive and on-demand mechanisms are supported? 1779 Q#38 What performance limits exist under high proactive monitoring 1780 rates? 1782 Q#39 Can excessively high proactive monitoring rates impact control 1783 plane performance or cause control plane instability? 1785 Q#40 Ask the prior questions for each of the following. 1787 A. MPLS OAM 1789 B. Pseudowire OAM 1791 C. MPLS-TP OAM 1793 D. Layer-2 OAM Interworking 1795 See Section 2.6.2. 1797 4. Forwarding Compliance and Performance Testing 1799 Packet rate performance of equipment supporting a large number of 10 1800 Gb/s or 100 Gb/s links is not possible using desktop computers or 1801 workstations. The use of high end workstations as a source of test 1802 traffic was barely viable 20 years ago, but is no longer at all 1803 viable. Though custom microcode has been used on specialized router 1804 forwarding cards to serve the purpose of generating test traffic and 1805 measuring it, for the most part performance testing will require 1806 specialized test equipment. There are multiple sources of suitable 1807 equipment. 1809 The set of tests listed here do not correspond one-to-one to the set 1810 of questions in Section 3. The same categorization is used and these 1811 tests largely serve to validate answers provided to the prior 1812 questions, and can also provide answers where a supplier is unwilling 1813 to disclose compliance or performance. 1815 Performance testing is the domain of the IETF Benchmark Methodology 1816 Working Group (BMWG). Below are brief descriptions of conformance 1817 and performance tests. Some very basic tests are specified in 1818 [RFC5695] which partially cover only the basic performance test T#3. 1820 The following tests should be performed by the systems designer, or 1821 deployer, or performed by the supplier on their behalf if it is not 1822 practical for the potential customer to perform the tests directly. 1823 These tests are grouped into broad categories. 1825 The tests in Section 4.1 should be repeated under various conditions 1826 to retest basic performance when critical capabilities are enabled. 1827 Complete repetition of the performance tests enabling each capability 1828 and combinations of capabilities would be very time intensive, 1829 therefore a reduced set of performance tests can be used to gauge the 1830 impact of enabling specific capabilities. 1832 4.1. Basic Compliance 1834 T#1 Test forwarding at a high rate for packets with varying number 1835 of label entries. While packets with more than a dozen label 1836 entries are unlikely to be used in any practical scenario today, 1837 it is useful to know if limitations exists. 1839 T#2 For each of the questions listed under "Basic Compliance" in 1840 Section 3, verify the claimed compliance. For any functionality 1841 considered critical to a deployment, where applicable 1842 performance using each capability under load should be verified 1843 in addition to basic compliance. 1845 4.2. Basic Performance 1847 T#3 Test packet forwarding at full line rate with small packets. 1848 See [RFC5695]. The most likely case to fail is the smallest 1849 packet size. Also test with packet sizes in four byte 1850 increments ranging from payload sizes or 40 to 128 bytes. 1852 T#4 If the prior tests did not succeed for all packet sizes, then 1853 perform the following tests. 1855 A. Increase the packet size by 4 bytes until a size is found 1856 that can be forwarded at full rate. 1858 B. Inject bursts of consecutive small packets into a stream of 1859 larger packets. Allow some time for recovery between 1860 bursts. Increase the number of packets in the burst until 1861 packets are dropped. 1863 T#5 Send test traffic where a swap operation is required. Also set 1864 up multiple LSP carried over other LSP where the device under 1865 test (DUT) is the egress of these LSP. Create test packets such 1866 that the swap operation is performed after pop operations, 1867 increasing the number of pop operations until forwarding of 1868 small packets at full line rate can no longer be supported. 1869 Also check to see how many pop operations can be supported 1870 before the full set of counters can no longer be maintained. 1871 This requirement is particularly relevant for MPLS-TP. 1873 T#6 Send all traffic on one LSP and see if the counters become 1874 inaccurate. Often counters on silicon are much smaller than the 1875 64 bit packet and byte counters in IETF MIB. System developers 1876 should consider what counter polling rate is necessary to 1877 maintain accurate counters and whether those polling rates are 1878 practical. Relevant MIBs for MPLS are discussed in [RFC4221] 1879 and [RFC6639]. 1881 4.3. Multipath Capabilities and Performance 1883 Multipath capabilities do not apply to MPLS-TP but apply to MPLS and 1884 apply if MPLS-TP is carried in MPLS. 1886 T#7 Send traffic at a rate well exceeding the capacity of a single 1887 multipath component link, and where entropy exists only below 1888 the top of stack. If only the top label is used this test will 1889 fail immediately. 1891 T#8 Move the labels with entropy down in the stack until either the 1892 full forwarding rate can no longer be supported or most or all 1893 packets try to use the same component link. 1895 T#9 Repeat the two tests above with the entropy contained in IP 1896 headers or IP payload fields below the label stack rather than 1897 in the label stack. Test with the set of IP headers or IP 1898 payload fields considered relevant to the deployment or to the 1899 target market. 1901 T#10 Determine whether traffic that contains a pseudowire control 1902 word is interpreted as IP traffic. Information in the payload 1903 MUST NOT be used in the load balancing if the first nibble of 1904 the packet is not 4 or 6 (IPv4 or IPv6). 1906 T#11 Determine whether special purpose labels and extended special 1907 purpose labels are excluded from the label stack hash. They 1908 MUST be excluded. 1910 T#12 Perform testing in the presence of combinations of: 1912 A. Very large microflows. 1914 B. Relatively short lived high capacity flows. 1916 C. Extremely large numbers of flows. 1918 D. Very short lived small flows. 1920 4.4. Pseudowire Capabilities and Performance 1922 T#13 Ensure that pseudowire can be set up with a pseudowire label 1923 and pseudowire control word added at ingress and the pseudowire 1924 label and pseudowire control word removed at egress. 1926 T#14 For pseudowire that contains variable length payload packets, 1927 repeat performance tests listed under "Basic Performance" for 1928 pseudowire ingress and egress functions. 1930 T#15 Repeat pseudowire performance tests with and without a 1931 pseudowire control word. 1933 T#16 Determine whether pseudowire can be set up with a pseudowire 1934 label, flow label, and pseudowire control word added at ingress 1935 and the pseudowire label, flow label, and pseudowire control 1936 word removed at egress. 1938 T#17 Determine which payload fields are used to create the flow 1939 label and whether the set of fields and algorithm provide 1940 sufficient entropy for load balancing. 1942 T#18 Repeat pseudowire performance tests with flow labels included. 1944 4.5. Entropy Label Support and Performance 1946 T#19 Determine whether entropy labels can be added at ingress and 1947 removed at egress. 1949 T#20 Determine which fields are used to create an entropy label. 1950 Labels further down in the stack, including entropy labels 1951 further down and IP headers or IP payload fields where 1952 applicable should be used. Determine whether the set of fields 1953 and algorithm provide sufficient entropy for load balancing. 1955 T#21 Repeat performance tests under "Basic Performance" when entropy 1956 labels are used, where ingress or egress is the device under 1957 test (DUT). 1959 T#22 Determine whether an ELI is detected when acting as a midpoint 1960 LSR and whether the search for further information on which to 1961 base the load balancing is used. Information below the entropy 1962 label SHOULD NOT be used. 1964 T#23 Ensure that the entropy label indicator and entropy label (ELI 1965 and EL) are removed from the label stack during UHP and PHP 1966 operations. 1968 T#24 Insure that operations on the TC field when adding and removing 1969 entropy label are correctly carried out. If TC is changed 1970 during a swap operation, the ability to transfer that change 1971 MUST be provided. The ability to suppress the transfer of TC 1972 MUST also be provided. See "pipe", "short pipe", and "uniform" 1973 models in [RFC3443]. 1975 T#25 Repeat performance tests for midpoint LSR with entropy labels 1976 found at various label stack depths. 1978 4.6. DoS Protection 1980 T#26 Actively attack LSR under high protocol churn load and 1981 determine control plane performance impact or successful DoS 1982 under test conditions. Specifically test for the following. 1984 A. TCP SYN attack against control plane and management plane 1985 protocols using TCP, including CLI access (typically SSH 1986 protected login), NETCONF, etc. 1988 B. High traffic volume attack against control plane and 1989 management plane protocols not using TCP. 1991 C. Attacks which can be performed from a compromised 1992 management subnet computer, but not one with authentication 1993 keys. 1995 D. Attacks which can be performed from a compromised peer 1996 within the control plane (internal domain and external 1997 domain). Assume that per peering keys and per router ID 1998 keys rather than network wide keys are in use. 2000 See Section 2.6.1. 2002 4.7. OAM Capabilities and Performance 2004 T#27 Determine maximum sustainable rates of BFD traffic. If BFD 2005 requires CPU intervention, determine both maximum rates and CPU 2006 loading when multiple interfaces are active. 2008 T#28 Verify LSP Ping and LSP Traceroute capability. 2010 T#29 Determine maximum rates of MPLS-TP CC-CV traffic. If CC-CV 2011 requires CPU intervention, determine both maximum rates and CPU 2012 loading when multiple interfaces are active. 2014 T#30 Determine MPLS-TP DM precision. 2016 T#31 Determine MPLS-TP LM accuracy. 2018 T#32 Verify MPLS-TP AIS/RDI and PSC functionality, protection speed, 2019 and AIS/RDI notification speed when a large number of 2020 Management Entities (ME) must be notified with AIS/RDI. 2022 5. Acknowledgements 2024 Numerous very useful comments have been received in private email. 2025 Some of these contributions are acknowledged here, approximately in 2026 chronologic order. 2028 Paul Doolan provided a brief review resulting in a number of 2029 clarifications, most notably regarding on-chip vs. system buffering, 2030 100 Gb/s link speed assumptions in the 150 Mpps figure, and handling 2031 of large microflows. Pablo Frank reminded us of the sawtooth effect 2032 in PPS vs. packet size graphs, prompting the addition of a few 2033 paragraphs on this. Comments from Lou Berger at IETF-85 prompted the 2034 addition of Section 2.7. 2036 Valuable comments were received on the BMWG mailing list. Jay 2037 Karthik pointed out testing methodology hints that after discussion 2038 were deemed out of scope and were removed but may benefit later work 2039 in BMWG. 2041 Nabil Bitar pointed out the need to cover QoS (Differentiated 2042 Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM. Nabil 2043 also provided a number of clarifications to the questions and tests 2044 in Section 3 and Section 4. 2046 Gregory Mirsky and Thomas Beckhaus provided useful comments during 2047 the MPLS RT review. 2049 Tal Mizrahi provided comments that prompted clarifications regarding 2050 timestamp processing, local delivery of packets, and the need for 2051 hardware assistance in processing OAM traffic. 2053 6. IANA Considerations 2055 This memo includes no request to IANA. 2057 7. Security Considerations 2059 This document reviews forwarding behavior specified elsewhere and 2060 points out compliance and performance requirements. As such it 2061 introduces no new security requirements or concerns. 2063 Knowledge of potential performance shortcomings may serve to help new 2064 implementations avoid pitfalls. It is unlikely that such knowledge 2065 could be the basis of new denial of service as these pitfalls are 2066 already widely known in the service provider community and among 2067 leading equipment suppliers. In practice extreme data and packet 2068 rate are needed to affect existing equipment and to affect networks 2069 that may be still vulnerable due to failure to implement adequate 2070 protection. The extreme data and packet rates make this type of 2071 denial of service unlikely and make undetectable denial of service of 2072 this type impossible. 2074 8. References 2076 8.1. Normative References 2078 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2079 Requirement Levels", BCP 14, RFC 2119, March 1997. 2081 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2082 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2083 Encoding", RFC 3032, January 2001. 2085 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2086 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2087 Tunnels", RFC 3209, December 2001. 2089 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2090 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2091 Protocol Label Switching (MPLS) Support of Differentiated 2092 Services", RFC 3270, May 2002. 2094 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2095 in Multi-Protocol Label Switching (MPLS) Networks", 2096 RFC 3443, January 2003. 2098 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 2099 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 2100 May 2005. 2102 [RFC4182] Rosen, E., "Removing a Restriction on the use of MPLS 2103 Explicit NULL", RFC 4182, September 2005. 2105 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 2106 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 2108 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2109 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2110 Use over an MPLS PSN", RFC 4385, February 2006. 2112 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2113 Marking in MPLS", RFC 5129, January 2008. 2115 [RFC5586] Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic 2116 Associated Channel", RFC 5586, June 2009. 2118 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 2119 J., and S. Amante, "Flow-Aware Transport of Pseudowires 2120 over an MPLS Packet Switched Network", RFC 6391, 2121 November 2011. 2123 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2124 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2125 RFC 6790, November 2012. 2127 8.2. Informative References 2129 [ACK-compression] 2130 "Observations and Dynamics of a Congestion Control 2131 Algorithm: The Effects of Two-Way Traffic", Proc. ACM 2132 SIGCOMM, ACM Computer Communications Review (CCR) Vol 21, 2133 No 4, 1991, pp.133-147., 1991. 2135 [ATM-EPD-and-PPD] 2136 "Dynamics of TCP Traffic over ATM Networks", IEEE Journal 2137 of Special Areas of Communication Vol 13 No 4, May 1995, 2138 pp. 633-641., May 1995. 2140 [I-D.ietf-mpls-special-purpose-labels] 2141 Kompella, K., Andersson, L., and A. Farrel, "Allocating 2142 and Retiring Special Purpose MPLS Labels", 2143 draft-ietf-mpls-special-purpose-labels-03 (work in 2144 progress), July 2013. 2146 [I-D.ietf-pwe3-vccv-impl-survey-results] 2147 Malis, A., "The Pseudowire (PW) & Virtual Circuit 2148 Connectivity Verification (VCCV) Implementation Survey 2149 Results", draft-ietf-pwe3-vccv-impl-survey-results-03 2150 (work in progress), October 2013. 2152 [I-D.ietf-tictoc-1588overmpls] 2153 Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. 2154 Montini, "Transporting Timing messages over MPLS 2155 Networks", draft-ietf-tictoc-1588overmpls-05 (work in 2156 progress), June 2013. 2158 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 2159 September 1981. 2161 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2162 "Definition of the Differentiated Services Field (DS 2163 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2164 December 1998. 2166 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 2167 and W. Weiss, "An Architecture for Differentiated 2168 Services", RFC 2475, December 1998. 2170 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 2171 "Assured Forwarding PHB Group", RFC 2597, June 1999. 2173 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2174 Label Switching Architecture", RFC 3031, January 2001. 2176 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2177 of Explicit Congestion Notification (ECN) to IP", 2178 RFC 3168, September 2001. 2180 [RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for 2181 Multiprotocol Label Switching Architecture (MPLS) 2182 Operation and Maintenance (OAM) Functions", RFC 3429, 2183 November 2002. 2185 [RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching 2186 (GMPLS) Signaling Functional Description", RFC 3471, 2187 January 2003. 2189 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 2190 Edge (PWE3) Architecture", RFC 3985, March 2005. 2192 [RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 2193 Provider-Provisioned Virtual Private Networks (PPVPNs)", 2194 RFC 4110, July 2005. 2196 [RFC4124] Le Faucheur, F., "Protocol Extensions for Support of 2197 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 2198 June 2005. 2200 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2201 Hierarchy with Generalized Multi-Protocol Label Switching 2202 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 2204 [RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol 2205 Label Switching (MPLS) Management Overview", RFC 4221, 2206 November 2005. 2208 [RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. 2209 Matsushima, "Operations and Management (OAM) Requirements 2210 for Multi-Protocol Label Switched (MPLS) Networks", 2211 RFC 4377, February 2006. 2213 [RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol 2214 Label Switched (MPLS) Data Plane Failures", RFC 4379, 2215 February 2006. 2217 [RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual 2218 Private Networks (L2VPNs)", RFC 4664, September 2006. 2220 [RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, 2221 "Extensions to Resource Reservation Protocol - Traffic 2222 Engineering (RSVP-TE) for Point-to-Multipoint TE Label 2223 Switched Paths (LSPs)", RFC 4875, May 2007. 2225 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 2226 Cost Multipath Treatment in MPLS Networks", BCP 128, 2227 RFC 4928, June 2007. 2229 [RFC4950] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP 2230 Extensions for Multiprotocol Label Switching", RFC 4950, 2231 August 2007. 2233 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 2234 Specification", RFC 5036, October 2007. 2236 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., and C. 2237 Pignataro, "The Generalized TTL Security Mechanism 2238 (GTSM)", RFC 5082, October 2007. 2240 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 2241 Connectivity Verification (VCCV): A Control Channel for 2242 Pseudowires", RFC 5085, December 2007. 2244 [RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) 2245 Report on MPLS Architectural Considerations for a 2246 Transport Profile", RFC 5317, February 2009. 2248 [RFC5332] Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS 2249 Multicast Encapsulations", RFC 5332, August 2008. 2251 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2252 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2253 Class" Field", RFC 5462, February 2009. 2255 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 2256 Benchmarking Methodology for IP Flows", RFC 5695, 2257 November 2009. 2259 [RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for 2260 Operations, Administration, and Maintenance (OAM) in MPLS 2261 Transport Networks", RFC 5860, May 2010. 2263 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 2264 (BFD)", RFC 5880, June 2010. 2266 [RFC5884] Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow, 2267 "Bidirectional Forwarding Detection (BFD) for MPLS Label 2268 Switched Paths (LSPs)", RFC 5884, June 2010. 2270 [RFC5885] Nadeau, T. and C. Pignataro, "Bidirectional Forwarding 2271 Detection (BFD) for the Pseudowire Virtual Circuit 2272 Connectivity Verification (VCCV)", RFC 5885, June 2010. 2274 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 2275 Time Protocol Version 4: Protocol and Algorithms 2276 Specification", RFC 5905, June 2010. 2278 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 2279 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 2280 Acronym in the IETF", BCP 161, RFC 6291, June 2011. 2282 [RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., 2283 Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, 2284 Administration, and Maintenance (OAM) Message Mapping", 2285 RFC 6310, July 2011. 2287 [RFC6371] Busi, I. and D. Allan, "Operations, Administration, and 2288 Maintenance Framework for MPLS-Based Transport Networks", 2289 RFC 6371, September 2011. 2291 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 2292 Measurement for MPLS Networks", RFC 6374, September 2011. 2294 [RFC6375] Frost, D. and S. Bryant, "A Packet Loss and Delay 2295 Measurement Profile for MPLS-Based Transport Networks", 2296 RFC 6375, September 2011. 2298 [RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, 2299 "Label Distribution Protocol Extensions for Point-to- 2300 Multipoint and Multipoint-to-Multipoint Label Switched 2301 Paths", RFC 6388, November 2011. 2303 [RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for 2304 Performing Label Switched Path Ping (LSP Ping) over MPLS 2305 Tunnels", RFC 6424, November 2011. 2307 [RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, 2308 S., and T. Nadeau, "Detecting Data-Plane Failures in 2309 Point-to-Multipoint MPLS - Extensions to LSP Ping", 2310 RFC 6425, November 2011. 2312 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2313 On-Demand Connectivity Verification and Route Tracing", 2314 RFC 6426, November 2011. 2316 [RFC6427] Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S., 2317 and D. Ward, "MPLS Fault Management Operations, 2318 Administration, and Maintenance (OAM)", RFC 6427, 2319 November 2011. 2321 [RFC6428] Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive 2322 Connectivity Verification, Continuity Check, and Remote 2323 Defect Indication for the MPLS Transport Profile", 2324 RFC 6428, November 2011. 2326 [RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., 2327 and X. Dai, "MPLS Transport Profile Lock Instruct and 2328 Loopback Functions", RFC 6435, November 2011. 2330 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 2331 for Equal Cost Multipath Routing and Link Aggregation in 2332 Tunnels", RFC 6438, November 2011. 2334 [RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, 2335 "Pseudowire Status for Static Pseudowires", RFC 6478, 2336 May 2012. 2338 [RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching 2339 Transport Profile (MPLS-TP) MIB-Based Management 2340 Overview", RFC 6639, June 2012. 2342 [RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, 2343 Administration, and Maintenance (OAM) Toolset for MPLS- 2344 Based Transport Networks", RFC 6669, July 2012. 2346 [RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a 2347 Single Solution for MPLS Transport Profile (MPLS-TP) 2348 Operations, Administration, and Maintenance (OAM)", 2349 RFC 6670, July 2012. 2351 [RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security 2352 Mechanism (GTSM) for the Label Distribution Protocol 2353 (LDP)", RFC 6720, August 2012. 2355 [RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label 2356 Switched Path (LSP) Ping for Pseudowire Forwarding 2357 Equivalence Classes (FECs) Advertised over IPv6", 2358 RFC 6829, January 2013. 2360 [RFC7023] Mohan, D., Bitar, N., Sajassi, A., DeLord, S., Niger, P., 2361 and R. Qiu, "MPLS and Ethernet Operations, Administration, 2362 and Maintenance (OAM) Interworking", RFC 7023, 2363 October 2013. 2365 Appendix A. Organization of References Section 2367 The References section is split into Normative and Informative 2368 subsections. References that directly specify forwarding 2369 encapsulations or behaviors are listed as normative. References 2370 which describe signaling only, though normative with respect to 2371 signaling, are listed as informative. They are informative with 2372 respect to MPLS forwarding. 2374 Authors' Addresses 2376 Curtis Villamizar (editor) 2377 Outer Cape Cod Network Consulting, LLC 2379 Email: curtis@occnc.com 2381 Kireeti Kompella 2382 Contrail Systems 2384 Email: kireeti.kompella@gmail.com 2385 Shane Amante 2386 Level 3 Communications, Inc. 2387 1025 Eldorado Blvd 2388 Broomfield, CO 80021 2390 Email: shane@level3.net 2392 Andrew Malis 2393 Verizon 2394 60 Sylvan Road 2395 Waltham, MA 02451 2397 Phone: +1 781-466-2362 2398 Email: andrew.g.malis@verizon.com 2400 Carlos Pignataro 2401 Cisco Systems 2402 7200-12 Kit Creek Road 2403 Research Triangle Park, NC 27709 2404 US 2406 Email: cpignata@cisco.com