idnits 2.17.00 (12 Aug 2021) /tmp/idnits65050/draft-ietf-mpls-forwarding-01.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (October 9, 2013) is 3146 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: draft-ietf-mpls-special-purpose-labels has been published as RFC 7274 == Outdated reference: draft-ietf-pwe3-mpls-eth-oam-iwk has been published as RFC 7023 == Outdated reference: draft-ietf-pwe3-vccv-impl-survey-results has been published as RFC 7079 == Outdated reference: A later version (-07) exists of draft-ietf-tictoc-1588overmpls-05 -- Obsolete informational reference (is this intentional?): RFC 4379 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6424 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6829 (Obsoleted by RFC 8029) Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPLS C. Villamizar, Ed. 3 Internet-Draft OCCNC 4 Intended status: Informational K. Kompella 5 Expires: April 12, 2014 Contrail Systems 6 S. Amante 7 Level 3 Communications, Inc. 8 A. Malis 9 Verizon 10 C. Pignataro 11 Cisco 12 October 9, 2013 14 MPLS Forwarding Compliance and Performance Requirements 15 draft-ietf-mpls-forwarding-01 17 Abstract 19 This document provides guidelines for implementers regarding MPLS 20 forwarding and a basis for evaluations of forwarding implementations. 21 Guidelines cover many aspects of MPLS forwarding. Topics are 22 highlighted where implementers might potentially overlook practical 23 requirements which are unstated or under emphasized or are optional 24 for conformance to RFCs but are often considered mandatory by 25 providers. 27 Status of this Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on April 12, 2014. 44 Copyright Notice 46 Copyright (c) 2013 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction and Document Scope . . . . . . . . . . . . . . . 4 62 1.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.2. Use of Requirements Language . . . . . . . . . . . . . . . 8 64 1.3. Apparent Misconceptions . . . . . . . . . . . . . . . . . 9 65 1.4. Target Audience . . . . . . . . . . . . . . . . . . . . . 10 66 2. Forwarding Issues . . . . . . . . . . . . . . . . . . . . . . 10 67 2.1. Forwarding Basics . . . . . . . . . . . . . . . . . . . . 11 68 2.1.1. MPLS Special Purpose Labels . . . . . . . . . . . . . 12 69 2.1.2. MPLS Differentiated Services . . . . . . . . . . . . . 12 70 2.1.3. Time Synchronization . . . . . . . . . . . . . . . . . 13 71 2.1.4. Uses of Multiple Label Stack Entries . . . . . . . . . 14 72 2.1.5. MPLS Link Bundling . . . . . . . . . . . . . . . . . . 15 73 2.1.6. MPLS Hierarchy . . . . . . . . . . . . . . . . . . . . 15 74 2.1.7. MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . . 15 75 2.1.8. Pseudowire Encapsulation . . . . . . . . . . . . . . . 16 76 2.1.8.1. Pseudowire Sequence Number . . . . . . . . . . . . 16 77 2.1.9. Layer-2 and Layer-3 VPN . . . . . . . . . . . . . . . 17 78 2.2. MPLS Multicast . . . . . . . . . . . . . . . . . . . . . . 18 79 2.3. Packet Rates . . . . . . . . . . . . . . . . . . . . . . . 18 80 2.4. MPLS Multipath Techniques . . . . . . . . . . . . . . . . 20 81 2.4.1. Pseudowire Control Word . . . . . . . . . . . . . . . 21 82 2.4.2. Large Microflows . . . . . . . . . . . . . . . . . . . 22 83 2.4.3. Pseudowire Flow Label . . . . . . . . . . . . . . . . 22 84 2.4.4. MPLS Entropy Label . . . . . . . . . . . . . . . . . . 22 85 2.4.5. Fields Used for Multipath Load Balance . . . . . . . . 23 86 2.4.5.1. MPLS Fields in Multipath . . . . . . . . . . . . . 23 87 2.4.5.2. IP Fields in Multipath . . . . . . . . . . . . . . 25 88 2.4.5.3. Fields Used in Flow Label . . . . . . . . . . . . 26 89 2.4.5.4. Fields Used in Entropy Label . . . . . . . . . . . 27 90 2.5. MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . . 27 91 2.6. Local Delivery of Packets . . . . . . . . . . . . . . . . 27 92 2.6.1. DoS Protection . . . . . . . . . . . . . . . . . . . . 28 93 2.6.2. MPLS OAM . . . . . . . . . . . . . . . . . . . . . . . 30 94 2.6.3. Pseudowire OAM . . . . . . . . . . . . . . . . . . . . 31 95 2.6.4. MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . . 31 96 2.6.5. MPLS OAM and Layer-2 OAM Interworking . . . . . . . . 32 97 2.6.6. Extent of OAM Support by Hardware . . . . . . . . . . 33 98 2.7. Number and Size of Flows . . . . . . . . . . . . . . . . . 34 99 3. Questions for Suppliers . . . . . . . . . . . . . . . . . . . 34 100 3.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . . 35 101 3.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 36 102 3.3. Multipath Capabilities and Performance . . . . . . . . . . 37 103 3.4. Pseudowire Capabilities and Performance . . . . . . . . . 37 104 3.5. Entropy Label Support and Performance . . . . . . . . . . 38 105 3.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . . 38 106 3.7. OAM Capabilities and Performance . . . . . . . . . . . . . 38 107 4. Forwarding Compliance and Performance Testing . . . . . . . . 39 108 4.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . . 39 109 4.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 40 110 4.3. Multipath Capabilities and Performance . . . . . . . . . . 41 111 4.4. Pseudowire Capabilities and Performance . . . . . . . . . 41 112 4.5. Entropy Label Support and Performance . . . . . . . . . . 42 113 4.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . . 43 114 4.7. OAM Capabilities and Performance . . . . . . . . . . . . . 43 115 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 44 116 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 44 117 7. Security Considerations . . . . . . . . . . . . . . . . . . . 44 118 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 45 119 8.1. Normative References . . . . . . . . . . . . . . . . . . . 45 120 8.2. Informative References . . . . . . . . . . . . . . . . . . 46 121 Appendix A. Organization of References Section . . . . . . . . . 51 122 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 51 124 1. Introduction and Document Scope 126 The initial purpose of this document was to address concerns raised 127 on the MPLS WG mailing list about shortcomings in implementations of 128 MPLS forwarding. Documenting existing misconceptions and potential 129 pitfalls might potentially avoid repeating past mistakes. The 130 document has grown to address a broad set of forwarding requirements. 132 The focus of this document is MPLS forwarding, base pseudowire 133 forwarding, and MPLS OAM. The use of pseudowire control word, and 134 sequence number are discussed. Specific pseudowire AC and NSP are 135 out of scope. Specific pseudowire applications, such as various 136 forms of VPN, are out of scope. 138 MPLS support for multipath techniques is considered essential by many 139 service providers and is useful for other high capacity networks. In 140 order to obtain sufficient entropy from MPLS traffic service 141 providers and others find it essential for the MPLS implementation to 142 interpret the MPLS payload as IPv4 or IPv6 based on the contents of 143 the first nibble of payload. The use of IP addresses, the IP 144 protocol field, and UDP and TCP port number fields in multipath load 145 balancing are considered within scope. The use of any other IP 146 protocol fields, such as tunneling protocols carried within IP, are 147 out of scope. 149 Implementation details are a local matter and are out of scope. Most 150 interfaces today operate at 1 Gb/s or greater. It is assumed that 151 all forwarding operations are implemented in specialized forwarding 152 hardware rather than on a general purpose processor. This is often 153 referred to as "fast path" and "slow path" processing. Some 154 recommendations are made regarding implementing control or management 155 plane functionality in specialized hardware or with limited 156 assistance from specialized hardware. This advise is based on 157 expected control or management protocol loads and on the need for 158 denial of service (DoS) protection. 160 1.1. Acronyms 162 The following acronyms are used. 164 AC Attachment Circuit ([RFC3985]) 166 ACH Associated Channel Header (pseudowires) 168 ACK Acknowledgement (TCP flag and type of TCP packet) 169 AIS Alarm Indication Signal (MPLS-TP OAM) 171 ATM Asynchronous Transfer Mode (legacy switched circuits) 173 BFD Bidirectional Forwarding Detection 175 BGP Border Gateway Protocol 177 CC-CV Connectivity Check and Connectivity Verification 179 CE Customer Edge (LDP) 181 CPU Central Processing Unit (computer or microprocessor) 183 CT Class Type ([RFC4124]) 185 CW Control Word ([RFC4385]) 187 DCCP Datagram Congestion Control Protocol 189 DDoS Distributed Denial of Service 191 DM Delay Measurement (MPLS-TP OAM) 193 DSCP Differentiated Services Code Point ([RFC2474]) 195 DWDM Dense Wave Division Multiplexing 197 DoS Denial of Service 199 E-LSP EXP-Inferred-PSC LSP ([RFC3270]) 201 EBGP External BGP 203 ECMP Equal Cost Multi-Path 205 ECN Explicit Congestion Notification ([RFC3168] and [RFC5129]) 207 EL Entropy Label ([RFC6790]) 209 ELI Entropy Label Indicator ([RFC6790]) 211 EXP Experimental (field in MPLS renamed to TC in [RFC5462]) 213 FEC Forwarding Equivalence Classes (LDP), also Forward Error 214 Correction in other context 216 FR Frame Relay (legacy switched circuits) 218 FRR Fast Reroute ([RFC4090]) 220 G-ACh Generic Associated Channel ([RFC5586]) 222 GAL Generic Associated Channel Label ([RFC5586]) 224 GFP Generic Framing Protocol (used in OTN) 226 GMPLS Generalized MPLS ([RFC3471]) 228 GTSM Generalized TTL Security Mechanism ([RFC5082]) 230 Gb/s Gigabits per second (billion bits per second) 232 IANA Internet Assigned Numbers Authority 234 ILM Incoming Label Map ([RFC3031]) 236 IP Internet Protocol 238 IPVPN Internet Protocol VPN 240 IPv4 Internet Protocol version 4 242 IPv6 Internet Protocol version 6 244 L-LSP Label-Only-Inferred-PSC LSP ([RFC3270]) 246 L2VPN Layer 2 VPN 248 LDP Label Distribution Protocol ([RFC5036]) 250 LER Label Edge Router ([RFC3031]) 252 LM Loss Measurement (MPLS-TP OAM) 254 LSP Label Switched Path ([RFC3031]) 256 LSR Label Switching Router ([RFC3031]) 258 MP2MP Multipoint to Point 260 MPLS MultiProtocol Label Switching ([RFC3031]) 261 MPLS-TP MPLS Transport Profile ([RFC5317]) 263 Mb/s Megabits per second (million bits per second) 265 NSP Native Service Processing ([RFC3985]) 267 NTP Network Time Protocol 269 OAM Operations, Administration, and Maintenance ([RFC6291]) 271 OOB Out-of-band (not carried within a data channel) 273 OTN Optical Transport Network 275 P Provider router (LDP) 277 P2MP Point to Multi-Point 279 PE Provider Edge router (LDP) 281 PHB Per-Hop-Behavior ([RFC2475]) 283 PHP Penultimate Hop Popping ([RFC3443]) 285 POS Packet over SONET 287 PSC This acronym has multiple interpretations. 289 1. Packet Switch Capable ([RFC3471] 291 2. PHB Scheduling Class ([RFC3270]) 293 3. Protection State Coordination (MPLS-TP linear protection) 295 PTP Precision Time Protocol 297 PW Pseudowire 299 QoS Quality of Service 301 RA Router Alert ([RFC3032]) 303 RDI Remote Defect Indication (MPLS-TP OAM) 305 RSVP-TE RSVP Traffic Engineering 306 RTP Real-Time Transport Protocol 308 SCTP Stream Control Transmission Protocol 310 SDH Synchronous Data Hierarchy (European SONET, a form of TDM) 312 SONET Synchronous Optical Network (US SDH, a form of TDM) 314 T-LDP Targeted LDP (LDP sessions over more than one hop) 316 TC Traffic Class ([RFC5462]) 318 TCP Transmission Control Protocol 320 TDM Time-Division Multiplexing (legacy encapsulations) 322 TOS Type of Service (see [RFC2474]) 324 TTL Time-to-live (a field in IP and MPLS headers) 326 UDP User Datagram Protocol 328 UHP Ultimate Hop Popping (opposite of PHP) 330 VCCV Virtual Circuit Connectivity Verification ([RFC5085]) 332 VLAN Virtual Local Area Network (Ethernet) 334 VOQ Virtual Output Queuing (switch fabric design) 336 VPN Virtual Private Network 338 WG Working Group 340 1.2. Use of Requirements Language 342 This document is informational. The key words "MUST", "MUST NOT", 343 "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", 344 "RECOMMENDED", "MAY", and "OPTIONAL" are used only where the 345 requirement is specified in an existing RFC. These keywords SHOULD 346 be interpreted as described in RFC 2119 [RFC2119]. 348 Advice given in this document does not use the upper case RFC 2119 349 keywords, except where explicitly noted that the keywords indicate 350 that operator experiences indicate a requirement, but there are no 351 existing RFC requirements. Such advice may be ignored by 352 implementations. Similarly, implementations not claiming conformance 353 to specific RFCs may ignore the requirements of those RFCs. In both 354 cases, implementers should consider the risk of doing so. 356 1.3. Apparent Misconceptions 358 In early generations of forwarding silicon (which might now be behind 359 us), there apparently were some misconceptions about MPLS. The 360 following statements provide clarifications. 362 1. There are practical reasons to have more than one or two labels 363 in an MPLS label stack. Under some circumstances the label stack 364 can become quite deep. See Section 2.1. 366 2. The label stack MUST be considered to be arbitrarily deep. 367 Section 3.27.4. "Hierarchy: LSP Tunnels within LSPs" of RFC3031 368 states "The label stack mechanism allows LSP tunneling to nest to 369 any depth." [RFC3031] If a bottom of the label stack cannot be 370 found, but sufficient number of labels exist to forward, an LSR 371 MUST forward the packet. An LSR MUST NOT assume the packet is 372 malformed unless the end of packet is found before bottom of 373 stack. See Section 2.1. 375 3. In networks where deep label stacks are encountered, they are not 376 rare. Full packet rate performance is required regardless of 377 label stack depth, except where multiple pop operations are 378 required. See Section 2.1. 380 4. Research has shown that long bursts of short packets with 40 byte 381 or 44 byte IP payload sizes in these bursts are quite common. 382 This is due to TCP ACK compression [ACK-compression]. The 383 following two sub-bullets constitutes advice that reflects very 384 common hard requirements of providers. Implementers may ignore 385 this advice but should consider the risk of doing so. 387 A. A forwarding engine SHOULD, if practical, be able to sustain 388 an arbitrarily long sequence of small packets arriving at 389 full interface rate. 391 B. If indefinite full packet rate for small packets is not 392 practical, a forwarding engine MUST be able to buffer a long 393 sequence of small packets inbound to the on-chip decision 394 engine and sustain full interface rate for some reasonable 395 average packet rate. Absent this small on-chip buffering, 396 QoS agnostic packet drops can occur. 398 See Section 2.3. 400 5. The implementer and system designer MUST support pseudowire 401 control word (CW) if MPLS-TP is supported or if ACH [RFC5586] is 402 being used on a pseudowire. The implementer and system designer 403 SHOULD support pseudowire CW even if MPLS-TP and ACH [RFC5586] 404 are not used, using instead CW and VCCV Type 1 [RFC5085] to allow 405 the use of multipath in the underlying network topology without 406 impacting the PW traffic. 407 [I-D.ietf-pwe3-vccv-impl-survey-results] does note that there are 408 still some deployments where the CW is not always used. It also 409 notes that many service providers do enable the CW. See 410 Section 2.4.1 for more discussion on why deployments SHOULD 411 enable the pseudowire CW. 413 6. The implementer and system designer SHOULD support adding a 414 pseudowire Flow Label [RFC6391]. Deployments MAY enable this 415 feature for appropriate pseudowire types. See Section 2.4.3. 417 7. The implementer and system designer SHOULD support adding an MPLS 418 entropy label [RFC6790]. Deployments MAY enable this feature. 419 See Section 2.4.4. 421 1.4. Target Audience 423 This document is intended for multiple audiences: implementer 424 (implementing MPLS forwarding in silicon or in software); systems 425 designer (putting together a MPLS forwarding systems); deployer 426 (running an MPLS network). These guidelines are intended to serve 427 the following purposes: 429 1. Explain what to do and what not to do when a deep label stack is 430 encountered. (audience: implementer) 432 2. Highlight pitfalls to look for when implementing an MPLS 433 forwarding chip. (audience: implementer) 435 3. Provide a checklist of features and performance specifications to 436 request. (audience: systems designer, deployer) 438 4. Provide a set of tests to perform. (audience: systems designer, 439 deployer). 441 The implementer, systems designer, and deployer have a transitive 442 supplier customer relationship. It is in the best interest of the 443 supplier to review their product against their customer's checklist 444 and customer's customer's checklist if applicable. 446 2. Forwarding Issues 448 A brief review of forwarding issues is provided in the subsections 449 that follow. This section provides some background on why some of 450 these requirements exist. The questions to ask of suppliers and 451 testing is covered in the following sections, Section 3 and 452 Section 4. 454 2.1. Forwarding Basics 456 Basic MPLS architecture and MPLS encapsulation, and therefore packet 457 forwarding are defined in [RFC3031] and [RFC3032]. RFC3031 and 458 RFC3032 are somewhat LDP centric. RSVP-TE supports traffic 459 engineering (TE) and fast reroute, features that LDP lacks. The base 460 document for RSVP-TE based MPLS is [RFC3209]. 462 A few RFCs update RFC3032. Those with impact on forwarding include 463 the following. 465 1. TTL processing is clarified in [RFC3443]. 467 2. The use of MPLS Explicit NULL is modified in [RFC4182]. 469 3. Differentiated Services is supported by [RFC3270] and [RFC4124]. 470 The "EXP" field is renamed to "Traffic Class" in [RFC5462], 471 removing any misconception that it was available for 472 experimentation or could be ignored. 474 4. ECN is supported by [RFC5129]. 476 5. The MPLS G-ACh and GAL are defined in [RFC5586]. 478 Other RFCs have implications to MPLS Forwarding and do not update 479 RFC3032 or RFC3209, including: 481 1. The pseudowire (PW) Associated Channel Header (ACH), defined by 482 [RFC5085], later generalized by the MPLS G-ACh [RFC5586]. 484 2. The entropy label indicator (ELI) and entropy label (EL) are 485 defined by [RFC6790]. 487 A few RFCs update RFC3209. Those that are listed as updating RFC3209 488 generally impact only RSVP-TE signaling. Forwarding is modified by 489 major extension built upon RFC3209. 491 RFCs which impact forwarding are discussed in the following 492 subsections. 494 2.1.1. MPLS Special Purpose Labels 496 [RFC3032] specifies that label values 0-15 are special purpose labels 497 with special meanings. Three values of NULL label are defined (two 498 of which are later updated by [RFC4182]) and a router-alert label is 499 defined. The original intent was that special purpose labels, except 500 the NULL labels, could be sent to the routing engine CPU rather than 501 be processed in forwarding hardware. Hardware support is required by 502 new RFCs such as those defining entropy label and OAM processed as a 503 result of receiving a GAL. For new special purpose labels, some 504 accommodation is needed for LSR that will send the labels to a 505 general purpose CPU or other highly programmable hardware. For 506 example, ELI will only be sent to LSR which have signaled support for 507 [RFC6790] and high OAM packet rate must be negotiated among 508 endpoints. 510 [RFC3429] reserves a label for ITU-T Y.1711, however Y.1711 does not 511 work with multipath and its use is strongly discouraged. 513 The current list of special purpose labels can be found on the 514 "Multiprotocol Label Switching Architecture (MPLS) Label Values" 515 registry reachable at IANA's pages at . 517 [I-D.ietf-mpls-special-purpose-labels] introduces an IANA "Extended 518 Special Purpose MPLS Label Values" registry and makes use of the 519 "extension" label, label 15, to indicate that the next label is an 520 extended special purpose label and requires special handling. The 521 range of only 16 values for special purpose labels allows a table to 522 be used. The range of extended special purpose labels with 20 bits 523 available for use may have to be handled in some other way in the 524 unlikely event that in the future the range of currently reserved 525 values 256-1048575 are used. If only the standards action range, 16- 526 239, and the experimental range, 240-255, are used, then a table of 527 256 entries can be used. 529 Unknown special purpose labels and unknown extended special purpose 530 labels are handled the same. When an unknown special purpose label 531 is encountered or a special purpose label not directly handled in 532 forwarding hardware is encountered, the packet should be sent to a 533 general purpose CPU by default. If this capability is supported, 534 there must be an option to either drop or rate limit such packets on 535 a per special purpose label value basis. 537 2.1.2. MPLS Differentiated Services 539 [RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence 540 (Prec) fields and replaces them with the Differentiated Services 541 Field more commonly known as the Differentiated Services Code Point 542 (DSCP) field. [RFC2475] defines the Differentiated Services 543 architecture, which in other forum is often called a Quality of 544 Service (QoS) architecture. 546 MPLS uses the Traffic Class (TC) field to support Differentiated 547 Services [RFC5462]. There are two primary documents describing how 548 DSCP is mapped into TC. 550 1. [RFC3270] defines E-LSP and L-LSP. E-LSP use a static mapping of 551 DSCP into TC. L-LSP uses a per LSP mapping of DSCP into TC, with 552 one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use 553 multiple Per-Hop Behavior (PHB) values. For example, the Assured 554 Forwarding service defines three PSC, each with three PHB 555 [RFC2597]. 557 2. [RFC4124] defines assignment of a class-type (CT) to an LSP, 558 where a per CT static mapping of TC to PHB is used. [RFC4124] 559 provides a means to support up to eight E-LSP-like mappings of 560 DSCP to TC. 562 To meet Differentiated Services requirements specified in [RFC3270], 563 the following forwarding requirements must be met. An ingress LER 564 MUST be able to select an LSP and then apply a per LSP map of DSCP 565 into TC. A midpoint LSR MUST be able to apply a per LSP map of TC to 566 PHB. The number of mappings supported will be far less than the 567 number of LSP supported. 569 To meet Differentiated Services requirements specified in [RFC4124], 570 the following forwarding requirements must be met. An ingress LER 571 MUST be able to select an LSP and then apply a per LSP map of DSCP 572 into TC. A midpoint LSR MUST be able to apply a per LSP map to CT 573 map and then use Class Type (CT) to map TC to PHB. Since there are 574 only eight allowed values of CT, only eight maps of TC to PHB need to 575 be supported. The LSP label can be used directly to find the TC to 576 PHB mapping, as is needed to support [RFC3270] L-LSP. 578 While support for [RFC4124] and not [RFC3270] would allow support for 579 only eight mappings of TC to PHB, it is common to support both and 580 simply state a limit on the number of unique TC to PHB mappings which 581 can be supported. 583 2.1.3. Time Synchronization 585 PTP or NTP may be carried over MPLS [I-D.ietf-tictoc-1588overmpls]. 586 Generally NTP will be carried within IP with IP carried in MPLS 587 [RFC5905]. Both PTP and NTP benefit from accurate time stamping of 588 incoming packets and the ability to insert accurate time stamps in 589 outgoing packets. PTP correction which occurs when forwarding 590 requires updating a timestamp compensation field based on the 591 difference between packet arrival at an LSR and packet transmit time 592 at that same LSR. 594 Since the label stack depth may vary, hardware should allow a 595 timestamp to be placed in an outgoing packet at any specified byte 596 position. It may be necessary to modify layer-2 checksums or frame 597 check sequences after insertion. PTP and NTP timestamp formats 598 differ slightly. If NTP or PTP is carried over UDP/IP or UDP/IP/ 599 MPLS, the UDP checksum will also have to be updated. 601 Accurate time synchronization in addition to being generally useful 602 is required for MPLS-TP delay measurement (DM) OAM. See 603 Section 2.6.4. 605 2.1.4. Uses of Multiple Label Stack Entries 607 MPLS deployments in the early part of the prior decade (circa 2000) 608 tended to support either LDP or RSVP-TE. LDP was favored by some for 609 its ability to scale to a very large number of PE devices at the edge 610 of the network, without adding deployment complexity. RSVP-TE was 611 favored, generally in the network core, where traffic engineering 612 and/or fast reroute were considered important. 614 Both LDP and RSVP-TE are used simultaneously within major Service 615 Provider networks using a technique known as "LDP over RSVP-TE 616 Tunneling". This technique allows service providers to carry LDP 617 tunnels, originating and terminating at PEs, inside of RSVP-TE 618 tunnels, generally between Inter-City P routers, to take advantage of 619 Traffic Engineering and Fast Re-Route on more expensive Inter-City 620 and Inter-Continental Transport paths. LDP over RSVP-TE tunneling 621 requires a minimum of two MPLS labels: one each for LDP and RSVP-TE. 623 The use of MPLS FRR [RFC4090] might add one more label to MPLS 624 traffic, but only when FRR protection is in use (active). If LDP 625 over RSVP-TE is in use, and FRR protection is in use, then at least 626 three MPLS labels are present on the label stack on the links through 627 which the Bypass LSP traverses. FRR is covered in Section 2.1.7. 629 LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN 630 services that are deployed in the vast majority of service providers. 631 These VPN services added yet another label, bringing the label stack 632 depth (when FRR is active) to four. 634 Pseudowires and VPN are discussed in further detail in Section 2.1.8 635 and Section 2.1.9. 637 MPLS hierarchy as described in [RFC4206] can in principle add up to 638 four additional labels. MPLS hierarchy is discussed in 639 Section 2.1.6. 641 Other features such as Entropy Label (discussed in Section 2.4.4) and 642 Flow Label (discussed in Section 2.4.3) can add additional labels to 643 the label stack. 645 Although theoretical scenarios can easily result in eight or more 646 labels, such cases are rare if they occur at all today. For the 647 purpose of forwarding, only the top label needs to be examined if PHP 648 is used, a few more if UHP is used (see Section 2.5). For deep label 649 stacks, quite a few labels may have to be examined for the purpose of 650 load balancing across parallel links (see Section 2.4), however this 651 depth can be bounded by a provider through use of Entropy Label. 653 2.1.5. MPLS Link Bundling 655 MPLS Link Bundling was the first RFC to address the need for multiple 656 parallel links between nodes [RFC4201]. MPLS Link Bundling is 657 notable in that it tried not to change MPLS forwarding, except in 658 specifying the "All-Ones" component link. MPLS Link Bundling is 659 seldom if ever deployed. Instead multipath techniques described in 660 Section 2.4 are used. 662 2.1.6. MPLS Hierarchy 664 MPLS hierarchy is defined in [RFC4206]. Although RFC4206 is 665 considered part of GMPLS, the Packet Switching Capable (PSC) portion 666 of the MPLS hierarchy are applicable to MPLS and may be supported in 667 an otherwise GMPLS free implementation. The MPLS PSC hierarchy 668 remains the most likely means of providing further scaling in an 669 RSVP-TE MPLS network, particularly where the network is designed to 670 provide RSVP-TE connectivity to the edges. This is the case for 671 envisioned MPLS-TP networks. The use of the MPLS PSC hierarchy can 672 add as many as four labels to a label stack, though it is likely that 673 only one layer of PSC will be used in the near future. 675 2.1.7. MPLS Fast Reroute (FRR) 677 Fast reroute is defined by [RFC4090]. Two significantly different 678 methods are defined in RFC4090, the "One-to-One Backup" method which 679 uses the "Detour LSP" and the " Facility Backup" which uses a "bypass 680 tunnel". These are commonly referred to as the detour and bypass 681 methods respectively. 683 The detour method makes use of a presignaled LSP. Hardware 684 assistance is needed for detour FRR only if necessary to accomplish 685 local repair of a large number of LSP within the 10s of milliseconds 686 target. For each affected LSP a swap operation must be reprogrammed 687 or otherwise switched over. The use of detour FRR doubles the number 688 of LSP terminating at any given hop and will increase the number of 689 LSP within a network by a factor dependent on the average detour path 690 length. 692 The bypass method makes use of a tunnel that is unused when no fault 693 exists but may carry many LSP when a local repair is required. There 694 is no presignaling indicating which working LSP will be diverted into 695 any specific bypass LSP. The merge LSR (egress LSR of the bypass 696 LSP) MUST use platform label space (as defined in [RFC3031]) so that 697 an LSP working path on any give interface can be backed up using a 698 bypass LSP terminating on any other interface. Hardware assistance 699 is needed if necessary to accomplish local repair of a large number 700 of LSP within the 10s of milliseconds target. For each affected LSP 701 a swap operation must be reprogrammed or otherwise switched over with 702 an additional push of the bypass LSP label. The use of platform 703 label space impacts the size of the LSR ILM for LSR with a very large 704 number of interfaces. 706 2.1.8. Pseudowire Encapsulation 708 The pseudowire (PW) architecture is defined in [RFC3985]. A 709 pseudowire, when carried over MPLS, adds one or more additional label 710 entries to the MPLS label stack. A PW Control Word is defined in 711 [RFC4385] with motivation for defining the control word in [RFC4928]. 712 The PW Associated Channel defined in [RFC4385] is used for OAM in 713 [RFC5085]. The PW Flow Label is defined in [RFC6391] and is 714 discussed further in this document in Section 2.4.3. 716 There are numerous pseudowire encapsulations, supporting emulation of 717 services such as Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over 718 packet switched networks (PSNs) using IP or MPLS. 720 The pseudowire encapsulation is out of scope for this document. 721 Pseudowire impact on MPLS forwarding at midpoint LSR is within scope. 722 The impact on ingress MPLS push and egress MPLS UHP pop are within 723 scope. While pseudowire encapsulation is out of scope, some advice 724 is given on sequence number support. 726 2.1.8.1. Pseudowire Sequence Number 728 Pseudowire (PW) sequence number support is most important for PW 729 payload types with a high expectation of in-order delivery. 730 Resequencing support, rather than dropping at egress on out of order 731 arrival, is most important for PW payload types with a high 732 expectation of lossless delivery. For example, TDM payloads require 733 sequence number support and require resequencing support. The same 734 is true of ATM CBR service. ATM VBR or ABR may have somewhat relaxed 735 requirements, but generally require ATM Early Packet Discard (EPD) or 736 ATM Partial Packet Discard (PPD) [ATM-EPD-and-PPD]. Though sequence 737 number support and resequencing support are beneficial to PW packet 738 oriented payloads such as FR and Ethernet, they are highly desirable 739 but not as strongly required. 741 Packet reorder should be rare except in a small number of 742 circumstances, most of which are due to network design or equipment 743 design errors: 745 1. The most common case is where reordering occurs is rare, 746 occurring only when a network or equipment fault forces traffic 747 on a new path with different delay. The packet loss that 748 accompanies a network or equipment fault is generally more 749 disruptive than any reordering which may occur. 751 2. A path change can be caused by reasons other than a network or 752 equipment fault, such as administrative routing change. This may 753 result in packet reordering but generally without any packet 754 loss. 756 3. If the edge is not using pseudowire control word (CW) and the 757 core is using multipath, reordering will be far more common. If 758 this is occurring, the best solution is to use CW on the edge, 759 rather than try to fix the reordering using resequencing. 761 4. Another avoidable case is where some core equipment has multipath 762 and for some reason insists on periodically installing a new 763 random number as the multipath hash seed. If supporting MPLS-TP, 764 equipment MUST provide a means to disable periodic hash reseeding 765 and deployments MUST disable periodic hash reseeding. Even if 766 not supporting MPLS-TP, equipment should provide a means to 767 disable periodic hash reseeding and deployments should disable 768 periodic hash reseeding. 770 2.1.9. Layer-2 and Layer-3 VPN 772 Layer-2 VPN [RFC4664] and Layer-3 VPN [RFC4110] add one or more label 773 entry to the MPLS label stack. VPN encapsulations are out of scope 774 for this document. Its impact on forwarding at midpoint LSR are 775 within scope. 777 Any of these services may be used on an MPLS entropy label enabled 778 ingress and egress (see Section 2.4.4 for discussion of entropy 779 label) which would add an additional label to the MPLS label stack. 780 The need to provide a useful entropy label value impacts the 781 requirements of the VPN ingress LER but is out of scope for this 782 document. 784 2.2. MPLS Multicast 786 MPLS Multicast encapsulation is clarified in [RFC5332]. MPLS 787 Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388]. 789 [RFC4875] defines a root initiated RSVP-TE LSP setup rather than leaf 790 initiated join used in IP multicast. [RFC6388] defines a leaf 791 initiated LDP setup. Both [RFC4875] and [RFC6388] define point to 792 multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint to 793 multipoint (MP2MP) LSP setup. 795 The P2MP LSP have a single source. An LSR may be a leaf node, an 796 intermediate node, or a "bud" node. A bud serves as both a leaf and 797 intermediate. At a leaf an MPLS pop is performed. The payload may 798 be a IP Multicast packet that requires further replication. At an 799 intermediate node a MPLS swap operation is performed. The bud 800 requires that both a pop operation and a swap operation be performed 801 for the same incoming packet. 803 One strategy to support P2MP functionality is to pop at the LSR 804 interface serving as ingress to the P2MP traffic and then optionally 805 push labels at each LSR interface serving as egress to the P2MP 806 traffic at that same LSR. A given LSR egress chip may support 807 multiple egress interfaces, each of which requires a copy, but each 808 with a different set of added labels and layer-2 encapsulation. Some 809 physical interfaces may have multiple sub-interfaces (such as 810 Ethernet VLAN or channelized interfaces) each requiring a copy. 812 If packet replication is performed at LSR ingress, then the ingress 813 interface performance may suffer. If the packet replication is 814 performed within a LSR switching fabric and at LSR egress, congestion 815 of egress interfaces cannot make use of backpressure to ingress 816 interfaces using techniques such as virtual output queuing (VOQ). If 817 buffering is primarily supported at egress, then the need for 818 backpressure is minimized. There may be no good solution for high 819 volumes of multicast traffic if VOQ is used. 821 MP2MP LSP differ in that any branch may provide an input, including a 822 leaf. Packets must be replicated onto all other branches. This 823 forwarding is often implemented as multiple P2MP forwarding trees, 824 one for each potential input interface at a given LSR. 826 2.3. Packet Rates 828 While average packet size of Internet traffic may be large, long 829 sequences of small packets have both been predicted in theory and 830 observed in practice. Traffic compression and TCP ACK compression 831 can conspire to create long sequences of packets of 40-44 bytes in 832 payload length. If carried over Ethernet, the 64 byte minimum 833 payload applies, yielding a packet rate of approximately 150 Mpps 834 (million packets per second) for the duration of the burst on a 835 nominal 100 Gb/s link. The peak rate is higher for other 836 encapsulations and can be as high as 250 Mpps (for example IP or MPLS 837 encapsulated using GFP over OTN ODU4). 839 It is also possible that the packet rates for a minimum payload size, 840 such as 64 byte (64B) payload for Ethernet, is acceptable, but the 841 rate declines for other packet sizes, such as 65B payload. There are 842 other packet rates of interest besides TCP ACK. For example, a TCP 843 ACK carried over an Ethernet PW over MPLS over Ethernet may occupy 844 82B or 82B plus an increment of 4B if additional MPLS labels are 845 present. 847 A graph of packet rate vs. packet size often displays a sawtooth. 848 The sawtooth is commonly due to a memory bottleneck and memory 849 widths, sometimes internal cache, but often a very wide external 850 buffer memory interface. In some cases it may be due to a fabric 851 transfer width. A fine packing, rounding up to the nearest 8B or 16B 852 will result in a fine sawtooth with small degradation for 65B, and 853 even less for 82B packets. A course packing, rounding up to 64B can 854 yield a sharper drop in performance for 65B packets, or perhaps more 855 important, a larger drop for 82B packets. 857 The loss of some TCP ACK packets are not the primary concern when 858 such a burst occurs. When a burst occurs, any other packets, 859 regardless of packet length and packet QoS are dropped once on-chip 860 input buffers prior to the decision engine are exceeded. Buffers in 861 front of the packet decision engine are often very small or non- 862 existent (less than one packet of buffer) causing significant QoS 863 agnostic packet drop. 865 Internet service providers and content providers at one time 866 specified full rate forwarding with 40 byte payload packets as a 867 requirement. Today, this requirement often can be waived if the 868 provider can be convinced that when long sequence of short packets 869 occur no packets will be dropped. 871 Many equipment suppliers have pointed out that the extra cost in 872 designing hardware capable of processing the minimum size packets at 873 full line rate is significant for very high speed interfaces. If 874 hardware is not capable of processing the minimum size packets at 875 full line rate, then that hardware MUST be capable of handling large 876 burst of small packets, a condition which is often observed. This 877 level of performance is necessary to meet Differentiated Services 879 [RFC2475] requirements for without it, packets are lost prior to 880 inspection of the IP DSCP field [RFC2474] or MPLS TC field [RFC5462]. 882 With adequate on-chip buffers before the packet decision engine, an 883 LSR can absorb a long sequence of short packets. Even if the output 884 is slowed to the point where light congestion occurs, the packets, 885 having cleared the decision process, can make use of larger VOQ or 886 output side buffers and be dealt with according to configured QoS 887 treatment, rather than dropped completely at random. 889 These on-chip buffers need not contribute significant delay since 890 they are only used when the packet decision engine is unable to keep 891 up, not in response to congestion, plus these buffers are quite 892 small. For example, an on-chip buffer capable of handling 4K packets 893 of 64 bytes in length, or 256KB, corresponds to 2 msec on a 10 Mb/s 894 link and 0.2 usec on a 100 Gb/s link. If the packet decision engine 895 is capable of handling packets at 90% of the full rate for small 896 packets, then the maximum added delay is 0.2 msec and 20 nsec 897 respectively, and this delay only applies if a 4K burst of short 898 packets occurs. When no burst of short packets was being processed, 899 no delay is added. 901 Packet rate requirements apply regardless of which network tier 902 equipment is deployed in. Whether deployed in the network core or 903 near the network edges, one of the two conditions MUST be met if 904 Differentiated Services requirements are to be met: 906 1. Packets must be processed at full line rate with minimum sized 907 packets. -OR- 909 2. Packets must be processed at a rate well under generally accepted 910 average packet sizes, with sufficient buffering prior to the 911 packet decision engine to accommodate long bursts of small 912 packets. 914 2.4. MPLS Multipath Techniques 916 In any large provider, service providers and content providers, hash 917 based multipath techniques are used in the core and in the edge. In 918 many of these providers hash based multipath is also used in the 919 larger metro networks. 921 The most common multipath techniques are ECMP applied at the IP 922 forwarding level, Ethernet LAG with inspection of the IP payload, and 923 multipath on links carrying both IP and MPLS, where the IP header is 924 inspected below the MPLS label stack. In most core networks, the 925 vast majority of traffic is MPLS encapsulated. 927 In order to support an adequately balanced load distribution across 928 multiple links, IP header information must be used. Common practice 929 today is to reinspect the IP headers at each LSR and use the label 930 stack and IP header information in a hash performed at each LSR. 931 Further details are provided in Section 2.4.5. 933 The use of this technique is so ubiquitous in provider networks that 934 lack of support for multipath makes any product unsuitable for use in 935 large core networks. This will continue to be the case in the near 936 future, even as deployment of MPLS entropy label begins to relax the 937 core LSR multipath performance requirements given the existing 938 deployed base of edge equipment without the ability to add an entropy 939 label. 941 A generation of edge equipment supporting the ability to add an MPLS 942 entropy label is needed before the performance requirements for core 943 LSR can be relaxed. However, it is likely that two generations of 944 deployment in the future will allow core LSR to support full packet 945 rate only when a relatively small number of MPLS labels need to be 946 inspected before hashing. For now, don't count on it. 948 Common practice today is to reinspect the packet at each LSR and use 949 information from the packet combined plus a hash seed that is 950 selected by each LSR. Where flow labels or entropy labels are used, 951 a hash seed must be used when creating these labels. 953 2.4.1. Pseudowire Control Word 955 Within the core of a network some form of multipath is almost certain 956 to be used. Multipath techniques deployed today are likely to be 957 looking beneath the label stack for an opportunity to hash on IP 958 addresses. 960 A pseudowire encapsulated at a network edge must have a means to 961 prevent reordering within the core if the pseudowire will be crossing 962 a network core, or any part of a network topology where multipath is 963 used (see [RFC4385] and [RFC4928]). 965 Not supporting the ability to encapsulate a pseudowire with a control 966 word may lock a product out from consideration. A pseudowire 967 capability without control word support might be sufficient for 968 applications that are strictly both intra-metro and low bandwidth. 969 However a provider with other applications will very likely not 970 tolerate having equipment which can only support a subset of their 971 pseudowire needs. 973 2.4.2. Large Microflows 975 Where multipath makes use of a simple hash and simple load balance 976 such as modulo or other fixed allocation (see Section 2.4) the 977 presence of large microflows that each consumes 10% of the capacity 978 of a component link of a potentially congested composite link, one 979 such microflow can upset the traffic balance and more than one can in 980 effect reduce the effective capacity of the entire composite link by 981 more than 10%. 983 When even a very small number of large microflows are present, there 984 is a significant probability that more than one of these large 985 microflows could fall on the same component link. If the traffic 986 contribution from large microflows is small, the probability for 987 three or more large microflows on the same component link drops 988 significantly. Therefore in a network where a significant number of 989 parallel 10 Gb/s links exists, even a 1 Gb/s pseudowire or other 990 large microflow that could not otherwise be subdivided into smaller 991 flows should carry a flow label or entropy label if possible. 993 Active management of the hash space to better accommodate large 994 microflows has been implemented and deployed in the past, however 995 such techniques are out of scope for this document. 997 2.4.3. Pseudowire Flow Label 999 Unlike a pseudowire control word, a pseudowire flow label [RFC6391], 1000 is required only for relatively large capacity pseudowires. There 1001 are many cases where a pseudowire flow label makes sense. Any 1002 service such as a VPN which carries IP traffic within a pseudowire 1003 can make use of a pseudowire flow label. 1005 Any pseudowire carried over MPLS which makes use of the pseudowire 1006 control word and does not carry a flow label is in effect a single 1007 microflow (in [RFC2475] terms) and may result in the types of 1008 problems described in Section 2.4.2. 1010 2.4.4. MPLS Entropy Label 1012 The MPLS entropy label simplifies flow group identification [RFC6790] 1013 at midpoint LSR. Prior to the MPLS entropy label midpoint LSR needed 1014 to inspect the entire label stack and often the IP headers to provide 1015 an adequate distribution of traffic when using multipath techniques 1016 (see Section 2.4.5). With the use of MPLS entropy label, a hash can 1017 be performed closer to network edges, placed in the label stack, and 1018 used by midpoint LSR without fully reinspecting the label stack and 1019 inspecting the payload. 1021 The MPLS entropy label is capable of avoiding full label stack and 1022 payload inspection within the core where performance levels are most 1023 difficult to achieve (see Section 2.3). The label stack inspection 1024 can be terminated as soon as the first entropy label is encountered, 1025 which is generally after a small number of labels are inspected. 1027 In order to provide these benefits in the core, LSR closer to the 1028 edge must be capable of adding an entropy label. This support may 1029 not be required in the access tier, the tier closest to the customer, 1030 but is likely to be required in the edge or the border to the network 1031 core. LSR peering with external networks will also need to be able 1032 to add an entropy label on incoming traffic. 1034 2.4.5. Fields Used for Multipath Load Balance 1036 The most common multipath techniques are based on a hash over a set 1037 of fields. Regardless of whether a hash is used or some other method 1038 is used, the there is a limited set of fields which can safely be 1039 used for multipath. 1041 2.4.5.1. MPLS Fields in Multipath 1043 If the "outer" or "first" layer of encapsulation is MPLS, then label 1044 stack entries are used in the hash. Within a finite amount of time 1045 (and for small packets arriving at high speed that time can be quite 1046 limited) only a finite number of label entries can be inspected. 1047 Pipelined or parallel architectures improve this, but the limit is 1048 still finite. 1050 The following guidelines are provided for use of MPLS fields in 1051 multipath load balancing. 1053 1. Only the 20 bit label field SHOULD be used. The TTL field SHOULD 1054 NOT be used. The S bit MUST NOT be used. The TC field (formerly 1055 EXP) MUST NOT be used. See text following this list for reasons. 1057 2. If an ELI label is found, then if the LSR supports entropy label, 1058 the EL label field in the next label entry (the EL) SHOULD be 1059 used and label entries below that label SHOULD NOT be used and 1060 the MPLS payload SHOULD NOT be used. See below this list for 1061 reasons. 1063 3. Special purpose labels (label values 0-15) MUST NOT be used. 1064 Extended special purpose labels (any label following label 15) 1065 MUST NOT be used. In particular, GAL and RA MUST NOT be used so 1066 that OAM traffic follows the same path as payload packets with 1067 the same label stack. 1069 4. The most entropy is generally found in the label stack entries 1070 near the bottom of the label stack (innermost label, closest to 1071 S=1 bit). If the entire label stack cannot be used (or entire 1072 stack up to an EL), then it is better to use as many labels as 1073 possible closest to the bottom of stack. 1075 5. If no ELI is encountered, and the first nibble of payload 1076 contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support 1077 the ability to interpret the payload as IPv4 or IPv6 and extract 1078 and use appropriate fields from the IP headers. This feature is 1079 considered a hard requirement by many service providers. If 1080 supported, there MUST be a way to disable it (if, for example, PW 1081 without CW are used). This ability to disable this feature is 1082 considered a hard requirement by many service providers. 1083 Therefore an implementation has a very strong incentive to 1084 support both options. 1086 6. A label which is popped at egress (UHP pop) SHOULD NOT be used. 1087 A label which is popped at the penultimate hop (PHP pop) SHOULD 1088 be used. 1090 Apparently some chips have made use of the TC (formerly EXP) bits as 1091 a source of entropy. This is very harmful since it will reorder 1092 Assured Forwarding (AF) traffic [RFC2597] when a subset does not 1093 conform to the configured rates and is remarked but not dropped at a 1094 prior LSR. Traffic which uses MPLS ECN [RFC5129] can also be 1095 reordered if TC is used for entropy. Therefore, as stated in the 1096 guidelines above, the TC field (formerly EXP) MUST NOT be used in 1097 multipath load balancing as it violates Differentiated Services 1098 Ordered Aggregate (OA) requirements in these two instances. 1100 Use of the MPLS label entry S bit would result in putting OAM traffic 1101 on a different path if the addition of a GAL at the bottom of stack 1102 removed the S bit from the prior label. 1104 If an ELI label is found, then if the LSR supports entropy label, the 1105 EL label field in the next label entry (the EL) SHOULD be used and 1106 the search for additional entropy within the packet SHOULD be 1107 terminated. Failure to terminate the search will impact client 1108 MPLS-TP LSP carried within server MPLS LSP. A network operator has 1109 the option to use administrative attributes as a means to identify 1110 LSR which do not terminate the entropy search at the first EL. 1111 Administrative attributes are defined in [RFC3209]. Some 1112 configuration is required to support this. 1114 If the label removed by a PHP pop is not used, then for any PW for 1115 which CW is used, there is no basis for multipath load split. In 1116 some networks it is infeasible to put all PW traffic on one component 1117 link. Any PW which does not use CW will be improperly split 1118 regardless of whether the label removed by a PHP pop is used. 1119 Therefore the PHP pop label SHOULD be used as recommended above. 1121 2.4.5.2. IP Fields in Multipath 1123 Inspecting the IP payload provides the most entropy in provider 1124 networks. The practice of looking past the bottom of stack label for 1125 an IP payload is well accepted and documented in [RFC4928] and in 1126 other RFCs. 1128 Where IP is mentioned in the document, both IPv4 and IPv6 apply. All 1129 LSRs MUST fully support IPv6. 1131 When information in the IP header is used, the following guidelines 1132 apply: 1134 1. Both the IP source address and IP destination address SHOULD be 1135 used. There MAY be an option to reverse the order of these 1136 addresses, improving the ability to provide symmetric paths in 1137 some cases. Many service providers require that both addresses 1138 be used. 1140 2. Implementations SHOULD allow inspection of the IP protocol field 1141 and use of the UDP or TCP port numbers. For many service 1142 providers this feature is considered mandatory, particularly for 1143 enterprise, data center, or edge equipment. If this feature is 1144 provided, it SHOULD be possible to disable use of TCP and UDP 1145 ports. Many service providers consider it a hard requirement 1146 that use of UDP and TCP ports can be disabled. Therefore there 1147 is a stong incentive for implementations to provide both options. 1149 3. Equipment suppliers MUST NOT make assumptions that because the IP 1150 version field is equal to 4 (an IPv4 packet) that the IP protocol 1151 will either be TCP (IP protocol 6) or UDP (IP protocol 17) and 1152 blindly fetch the data at the offset where the TCP or UDP ports 1153 would be found. With IPv6, TCP and UDP port numbers are not at 1154 fixed offsets. With IPv4 packets carrying IP options, TCP and 1155 UDP port numbers are not at fixed offsets. 1157 4. The IPv6 header flow field SHOULD be used. This is the explicit 1158 purpose of the IPv6 flow field, however observed flow fields 1159 rarely contains a non-zero value. Some uses of the flow field 1160 have been defined such as [RFC6438]. In the absence of MPLS 1161 encapsulation, the IPv6 flow field can serve a role equivalent to 1162 entropy label. 1164 5. Support for other protocols that share a common Layer-4 header 1165 such as RTP, UDP-lite, SCTP and DCCP SHOULD be provided, 1166 particularly for edge or access equipment where additional 1167 entropy may be needed. Equipment SHOULD also use RTP, UDP-lite, 1168 SCTP and DCCP headers when creating an entropy label. 1170 6. The following IP header fields should not or must not be used: 1172 A. Similar to avoiding TC in MPLS, the IP DSCP, and ECN bits 1173 MUST NOT be used. 1175 B. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used. 1177 C. Note that the IP TOS field was deprecated ([RFC0791] was 1178 updated by [RFC2474]). No part of the IP DSCP field can be 1179 used (formerly IP PREC and IP TOS bits). 1181 7. Some IP encapsulations support tunneling, such as IP-in-IP, GRE, 1182 L2TPv3, and IPSEC. These provide a greater source of entropy 1183 which some provider networks carrying large amounts of tunneled 1184 traffic may need. The use of tunneling header information is out 1185 of scope for this document. 1187 This document makes the following recommendations. These 1188 recommendations are not required to claim compliance to any existing 1189 RFC therefore implementers are free to ignore them, but due to 1190 service provider requirements should consider the risk of doing so. 1191 The use of IP addresses MUST be supported and TCP and UDP ports 1192 (conditional on the protocol field and properly located) MUST be 1193 supported. The ability to disable use of UDP and TCP ports MUST be 1194 available. Though potentially very useful in some networks, it is 1195 uncommon to support using payloads of tunneling protocols carried 1196 over IP. Though the use of tunneling protocol header information is 1197 out of scope for this document, it is not discouraged. 1199 2.4.5.3. Fields Used in Flow Label 1201 The ingress to a pseudowire (PW) can extract information from the 1202 payload being encapsulated to create a flow label. [RFC6391] 1203 references IP carried in Ethernet as an example. The Native Service 1204 Processing (NSP) function defined in [RFC3985] differs with 1205 pseudowire type. It is in the NSP function where information for a 1206 specific type of PW can be extracted for use in a flow label. Which 1207 fields to use for any given PW NSP is out of scope for this document. 1209 2.4.5.4. Fields Used in Entropy Label 1211 An entropy label is added at the ingress to an LSP. The payload 1212 being encapsulated is most often MPLS, a PW, or IP. The payload type 1213 is identified by the layer-2 encapsulation (Ethernet, GFP, POS, etc). 1215 If the payload is MPLS, then the information used to create an 1216 entropy label is the same information used for local load balancing 1217 (see Section 2.4.5.1). This information MUST be extracted for use in 1218 generating an entropy label even if the LSR local egress interface is 1219 not a multipath. 1221 Of the non-MPLS payload types, only payloads that are forwarded are 1222 of interest. For example, ARP is not forwarded and CNLP (used only 1223 for ISIS) is not forwarded. 1225 The non-MPLS payload type of greatest interest are IPv4 and IPv6. 1226 The guidelines in Section 2.4.5.2 apply to fields used to create and 1227 entropy label. 1229 The IP tunneling protocols mentioned in Section 2.4.5.2 may be more 1230 applicable to generation of an entropy label at edge or access where 1231 deep packet inspection is practical due to lower interface speeds 1232 than in the core where deep packet inspection may be impractical. 1234 2.5. MPLS-TP and UHP 1236 MPLS-TP introduces forwarding demands that will be extremely 1237 difficult to meet in a core network. Most troublesome is the 1238 requirement for Ultimate Hop Popping (UHP, the opposite of 1239 Penultimate Hop Popping or PHP). Using UHP opens the possibility of 1240 one or more MPLS pop operation plus an MPLS swap operation for each 1241 packet. The potential for multiple lookups and multiple counter 1242 instances per packet exists. 1244 As networks grow and tunneling of LDP LSPs into RSVP-TE LSPs is used, 1245 and/or RSVP-TE hierarchy is used, the requirement to perform one or 1246 two or more MPLS pop operations plus a MPLS swap operation (and 1247 possibly a push or two) increases. If MPLS-TP LM (link monitoring) 1248 OAM is enabled at each layer, then a packet and byte count MUST be 1249 maintained for each pop and swap operation so as to offer OAM for 1250 each layer. 1252 2.6. Local Delivery of Packets 1254 There are a number of situations in which packets are destined to a 1255 local address or where a return packet must be generated. There is a 1256 need to mitigate the potential for outage as a result of either 1257 attacks on network infrastructure, or in some cases unintentional 1258 misconfiguration resulting in processor overload. Some hardware 1259 assistance is needed for all traffic destined to the general purpose 1260 CPU that is used in MPLS control protocol processing or network 1261 management protocol processing and in most cases to other general 1262 purpose CPUs residing on an LSR. This is due to the ease of 1263 overwhelming such a processor with traffic arriving on LSR high speed 1264 interfaces, whether the traffic is malicious or not. 1266 Denial of service (DoS) protection is an area requiring hardware 1267 support that is often overlooked or inadequately considered. 1268 Hardware assist is also needed for OAM, particularly the more 1269 demanding MPLS-TP OAM. 1271 2.6.1. DoS Protection 1273 Modern equipment supports a number of control plane and management 1274 plane protocols. Generally no single means of protecting network 1275 equipment from denial of service (DoS) attacks is sufficient, 1276 particularly for high speed interfaces. This problem is not specific 1277 to MPLS, but is a topic that cannot be ignored when implementing or 1278 evaluating MPLS implementations. 1280 Two types of protections are often cited as primary means of 1281 protecting against attacks of all kinds. 1283 Isolated Control/Management Traffic 1284 Control and Management traffic can be carried out-of-band (OOB), 1285 meaning not intermixed with payload. For MPLS, use of G-ACh and 1286 GAL to carry control and management traffic provides a means of 1287 isolation from potentially malicious payload. Used alone, the 1288 compromise of a single node, including a small computer at a 1289 network operations center, could compromise an entire network. 1290 Implementations which send all G-ACh/GAL traffic directly to a 1291 routing engine CPU are subject to DoS attack as a result of such 1292 a compromise. 1294 Cryptographic Authentication 1295 Cryptographic authentication can very effectively prevent 1296 malicious injection of control or management traffic. 1297 Cryptographic authentication can is some circumstances be subject 1298 to DoS attack by overwhelming the capacity of the decryption with 1299 a high volume of malicious traffic. For very low speed 1300 interfaces, cryptographic authentication can be performed by the 1301 general purpose CPU used as a routing engine. For all other 1302 cases, cryptographic hardware may be needed. For very high speed 1303 interfaces, even cryptographic hardware can be overwhelmed. 1305 Some control and management protocols are often carried with payload 1306 traffic. This is commonly the case with BGP, T-LDP, and SNMP. It is 1307 often the case with RSVP-TE. Even when carried over G-ACh/GAL 1308 additional measures can reduce the potential for a minor breach to be 1309 leveraged to a full network attack. 1311 Some of the additional protections are supported by hardware packet 1312 filtering. 1314 GTSM 1315 [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop 1316 Limit fields to insure control traffic that can only originate 1317 from an immediate neighbor is not forged and originating from a 1318 distant source. GTSM can be applied to many control protocols 1319 which are routable, for example LDP [RFC6720]. 1321 IP Filtering 1322 At the very minimum, packet filtering plus classification and use 1323 of multiple queues supporting rate limiting is needed for traffic 1324 that could potentially be sent to a general purpose CPU used as a 1325 routing engine. The first level of filtering only allows 1326 connections to be initiated from specific IP prefixes to specific 1327 destination ports and then preferably passes traffic directly to 1328 a cryptographic engine and/or rate limits. The second level of 1329 filtering passes connected traffic, such as TCP connections 1330 having received at least one authenticated SYN or having been 1331 locally initiated. The second level of filtering only passes 1332 traffic to specific address and port pairs to be checked for 1333 cryptographic authentication. 1335 The cryptographic authentication is generally the last resort in DoS 1336 attack mitigation. If a packet must be first sent to a general 1337 purpose CPU, then sent to a cryptographic engine, a DoS attack is 1338 possible on high speed interfaces. Only where hardware can identify 1339 a signature and the portion of packet covered by the signature is 1340 cryptographic authentication highly beneficial in protecting against 1341 DoS attacks. 1343 For chips supporting multiple 100 Gb/s interfaces, only a very large 1344 number of parallel cryptographic engines can provide the processing 1345 capacity to handle a large scale DoS or distributed DoS (DDoS) 1346 attack. For many forwarding chips this much processing power 1347 requires significant chip real estate and power, and therefore 1348 reduces system space and power density. For this reason, 1349 cryptographic authentication is not considered a viable first line of 1350 defense. 1352 For some networks the first line of defense is some means of 1353 supporting OOB control and management traffic. In the past this OOB 1354 channel might make use of overhead bits in SONET or OTN or a 1355 dedicated DWDM wavelength. G-ACh and GAL provide an alternative OOB 1356 mechanism which is independent of underlying layers. In other 1357 networks, including most IP/MPLS networks, perimeter filtering serves 1358 a similar purpose, though less effective without extreme vigilance. 1360 A second line of defense is filtering, including GTSM. For protocols 1361 such as EBGP, GTSM and other filtering is often the first line of 1362 defense. Cryptographic authentication is usually the last line of 1363 defense and insufficient by itself to mitigate DoS or DDoS attacks. 1365 2.6.2. MPLS OAM 1367 [RFC4377] defines requirements for MPLS OAM that predate MPLS-TP. 1368 [RFC4379] defines what is commonly referred to as LSP Ping and LSP 1369 Traceroute. [RFC4379] is updated by [RFC6424] supporting MPLS 1370 tunnels and stitched LSP and P2MP LSP. [RFC4379] is updated by 1371 [RFC6425] supporting P2MP LSP. [RFC4379] is updated by [RFC6426] to 1372 support MPLS-TP connectivity verification (CV) and route tracing. 1374 [RFC4950] extends the ICMP format to support TTL expiration that may 1375 occur when using IP traceroute within an MPLS tunnel. The ICMP 1376 message generation can be implemented in forwarding hardware, but if 1377 sent to a general purpose CPU must be rate limited to avoid a 1378 potential denial or service (DoS) attack. 1380 [RFC5880] defines Bidirectional Forwarding Detection (BFD), a 1381 protocol intended to detect faults in the bidirectional path between 1382 two forwarding engines. [RFC5884] and [RFC5885] define BFD for MPLS. 1383 BFD can provide failure detection on any kind of path between 1384 systems, including direct physical links, virtual circuits, tunnels, 1385 MPLS Label Switched Paths (LSPs), multihop routed paths, and 1386 unidirectional links as long as there is some return path. 1388 The processing requirements for BFD are less than for LSP Ping, 1389 making BFD somewhat better suited for relatively high rate proactive 1390 monitoring. BFD does not verify that the data plane matches the 1391 control plane, where LSP Ping does. LSP Ping is somewhat better 1392 suited for on-demand monitoring including relatively low rate 1393 periodic verification of data plane and as a diagnostic tool. 1395 Hardware assistance is often provided for BFD response where BFD 1396 setup or parameter change is not involved and may be necessary for 1397 relatively high rate proactive monitoring. If both BFD and LSP Ping 1398 are recognized in filtering prior to passing traffic to a general 1399 purpose CPU, appropriate DoS protection can be applied (see 1400 Section 2.6.1). Failure to recognize BFD and LSP Ping and at least 1401 rate limit creates the potential for misconfiguration to cause 1402 outages rather than cause errors in the misconfigured OAM. 1404 2.6.3. Pseudowire OAM 1406 Pseudowire OAM makes use of the control channel provided by Virtual 1407 Circuit Connectivity Verification (VCCV) [RFC5085]. VCCV makes use 1408 of the Pseudowire Control Word. BFD support over VCCV is defined by 1409 [RFC5885]. [RFC5885] is updated by [RFC6478] in support of static 1410 pseudowires. [RFC4379] is updated by [RFC6829] supporting LSP Ping 1411 for Pseudowire FEC advertised over IPv6. 1413 G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control 1414 channel and applies to any MPLS-TP end points, including Pseudowire. 1415 See Section 2.6.4 for an overview of MPLS-TP OAM. 1417 2.6.4. MPLS-TP OAM 1419 [RFC6669] summarizes the MPLS-TP OAM toolset, the set of protocols 1420 supporting the MPLS-TP OAM requirements specified in [RFC5860] and 1421 supported by the MPLS-TP OAM framework defined in [RFC6371]. 1423 The MPLS-TP OAM toolset includes: 1425 CC-CV 1426 [RFC6428] defines BFD extensions to support proactive 1427 Connectivity Check and Connectivity Verification (CC-CV) 1428 applications. [RFC6426] provides LSP ping extensions that are 1429 used to implement on-demand connectivity verification. 1431 RDI 1432 Remote Defect Indication (RDI) is triggered by failure of 1433 proactive CC-CV, which is BFD based. For fast RDI initiation, 1434 RDI SHOULD be initiated and handled by hardware if BFD is handled 1435 in forwarding hardware. [RFC6428] provides an extension for BFD 1436 that includes the RDI indication in the BFD format and a 1437 specification of how this indication is to be used. 1439 Route Tracing 1440 [RFC6426] specifies that the LSP ping enhancements for MPLS-TP 1441 on-demand connectivity verification include information on the 1442 use of LSP ping for route tracing of an MPLS-TP path. 1444 Alarm Reporting 1445 [RFC6427] describes the details of a new protocol supporting 1446 Alarm Indication Signal (AIS), Link Down Indication, and fault 1447 management. Failure to support this functionality in forwarding 1448 hardware can potentially result in failure to meet protection 1449 recovery time requirements and is therefore strongly recommended. 1451 Lock Instruct 1452 Lock instruct is initiated on-demand and therefore need not be 1453 implemented in forwarding hardware. [RFC6435] defines a lock 1454 instruct protocol. 1456 Lock Reporting 1457 [RFC6427] covers lock reporting. Lock reporting need not be 1458 implemented in forwarding hardware. 1460 Diagnostic 1461 [RFC6435] defines protocol support for loopback. Loopback 1462 initiation is on-demand and therefore need not be implemented in 1463 forwarding hardware. Loopback of packet traffic SHOULD be 1464 implemented in forwarding hardware on high speed interfaces. 1466 Packet Loss and Delay Measurement 1467 [RFC6374] and [RFC6375] define a protocol and profile for packet 1468 loss measurement (LM) and delay measurement (DM). LM requires a 1469 very accurate capture and insertion of packet and byte counters 1470 when a packet is transmitted and capture of packet and byte 1471 counters when a packet is received. This capture and insertion 1472 MUST be implemented in forwarding hardware for LM OAM if high 1473 accuracy is needed. DM requires very accurate capture and 1474 insertion of a timestamp on transmission and capture of timestamp 1475 when a packet is received. This timestamp capture and insertion 1476 MUST be implemented in forwarding hardware for DM OAM if high 1477 accuracy is needed. 1479 See Section 2.6.2 for discussion of hardware support necessary for 1480 BFD and LSP Ping. 1482 CC-CV and alarm reporting is tied to protection and therefore SHOULD 1483 be supported in forwarding hardware in order to provide protection 1484 for a large number of affected LSP within target response intervals. 1485 Since CC-CV is supported by BFD, for MPLS-TP providing hardware 1486 assistance for BFD processing helps insure that protection recovery 1487 time requirements can be met even for faults affecting a large number 1488 of LSP. 1490 2.6.5. MPLS OAM and Layer-2 OAM Interworking 1492 [RFC6670] provides the reasons for selecting a single MPLS-TP OAM 1493 solution and examines the consequences were ITU-T to develop a second 1494 OAM solution that is based on Ethernet encodings and mechanisms. 1496 [RFC6310] and [I-D.ietf-pwe3-mpls-eth-oam-iwk] specifies the mapping 1497 of defect states between many types of hardware Attachment Circuits 1498 (ACs) and associated Pseudowires (PWs). This functionality SHOULD be 1499 supported in forwarding hardware. 1501 It is beneficial if an MPLS OAM implementation can interwork with the 1502 underlying server layer and provide a means to interwork with a 1503 client layer. For example, [RFC6427] specifies an inter-layer 1504 propagation of AIS and LDI from MPLS server layer to client MPLS 1505 layers. Where the server layer is a Layer-2, such as Ethernet, PPP 1506 over SONET/SDH, or GFP over OTN, interwork among layers is also 1507 beneficial. For high speed interfaces, supporting this interworking 1508 in forwarding hardware helps insure that protection based on this 1509 interworking can meet recovery time requirements even for faults 1510 affecting a large number of LSP. 1512 2.6.6. Extent of OAM Support by Hardware 1514 Where certain requirements must be met, such as relatively high CC-CV 1515 rates and a large number of interfaces, or strict protection recovery 1516 time requirements and a moderate number of affected LSP, some OAM 1517 functionality must be supported by forwarding hardware. In other 1518 cases, such as highly accurate LM and DM OAM or strict protection 1519 recovery time requirements with a large number of affected LSP, OAM 1520 functionality must be entirely implemented in forwarding hardware. 1522 Where possible, implementation in forwarding hardware should be in 1523 programmable hardware such that if standards are later changed or 1524 extended these changes are likely to be accommodated with hardware 1525 reprogramming rather than replacement. 1527 For some functionality there is a strong case for an implementation 1528 in dedicated forwarding hardware. Examples include packet and byte 1529 counters needed for LM OAM as well as needed for management 1530 protocols. Similarly the capture and insertion of packet and byte 1531 counts or timestamps needed for transmitted LM or DM or time 1532 synchronization packets MUST be implemented in forwarding hardware if 1533 high accuracy is required. 1535 For some functions there is a strong case to provide limited support 1536 in forwarding hardware but may make use of an external general 1537 purpose processor if performance criteria can be met. For example 1538 origination of RDI triggered by CC-CV, response to RDI, and PSC 1539 functionality may be supported by hardware, but expansion to a large 1540 number of client LSP and transmission of AIS or RDI to the client LSP 1541 may occur in a general purpose processor. Some forwarding hardware 1542 supports one or more on-chip general purpose processors which may be 1543 well suited for such a role. 1545 The customer (system supplier or provider) should not dictate design, 1546 but should independently validate target functionality and 1547 performance. However, it is not uncommon for service providers and 1548 system implementers to insist on reviewing design details (under NDA) 1549 due to past experiences with suppliers and to reject suppliers who 1550 are unwilling to provide details. 1552 2.7. Number and Size of Flows 1554 Service provider networks may carry up to hundreds of millions of 1555 flows on 10 Gb/s links. Most flows are very short lived, many under 1556 a second. A subset of the flows are low capacity and somewhat long 1557 lived. When Internet traffic dominates capacity a very small subset 1558 of flows are high capacity and/or very long lived. 1560 Two types of limitations with regard to number and size of flows have 1561 been observed. 1563 1. Some hardware cannot handle some very large flows because of 1564 internal paths which are limited, such as per packet backplane 1565 paths or paths internal or external to chips such as buffer 1566 memory paths. Such designs can handle aggregates of smaller 1567 flows. Some hardware with acknowledged limitations has been 1568 successfully deployed but may be increasingly problematic if the 1569 capacity of large microflows in deployed networks continues to 1570 grow. 1572 2. Some hardware approaches cannot handle a large number of flows, 1573 or a large number of large flows due to attempting to count per 1574 flow, rather than deal with aggregates of flows. Hash techniques 1575 scale with regard to number of flows due to a fixed hash size 1576 with many flows falling into the same hash bucket. Techniques 1577 that identify individual flows have been implemented but have 1578 never successfully deployed for Internet traffic. 1580 3. Questions for Suppliers 1582 The following questions should be asked of a supplier. These 1583 questions are grouped into broad categories. The questions 1584 themselves are intended to be an open ended question to the supplier. 1585 The tests in Section 4 are intended to verify whether the supplier 1586 disclosed any compliance or performance limitations completely and 1587 accurately. 1589 3.1. Basic Compliance 1591 Q#1 Can the implementation forward packets with an arbitrarily 1592 large stack depth? What limitations exist, and under what 1593 circumstances do further limitations come into play (such as 1594 high packet rate or specific features enabled or specific types 1595 of packet processing)? See Section 2.1. 1597 Q#2 Is the entire set of basic MPLS functionality described in 1598 Section 2.1 supported? 1600 Q#3 Are the set of MPLS special purpose labels handled correctly 1601 and with adequate performance? Are extended special purpose 1602 labels handled correctly and with adequate performance? See 1603 Section 2.1.1. 1605 Q#4 Are mappings of label value and TC to PHB handled correctly, 1606 including RFC3270 L-LSP mappings and RFC4124 CT mappings to 1607 PHB? See Section 2.1.2. 1609 Q#5 Is time synchronization adequately supported in forwarding 1610 hardware? 1612 A. Are both PTP and NTP formats supported? 1614 B. Is the accuracy of timestamp insertion and incoming 1615 stamping sufficient? 1617 See Section 2.1.3. 1619 Q#6 Is link bundling supported? 1621 A. Can LSP be pinned to specific components? 1623 B. Is the "all-ones" component link supported? 1625 See Section 2.1.5. 1627 Q#7 Is MPLS hierarchy supported? 1629 A. Are both PHP and UHP supported? What limitations exist on 1630 the number of pop operations with UHP? 1632 B. Are the pipe, short-pipe, and uniform models supported? 1633 Are TTL and TC values updated correctly at egress where 1634 applicable? 1636 See Section 2.1.6 1638 Q#8 Are pseudowire sequence numbers handled correctly? See 1639 Section 2.1.8.1. 1641 Q#9 Is VPN LER functionality handled correctly and without 1642 performance issues? See Section 2.1.9. 1644 Q#10 Is MPLS multicast (P2MP and MP2MP) handled correctly? 1646 A. Are packets dropped on uncongested outputs if some outputs 1647 are congested? 1649 B. Is performance limited in high fanout situations? 1651 See Section 2.2. 1653 3.2. Basic Performance 1655 Q#11 Can very small packets be forwarded at full line rate on all 1656 interfaces indefinitely? What limitations exist, and under 1657 what circumstances do further limitations come into play (such 1658 as specific features enabled or specific types of packet 1659 processing)? 1661 Q#12 Customers must decide whether to relax the prior requirement 1662 and to what extent. If the answer to the prior question 1663 indicates that limitations exist, then: 1665 A. What is the smallest packet size where full line rate 1666 forwarding can be supported? 1668 B. What is the longest burst of full rate small packets that 1669 can be supported? 1671 Specify circumstances (such as specific features enabled or 1672 specific types of packet processing) often impact these rates 1673 and burst sizes. 1675 Q#13 How many pop operations can be supported along with a swap 1676 operation at full line rate while maintaining per LSP packet 1677 and byte counts for each pop and swap? This requirement is 1678 particularly relevant for MPLS-TP. 1680 Q#14 How many label push operations can be supported. While this 1681 limitation is rarely an issue, it applies to both PHP and UHP, 1682 unlike the pop limit which applies to UHP. 1684 Q#15 For a worst case where all packets arrive on one LSP, what is 1685 the counter overflow time? Are any means provided to avoid 1686 polling all counters at short intervals? This applies to both 1687 MPLS and MPLS-TP. 1689 3.3. Multipath Capabilities and Performance 1691 Multipath capabilities and performance do not apply to MPLS-TP but 1692 apply to MPLS and apply if MPLS-TP is carried in MPLS. 1694 Q#16 How are large microflows accommodated? Is there active 1695 management of the hash space mapping to output ports? See 1696 Section 2.4.2. 1698 Q#17 How many MPLS labels can be included in a hash based on the 1699 MPLS label stack? 1701 Q#18 Is packet rate performance decreased beyond some number of 1702 labels? 1704 Q#19 Can the IP header and payload information below the MPLS stack 1705 be used in the hash? If so, which IP fields, payload types and 1706 payload fields are supported? 1708 Q#20 At what maximum MPLS label stack depth can Bottom of Stack and 1709 an IP header appear without impacting packet rate performance? 1711 Q#21 Are special purpose labels excluded from the label stack hash? 1712 Are extended purpose labels excluded from the label stack hash? 1713 See Section 2.4.5.1. 1715 Q#22 How is multipath performance affected by very large flows or an 1716 extremely large number of flows, or by very short lived flows? 1717 See Section 2.7. 1719 3.4. Pseudowire Capabilities and Performance 1721 Q#23 Is the pseudowire control word supported? 1723 Q#24 What is the maximum rate of pseudowire encapsulation and 1724 decapsulation? Apply the same questions as in Base Performance 1725 for any packet based pseudowire such as IP VPN or Ethernet. 1727 Q#25 Does inclusion of a pseudowire control word impact performance? 1728 Q#26 Are flow labels supported? 1730 Q#27 If so, what fields are hashed on for the flow label for 1731 different types of pseudowires? 1733 Q#28 Does inclusion of a flow label impact performance? 1735 3.5. Entropy Label Support and Performance 1737 Q#29 Can an entropy label be added when acting as in ingress LER and 1738 can it be removed when acting as an egress LER? 1740 Q#30 If so, what fields are hashed on for the entropy label? 1742 Q#31 Does adding or removing an entropy label impact packet rate 1743 performance? 1745 Q#32 Can an entropy label be detected in the label stack, used in 1746 the hash, and properly terminate the search for further 1747 information to hash on? 1749 Q#33 Does using an entropy label have any negative impact on 1750 performance? It should have no impact or a positive impact. 1752 3.6. DoS Protection 1754 Q#34 For each control and management plane protocol in use, what 1755 measures are taken to provide DoS attack hardening? 1757 Q#35 Have DoS attack tests been performed? 1759 Q#36 Can compromise of an internal computer on a management subnet 1760 be leveraged for any form of attack including DoS attack? 1762 3.7. OAM Capabilities and Performance 1764 Q#37 What OAM proactive and on-demand mechanisms are supported? 1766 Q#38 What performance limits exist under high proactive monitoring 1767 rates? 1769 Q#39 Can excessively high proactive monitoring rates impact control 1770 plane performance or cause control plane instability? 1772 Q#40 Ask the prior questions for each of the following. 1774 A. MPLS OAM 1775 B. Pseudowire OAM 1777 C. MPLS-TP OAM 1779 D. Layer-2 OAM Interworking 1781 See Section 2.6.2. 1783 4. Forwarding Compliance and Performance Testing 1785 Packet rate performance of equipment supporting a large number of 10 1786 Gb/s or 100 Gb/s links is not possible using desktop computers or 1787 workstations. The use of high end workstations as a source of test 1788 traffic was barely viable 20 years ago, but is no longer at all 1789 viable. Though custom microcode has been used on specialized router 1790 forwarding cards to serve the purpose of generating test traffic and 1791 measuring it, for the most part performance testing will require 1792 specialized test equipment. There are multiple sources of suitable 1793 equipment. 1795 The set of tests listed here do not correspond one-to-one to the set 1796 of questions in Section 3. The same categorization is used and these 1797 tests largely serve to validate answers provided to the prior 1798 questions, and can also provide answers where a supplier is unwilling 1799 to disclose compliance or performance. 1801 Performance testing is the domain of the IETF Benchmark Methodology 1802 Working Group (BMWG). Below are brief descriptions of conformance 1803 and performance tests. Some very basic tests are specified in 1804 [RFC5695] which partially cover only the basic performance test T#3. 1806 The following tests should be performed by the systems designer, or 1807 deployer, or performed by the supplier on their behalf if it is not 1808 practical for the potential customer to perform the tests directly. 1809 These tests are grouped into broad categories. 1811 The tests in Section 4.1 should be repeated under various conditions 1812 to retest basic performance when critical capabilities are enabled. 1813 Complete repetition of the performance tests enabling each capability 1814 and combinations of capabilities would be very time intensive, 1815 therefore a reduced set of performance tests can be used to gauge the 1816 impact of enabling specific capabilities. 1818 4.1. Basic Compliance 1819 T#1 Test forwarding at a high rate for packets with varying number 1820 of label entries. While packets with more than a dozen label 1821 entries are unlikely to be used in any practical scenario today, 1822 it is useful to know if limitations exists. 1824 T#2 For each of the questions listed under "Basic Compliance" in 1825 Section 3, verify the claimed compliance. For any functionality 1826 considered critical to a deployment, where applicable 1827 performance using each capability under load should be verified 1828 in addition to basic compliance. 1830 4.2. Basic Performance 1832 T#3 Test packet forwarding at full line rate with small packets. 1833 See [RFC5695]. The most likely case to fail is the smallest 1834 packet size. Also test with packet sizes in four byte 1835 increments ranging from payload sizes or 40 to 128 bytes. 1837 T#4 If the prior tests did not succeed for all packet sizes, then 1838 perform the following tests. 1840 A. Increase the packet size by 4 bytes until a size is found 1841 that can be forwarded at full rate. 1843 B. Inject bursts of consecutive small packets into a stream of 1844 larger packets. Allow some time for recovery between 1845 bursts. Increase the number of packets in the burst until 1846 packets are dropped. 1848 T#5 Send test traffic where a swap operation is required. Also set 1849 up multiple LSP carried over other LSP where the device under 1850 test (DUT) is the egress of these LSP. Create test packets such 1851 that the swap operation is performed after pop operations, 1852 increasing the number of pop operations until forwarding of 1853 small packets at full line rate can no longer be supported. 1854 Also check to see how many pop operations can be supported 1855 before the full set of counters can no longer be maintained. 1856 This requirement is particularly relevant for MPLS-TP. 1858 T#6 Send all traffic on one LSP and see if the counters become 1859 inaccurate. Often counters on silicon are much smaller than the 1860 64 bit packet and byte counters in IETF MIB. System developers 1861 should consider what counter polling rate is necessary to 1862 maintain accurate counters and whether those polling rates are 1863 practical. Relevant MIBs for MPLS are discussed in [RFC4221] 1864 and [RFC6639]. 1866 4.3. Multipath Capabilities and Performance 1868 Multipath capabilities do not apply to MPLS-TP but apply to MPLS and 1869 apply if MPLS-TP is carried in MPLS. 1871 T#7 Send traffic at a rate well exceeding the capacity of a single 1872 multipath component link, and where entropy exists only below 1873 the top of stack. If only the top label is used this test will 1874 fail immediately. 1876 T#8 Move the labels with entropy down in the stack until either the 1877 full forwarding rate can no longer be supported or most or all 1878 packets try to use the same component link. 1880 T#9 Repeat the two tests above with the entropy contained in IP 1881 headers or IP payload fields below the label stack rather than 1882 in the label stack. Test with the set of IP headers or IP 1883 payload fields considered relevant to the deployment or to the 1884 target market. 1886 T#10 Determine whether traffic that contains a pseudowire control 1887 word is interpreted as IP traffic. Information in the payload 1888 MUST NOT be used in the load balancing if the first nibble of 1889 the packet is not 4 or 6 (IPv4 or IPv6). 1891 T#11 Determine whether special purpose labels and extended special 1892 purpose labels are excluded from the label stack hash. They 1893 MUST be excluded. 1895 T#12 Perform testing in the presence of combinations of: 1897 A. Very large microflows. 1899 B. Relatively short lived high capacity flows. 1901 C. Extremely large numbers of flows. 1903 D. Very short lived small flows. 1905 4.4. Pseudowire Capabilities and Performance 1907 T#13 Ensure that pseudowire can be set up with a pseudowire label 1908 and pseudowire control word added at ingress and the pseudowire 1909 label and pseudowire control word removed at egress. 1911 T#14 For pseudowire that contains variable length payload packets, 1912 repeat performance tests listed under "Basic Performance" for 1913 pseudowire ingress and egress functions. 1915 T#15 Repeat pseudowire performance tests with and without a 1916 pseudowire control word. 1918 T#16 Determine whether pseudowire can be set up with a pseudowire 1919 label, flow label, and pseudowire control word added at ingress 1920 and the pseudowire label, flow label, and pseudowire control 1921 word removed at egress. 1923 T#17 Determine which payload fields are used to create the flow 1924 label and whether the set of fields and algorithm provide 1925 sufficient entropy for load balancing. 1927 T#18 Repeat pseudowire performance tests with flow labels included. 1929 4.5. Entropy Label Support and Performance 1931 T#19 Determine whether entropy labels can be added at ingress and 1932 removed at egress. 1934 T#20 Determine which fields are used to create an entropy label. 1935 Labels further down in the stack, including entropy labels 1936 further down and IP headers or IP payload fields where 1937 applicable should be used. Determine whether the set of fields 1938 and algorithm provide sufficient entropy for load balancing. 1940 T#21 Repeat performance tests under "Basic Performance" when entropy 1941 labels are used, where ingress or egress is the device under 1942 test (DUT). 1944 T#22 Determine whether an ELI is detected when acting as a midpoint 1945 LSR and whether the search for further information on which to 1946 base the load balancing is used. Information below the entropy 1947 label SHOULD NOT be used. 1949 T#23 Ensure that the entropy label indicator and entropy label (ELI 1950 and EL) are removed from the label stack during UHP and PHP 1951 operations. 1953 T#24 Insure that operations on the TC field when adding and removing 1954 entropy label are correctly carried out. If TC is changed 1955 during a swap operation, the ability to transfer that change 1956 MUST be provided. The ability to suppress the transfer of TC 1957 MUST also be provided. See "pipe", "short pipe", and "uniform" 1958 models in [RFC3443]. 1960 T#25 Repeat performance tests for midpoint LSR with entropy labels 1961 found at various label stack depths. 1963 4.6. DoS Protection 1965 T#26 Actively attack LSR under high protocol churn load and 1966 determine control plane performance impact or successful DoS 1967 under test conditions. Specifically test for the following. 1969 A. TCP SYN attack against control plane and management plane 1970 protocols using TCP, including CLI access (typically SSH 1971 protected login), NETCONF, etc. 1973 B. High traffic volume attack against control plane and 1974 management plane protocols not using TCP. 1976 C. Attacks which can be performed from a compromised 1977 management subnet computer, but not one with authentication 1978 keys. 1980 D. Attacks which can be performed from a compromised peer 1981 within the control plane (internal domain and external 1982 domain). Assume that per peering keys and per router ID 1983 keys rather than network wide keys are in use. 1985 See Section 2.6.1. 1987 4.7. OAM Capabilities and Performance 1989 T#27 Determine maximum sustainable rates of BFD traffic. If BFD 1990 requires CPU intervention, determine both maximum rates and CPU 1991 loading when multiple interfaces are active. 1993 T#28 Verify LSP Ping and LSP Traceroute capability. 1995 T#29 Determine maximum rates of MPLS-TP CC-CV traffic. If CC-CV 1996 requires CPU intervention, determine both maximum rates and CPU 1997 loading when multiple interfaces are active. 1999 T#30 Determine MPLS-TP DM precision. 2001 T#31 Determine MPLS-TP LM accuracy. 2003 T#32 Verify MPLS-TP AIS/RDI and PSC functionality, protection speed, 2004 and AIS/RDI notification speed when a large number of 2005 Management Entities (ME) must be notified with AIS/RDI. 2007 5. Acknowledgements 2009 Numerous very useful comments have been received in private email. 2010 Some of these contributions are acknowledged here, approximately in 2011 chronologic order. 2013 Paul Doolan provided a brief review resulting in a number of 2014 clarifications, most notably regarding on-chip vs. system buffering, 2015 100 Gb/s link speed assumptions in the 150 Mpps figure, and handling 2016 of large microflows. Pablo Frank reminded us of the sawtooth effect 2017 in PPS vs. packet size graphs, prompting the addition of a few 2018 paragraphs on this. Comments from Lou Berger at IETF-85 prompted the 2019 addition of Section 2.7. 2021 Valuable comments were received on the BMWG mailing list. Jay 2022 Karthik pointed out testing methodology hints that after discussion 2023 were deemed out of scope and were removed but may benefit later work 2024 in BMWG. 2026 Nabil Bitar pointed out the need to cover QoS (Differentiated 2027 Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM. Nabil 2028 also provided a number of clarifications to the questions and tests 2029 in Section 3 and Section 4. 2031 Gregory Mirsky and Thomas Beckhaus provided useful comments during 2032 the MPLS RT review. 2034 Tal Mizrahi provided comments that prompted clarifications regarding 2035 timestamp processing, local delivery of packets, and the need for 2036 hardware assistance in processing OAM traffic. 2038 6. IANA Considerations 2040 This memo includes no request to IANA. 2042 7. Security Considerations 2044 This document reviews forwarding behavior specified elsewhere and 2045 points out compliance and performance requirements. As such it 2046 introduces no new security requirements or concerns. 2048 Knowledge of potential performance shortcomings may serve to help new 2049 implementations avoid pitfalls. It is unlikely that such knowledge 2050 could be the basis of new denial of service as these pitfalls are 2051 already widely known in the service provider community and among 2052 leading equipment suppliers. In practice extreme data and packet 2053 rate are needed to affect existing equipment and to affect networks 2054 that may be still vulnerable due to failure to implement adequate 2055 protection. The extreme data and packet rates make this type of 2056 denial of service unlikely and make undetectable denial of service of 2057 this type impossible. 2059 8. References 2061 8.1. Normative References 2063 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2064 Requirement Levels", BCP 14, RFC 2119, March 1997. 2066 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2067 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2068 Encoding", RFC 3032, January 2001. 2070 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2071 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2072 Tunnels", RFC 3209, December 2001. 2074 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2075 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2076 Protocol Label Switching (MPLS) Support of Differentiated 2077 Services", RFC 3270, May 2002. 2079 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2080 in Multi-Protocol Label Switching (MPLS) Networks", 2081 RFC 3443, January 2003. 2083 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 2084 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 2085 May 2005. 2087 [RFC4182] Rosen, E., "Removing a Restriction on the use of MPLS 2088 Explicit NULL", RFC 4182, September 2005. 2090 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 2091 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 2093 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2094 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2095 Use over an MPLS PSN", RFC 4385, February 2006. 2097 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2098 Marking in MPLS", RFC 5129, January 2008. 2100 [RFC5586] Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic 2101 Associated Channel", RFC 5586, June 2009. 2103 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 2104 J., and S. Amante, "Flow-Aware Transport of Pseudowires 2105 over an MPLS Packet Switched Network", RFC 6391, 2106 November 2011. 2108 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2109 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2110 RFC 6790, November 2012. 2112 8.2. Informative References 2114 [ACK-compression] 2115 "Observations and Dynamics of a Congestion Control 2116 Algorithm: The Effects of Two-Way Traffic", Proc. ACM 2117 SIGCOMM, ACM Computer Communications Review (CCR) Vol 21, 2118 No 4, 1991, pp.133-147., 1991. 2120 [ATM-EPD-and-PPD] 2121 "Dynamics of TCP Traffic over ATM Networks", IEEE Journal 2122 of Special Areas of Communication Vol 13 No 4, May 1995, 2123 pp. 633-641., May 1995. 2125 [I-D.ietf-mpls-special-purpose-labels] 2126 Kompella, K., Andersson, L., and A. Farrel, "Allocating 2127 and Retiring Special Purpose MPLS Labels", 2128 draft-ietf-mpls-special-purpose-labels-03 (work in 2129 progress), July 2013. 2131 [I-D.ietf-pwe3-mpls-eth-oam-iwk] 2132 Mohan, D., Bitar, N., Sajassi, A., DeLord, S., Niger, P., 2133 and R. Qiu, "MPLS and Ethernet OAM Interworking", 2134 draft-ietf-pwe3-mpls-eth-oam-iwk-08 (work in progress), 2135 July 2013. 2137 [I-D.ietf-pwe3-vccv-impl-survey-results] 2138 Malis, A., "The Pseudowire (PW) & Virtual Circuit 2139 Connectivity Verification (VCCV) Implementation Survey 2140 Results", draft-ietf-pwe3-vccv-impl-survey-results-03 2141 (work in progress), October 2013. 2143 [I-D.ietf-tictoc-1588overmpls] 2144 Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. 2145 Montini, "Transporting Timing messages over MPLS 2146 Networks", draft-ietf-tictoc-1588overmpls-05 (work in 2147 progress), June 2013. 2149 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 2150 September 1981. 2152 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2153 "Definition of the Differentiated Services Field (DS 2154 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2155 December 1998. 2157 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 2158 and W. Weiss, "An Architecture for Differentiated 2159 Services", RFC 2475, December 1998. 2161 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 2162 "Assured Forwarding PHB Group", RFC 2597, June 1999. 2164 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2165 Label Switching Architecture", RFC 3031, January 2001. 2167 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2168 of Explicit Congestion Notification (ECN) to IP", 2169 RFC 3168, September 2001. 2171 [RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for 2172 Multiprotocol Label Switching Architecture (MPLS) 2173 Operation and Maintenance (OAM) Functions", RFC 3429, 2174 November 2002. 2176 [RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching 2177 (GMPLS) Signaling Functional Description", RFC 3471, 2178 January 2003. 2180 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 2181 Edge (PWE3) Architecture", RFC 3985, March 2005. 2183 [RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 2184 Provider-Provisioned Virtual Private Networks (PPVPNs)", 2185 RFC 4110, July 2005. 2187 [RFC4124] Le Faucheur, F., "Protocol Extensions for Support of 2188 Diffserv-aware MPLS Traffic Engineering", RFC 4124, 2189 June 2005. 2191 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2192 Hierarchy with Generalized Multi-Protocol Label Switching 2193 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 2195 [RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol 2196 Label Switching (MPLS) Management Overview", RFC 4221, 2197 November 2005. 2199 [RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. 2200 Matsushima, "Operations and Management (OAM) Requirements 2201 for Multi-Protocol Label Switched (MPLS) Networks", 2202 RFC 4377, February 2006. 2204 [RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol 2205 Label Switched (MPLS) Data Plane Failures", RFC 4379, 2206 February 2006. 2208 [RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual 2209 Private Networks (L2VPNs)", RFC 4664, September 2006. 2211 [RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, 2212 "Extensions to Resource Reservation Protocol - Traffic 2213 Engineering (RSVP-TE) for Point-to-Multipoint TE Label 2214 Switched Paths (LSPs)", RFC 4875, May 2007. 2216 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 2217 Cost Multipath Treatment in MPLS Networks", BCP 128, 2218 RFC 4928, June 2007. 2220 [RFC4950] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP 2221 Extensions for Multiprotocol Label Switching", RFC 4950, 2222 August 2007. 2224 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 2225 Specification", RFC 5036, October 2007. 2227 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., and C. 2228 Pignataro, "The Generalized TTL Security Mechanism 2229 (GTSM)", RFC 5082, October 2007. 2231 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 2232 Connectivity Verification (VCCV): A Control Channel for 2233 Pseudowires", RFC 5085, December 2007. 2235 [RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) 2236 Report on MPLS Architectural Considerations for a 2237 Transport Profile", RFC 5317, February 2009. 2239 [RFC5332] Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS 2240 Multicast Encapsulations", RFC 5332, August 2008. 2242 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2243 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2244 Class" Field", RFC 5462, February 2009. 2246 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 2247 Benchmarking Methodology for IP Flows", RFC 5695, 2248 November 2009. 2250 [RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for 2251 Operations, Administration, and Maintenance (OAM) in MPLS 2252 Transport Networks", RFC 5860, May 2010. 2254 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 2255 (BFD)", RFC 5880, June 2010. 2257 [RFC5884] Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow, 2258 "Bidirectional Forwarding Detection (BFD) for MPLS Label 2259 Switched Paths (LSPs)", RFC 5884, June 2010. 2261 [RFC5885] Nadeau, T. and C. Pignataro, "Bidirectional Forwarding 2262 Detection (BFD) for the Pseudowire Virtual Circuit 2263 Connectivity Verification (VCCV)", RFC 5885, June 2010. 2265 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 2266 Time Protocol Version 4: Protocol and Algorithms 2267 Specification", RFC 5905, June 2010. 2269 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 2270 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 2271 Acronym in the IETF", BCP 161, RFC 6291, June 2011. 2273 [RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., 2274 Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, 2275 Administration, and Maintenance (OAM) Message Mapping", 2276 RFC 6310, July 2011. 2278 [RFC6371] Busi, I. and D. Allan, "Operations, Administration, and 2279 Maintenance Framework for MPLS-Based Transport Networks", 2280 RFC 6371, September 2011. 2282 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 2283 Measurement for MPLS Networks", RFC 6374, September 2011. 2285 [RFC6375] Frost, D. and S. Bryant, "A Packet Loss and Delay 2286 Measurement Profile for MPLS-Based Transport Networks", 2287 RFC 6375, September 2011. 2289 [RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, 2290 "Label Distribution Protocol Extensions for Point-to- 2291 Multipoint and Multipoint-to-Multipoint Label Switched 2292 Paths", RFC 6388, November 2011. 2294 [RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for 2295 Performing Label Switched Path Ping (LSP Ping) over MPLS 2296 Tunnels", RFC 6424, November 2011. 2298 [RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, 2299 S., and T. Nadeau, "Detecting Data-Plane Failures in 2300 Point-to-Multipoint MPLS - Extensions to LSP Ping", 2301 RFC 6425, November 2011. 2303 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2304 On-Demand Connectivity Verification and Route Tracing", 2305 RFC 6426, November 2011. 2307 [RFC6427] Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S., 2308 and D. Ward, "MPLS Fault Management Operations, 2309 Administration, and Maintenance (OAM)", RFC 6427, 2310 November 2011. 2312 [RFC6428] Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive 2313 Connectivity Verification, Continuity Check, and Remote 2314 Defect Indication for the MPLS Transport Profile", 2315 RFC 6428, November 2011. 2317 [RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., 2318 and X. Dai, "MPLS Transport Profile Lock Instruct and 2319 Loopback Functions", RFC 6435, November 2011. 2321 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 2322 for Equal Cost Multipath Routing and Link Aggregation in 2323 Tunnels", RFC 6438, November 2011. 2325 [RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, 2326 "Pseudowire Status for Static Pseudowires", RFC 6478, 2327 May 2012. 2329 [RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching 2330 Transport Profile (MPLS-TP) MIB-Based Management 2331 Overview", RFC 6639, June 2012. 2333 [RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, 2334 Administration, and Maintenance (OAM) Toolset for MPLS- 2335 Based Transport Networks", RFC 6669, July 2012. 2337 [RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a 2338 Single Solution for MPLS Transport Profile (MPLS-TP) 2339 Operations, Administration, and Maintenance (OAM)", 2340 RFC 6670, July 2012. 2342 [RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security 2343 Mechanism (GTSM) for the Label Distribution Protocol 2344 (LDP)", RFC 6720, August 2012. 2346 [RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label 2347 Switched Path (LSP) Ping for Pseudowire Forwarding 2348 Equivalence Classes (FECs) Advertised over IPv6", 2349 RFC 6829, January 2013. 2351 Appendix A. Organization of References Section 2353 The References section is split into Normative and Informative 2354 subsections. References that directly specify forwarding 2355 encapsulations or behaviors are listed as normative. References 2356 which describe signaling only, though normative with respect to 2357 signaling, are listed as informative. They are informative with 2358 respect to MPLS forwarding. 2360 Authors' Addresses 2362 Curtis Villamizar (editor) 2363 Outer Cape Cod Network Consulting, LLC 2365 Email: curtis@occnc.com 2367 Kireeti Kompella 2368 Contrail Systems 2370 Email: kireeti.kompella@gmail.com 2372 Shane Amante 2373 Level 3 Communications, Inc. 2374 1025 Eldorado Blvd 2375 Broomfield, CO 80021 2377 Email: shane@level3.net 2378 Andrew Malis 2379 Verizon 2380 60 Sylvan Road 2381 Waltham, MA 02451 2383 Phone: +1 781-466-2362 2384 Email: andrew.g.malis@verizon.com 2386 Carlos Pignataro 2387 Cisco Systems 2388 7200-12 Kit Creek Road 2389 Research Triangle Park, NC 27709 2390 US 2392 Email: cpignata@cisco.com