idnits 2.17.00 (12 Aug 2021) /tmp/idnits10411/draft-ietf-mpls-forwarding-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (November 14, 2013) is 3110 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 522 == Outdated reference: draft-ietf-mpls-special-purpose-labels has been published as RFC 7274 == Outdated reference: draft-ietf-pwe3-vccv-impl-survey-results has been published as RFC 7079 == Outdated reference: A later version (-07) exists of draft-ietf-tictoc-1588overmpls-05 -- Obsolete informational reference (is this intentional?): RFC 4379 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6424 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6829 (Obsoleted by RFC 8029) Summary: 0 errors (**), 0 flaws (~~), 5 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPLS C. Villamizar, Ed. 3 Internet-Draft OCCNC 4 Intended status: Informational K. Kompella 5 Expires: May 18, 2014 Juniper Networks 6 S. Amante 7 Apple Inc. 8 A. Malis 9 Consultant 10 C. Pignataro 11 Cisco 12 November 14, 2013 14 MPLS Forwarding Compliance and Performance Requirements 15 draft-ietf-mpls-forwarding-03 17 Abstract 19 This document provides guidelines for implementers regarding MPLS 20 forwarding and a basis for evaluations of forwarding implementations. 21 Guidelines cover many aspects of MPLS forwarding. Topics are 22 highlighted where implementers might otherwise overlook practical 23 requirements which are unstated or under emphasized or are optional 24 for conformance to RFCs but are often considered mandatory by 25 providers. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on May 18, 2014. 44 Copyright Notice 46 Copyright (c) 2013 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction and Document Scope . . . . . . . . . . . . . . . 3 62 1.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . . . 4 63 1.2. Use of Requirements Language . . . . . . . . . . . . . . 8 64 1.3. Apparent Misconceptions . . . . . . . . . . . . . . . . . 8 65 1.4. Target Audience . . . . . . . . . . . . . . . . . . . . . 9 66 2. Forwarding Issues . . . . . . . . . . . . . . . . . . . . . . 10 67 2.1. Forwarding Basics . . . . . . . . . . . . . . . . . . . . 10 68 2.1.1. MPLS Special Purpose Labels . . . . . . . . . . . . . 11 69 2.1.2. MPLS Differentiated Services . . . . . . . . . . . . 12 70 2.1.3. Time Synchronization . . . . . . . . . . . . . . . . 13 71 2.1.4. Uses of Multiple Label Stack Entries . . . . . . . . 14 72 2.1.5. MPLS Link Bundling . . . . . . . . . . . . . . . . . 15 73 2.1.6. MPLS Hierarchy . . . . . . . . . . . . . . . . . . . 15 74 2.1.7. MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . . 15 75 2.1.8. Pseudowire Encapsulation . . . . . . . . . . . . . . 16 76 2.1.8.1. Pseudowire Sequence Number . . . . . . . . . . . 16 77 2.1.9. Layer-2 and Layer-3 VPN . . . . . . . . . . . . . . . 17 78 2.2. MPLS Multicast . . . . . . . . . . . . . . . . . . . . . 17 79 2.3. Packet Rates . . . . . . . . . . . . . . . . . . . . . . 18 80 2.4. MPLS Multipath Techniques . . . . . . . . . . . . . . . . 20 81 2.4.1. Pseudowire Control Word . . . . . . . . . . . . . . . 21 82 2.4.2. Large Microflows . . . . . . . . . . . . . . . . . . 22 83 2.4.3. Pseudowire Flow Label . . . . . . . . . . . . . . . . 22 84 2.4.4. MPLS Entropy Label . . . . . . . . . . . . . . . . . 22 85 2.4.5. Fields Used for Multipath Load Balance . . . . . . . 23 86 2.4.5.1. MPLS Fields in Multipath . . . . . . . . . . . . 23 87 2.4.5.2. IP Fields in Multipath . . . . . . . . . . . . . 25 88 2.4.5.3. Fields Used in Flow Label . . . . . . . . . . . . 26 89 2.4.5.4. Fields Used in Entropy Label . . . . . . . . . . 26 90 2.5. MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . . 27 91 2.6. Local Delivery of Packets . . . . . . . . . . . . . . . . 27 92 2.6.1. DoS Protection . . . . . . . . . . . . . . . . . . . 28 93 2.6.2. MPLS OAM . . . . . . . . . . . . . . . . . . . . . . 30 94 2.6.3. Pseudowire OAM . . . . . . . . . . . . . . . . . . . 31 95 2.6.4. MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . . 31 96 2.6.5. MPLS OAM and Layer-2 OAM Interworking . . . . . . . . 32 97 2.6.6. Extent of OAM Support by Hardware . . . . . . . . . . 33 98 2.7. Number and Size of Flows . . . . . . . . . . . . . . . . 34 99 3. Questions for Suppliers . . . . . . . . . . . . . . . . . . . 34 100 3.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . 34 101 3.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 36 102 3.3. Multipath Capabilities and Performance . . . . . . . . . 37 103 3.4. Pseudowire Capabilities and Performance . . . . . . . . . 37 104 3.5. Entropy Label Support and Performance . . . . . . . . . . 38 105 3.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . 38 106 3.7. OAM Capabilities and Performance . . . . . . . . . . . . 38 107 4. Forwarding Compliance and Performance Testing . . . . . . . . 39 108 4.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . 39 109 4.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 40 110 4.3. Multipath Capabilities and Performance . . . . . . . . . 40 111 4.4. Pseudowire Capabilities and Performance . . . . . . . . . 41 112 4.5. Entropy Label Support and Performance . . . . . . . . . . 42 113 4.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . 42 114 4.7. OAM Capabilities and Performance . . . . . . . . . . . . 43 115 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 43 116 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 44 117 7. Security Considerations . . . . . . . . . . . . . . . . . . . 44 118 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 44 119 8.1. Normative References . . . . . . . . . . . . . . . . . . 45 120 8.2. Informative References . . . . . . . . . . . . . . . . . 46 121 Appendix A. Organization of References Section . . . . . . . . . 51 122 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 51 124 1. Introduction and Document Scope 126 The initial purpose of this document was to address concerns raised 127 on the MPLS WG mailing list about shortcomings in implementations of 128 MPLS forwarding. Documenting existing misconceptions and potential 129 pitfalls might potentially avoid repeating past mistakes. The 130 document has grown to address a broad set of forwarding requirements. 132 The focus of this document is MPLS forwarding, base pseudowire 133 forwarding, and MPLS OAM. The use of pseudowire control word, and 134 sequence number are discussed. Specific pseudowire AC and NSP are 135 out of scope. Specific pseudowire applications, such as various 136 forms of VPN, are out of scope. 138 MPLS support for multipath techniques is considered essential by many 139 service providers and is useful for other high capacity networks. In 140 order to obtain sufficient entropy from MPLS traffic service 141 providers and others find it essential for the MPLS implementation to 142 interpret the MPLS payload as IPv4 or IPv6 based on the contents of 143 the first nibble of payload. The use of IP addresses, the IP 144 protocol field, and UDP and TCP port number fields in multipath load 145 balancing are considered within scope. The use of any other IP 146 protocol fields, such as tunneling protocols carried within IP, are 147 out of scope. 149 Implementation details are a local matter and are out of scope. Most 150 interfaces today operate at 1 Gb/s or greater. It is assumed that 151 all forwarding operations are implemented in specialized forwarding 152 hardware rather than on a general purpose processor. This is often 153 referred to as "fast path" and "slow path" processing. Some 154 recommendations are made regarding implementing control or management 155 plane functionality in specialized hardware or with limited 156 assistance from specialized hardware. This advise is based on 157 expected control or management protocol loads and on the need for 158 denial of service (DoS) protection. 160 1.1. Acronyms 162 The following acronyms are used. 164 AC Attachment Circuit ([RFC3985]) 166 ACH Associated Channel Header (pseudowires) 168 ACK Acknowledgement (TCP flag and type of TCP packet) 170 AIS Alarm Indication Signal (MPLS-TP OAM) 172 ATM Asynchronous Transfer Mode (legacy switched circuits) 174 BFD Bidirectional Forwarding Detection 176 BGP Border Gateway Protocol 178 CC-CV Connectivity Check and Connectivity Verification 180 CE Customer Edge (LDP) 182 CPU Central Processing Unit (computer or microprocessor) 184 CT Class Type ([RFC4124]) 186 CW Control Word ([RFC4385]) 187 DCCP Datagram Congestion Control Protocol 189 DDoS Distributed Denial of Service 191 DM Delay Measurement (MPLS-TP OAM) 193 DSCP Differentiated Services Code Point ([RFC2474]) 195 DWDM Dense Wave Division Multiplexing 197 DoS Denial of Service 199 E-LSP EXP-Inferred-PSC LSP ([RFC3270]) 201 EBGP External BGP 203 ECMP Equal Cost Multi-Path 205 ECN Explicit Congestion Notification ([RFC3168] and [RFC5129]) 207 EL Entropy Label ([RFC6790]) 209 ELI Entropy Label Indicator ([RFC6790]) 211 EXP Experimental (field in MPLS renamed to TC in [RFC5462]) 213 FEC Forwarding Equivalence Classes (LDP), also Forward Error 214 Correction in other context 216 FR Frame Relay (legacy switched circuits) 218 FRR Fast Reroute ([RFC4090]) 220 G-ACh Generic Associated Channel ([RFC5586]) 222 GAL Generic Associated Channel Label ([RFC5586]) 224 GFP Generic Framing Protocol (used in OTN) 226 GMPLS Generalized MPLS ([RFC3471]) 228 GTSM Generalized TTL Security Mechanism ([RFC5082]) 230 Gb/s Gigabits per second (billion bits per second) 232 IANA Internet Assigned Numbers Authority 234 ILM Incoming Label Map ([RFC3031]) 235 IP Internet Protocol 237 IPVPN Internet Protocol VPN 239 IPv4 Internet Protocol version 4 241 IPv6 Internet Protocol version 6 243 L-LSP Label-Only-Inferred-PSC LSP ([RFC3270]) 245 L2VPN Layer 2 VPN 247 LDP Label Distribution Protocol ([RFC5036]) 249 LER Label Edge Router ([RFC3031]) 251 LM Loss Measurement (MPLS-TP OAM) 253 LSP Label Switched Path ([RFC3031]) 255 LSR Label Switching Router ([RFC3031]) 257 MP2MP Multipoint to Point 259 MPLS MultiProtocol Label Switching ([RFC3031]) 261 MPLS-TP MPLS Transport Profile ([RFC5317]) 263 Mb/s Megabits per second (million bits per second) 265 NSP Native Service Processing ([RFC3985]) 267 NTP Network Time Protocol 269 OAM Operations, Administration, and Maintenance ([RFC6291]) 271 OOB Out-of-band (not carried within a data channel) 273 OTN Optical Transport Network 275 P Provider router (LDP) 277 P2MP Point to Multi-Point 279 PE Provider Edge router (LDP) 281 PHB Per-Hop-Behavior ([RFC2475]) 282 PHP Penultimate Hop Popping ([RFC3443]) 284 POS Packet over SONET 286 PSC This acronym has multiple interpretations. 288 1. Packet Switch Capable ([RFC3471] 290 2. PHB Scheduling Class ([RFC3270]) 292 3. Protection State Coordination (MPLS-TP linear protection) 294 PTP Precision Time Protocol 296 PW Pseudowire 298 QoS Quality of Service 300 RA Router Alert ([RFC3032]) 302 RDI Remote Defect Indication (MPLS-TP OAM) 304 RSVP-TE RSVP Traffic Engineering 306 RTP Real-Time Transport Protocol 308 SCTP Stream Control Transmission Protocol 310 SDH Synchronous Data Hierarchy (European SONET, a form of TDM) 312 SONET Synchronous Optical Network (US SDH, a form of TDM) 314 T-LDP Targeted LDP (LDP sessions over more than one hop) 316 TC Traffic Class ([RFC5462]) 318 TCP Transmission Control Protocol 320 TDM Time-Division Multiplexing (legacy encapsulations) 322 TOS Type of Service (see [RFC2474]) 324 TTL Time-to-live (a field in IP and MPLS headers) 326 UDP User Datagram Protocol 328 UHP Ultimate Hop Popping (opposite of PHP) 329 VCCV Virtual Circuit Connectivity Verification ([RFC5085]) 331 VLAN Virtual Local Area Network (Ethernet) 333 VOQ Virtual Output Queuing (switch fabric design) 335 VPN Virtual Private Network 337 WG Working Group 339 1.2. Use of Requirements Language 341 This document is informational. The upper case [RFC2119] key words 342 are not used in this document, except in the following cases. 344 1. RFC 2119 keywords are used where requirements stated in this 345 document are called for in referenced RFCs. In most cases the 346 RFC containing the requirement is cited within the statement 347 using an RFC 2119 keyword. 349 2. RFC 2119 keywords are used where explicitly noted that the 350 keywords indicate that operator experiences indicate a 351 requirement, but there are no existing RFC requirements. 353 Advice provided by this document may be ignored by implementations. 354 Similarly, implementations not claiming conformance to specific RFCs 355 may ignore the requirements of those RFCs. In both cases, 356 implementers should consider the risk of doing so. 358 1.3. Apparent Misconceptions 360 In early generations of forwarding silicon (which might now be behind 361 us), there apparently were some misconceptions about MPLS. The 362 following statements provide clarifications. 364 1. There are practical reasons to have more than one or two labels 365 in an MPLS label stack. Under some circumstances the label stack 366 can become quite deep. See Section 2.1. 368 2. The label stack MUST be considered to be arbitrarily deep. 369 Section 3.27.4. "Hierarchy: LSP Tunnels within LSPs" of RFC3031 370 states "The label stack mechanism allows LSP tunneling to nest to 371 any depth." [RFC3031] If a bottom of the label stack cannot be 372 found, but sufficient number of labels exist to forward, an LSR 373 MUST forward the packet. An LSR MUST NOT assume the packet is 374 malformed unless the end of packet is found before bottom of 375 stack. See Section 2.1. 377 3. In networks where deep label stacks are encountered, they are not 378 rare. Full packet rate performance is required regardless of 379 label stack depth, except where multiple pop operations are 380 required. See Section 2.1. 382 4. Research has shown that long bursts of short packets with 40 byte 383 or 44 byte IP payload sizes in these bursts are quite common. 384 This is due to TCP ACK compression [ACK-compression]. The 385 following two sub-bullets constitutes advice that reflects very 386 common hard requirements of providers. Implementers may ignore 387 this advice but should consider the risk of doing so. 389 a. A forwarding engine SHOULD, if practical, be able to sustain 390 an arbitrarily long sequence of small packets arriving at 391 full interface rate. 393 b. If indefinite full packet rate for small packets is not 394 practical, a forwarding engine MUST be able to buffer a long 395 sequence of small packets inbound to the on-chip decision 396 engine and sustain full interface rate for some reasonable 397 average packet rate. Absent this small on-chip buffering, 398 QoS agnostic packet drops can occur. 400 See Section 2.3. 402 5. The implementer and system designer MUST support pseudowire 403 control word (CW) if MPLS-TP is supported or if ACH [RFC5586] is 404 being used on a pseudowire. The implementer and system designer 405 SHOULD support pseudowire CW even if MPLS-TP and ACH [RFC5586] 406 are not used, using instead CW and VCCV Type 1 [RFC5085] to allow 407 the use of multipath in the underlying network topology without 408 impacting the PW traffic. 409 [I-D.ietf-pwe3-vccv-impl-survey-results] does note that there are 410 still some deployments where the CW is not always used. It also 411 notes that many service providers do enable the CW. See 412 Section 2.4.1 for more discussion on why deployments SHOULD 413 enable the pseudowire CW. 415 6. The implementer and system designer SHOULD support adding a 416 pseudowire Flow Label [RFC6391]. Deployments MAY enable this 417 feature for appropriate pseudowire types. See Section 2.4.3. 419 7. The implementer and system designer SHOULD support adding an MPLS 420 entropy label [RFC6790]. Deployments MAY enable this feature. 421 See Section 2.4.4. 423 1.4. Target Audience 424 This document is intended for multiple audiences: implementer 425 (implementing MPLS forwarding in silicon or in software); systems 426 designer (putting together a MPLS forwarding systems); deployer 427 (running an MPLS network). These guidelines are intended to serve 428 the following purposes: 430 1. Explain what to do and what not to do when a deep label stack is 431 encountered. (audience: implementer) 433 2. Highlight pitfalls to look for when implementing an MPLS 434 forwarding chip. (audience: implementer) 436 3. Provide a checklist of features and performance specifications to 437 request. (audience: systems designer, deployer) 439 4. Provide a set of tests to perform. (audience: systems designer, 440 deployer). 442 The implementer, systems designer, and deployer have a transitive 443 supplier customer relationship. It is in the best interest of the 444 supplier to review their product against their customer's checklist 445 and customer's customer's checklist if applicable. 447 2. Forwarding Issues 449 A brief review of forwarding issues is provided in the subsections 450 that follow. This section provides some background on why some of 451 these requirements exist. The questions to ask of suppliers is 452 covered in Section 3. Some guidelines for testing are provided in 453 Section 4. 455 2.1. Forwarding Basics 457 Basic MPLS architecture and MPLS encapsulation, and therefore packet 458 forwarding are defined in [RFC3031] and [RFC3032]. RFC3031 and 459 RFC3032 are somewhat LDP centric. RSVP-TE supports traffic 460 engineering (TE) and fast reroute, features that LDP lacks. The base 461 document for RSVP-TE based MPLS is [RFC3209]. 463 A few RFCs update RFC3032. Those with impact on forwarding include 464 the following. 466 1. TTL processing is clarified in [RFC3443]. 468 2. The use of MPLS Explicit NULL is modified in [RFC4182]. 470 3. Differentiated Services is supported by [RFC3270] and [RFC4124]. 471 The "EXP" field is renamed to "Traffic Class" in [RFC5462], 472 removing any misconception that it was available for 473 experimentation or could be ignored. 475 4. ECN is supported by [RFC5129]. 477 5. The MPLS G-ACh and GAL are defined in [RFC5586]. 479 6. [RFC5332] redefines the two data link layer codepoints for MPLS 480 packets. 482 Tunneling encapsulations which may carry MPLS, such as MPLS in GRE, 483 L2TP, or LDP, are out of scope. 485 Other RFCs have implications to MPLS Forwarding and do not update 486 RFC3032 or RFC3209, including: 488 1. The pseudowire (PW) Associated Channel Header (ACH), defined by 489 [RFC5085], later generalized by the MPLS G-ACh [RFC5586]. 491 2. The entropy label indicator (ELI) and entropy label (EL) are 492 defined by [RFC6790]. 494 A few RFCs update RFC3209. Those that are listed as updating RFC3209 495 generally impact only RSVP-TE signaling. Forwarding is modified by 496 major extension built upon RFC3209. 498 RFCs which impact forwarding are discussed in the following 499 subsections. 501 2.1.1. MPLS Special Purpose Labels 503 [RFC3032] specifies that label values 0-15 are special purpose labels 504 with special meanings. Three values of NULL label are defined (two 505 of which are later updated by [RFC4182]) and a router-alert label is 506 defined. The original intent was that special purpose labels, except 507 the NULL labels, could be sent to the routing engine CPU rather than 508 be processed in forwarding hardware. Hardware support is required by 509 new RFCs such as those defining entropy label and OAM processed as a 510 result of receiving a GAL. For new special purpose labels, some 511 accommodation is needed for LSR that will send the labels to a 512 general purpose CPU or other highly programmable hardware. For 513 example, ELI will only be sent to LSR which have signaled support for 514 [RFC6790] and high OAM packet rate must be negotiated among 515 endpoints. 517 [RFC3429] reserves a label for ITU-T Y.1711, however Y.1711 does not 518 work with multipath and its use is strongly discouraged. 520 The current list of special purpose labels can be found on the 521 "Multiprotocol Label Switching Architecture (MPLS) Label Values" 522 registry reachable at IANA's pages at [1]. 524 [I-D.ietf-mpls-special-purpose-labels] introduces an IANA "Extended 525 Special Purpose MPLS Label Values" registry and makes use of the 526 "extension" label, label 15, to indicate that the next label is an 527 extended special purpose label and requires special handling. The 528 range of only 16 values for special purpose labels allows a table to 529 be used. The range of extended special purpose labels with 20 bits 530 available for use may have to be handled in some other way in the 531 unlikely event that in the future the range of currently reserved 532 values 256-1048575 are used. If only the standards action range, 533 16-239, and the experimental range, 240-255, are used, then a table 534 of 256 entries can be used. 536 Unknown special purpose labels and unknown extended special purpose 537 labels are handled the same. When an unknown special purpose label 538 is encountered or a special purpose label not directly handled in 539 forwarding hardware is encountered, the packet should be sent to a 540 general purpose CPU by default. If this capability is supported, 541 there must be an option to either drop or rate limit such packets on 542 a per special purpose label value basis. 544 2.1.2. MPLS Differentiated Services 546 [RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence 547 (Prec) fields and replaces them with the Differentiated Services 548 Field more commonly known as the Differentiated Services Code Point 549 (DSCP) field. [RFC2475] defines the Differentiated Services 550 architecture, which in other forum is often called a Quality of 551 Service (QoS) architecture. 553 MPLS uses the Traffic Class (TC) field to support Differentiated 554 Services [RFC5462]. There are two primary documents describing how 555 DSCP is mapped into TC. 557 1. [RFC3270] defines E-LSP and L-LSP. E-LSP use a static mapping of 558 DSCP into TC. L-LSP uses a per LSP mapping of DSCP into TC, with 559 one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use 560 multiple Per-Hop Behavior (PHB) values. For example, the Assured 561 Forwarding service defines three PSC, each with three PHB 562 [RFC2597]. 564 2. [RFC4124] defines assignment of a class-type (CT) to an LSP, 565 where a per CT static mapping of TC to PHB is used. [RFC4124] 566 provides a means to support up to eight E-LSP-like mappings of 567 DSCP to TC. 569 To meet Differentiated Services requirements specified in [RFC3270], 570 the following forwarding requirements must be met. An ingress LER 571 MUST be able to select an LSP and then apply a per LSP map of DSCP 572 into TC. A midpoint LSR MUST be able to apply a per LSP map of TC to 573 PHB. The number of mappings supported will be far less than the 574 number of LSP supported. 576 To meet Differentiated Services requirements specified in [RFC4124], 577 the following forwarding requirements must be met. An ingress LER 578 MUST be able to select an LSP and then apply a per LSP map of DSCP 579 into TC. A midpoint LSR MUST be able to apply a per LSP map to CT 580 map and then use Class Type (CT) to map TC to PHB. Since there are 581 only eight allowed values of CT, only eight maps of TC to PHB need to 582 be supported. The LSP label can be used directly to find the TC to 583 PHB mapping, as is needed to support [RFC3270] L-LSP. 585 While support for [RFC4124] and not [RFC3270] would allow support for 586 only eight mappings of TC to PHB, it is common to support both and 587 simply state a limit on the number of unique TC to PHB mappings which 588 can be supported. 590 2.1.3. Time Synchronization 592 PTP or NTP may be carried over MPLS [I-D.ietf-tictoc-1588overmpls]. 593 Generally NTP will be carried within IP with IP carried in MPLS 594 [RFC5905]. Both PTP and NTP benefit from accurate time stamping of 595 incoming packets and the ability to insert accurate time stamps in 596 outgoing packets. PTP correction which occurs when forwarding 597 requires updating a timestamp compensation field based on the 598 difference between packet arrival at an LSR and packet transmit time 599 at that same LSR. 601 Since the label stack depth may vary, hardware should allow a 602 timestamp to be placed in an outgoing packet at any specified byte 603 position. It may be necessary to modify layer-2 checksums or frame 604 check sequences after insertion. PTP and NTP timestamp formats 605 differ slightly. If NTP or PTP is carried over UDP/IP or UDP/IP/ 606 MPLS, the UDP checksum will also have to be updated. 608 Accurate time synchronization in addition to being generally useful 609 is required for MPLS-TP delay measurement (DM) OAM. See 610 Section 2.6.4. 612 2.1.4. Uses of Multiple Label Stack Entries 614 MPLS deployments in the early part of the prior decade (circa 2000) 615 tended to support either LDP or RSVP-TE. LDP was favored by some for 616 its ability to scale to a very large number of PE devices at the edge 617 of the network, without adding deployment complexity. RSVP-TE was 618 favored, generally in the network core, where traffic engineering and 619 /or fast reroute were considered important. 621 Both LDP and RSVP-TE are used simultaneously within major Service 622 Provider networks using a technique known as "LDP over RSVP-TE 623 Tunneling". This technique allows service providers to carry LDP 624 tunnels inside RSVP-TE tunnels. This makes it possible to take 625 advantage of the Traffic Engineering and Fast Re-Route on more 626 expensive Inter-City and Inter-Continental transport paths. The 627 ingress RSVP-TE PEs places many LDP tunnels on a single RSVP-TE LSP 628 and carries it to the egress RSVP-TE PE. The LDP PEs are situated 629 further from the core, for example within a metro network. LDP over 630 RSVP-TE tunneling requires a minimum of two MPLS labels: one each for 631 LDP and RSVP-TE. 633 The use of MPLS FRR [RFC4090] might add one more label to MPLS 634 traffic, but only when FRR protection is in use (active). If LDP 635 over RSVP-TE is in use, and FRR protection is in use, then at least 636 three MPLS labels are present on the label stack on the links through 637 which the Bypass LSP traverses. FRR is covered in Section 2.1.7. 639 LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN 640 services that are deployed by the vast majority of service providers. 641 These VPN services added yet another label, bringing the label stack 642 depth (when FRR is active) to four. 644 Pseudowires and VPN are discussed in further detail in Section 2.1.8 645 and Section 2.1.9. 647 MPLS hierarchy as described in [RFC4206] can in principle add up to 648 four additional labels. MPLS hierarchy is discussed in 649 Section 2.1.6. 651 Other features such as Entropy Label (discussed in Section 2.4.4) and 652 Flow Label (discussed in Section 2.4.3) can add additional labels to 653 the label stack. 655 Although theoretical scenarios can easily result in eight or more 656 labels, such cases are rare if they occur at all today. For the 657 purpose of forwarding, only the top label needs to be examined if PHP 658 is used, a few more if UHP is used (see Section 2.5). For deep label 659 stacks, quite a few labels may have to be examined for the purpose of 660 load balancing across parallel links (see Section 2.4), however this 661 depth can be bounded by a provider through use of Entropy Label. 663 2.1.5. MPLS Link Bundling 665 MPLS Link Bundling was the first RFC to address the need for multiple 666 parallel links between nodes [RFC4201]. MPLS Link Bundling is 667 notable in that it tried not to change MPLS forwarding, except in 668 specifying the "All-Ones" component link. MPLS Link Bundling is 669 seldom if ever deployed. Instead multipath techniques described in 670 Section 2.4 are used. 672 2.1.6. MPLS Hierarchy 674 MPLS hierarchy is defined in [RFC4206]. Although RFC4206 is 675 considered part of GMPLS, the Packet Switching Capable (PSC) portion 676 of the MPLS hierarchy are applicable to MPLS and may be supported in 677 an otherwise GMPLS free implementation. The MPLS PSC hierarchy 678 remains the most likely means of providing further scaling in an 679 RSVP-TE MPLS network, particularly where the network is designed to 680 provide RSVP-TE connectivity to the edges. This is the case for 681 envisioned MPLS-TP networks. The use of the MPLS PSC hierarchy can 682 add as many as four labels to a label stack, though it is likely that 683 only one layer of PSC will be used in the near future. 685 2.1.7. MPLS Fast Reroute (FRR) 687 Fast reroute is defined by [RFC4090]. Two significantly different 688 methods are defined in RFC4090, the "One-to-One Backup" method which 689 uses the "Detour LSP" and the " Facility Backup" which uses a "bypass 690 tunnel". These are commonly referred to as the detour and bypass 691 methods respectively. 693 The detour method makes use of a presignaled LSP. Hardware 694 assistance is needed for detour FRR only if necessary to accomplish 695 local repair of a large number of LSP within the 10s of milliseconds 696 target. For each affected LSP a swap operation must be reprogrammed 697 or otherwise switched over. The use of detour FRR doubles the number 698 of LSP terminating at any given hop and will increase the number of 699 LSP within a network by a factor dependent on the average detour path 700 length. 702 The bypass method makes use of a tunnel that is unused when no fault 703 exists but may carry many LSP when a local repair is required. There 704 is no presignaling indicating which working LSP will be diverted into 705 any specific bypass LSP. The merge LSR (egress LSR of the bypass 706 LSP) MUST use platform label space (as defined in [RFC3031]) so that 707 an LSP working path on any give interface can be backed up using a 708 bypass LSP terminating on any other interface. Hardware assistance 709 is needed if necessary to accomplish local repair of a large number 710 of LSP within the 10s of milliseconds target. For each affected LSP 711 a swap operation must be reprogrammed or otherwise switched over with 712 an additional push of the bypass LSP label. The use of platform 713 label space impacts the size of the LSR ILM for LSR with a very large 714 number of interfaces. 716 2.1.8. Pseudowire Encapsulation 718 The pseudowire (PW) architecture is defined in [RFC3985]. A 719 pseudowire, when carried over MPLS, adds one or more additional label 720 entries to the MPLS label stack. A PW Control Word is defined in 721 [RFC4385] with motivation for defining the control word in [RFC4928]. 722 The PW Associated Channel defined in [RFC4385] is used for OAM in 723 [RFC5085]. The PW Flow Label is defined in [RFC6391] and is 724 discussed further in this document in Section 2.4.3. 726 There are numerous pseudowire encapsulations, supporting emulation of 727 services such as Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over 728 packet switched networks (PSNs) using IP or MPLS. 730 The pseudowire encapsulation is out of scope for this document. 731 Pseudowire impact on MPLS forwarding at midpoint LSR is within scope. 732 The impact on ingress MPLS push and egress MPLS UHP pop are within 733 scope. While pseudowire encapsulation is out of scope, some advice 734 is given on sequence number support. 736 2.1.8.1. Pseudowire Sequence Number 738 Pseudowire (PW) sequence number support is most important for PW 739 payload types with a high expectation of in-order delivery. 740 Resequencing support, rather than dropping at egress on out of order 741 arrival, is most important for PW payload types with a high 742 expectation of lossless delivery. For example, TDM payloads require 743 sequence number support and require resequencing support. The same 744 is true of ATM CBR service. ATM VBR or ABR may have somewhat relaxed 745 requirements, but generally require ATM Early Packet Discard (EPD) or 746 ATM Partial Packet Discard (PPD) [ATM-EPD-and-PPD]. Though sequence 747 number support and resequencing support are beneficial to PW packet 748 oriented payloads such as FR and Ethernet, they are highly desirable 749 but not as strongly required. 751 Packet reorder should be rare except in a small number of 752 circumstances, most of which are due to network design or equipment 753 design errors: 755 1. The most common case is where reordering occurs is rare, 756 occurring only when a network or equipment fault forces traffic 757 on a new path with different delay. The packet loss that 758 accompanies a network or equipment fault is generally more 759 disruptive than any reordering which may occur. 761 2. A path change can be caused by reasons other than a network or 762 equipment fault, such as administrative routing change. This may 763 result in packet reordering but generally without any packet 764 loss. 766 3. If the edge is not using pseudowire control word (CW) and the 767 core is using multipath, reordering will be far more common. If 768 this is occurring, the best solution is to use CW on the edge, 769 rather than try to fix the reordering using resequencing. 771 4. Another avoidable case is where some core equipment has multipath 772 and for some reason insists on periodically installing a new 773 random number as the multipath hash seed. If supporting MPLS-TP, 774 equipment MUST provide a means to disable periodic hash reseeding 775 and deployments MUST disable periodic hash reseeding. Even if 776 not supporting MPLS-TP, equipment should provide a means to 777 disable periodic hash reseeding and deployments should disable 778 periodic hash reseeding. 780 2.1.9. Layer-2 and Layer-3 VPN 782 Layer-2 VPN [RFC4664] and Layer-3 VPN [RFC4110] add one or more label 783 entry to the MPLS label stack. VPN encapsulations are out of scope 784 for this document. Its impact on forwarding at midpoint LSR are 785 within scope. 787 Any of these services may be used on an MPLS entropy label enabled 788 ingress and egress (see Section 2.4.4 for discussion of entropy 789 label) which would add an additional two labels to the MPLS label 790 stack. The need to provide a useful entropy label value impacts the 791 requirements of the VPN ingress LER but is out of scope for this 792 document. 794 2.2. MPLS Multicast 796 MPLS Multicast encapsulation is clarified in [RFC5332]. MPLS 797 Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388]. 799 [RFC4875] defines a root initiated RSVP-TE LSP setup rather than leaf 800 initiated join used in IP multicast. [RFC6388] defines a leaf 801 initiated LDP setup. Both [RFC4875] and [RFC6388] define point to 802 multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint to 803 multipoint (MP2MP) LSP setup. 805 The P2MP LSP have a single source. An LSR may be a leaf node, an 806 intermediate node, or a "bud" node. A bud serves as both a leaf and 807 intermediate. At a leaf an MPLS pop is performed. The payload may 808 be a IP Multicast packet that requires further replication. At an 809 intermediate node a MPLS swap operation is performed. The bud 810 requires that both a pop operation and a swap operation be performed 811 for the same incoming packet. 813 One strategy to support P2MP functionality is to pop at the LSR 814 interface serving as ingress to the P2MP traffic and then optionally 815 push labels at each LSR interface serving as egress to the P2MP 816 traffic at that same LSR. A given LSR egress chip may support 817 multiple egress interfaces, each of which requires a copy, but each 818 with a different set of added labels and layer-2 encapsulation. Some 819 physical interfaces may have multiple sub-interfaces (such as 820 Ethernet VLAN or channelized interfaces) each requiring a copy. 822 If packet replication is performed at LSR ingress, then the ingress 823 interface performance may suffer. If the packet replication is 824 performed within a LSR switching fabric and at LSR egress, congestion 825 of egress interfaces cannot make use of backpressure to ingress 826 interfaces using techniques such as virtual output queuing (VOQ). If 827 buffering is primarily supported at egress, then the need for 828 backpressure is minimized. There may be no good solution for high 829 volumes of multicast traffic if VOQ is used. 831 MP2MP LSP differ in that any branch may provide an input, including a 832 leaf. Packets must be replicated onto all other branches. This 833 forwarding is often implemented as multiple P2MP forwarding trees, 834 one for each potential input interface at a given LSR. 836 2.3. Packet Rates 837 While average packet size of Internet traffic may be large, long 838 sequences of small packets have both been predicted in theory and 839 observed in practice. Traffic compression and TCP ACK compression 840 can conspire to create long sequences of packets of 40-44 bytes in 841 payload length. If carried over Ethernet, the 64 byte minimum 842 payload applies, yielding a packet rate of approximately 150 Mpps 843 (million packets per second) for the duration of the burst on a 844 nominal 100 Gb/s link. The peak rate for other encapsulations can be 845 as high as 250 Mpps (for example IP or MPLS encapsulated using GFP 846 over OTN ODU4). 848 It is possible that the packet rates achieved by a specific 849 implementation is acceptable for a minimum payload size, such as 64 850 byte (64B) payload for Ethernet, but the acheived rate declines to an 851 unacceptable level for other packet sizes, such as 65B payload. 852 There are other packet rates of interest besides TCP ACK. For 853 example, a TCP ACK carried over an Ethernet PW over MPLS over 854 Ethernet may occupy 82B or 82B plus an increment of 4B if additional 855 MPLS labels are present. 857 A graph of packet rate vs. packet size often displays a sawtooth. 858 The sawtooth is commonly due to a memory bottleneck and memory 859 widths, sometimes internal cache, but often a very wide external 860 buffer memory interface. In some cases it may be due to a fabric 861 transfer width. A fine packing, rounding up to the nearest 8B or 16B 862 will result in a fine sawtooth with small degradation for 65B, and 863 even less for 82B packets. A course packing, rounding up to 64B can 864 yield a sharper drop in performance for 65B packets, or perhaps more 865 important, a larger drop for 82B packets. 867 The loss of some TCP ACK packets are not the primary concern when 868 such a burst occurs. When a burst occurs, any other packets, 869 regardless of packet length and packet QoS are dropped once on-chip 870 input buffers prior to the decision engine are exceeded. Buffers in 871 front of the packet decision engine are often very small or non- 872 existent (less than one packet of buffer) causing significant QoS 873 agnostic packet drop. 875 Internet service providers and content providers at one time 876 specified full rate forwarding with 40 byte payload packets as a 877 requirement. Today, this requirement often can be waived if the 878 provider can be convinced that when long sequence of short packets 879 occur no packets will be dropped. 881 Many equipment suppliers have pointed out that the extra cost in 882 designing hardware capable of processing the minimum size packets at 883 full line rate is significant for very high speed interfaces. If 884 hardware is not capable of processing the minimum size packets at 885 full line rate, then that hardware MUST be capable of handling large 886 burst of small packets, a condition which is often observed. This 887 level of performance is necessary to meet Differentiated Services 888 [RFC2475] requirements for without it, packets are lost prior to 889 inspection of the IP DSCP field [RFC2474] or MPLS TC field [RFC5462]. 891 With adequate on-chip buffers before the packet decision engine, an 892 LSR can absorb a long sequence of short packets. Even if the output 893 is slowed to the point where light congestion occurs, the packets, 894 having cleared the decision process, can make use of larger VOQ or 895 output side buffers and be dealt with according to configured QoS 896 treatment, rather than dropped completely at random. 898 These on-chip buffers need not contribute significant delay since 899 they are only used when the packet decision engine is unable to keep 900 up, not in response to congestion, plus these buffers are quite 901 small. For example, an on-chip buffer capable of handling 4K packets 902 of 64 bytes in length, or 256KB, corresponds to 2 msec on a 10 Mb/s 903 link and 0.2 usec on a 100 Gb/s link. If the packet decision engine 904 is capable of handling packets at 90% of the full rate for small 905 packets, then the maximum added delay is 0.2 msec and 20 nsec 906 respectively, and this delay only applies if a 4K burst of short 907 packets occurs. When no burst of short packets was being processed, 908 no delay is added. 910 Packet rate requirements apply regardless of which network tier 911 equipment is deployed in. Whether deployed in the network core or 912 near the network edges, one of the two conditions MUST be met if 913 Differentiated Services requirements are to be met: 915 1. Packets must be processed at full line rate with minimum sized 916 packets. -OR- 918 2. Packets must be processed at a rate well under generally accepted 919 average packet sizes, with sufficient buffering prior to the 920 packet decision engine to accommodate long bursts of small 921 packets. 923 2.4. MPLS Multipath Techniques 925 In any large provider, service providers and content providers, hash 926 based multipath techniques are used in the core and in the edge. In 927 many of these providers hash based multipath is also used in the 928 larger metro networks. 930 The most common multipath techniques are ECMP applied at the IP 931 forwarding level, Ethernet LAG with inspection of the IP payload, and 932 multipath on links carrying both IP and MPLS, where the IP header is 933 inspected below the MPLS label stack. In most core networks, the 934 vast majority of traffic is MPLS encapsulated. 936 In order to support an adequately balanced load distribution across 937 multiple links, IP header information must be used. Common practice 938 today is to reinspect the IP headers at each LSR and use the label 939 stack and IP header information in a hash performed at each LSR. 940 Further details are provided in Section 2.4.5. 942 The use of this technique is so ubiquitous in provider networks that 943 lack of support for multipath makes any product unsuitable for use in 944 large core networks. This will continue to be the case in the near 945 future, even as deployment of MPLS entropy label begins to relax the 946 core LSR multipath performance requirements given the existing 947 deployed base of edge equipment without the ability to add an entropy 948 label. 950 A generation of edge equipment supporting the ability to add an MPLS 951 entropy label is needed before the performance requirements for core 952 LSR can be relaxed. However, it is likely that two generations of 953 deployment in the future will allow core LSR to support full packet 954 rate only when a relatively small number of MPLS labels need to be 955 inspected before hashing. For now, don't count on it. 957 Common practice today is to reinspect the packet at each LSR and use 958 information from the packet combined plus a hash seed that is 959 selected by each LSR. Where flow labels or entropy labels are used, 960 a hash seed must be used when creating these labels. 962 2.4.1. Pseudowire Control Word 964 Within the core of a network some form of multipath is almost certain 965 to be used. Multipath techniques deployed today are likely to be 966 looking beneath the label stack for an opportunity to hash on IP 967 addresses. 969 A pseudowire encapsulated at a network edge must have a means to 970 prevent reordering within the core if the pseudowire will be crossing 971 a network core, or any part of a network topology where multipath is 972 used (see [RFC4385] and [RFC4928]). 974 Not supporting the ability to encapsulate a pseudowire with a control 975 word may lock a product out from consideration. A pseudowire 976 capability without control word support might be sufficient for 977 applications that are strictly both intra-metro and low bandwidth. 978 However a provider with other applications will very likely not 979 tolerate having equipment which can only support a subset of their 980 pseudowire needs. 982 2.4.2. Large Microflows 984 Where multipath makes use of a simple hash and simple load balance 985 such as modulo or other fixed allocation (see Section 2.4) the 986 presence of large microflows that each consumes 10% of the capacity 987 of a component link of a potentially congested composite link, one 988 such microflow can upset the traffic balance and more than one can in 989 effect reduce the effective capacity of the entire composite link by 990 more than 10%. 992 When even a very small number of large microflows are present, there 993 is a significant probability that more than one of these large 994 microflows could fall on the same component link. If the traffic 995 contribution from large microflows is small, the probability for 996 three or more large microflows on the same component link drops 997 significantly. Therefore in a network where a significant number of 998 parallel 10 Gb/s links exists, even a 1 Gb/s pseudowire or other 999 large microflow that could not otherwise be subdivided into smaller 1000 flows should carry a flow label or entropy label if possible. 1002 Active management of the hash space to better accommodate large 1003 microflows has been implemented and deployed in the past, however 1004 such techniques are out of scope for this document. 1006 2.4.3. Pseudowire Flow Label 1008 Unlike a pseudowire control word, a pseudowire flow label [RFC6391], 1009 is required only for relatively large capacity pseudowires. There 1010 are many cases where a pseudowire flow label makes sense. Any 1011 service such as a VPN which carries IP traffic within a pseudowire 1012 can make use of a pseudowire flow label. 1014 Any pseudowire carried over MPLS which makes use of the pseudowire 1015 control word and does not carry a flow label is in effect a single 1016 microflow (in [RFC2475] terms) and may result in the types of 1017 problems described in Section 2.4.2. 1019 2.4.4. MPLS Entropy Label 1021 The MPLS entropy label simplifies flow group identification [RFC6790] 1022 at midpoint LSR. Prior to the MPLS entropy label midpoint LSR needed 1023 to inspect the entire label stack and often the IP headers to provide 1024 an adequate distribution of traffic when using multipath techniques 1025 (see Section 2.4.5). With the use of MPLS entropy label, a hash can 1026 be performed closer to network edges, placed in the label stack, and 1027 used by midpoint LSR without fully reinspecting the label stack and 1028 inspecting the payload. 1030 The MPLS entropy label is capable of avoiding full label stack and 1031 payload inspection within the core where performance levels are most 1032 difficult to achieve (see Section 2.3). The label stack inspection 1033 can be terminated as soon as the first entropy label is encountered, 1034 which is generally after a small number of labels are inspected. 1036 In order to provide these benefits in the core, LSR closer to the 1037 edge must be capable of adding an entropy label. This support may 1038 not be required in the access tier, the tier closest to the customer, 1039 but is likely to be required in the edge or the border to the network 1040 core. LSR peering with external networks will also need to be able 1041 to add an entropy label on incoming traffic. 1043 2.4.5. Fields Used for Multipath Load Balance 1045 The most common multipath techniques are based on a hash over a set 1046 of fields. Regardless of whether a hash is used or some other method 1047 is used, the there is a limited set of fields which can safely be 1048 used for multipath. 1050 2.4.5.1. MPLS Fields in Multipath 1052 If the "outer" or "first" layer of encapsulation is MPLS, then label 1053 stack entries are used in the hash. Within a finite amount of time 1054 (and for small packets arriving at high speed that time can be quite 1055 limited) only a finite number of label entries can be inspected. 1056 Pipelined or parallel architectures improve this, but the limit is 1057 still finite. 1059 The following guidelines are provided for use of MPLS fields in 1060 multipath load balancing. 1062 1. Only the 20 bit label field SHOULD be used. The TTL field SHOULD 1063 NOT be used. The S bit MUST NOT be used. The TC field (formerly 1064 EXP) MUST NOT be used. See text following this list for reasons. 1066 2. If an ELI label is found, then if the LSR supports entropy label, 1067 the EL label field in the next label entry (the EL) SHOULD be 1068 used and label entries below that label SHOULD NOT be used and 1069 the MPLS payload SHOULD NOT be used. See below this list for 1070 reasons. 1072 3. Special purpose labels (label values 0-15) MUST NOT be used. 1073 Extended special purpose labels (any label following label 15) 1074 MUST NOT be used. In particular, GAL and RA MUST NOT be used so 1075 that OAM traffic follows the same path as payload packets with 1076 the same label stack. 1078 4. The most entropy is generally found in the label stack entries 1079 near the bottom of the label stack (innermost label, closest to 1080 S=1 bit). If the entire label stack cannot be used (or entire 1081 stack up to an EL), then it is better to use as many labels as 1082 possible closest to the bottom of stack. 1084 5. If no ELI is encountered, and the first nibble of payload 1085 contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support 1086 the ability to interpret the payload as IPv4 or IPv6 and extract 1087 and use appropriate fields from the IP headers. This feature is 1088 considered a hard requirement by many service providers. If 1089 supported, there MUST be a way to disable it (if, for example, PW 1090 without CW are used). This ability to disable this feature is 1091 considered a hard requirement by many service providers. 1092 Therefore an implementation has a very strong incentive to 1093 support both options. 1095 6. A label which is popped at egress (UHP pop) SHOULD NOT be used. 1096 A label which is popped at the penultimate hop (PHP pop) SHOULD 1097 be used. 1099 Apparently some chips have made use of the TC (formerly EXP) bits as 1100 a source of entropy. This is very harmful since it will reorder 1101 Assured Forwarding (AF) traffic [RFC2597] when a subset does not 1102 conform to the configured rates and is remarked but not dropped at a 1103 prior LSR. Traffic which uses MPLS ECN [RFC5129] can also be 1104 reordered if TC is used for entropy. Therefore, as stated in the 1105 guidelines above, the TC field (formerly EXP) MUST NOT be used in 1106 multipath load balancing as it violates Differentiated Services 1107 Ordered Aggregate (OA) requirements in these two instances. 1109 Use of the MPLS label entry S bit would result in putting OAM traffic 1110 on a different path if the addition of a GAL at the bottom of stack 1111 removed the S bit from the prior label. 1113 If an ELI label is found, then if the LSR supports entropy label, the 1114 EL label field in the next label entry (the EL) SHOULD be used and 1115 the search for additional entropy within the packet SHOULD be 1116 terminated. Failure to terminate the search will impact client MPLS- 1117 TP LSP carried within server MPLS LSP. A network operator has the 1118 option to use administrative attributes as a means to identify LSR 1119 which do not terminate the entropy search at the first EL. 1120 Administrative attributes are defined in [RFC3209]. Some 1121 configuration is required to support this. 1123 If the label removed by a PHP pop is not used, then for any PW for 1124 which CW is used, there is no basis for multipath load split. In 1125 some networks it is infeasible to put all PW traffic on one component 1126 link. Any PW which does not use CW will be improperly split 1127 regardless of whether the label removed by a PHP pop is used. 1128 Therefore the PHP pop label SHOULD be used as recommended above. 1130 2.4.5.2. IP Fields in Multipath 1132 Inspecting the IP payload provides the most entropy in provider 1133 networks. The practice of looking past the bottom of stack label for 1134 an IP payload is well accepted and documented in [RFC4928] and in 1135 other RFCs. 1137 Where IP is mentioned in the document, both IPv4 and IPv6 apply. All 1138 LSRs MUST fully support IPv6. 1140 When information in the IP header is used, the following guidelines 1141 apply: 1143 1. Both the IP source address and IP destination address SHOULD be 1144 used. There MAY be an option to reverse the order of these 1145 addresses, improving the ability to provide symmetric paths in 1146 some cases. Many service providers require that both addresses 1147 be used. 1149 2. Implementations SHOULD allow inspection of the IP protocol field 1150 and use of the UDP or TCP port numbers. For many service 1151 providers this feature is considered mandatory, particularly for 1152 enterprise, data center, or edge equipment. If this feature is 1153 provided, it SHOULD be possible to disable use of TCP and UDP 1154 ports. Many service providers consider it a hard requirement 1155 that use of UDP and TCP ports can be disabled. Therefore there 1156 is a stong incentive for implementations to provide both options. 1158 3. Equipment suppliers MUST NOT make assumptions that because the IP 1159 version field is equal to 4 (an IPv4 packet) that the IP protocol 1160 will either be TCP (IP protocol 6) or UDP (IP protocol 17) and 1161 blindly fetch the data at the offset where the TCP or UDP ports 1162 would be found. With IPv6, TCP and UDP port numbers are not at 1163 fixed offsets. With IPv4 packets carrying IP options, TCP and 1164 UDP port numbers are not at fixed offsets. 1166 4. The IPv6 header flow field SHOULD be used. This is the explicit 1167 purpose of the IPv6 flow field, however observed flow fields 1168 rarely contains a non-zero value. Some uses of the flow field 1169 have been defined such as [RFC6438]. In the absence of MPLS 1170 encapsulation, the IPv6 flow field can serve a role equivalent to 1171 entropy label. 1173 5. Support for other protocols that share a common Layer-4 header 1174 such as RTP, UDP-lite, SCTP and DCCP SHOULD be provided, 1175 particularly for edge or access equipment where additional 1176 entropy may be needed. Equipment SHOULD also use RTP, UDP-lite, 1177 SCTP and DCCP headers when creating an entropy label. 1179 6. The following IP header fields should not or must not be used: 1181 a. Similar to avoiding TC in MPLS, the IP DSCP, and ECN bits 1182 MUST NOT be used. 1184 b. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used. 1186 c. Note that the IP TOS field was deprecated ([RFC0791] was 1187 updated by [RFC2474]). No part of the IP DSCP field can be 1188 used (formerly IP PREC and IP TOS bits). 1190 7. Some IP encapsulations support tunneling, such as IP-in-IP, GRE, 1191 L2TPv3, and IPSEC. These provide a greater source of entropy 1192 which some provider networks carrying large amounts of tunneled 1193 traffic may need. The use of tunneling header information is out 1194 of scope for this document. 1196 This document makes the following recommendations. These 1197 recommendations are not required to claim compliance to any existing 1198 RFC therefore implementers are free to ignore them, but due to 1199 service provider requirements should consider the risk of doing so. 1200 The use of IP addresses MUST be supported and TCP and UDP ports 1201 (conditional on the protocol field and properly located) MUST be 1202 supported. The ability to disable use of UDP and TCP ports MUST be 1203 available. Though potentially very useful in some networks, it is 1204 uncommon to support using payloads of tunneling protocols carried 1205 over IP. Though the use of tunneling protocol header information is 1206 out of scope for this document, it is not discouraged. 1208 2.4.5.3. Fields Used in Flow Label 1210 The ingress to a pseudowire (PW) can extract information from the 1211 payload being encapsulated to create a flow label. [RFC6391] 1212 references IP carried in Ethernet as an example. The Native Service 1213 Processing (NSP) function defined in [RFC3985] differs with 1214 pseudowire type. It is in the NSP function where information for a 1215 specific type of PW can be extracted for use in a flow label. Which 1216 fields to use for any given PW NSP is out of scope for this document. 1218 2.4.5.4. Fields Used in Entropy Label 1219 An entropy label is added at the ingress to an LSP. The payload 1220 being encapsulated is most often MPLS, a PW, or IP. The payload type 1221 is identified by the layer-2 encapsulation (Ethernet, GFP, POS, etc). 1223 If the payload is MPLS, then the information used to create an 1224 entropy label is the same information used for local load balancing 1225 (see Section 2.4.5.1). This information MUST be extracted for use in 1226 generating an entropy label even if the LSR local egress interface is 1227 not a multipath. 1229 Of the non-MPLS payload types, only payloads that are forwarded are 1230 of interest. For example, ARP is not forwarded and CNLP (used only 1231 for ISIS) is not forwarded. 1233 The non-MPLS payload type of greatest interest are IPv4 and IPv6. 1234 The guidelines in Section 2.4.5.2 apply to fields used to create and 1235 entropy label. 1237 The IP tunneling protocols mentioned in Section 2.4.5.2 may be more 1238 applicable to generation of an entropy label at edge or access where 1239 deep packet inspection is practical due to lower interface speeds 1240 than in the core where deep packet inspection may be impractical. 1242 2.5. MPLS-TP and UHP 1244 MPLS-TP introduces forwarding demands that will be extremely 1245 difficult to meet in a core network. Most troublesome is the 1246 requirement for Ultimate Hop Popping (UHP, the opposite of 1247 Penultimate Hop Popping or PHP). Using UHP opens the possibility of 1248 one or more MPLS pop operation plus an MPLS swap operation for each 1249 packet. The potential for multiple lookups and multiple counter 1250 instances per packet exists. 1252 As networks grow and tunneling of LDP LSPs into RSVP-TE LSPs is used, 1253 and/or RSVP-TE hierarchy is used, the requirement to perform one or 1254 two or more MPLS pop operations plus a MPLS swap operation (and 1255 possibly a push or two) increases. If MPLS-TP LM (link monitoring) 1256 OAM is enabled at each layer, then a packet and byte count MUST be 1257 maintained for each pop and swap operation so as to offer OAM for 1258 each layer. 1260 2.6. Local Delivery of Packets 1262 There are a number of situations in which packets are destined to a 1263 local address or where a return packet must be generated. There is a 1264 need to mitigate the potential for outage as a result of either 1265 attacks on network infrastructure, or in some cases unintentional 1266 misconfiguration resulting in processor overload. Some hardware 1267 assistance is needed for all traffic destined to the general purpose 1268 CPU that is used in MPLS control protocol processing or network 1269 management protocol processing and in most cases to other general 1270 purpose CPUs residing on an LSR. This is due to the ease of 1271 overwhelming such a processor with traffic arriving on LSR high speed 1272 interfaces, whether the traffic is malicious or not. 1274 Denial of service (DoS) protection is an area requiring hardware 1275 support that is often overlooked or inadequately considered. 1276 Hardware assist is also needed for OAM, particularly the more 1277 demanding MPLS-TP OAM. 1279 2.6.1. DoS Protection 1281 Modern equipment supports a number of control plane and management 1282 plane protocols. Generally no single means of protecting network 1283 equipment from denial of service (DoS) attacks is sufficient, 1284 particularly for high speed interfaces. This problem is not specific 1285 to MPLS, but is a topic that cannot be ignored when implementing or 1286 evaluating MPLS implementations. 1288 Two types of protections are often cited as primary means of 1289 protecting against attacks of all kinds. 1291 Isolated Control/Management Traffic 1292 Control and Management traffic can be carried out-of-band (OOB), 1293 meaning not intermixed with payload. For MPLS, use of G-ACh and 1294 GAL to carry control and management traffic provides a means of 1295 isolation from potentially malicious payload. Used alone, the 1296 compromise of a single node, including a small computer at a 1297 network operations center, could compromise an entire network. 1298 Implementations which send all G-ACh/GAL traffic directly to a 1299 routing engine CPU are subject to DoS attack as a result of such 1300 a compromise. 1302 Cryptographic Authentication 1303 Cryptographic authentication can very effectively prevent 1304 malicious injection of control or management traffic. 1305 Cryptographic authentication can is some circumstances be subject 1306 to DoS attack by overwhelming the capacity of the decryption with 1307 a high volume of malicious traffic. For very low speed 1308 interfaces, cryptographic authentication can be performed by the 1309 general purpose CPU used as a routing engine. For all other 1310 cases, cryptographic hardware may be needed. For very high speed 1311 interfaces, even cryptographic hardware can be overwhelmed. 1313 Some control and management protocols are often carried with payload 1314 traffic. This is commonly the case with BGP, T-LDP, and SNMP. It is 1315 often the case with RSVP-TE. Even when carried over G-ACh/GAL 1316 additional measures can reduce the potential for a minor breach to be 1317 leveraged to a full network attack. 1319 Some of the additional protections are supported by hardware packet 1320 filtering. 1322 GTSM 1323 [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop 1324 Limit fields to insure control traffic that can only originate 1325 from an immediate neighbor is not forged and originating from a 1326 distant source. GTSM can be applied to many control protocols 1327 which are routable, for example LDP [RFC6720]. 1329 IP Filtering 1330 At the very minimum, packet filtering plus classification and use 1331 of multiple queues supporting rate limiting is needed for traffic 1332 that could potentially be sent to a general purpose CPU used as a 1333 routing engine. The first level of filtering only allows 1334 connections to be initiated from specific IP prefixes to specific 1335 destination ports and then preferably passes traffic directly to 1336 a cryptographic engine and/or rate limits. The second level of 1337 filtering passes connected traffic, such as TCP connections 1338 having received at least one authenticated SYN or having been 1339 locally initiated. The second level of filtering only passes 1340 traffic to specific address and port pairs to be checked for 1341 cryptographic authentication. 1343 The cryptographic authentication is generally the last resort in DoS 1344 attack mitigation. If a packet must be first sent to a general 1345 purpose CPU, then sent to a cryptographic engine, a DoS attack is 1346 possible on high speed interfaces. Only where hardware can identify 1347 a signature and the portion of packet covered by the signature is 1348 cryptographic authentication highly beneficial in protecting against 1349 DoS attacks. 1351 For chips supporting multiple 100 Gb/s interfaces, only a very large 1352 number of parallel cryptographic engines can provide the processing 1353 capacity to handle a large scale DoS or distributed DoS (DDoS) 1354 attack. For many forwarding chips this much processing power 1355 requires significant chip real estate and power, and therefore 1356 reduces system space and power density. For this reason, 1357 cryptographic authentication is not considered a viable first line of 1358 defense. 1360 For some networks the first line of defense is some means of 1361 supporting OOB control and management traffic. In the past this OOB 1362 channel might make use of overhead bits in SONET or OTN or a 1363 dedicated DWDM wavelength. G-ACh and GAL provide an alternative OOB 1364 mechanism which is independent of underlying layers. In other 1365 networks, including most IP/MPLS networks, perimeter filtering serves 1366 a similar purpose, though less effective without extreme vigilance. 1368 A second line of defense is filtering, including GTSM. For protocols 1369 such as EBGP, GTSM and other filtering is often the first line of 1370 defense. Cryptographic authentication is usually the last line of 1371 defense and insufficient by itself to mitigate DoS or DDoS attacks. 1373 2.6.2. MPLS OAM 1375 [RFC4377] defines requirements for MPLS OAM that predate MPLS-TP. 1376 [RFC4379] defines what is commonly referred to as LSP Ping and LSP 1377 Traceroute. [RFC4379] is updated by [RFC6424] supporting MPLS 1378 tunnels and stitched LSP and P2MP LSP. [RFC4379] is updated by 1379 [RFC6425] supporting P2MP LSP. [RFC4379] is updated by [RFC6426] to 1380 support MPLS-TP connectivity verification (CV) and route tracing. 1382 [RFC4950] extends the ICMP format to support TTL expiration that may 1383 occur when using IP traceroute within an MPLS tunnel. The ICMP 1384 message generation can be implemented in forwarding hardware, but if 1385 sent to a general purpose CPU must be rate limited to avoid a 1386 potential denial or service (DoS) attack. 1388 [RFC5880] defines Bidirectional Forwarding Detection (BFD), a 1389 protocol intended to detect faults in the bidirectional path between 1390 two forwarding engines. [RFC5884] and [RFC5885] define BFD for MPLS. 1391 BFD can provide failure detection on any kind of path between 1392 systems, including direct physical links, virtual circuits, tunnels, 1393 MPLS Label Switched Paths (LSPs), multihop routed paths, and 1394 unidirectional links as long as there is some return path. 1396 The processing requirements for BFD are less than for LSP Ping, 1397 making BFD somewhat better suited for relatively high rate proactive 1398 monitoring. BFD does not verify that the data plane matches the 1399 control plane, where LSP Ping does. LSP Ping is somewhat better 1400 suited for on-demand monitoring including relatively low rate 1401 periodic verification of data plane and as a diagnostic tool. 1403 Hardware assistance is often provided for BFD response where BFD 1404 setup or parameter change is not involved and may be necessary for 1405 relatively high rate proactive monitoring. If both BFD and LSP Ping 1406 are recognized in filtering prior to passing traffic to a general 1407 purpose CPU, appropriate DoS protection can be applied (see 1408 Section 2.6.1). Failure to recognize BFD and LSP Ping and at least 1409 rate limit creates the potential for misconfiguration to cause 1410 outages rather than cause errors in the misconfigured OAM. 1412 2.6.3. Pseudowire OAM 1414 Pseudowire OAM makes use of the control channel provided by Virtual 1415 Circuit Connectivity Verification (VCCV) [RFC5085]. VCCV makes use 1416 of the Pseudowire Control Word. BFD support over VCCV is defined by 1417 [RFC5885]. [RFC5885] is updated by [RFC6478] in support of static 1418 pseudowires. [RFC4379] is updated by [RFC6829] supporting LSP Ping 1419 for Pseudowire FEC advertised over IPv6. 1421 G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control 1422 channel and applies to any MPLS-TP end points, including Pseudowire. 1423 See Section 2.6.4 for an overview of MPLS-TP OAM. 1425 2.6.4. MPLS-TP OAM 1427 [RFC6669] summarizes the MPLS-TP OAM toolset, the set of protocols 1428 supporting the MPLS-TP OAM requirements specified in [RFC5860] and 1429 supported by the MPLS-TP OAM framework defined in [RFC6371]. 1431 The MPLS-TP OAM toolset includes: 1433 CC-CV 1434 [RFC6428] defines BFD extensions to support proactive 1435 Connectivity Check and Connectivity Verification (CC-CV) 1436 applications. [RFC6426] provides LSP ping extensions that are 1437 used to implement on-demand connectivity verification. 1439 RDI 1440 Remote Defect Indication (RDI) is triggered by failure of 1441 proactive CC-CV, which is BFD based. For fast RDI initiation, 1442 RDI SHOULD be initiated and handled by hardware if BFD is handled 1443 in forwarding hardware. [RFC6428] provides an extension for BFD 1444 that includes the RDI indication in the BFD format and a 1445 specification of how this indication is to be used. 1447 Route Tracing 1448 [RFC6426] specifies that the LSP ping enhancements for MPLS-TP 1449 on-demand connectivity verification include information on the 1450 use of LSP ping for route tracing of an MPLS-TP path. 1452 Alarm Reporting 1453 [RFC6427] describes the details of a new protocol supporting 1454 Alarm Indication Signal (AIS), Link Down Indication, and fault 1455 management. Failure to support this functionality in forwarding 1456 hardware can potentially result in failure to meet protection 1457 recovery time requirements and is therefore strongly recommended. 1459 Lock Instruct 1460 Lock instruct is initiated on-demand and therefore need not be 1461 implemented in forwarding hardware. [RFC6435] defines a lock 1462 instruct protocol. 1464 Lock Reporting 1465 [RFC6427] covers lock reporting. Lock reporting need not be 1466 implemented in forwarding hardware. 1468 Diagnostic 1469 [RFC6435] defines protocol support for loopback. Loopback 1470 initiation is on-demand and therefore need not be implemented in 1471 forwarding hardware. Loopback of packet traffic SHOULD be 1472 implemented in forwarding hardware on high speed interfaces. 1474 Packet Loss and Delay Measurement 1475 [RFC6374] and [RFC6375] define a protocol and profile for packet 1476 loss measurement (LM) and delay measurement (DM). LM requires a 1477 very accurate capture and insertion of packet and byte counters 1478 when a packet is transmitted and capture of packet and byte 1479 counters when a packet is received. This capture and insertion 1480 MUST be implemented in forwarding hardware for LM OAM if high 1481 accuracy is needed. DM requires very accurate capture and 1482 insertion of a timestamp on transmission and capture of timestamp 1483 when a packet is received. This timestamp capture and insertion 1484 MUST be implemented in forwarding hardware for DM OAM if high 1485 accuracy is needed. 1487 See Section 2.6.2 for discussion of hardware support necessary for 1488 BFD and LSP Ping. 1490 CC-CV and alarm reporting is tied to protection and therefore SHOULD 1491 be supported in forwarding hardware in order to provide protection 1492 for a large number of affected LSP within target response intervals. 1493 Since CC-CV is supported by BFD, for MPLS-TP providing hardware 1494 assistance for BFD processing helps insure that protection recovery 1495 time requirements can be met even for faults affecting a large number 1496 of LSP. 1498 2.6.5. MPLS OAM and Layer-2 OAM Interworking 1500 [RFC6670] provides the reasons for selecting a single MPLS-TP OAM 1501 solution and examines the consequences were ITU-T to develop a second 1502 OAM solution that is based on Ethernet encodings and mechanisms. 1504 [RFC6310] and [RFC7023] specifies the mapping of defect states 1505 between many types of hardware Attachment Circuits (ACs) and 1506 associated Pseudowires (PWs). This functionality SHOULD be supported 1507 in forwarding hardware. 1509 It is beneficial if an MPLS OAM implementation can interwork with the 1510 underlying server layer and provide a means to interwork with a 1511 client layer. For example, [RFC6427] specifies an inter-layer 1512 propagation of AIS and LDI from MPLS server layer to client MPLS 1513 layers. Where the server layer is a Layer-2, such as Ethernet, PPP 1514 over SONET/SDH, or GFP over OTN, interwork among layers is also 1515 beneficial. For high speed interfaces, supporting this interworking 1516 in forwarding hardware helps insure that protection based on this 1517 interworking can meet recovery time requirements even for faults 1518 affecting a large number of LSP. 1520 2.6.6. Extent of OAM Support by Hardware 1522 Where certain requirements must be met, such as relatively high CC-CV 1523 rates and a large number of interfaces, or strict protection recovery 1524 time requirements and a moderate number of affected LSP, some OAM 1525 functionality must be supported by forwarding hardware. In other 1526 cases, such as highly accurate LM and DM OAM or strict protection 1527 recovery time requirements with a large number of affected LSP, OAM 1528 functionality must be entirely implemented in forwarding hardware. 1530 Where possible, implementation in forwarding hardware should be in 1531 programmable hardware such that if standards are later changed or 1532 extended these changes are likely to be accommodated with hardware 1533 reprogramming rather than replacement. 1535 For some functionality there is a strong case for an implementation 1536 in dedicated forwarding hardware. Examples include packet and byte 1537 counters needed for LM OAM as well as needed for management 1538 protocols. Similarly the capture and insertion of packet and byte 1539 counts or timestamps needed for transmitted LM or DM or time 1540 synchronization packets MUST be implemented in forwarding hardware if 1541 high accuracy is required. 1543 For some functions there is a strong case to provide limited support 1544 in forwarding hardware but may make use of an external general 1545 purpose processor if performance criteria can be met. For example 1546 origination of RDI triggered by CC-CV, response to RDI, and PSC 1547 functionality may be supported by hardware, but expansion to a large 1548 number of client LSP and transmission of AIS or RDI to the client LSP 1549 may occur in a general purpose processor. Some forwarding hardware 1550 supports one or more on-chip general purpose processors which may be 1551 well suited for such a role. 1553 The customer (system supplier or provider) should not dictate design, 1554 but should independently validate target functionality and 1555 performance. However, it is not uncommon for service providers and 1556 system implementers to insist on reviewing design details (under NDA) 1557 due to past experiences with suppliers and to reject suppliers who 1558 are unwilling to provide details. 1560 2.7. Number and Size of Flows 1562 Service provider networks may carry up to hundreds of millions of 1563 flows on 10 Gb/s links. Most flows are very short lived, many under 1564 a second. A subset of the flows are low capacity and somewhat long 1565 lived. When Internet traffic dominates capacity a very small subset 1566 of flows are high capacity and/or very long lived. 1568 Two types of limitations with regard to number and size of flows have 1569 been observed. 1571 1. Some hardware cannot handle some very large flows because of 1572 internal paths which are limited, such as per packet backplane 1573 paths or paths internal or external to chips such as buffer 1574 memory paths. Such designs can handle aggregates of smaller 1575 flows. Some hardware with acknowledged limitations has been 1576 successfully deployed but may be increasingly problematic if the 1577 capacity of large microflows in deployed networks continues to 1578 grow. 1580 2. Some hardware approaches cannot handle a large number of flows, 1581 or a large number of large flows due to attempting to count per 1582 flow, rather than deal with aggregates of flows. Hash techniques 1583 scale with regard to number of flows due to a fixed hash size 1584 with many flows falling into the same hash bucket. Techniques 1585 that identify individual flows have been implemented but have 1586 never successfully deployed for Internet traffic. 1588 3. Questions for Suppliers 1590 The following questions should be asked of a supplier. These 1591 questions are grouped into broad categories. The questions 1592 themselves are intended to be an open ended question to the supplier. 1593 The tests in Section 4 are intended to verify whether the supplier 1594 disclosed any compliance or performance limitations completely and 1595 accurately. 1597 3.1. Basic Compliance 1599 Q#1 Can the implementation forward packets with an arbitrarily 1600 large stack depth? What limitations exist, and under what 1601 circumstances do further limitations come into play (such as high 1602 packet rate or specific features enabled or specific types of 1603 packet processing)? See Section 2.1. 1605 Q#2 Is the entire set of basic MPLS functionality described in 1606 Section 2.1 supported? 1608 Q#3 Are the set of MPLS special purpose labels handled correctly 1609 and with adequate performance? Are extended special purpose 1610 labels handled correctly and with adequate performance? See 1611 Section 2.1.1. 1613 Q#4 Are mappings of label value and TC to PHB handled correctly, 1614 including RFC3270 L-LSP mappings and RFC4124 CT mappings to PHB? 1615 See Section 2.1.2. 1617 Q#5 Is time synchronization adequately supported in forwarding 1618 hardware? 1620 a. Are both PTP and NTP formats supported? 1622 b. Is the accuracy of timestamp insertion and incoming stamping 1623 sufficient? 1625 See Section 2.1.3. 1627 Q#6 Is link bundling supported? 1629 a. Can LSP be pinned to specific components? 1631 b. Is the "all-ones" component link supported? 1633 See Section 2.1.5. 1635 Q#7 Is MPLS hierarchy supported? 1637 a. Are both PHP and UHP supported? What limitations exist on 1638 the number of pop operations with UHP? 1640 b. Are the pipe, short-pipe, and uniform models supported? Are 1641 TTL and TC values updated correctly at egress where 1642 applicable? 1644 See Section 2.1.6 1646 Q#8 Are pseudowire sequence numbers handled correctly? See 1647 Section 2.1.8.1. 1649 Q#9 Is VPN LER functionality handled correctly and without 1650 performance issues? See Section 2.1.9. 1652 Q#10 Is MPLS multicast (P2MP and MP2MP) handled correctly? 1653 a. Are packets dropped on uncongested outputs if some outputs 1654 are congested? 1656 b. Is performance limited in high fanout situations? 1658 See Section 2.2. 1660 3.2. Basic Performance 1662 Q#11 Can very small packets be forwarded at full line rate on all 1663 interfaces indefinitely? What limitations exist, and under what 1664 circumstances do further limitations come into play (such as 1665 specific features enabled or specific types of packet 1666 processing)? 1668 Q#12 Customers must decide whether to relax the prior requirement and 1669 to what extent. If the answer to the prior question indicates 1670 that limitations exist, then: 1672 a. What is the smallest packet size where full line rate 1673 forwarding can be supported? 1675 b. What is the longest burst of full rate small packets that can 1676 be supported? 1678 Specify circumstances (such as specific features enabled or 1679 specific types of packet processing) often impact these rates and 1680 burst sizes. 1682 Q#13 How many pop operations can be supported along with a swap 1683 operation at full line rate while maintaining per LSP packet and 1684 byte counts for each pop and swap? This requirement is 1685 particularly relevant for MPLS-TP. 1687 Q#14 How many label push operations can be supported. While this 1688 limitation is rarely an issue, it applies to both PHP and UHP, 1689 unlike the pop limit which applies to UHP. 1691 Q#15 For a worst case where all packets arrive on one LSP, what is 1692 the counter overflow time? Are any means provided to avoid 1693 polling all counters at short intervals? This applies to both 1694 MPLS and MPLS-TP. 1696 3.3. Multipath Capabilities and Performance 1698 Multipath capabilities and performance do not apply to MPLS-TP but 1699 apply to MPLS and apply if MPLS-TP is carried in MPLS. 1701 Q#16 How are large microflows accommodated? Is there active 1702 management of the hash space mapping to output ports? See 1703 Section 2.4.2. 1705 Q#17 How many MPLS labels can be included in a hash based on the MPLS 1706 label stack? 1708 Q#18 Is packet rate performance decreased beyond some number of 1709 labels? 1711 Q#19 Can the IP header and payload information below the MPLS stack 1712 be used in the hash? If so, which IP fields, payload types and 1713 payload fields are supported? 1715 Q#20 At what maximum MPLS label stack depth can Bottom of Stack and 1716 an IP header appear without impacting packet rate performance? 1718 Q#21 Are special purpose labels excluded from the label stack hash? 1719 Are extended purpose labels excluded from the label stack hash? 1720 See Section 2.4.5.1. 1722 Q#22 How is multipath performance affected by very large flows or an 1723 extremely large number of flows, or by very short lived flows? 1724 See Section 2.7. 1726 3.4. Pseudowire Capabilities and Performance 1728 Q#23 Is the pseudowire control word supported? 1730 Q#24 What is the maximum rate of pseudowire encapsulation and 1731 decapsulation? Apply the same questions as in Base Performance 1732 for any packet based pseudowire such as IP VPN or Ethernet. 1734 Q#25 Does inclusion of a pseudowire control word impact performance? 1736 Q#26 Are flow labels supported? 1738 Q#27 If so, what fields are hashed on for the flow label for 1739 different types of pseudowires? 1741 Q#28 Does inclusion of a flow label impact performance? 1743 3.5. Entropy Label Support and Performance 1745 Q#29 Can an entropy label be added when acting as in ingress LER and 1746 can it be removed when acting as an egress LER? 1748 Q#30 If so, what fields are hashed on for the entropy label? 1750 Q#31 Does adding or removing an entropy label impact packet rate 1751 performance? 1753 Q#32 Can an entropy label be detected in the label stack, used in the 1754 hash, and properly terminate the search for further information 1755 to hash on? 1757 Q#33 Does using an entropy label have any negative impact on 1758 performance? It should have no impact or a positive impact. 1760 3.6. DoS Protection 1762 Q#34 For each control and management plane protocol in use, what 1763 measures are taken to provide DoS attack hardening? 1765 Q#35 Have DoS attack tests been performed? 1767 Q#36 Can compromise of an internal computer on a management subnet be 1768 leveraged for any form of attack including DoS attack? 1770 3.7. OAM Capabilities and Performance 1772 Q#37 What OAM proactive and on-demand mechanisms are supported? 1774 Q#38 What performance limits exist under high proactive monitoring 1775 rates? 1777 Q#39 Can excessively high proactive monitoring rates impact control 1778 plane performance or cause control plane instability? 1780 Q#40 Ask the prior questions for each of the following. 1782 a. MPLS OAM 1784 b. Pseudowire OAM 1786 c. MPLS-TP OAM 1788 d. Layer-2 OAM Interworking 1790 See Section 2.6.2. 1792 4. Forwarding Compliance and Performance Testing 1794 Packet rate performance of equipment supporting a large number of 10 1795 Gb/s or 100 Gb/s links is not possible using desktop computers or 1796 workstations. The use of high end workstations as a source of test 1797 traffic was barely viable 20 years ago, but is no longer at all 1798 viable. Though custom microcode has been used on specialized router 1799 forwarding cards to serve the purpose of generating test traffic and 1800 measuring it, for the most part performance testing will require 1801 specialized test equipment. There are multiple sources of suitable 1802 equipment. 1804 The set of tests listed here do not correspond one-to-one to the set 1805 of questions in Section 3. The same categorization is used and these 1806 tests largely serve to validate answers provided to the prior 1807 questions, and can also provide answers where a supplier is unwilling 1808 to disclose compliance or performance. 1810 Performance testing is the domain of the IETF Benchmark Methodology 1811 Working Group (BMWG). Below are brief descriptions of conformance 1812 and performance tests. Some very basic tests are specified in 1813 [RFC5695] which partially cover only the basic performance test T#3. 1815 The following tests should be performed by the systems designer, or 1816 deployer, or performed by the supplier on their behalf if it is not 1817 practical for the potential customer to perform the tests directly. 1818 These tests are grouped into broad categories. 1820 The tests in Section 4.1 should be repeated under various conditions 1821 to retest basic performance when critical capabilities are enabled. 1822 Complete repetition of the performance tests enabling each capability 1823 and combinations of capabilities would be very time intensive, 1824 therefore a reduced set of performance tests can be used to gauge the 1825 impact of enabling specific capabilities. 1827 4.1. Basic Compliance 1829 T#1 Test forwarding at a high rate for packets with varying number 1830 of label entries. While packets with more than a dozen label 1831 entries are unlikely to be used in any practical scenario today, 1832 it is useful to know if limitations exists. 1834 T#2 For each of the questions listed under "Basic Compliance" in 1835 Section 3, verify the claimed compliance. For any functionality 1836 considered critical to a deployment, where applicable performance 1837 using each capability under load should be verified in addition 1838 to basic compliance. 1840 4.2. Basic Performance 1842 T#3 Test packet forwarding at full line rate with small packets. 1843 See [RFC5695]. The most likely case to fail is the smallest 1844 packet size. Also test with packet sizes in four byte increments 1845 ranging from payload sizes or 40 to 128 bytes. 1847 T#4 If the prior tests did not succeed for all packet sizes, then 1848 perform the following tests. 1850 a. Increase the packet size by 4 bytes until a size is found 1851 that can be forwarded at full rate. 1853 b. Inject bursts of consecutive small packets into a stream of 1854 larger packets. Allow some time for recovery between bursts. 1855 Increase the number of packets in the burst until packets are 1856 dropped. 1858 T#5 Send test traffic where a swap operation is required. Also set 1859 up multiple LSP carried over other LSP where the device under 1860 test (DUT) is the egress of these LSP. Create test packets such 1861 that the swap operation is performed after pop operations, 1862 increasing the number of pop operations until forwarding of small 1863 packets at full line rate can no longer be supported. Also check 1864 to see how many pop operations can be supported before the full 1865 set of counters can no longer be maintained. This requirement is 1866 particularly relevant for MPLS-TP. 1868 T#6 Send all traffic on one LSP and see if the counters become 1869 inaccurate. Often counters on silicon are much smaller than the 1870 64 bit packet and byte counters in IETF MIB. System developers 1871 should consider what counter polling rate is necessary to 1872 maintain accurate counters and whether those polling rates are 1873 practical. Relevant MIBs for MPLS are discussed in [RFC4221] and 1874 [RFC6639]. 1876 4.3. Multipath Capabilities and Performance 1878 Multipath capabilities do not apply to MPLS-TP but apply to MPLS and 1879 apply if MPLS-TP is carried in MPLS. 1881 T#7 Send traffic at a rate well exceeding the capacity of a single 1882 multipath component link, and where entropy exists only below the 1883 top of stack. If only the top label is used this test will fail 1884 immediately. 1886 T#8 Move the labels with entropy down in the stack until either the 1887 full forwarding rate can no longer be supported or most or all 1888 packets try to use the same component link. 1890 T#9 Repeat the two tests above with the entropy contained in IP 1891 headers or IP payload fields below the label stack rather than in 1892 the label stack. Test with the set of IP headers or IP payload 1893 fields considered relevant to the deployment or to the target 1894 market. 1896 T#10 Determine whether traffic that contains a pseudowire control 1897 word is interpreted as IP traffic. Information in the payload 1898 MUST NOT be used in the load balancing if the first nibble of the 1899 packet is not 4 or 6 (IPv4 or IPv6). 1901 T#11 Determine whether special purpose labels and extended special 1902 purpose labels are excluded from the label stack hash. They MUST 1903 be excluded. 1905 T#12 Perform testing in the presence of combinations of: 1907 a. Very large microflows. 1909 b. Relatively short lived high capacity flows. 1911 c. Extremely large numbers of flows. 1913 d. Very short lived small flows. 1915 4.4. Pseudowire Capabilities and Performance 1917 T#13 Ensure that pseudowire can be set up with a pseudowire label and 1918 pseudowire control word added at ingress and the pseudowire label 1919 and pseudowire control word removed at egress. 1921 T#14 For pseudowire that contains variable length payload packets, 1922 repeat performance tests listed under "Basic Performance" for 1923 pseudowire ingress and egress functions. 1925 T#15 Repeat pseudowire performance tests with and without a 1926 pseudowire control word. 1928 T#16 Determine whether pseudowire can be set up with a pseudowire 1929 label, flow label, and pseudowire control word added at ingress 1930 and the pseudowire label, flow label, and pseudowire control word 1931 removed at egress. 1933 T#17 Determine which payload fields are used to create the flow label 1934 and whether the set of fields and algorithm provide sufficient 1935 entropy for load balancing. 1937 T#18 Repeat pseudowire performance tests with flow labels included. 1939 4.5. Entropy Label Support and Performance 1941 T#19 Determine whether entropy labels can be added at ingress and 1942 removed at egress. 1944 T#20 Determine which fields are used to create an entropy label. 1945 Labels further down in the stack, including entropy labels 1946 further down and IP headers or IP payload fields where applicable 1947 should be used. Determine whether the set of fields and 1948 algorithm provide sufficient entropy for load balancing. 1950 T#21 Repeat performance tests under "Basic Performance" when entropy 1951 labels are used, where ingress or egress is the device under test 1952 (DUT). 1954 T#22 Determine whether an ELI is detected when acting as a midpoint 1955 LSR and whether the search for further information on which to 1956 base the load balancing is used. Information below the entropy 1957 label SHOULD NOT be used. 1959 T#23 Ensure that the entropy label indicator and entropy label (ELI 1960 and EL) are removed from the label stack during UHP and PHP 1961 operations. 1963 T#24 Insure that operations on the TC field when adding and removing 1964 entropy label are correctly carried out. If TC is changed during 1965 a swap operation, the ability to transfer that change MUST be 1966 provided. The ability to suppress the transfer of TC MUST also 1967 be provided. See "pipe", "short pipe", and "uniform" models in 1968 [RFC3443]. 1970 T#25 Repeat performance tests for midpoint LSR with entropy labels 1971 found at various label stack depths. 1973 4.6. DoS Protection 1975 T#26 Actively attack LSR under high protocol churn load and determine 1976 control plane performance impact or successful DoS under test 1977 conditions. Specifically test for the following. 1979 a. TCP SYN attack against control plane and management plane 1980 protocols using TCP, including CLI access (typically SSH 1981 protected login), NETCONF, etc. 1983 b. High traffic volume attack against control plane and 1984 management plane protocols not using TCP. 1986 c. Attacks which can be performed from a compromised management 1987 subnet computer, but not one with authentication keys. 1989 d. Attacks which can be performed from a compromised peer within 1990 the control plane (internal domain and external domain). 1991 Assume that per peering keys and per router ID keys rather 1992 than network wide keys are in use. 1994 See Section 2.6.1. 1996 4.7. OAM Capabilities and Performance 1998 T#27 Determine maximum sustainable rates of BFD traffic. If BFD 1999 requires CPU intervention, determine both maximum rates and CPU 2000 loading when multiple interfaces are active. 2002 T#28 Verify LSP Ping and LSP Traceroute capability. 2004 T#29 Determine maximum rates of MPLS-TP CC-CV traffic. If CC-CV 2005 requires CPU intervention, determine both maximum rates and CPU 2006 loading when multiple interfaces are active. 2008 T#30 Determine MPLS-TP DM precision. 2010 T#31 Determine MPLS-TP LM accuracy. 2012 T#32 Verify MPLS-TP AIS/RDI and PSC functionality, protection speed, 2013 and AIS/RDI notification speed when a large number of Management 2014 Entities (ME) must be notified with AIS/RDI. 2016 5. Acknowledgements 2018 Numerous very useful comments have been received in private email. 2019 Some of these contributions are acknowledged here, approximately in 2020 chronologic order. 2022 Paul Doolan provided a brief review resulting in a number of 2023 clarifications, most notably regarding on-chip vs. system buffering, 2024 100 Gb/s link speed assumptions in the 150 Mpps figure, and handling 2025 of large microflows. Pablo Frank reminded us of the sawtooth effect 2026 in PPS vs. packet size graphs, prompting the addition of a few 2027 paragraphs on this. Comments from Lou Berger at IETF-85 prompted the 2028 addition of Section 2.7. 2030 Valuable comments were received on the BMWG mailing list. Jay 2031 Karthik pointed out testing methodology hints that after discussion 2032 were deemed out of scope and were removed but may benefit later work 2033 in BMWG. 2035 Nabil Bitar pointed out the need to cover QoS (Differentiated 2036 Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM. Nabil 2037 also provided a number of clarifications to the questions and tests 2038 in Section 3 and Section 4. 2040 Gregory Mirsky and Thomas Beckhaus provided useful comments during 2041 the MPLS RT review. 2043 Tal Mizrahi provided comments that prompted clarifications regarding 2044 timestamp processing, local delivery of packets, and the need for 2045 hardware assistance in processing OAM traffic. 2047 6. IANA Considerations 2049 This memo includes no request to IANA. 2051 7. Security Considerations 2053 This document reviews forwarding behavior specified elsewhere and 2054 points out compliance and performance requirements. As such it 2055 introduces no new security requirements or concerns. 2057 Knowledge of potential performance shortcomings may serve to help new 2058 implementations avoid pitfalls. It is unlikely that such knowledge 2059 could be the basis of new denial of service as these pitfalls are 2060 already widely known in the service provider community and among 2061 leading equipment suppliers. In practice extreme data and packet 2062 rate are needed to affect existing equipment and to affect networks 2063 that may be still vulnerable due to failure to implement adequate 2064 protection. The extreme data and packet rates make this type of 2065 denial of service unlikely and make undetectable denial of service of 2066 this type impossible. 2068 8. References 2069 8.1. Normative References 2071 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2072 Requirement Levels", BCP 14, RFC 2119, March 1997. 2074 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2075 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2076 Encoding", RFC 3032, January 2001. 2078 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2079 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2080 Tunnels", RFC 3209, December 2001. 2082 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2083 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2084 Protocol Label Switching (MPLS) Support of Differentiated 2085 Services", RFC 3270, May 2002. 2087 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2088 in Multi-Protocol Label Switching (MPLS) Networks", RFC 2089 3443, January 2003. 2091 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 2092 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, May 2093 2005. 2095 [RFC4182] Rosen, E., "Removing a Restriction on the use of MPLS 2096 Explicit NULL", RFC 4182, September 2005. 2098 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 2099 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 2101 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2102 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2103 Use over an MPLS PSN", RFC 4385, February 2006. 2105 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2106 Marking in MPLS", RFC 5129, January 2008. 2108 [RFC5586] Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic 2109 Associated Channel", RFC 5586, June 2009. 2111 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 2112 J., and S. Amante, "Flow-Aware Transport of Pseudowires 2113 over an MPLS Packet Switched Network", RFC 6391, November 2114 2011. 2116 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2117 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2118 RFC 6790, November 2012. 2120 8.2. Informative References 2122 [ACK-compression] 2123 , , , "Observations and Dynamics of a Congestion Control 2124 Algorithm: The Effects of Two-Way Traffic", Proc. ACM 2125 SIGCOMM, ACM Computer Communications Review (CCR) Vol 21, 2126 No 4, 1991, pp.133-147., 1991. 2128 [ATM-EPD-and-PPD] 2129 , "Dynamics of TCP Traffic over ATM Networks", IEEE 2130 Journal of Special Areas of Communication Vol 13 No 4, May 2131 1995, pp. 633-641., May 1995. 2133 [I-D.ietf-mpls-special-purpose-labels] 2134 Kompella, K., Andersson, L., and A. Farrel, "Allocating 2135 and Retiring Special Purpose MPLS Labels", draft-ietf- 2136 mpls-special-purpose-labels-03 (work in progress), July 2137 2013. 2139 [I-D.ietf-pwe3-vccv-impl-survey-results] 2140 Malis, A., "The Pseudowire (PW) & Virtual Circuit 2141 Connectivity Verification (VCCV) Implementation Survey 2142 Results", draft-ietf-pwe3-vccv-impl-survey-results-03 2143 (work in progress), October 2013. 2145 [I-D.ietf-tictoc-1588overmpls] 2146 Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. 2147 Montini, "Transporting Timing messages over MPLS 2148 Networks", draft-ietf-tictoc-1588overmpls-05 (work in 2149 progress), June 2013. 2151 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 2152 1981. 2154 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2155 "Definition of the Differentiated Services Field (DS 2156 Field) in the IPv4 and IPv6 Headers", RFC 2474, December 2157 1998. 2159 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 2160 and W. Weiss, "An Architecture for Differentiated 2161 Services", RFC 2475, December 1998. 2163 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 2164 "Assured Forwarding PHB Group", RFC 2597, June 1999. 2166 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2167 Label Switching Architecture", RFC 3031, January 2001. 2169 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2170 of Explicit Congestion Notification (ECN) to IP", RFC 2171 3168, September 2001. 2173 [RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for 2174 Multiprotocol Label Switching Architecture (MPLS) 2175 Operation and Maintenance (OAM) Functions", RFC 3429, 2176 November 2002. 2178 [RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching 2179 (GMPLS) Signaling Functional Description", RFC 3471, 2180 January 2003. 2182 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 2183 Edge (PWE3) Architecture", RFC 3985, March 2005. 2185 [RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 2186 Provider-Provisioned Virtual Private Networks (PPVPNs)", 2187 RFC 4110, July 2005. 2189 [RFC4124] Le Faucheur, F., "Protocol Extensions for Support of 2190 Diffserv-aware MPLS Traffic Engineering", RFC 4124, June 2191 2005. 2193 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2194 Hierarchy with Generalized Multi-Protocol Label Switching 2195 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 2197 [RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol 2198 Label Switching (MPLS) Management Overview", RFC 4221, 2199 November 2005. 2201 [RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. 2202 Matsushima, "Operations and Management (OAM) Requirements 2203 for Multi-Protocol Label Switched (MPLS) Networks", RFC 2204 4377, February 2006. 2206 [RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol 2207 Label Switched (MPLS) Data Plane Failures", RFC 4379, 2208 February 2006. 2210 [RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual 2211 Private Networks (L2VPNs)", RFC 4664, September 2006. 2213 [RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, 2214 "Extensions to Resource Reservation Protocol - Traffic 2215 Engineering (RSVP-TE) for Point-to-Multipoint TE Label 2216 Switched Paths (LSPs)", RFC 4875, May 2007. 2218 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 2219 Cost Multipath Treatment in MPLS Networks", BCP 128, RFC 2220 4928, June 2007. 2222 [RFC4950] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP 2223 Extensions for Multiprotocol Label Switching", RFC 4950, 2224 August 2007. 2226 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 2227 Specification", RFC 5036, October 2007. 2229 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., and C. 2230 Pignataro, "The Generalized TTL Security Mechanism 2231 (GTSM)", RFC 5082, October 2007. 2233 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 2234 Connectivity Verification (VCCV): A Control Channel for 2235 Pseudowires", RFC 5085, December 2007. 2237 [RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) 2238 Report on MPLS Architectural Considerations for a 2239 Transport Profile", RFC 5317, February 2009. 2241 [RFC5332] Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS 2242 Multicast Encapsulations", RFC 5332, August 2008. 2244 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2245 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2246 Class" Field", RFC 5462, February 2009. 2248 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 2249 Benchmarking Methodology for IP Flows", RFC 5695, November 2250 2009. 2252 [RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for 2253 Operations, Administration, and Maintenance (OAM) in MPLS 2254 Transport Networks", RFC 5860, May 2010. 2256 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 2257 (BFD)", RFC 5880, June 2010. 2259 [RFC5884] Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow, 2260 "Bidirectional Forwarding Detection (BFD) for MPLS Label 2261 Switched Paths (LSPs)", RFC 5884, June 2010. 2263 [RFC5885] Nadeau, T. and C. Pignataro, "Bidirectional Forwarding 2264 Detection (BFD) for the Pseudowire Virtual Circuit 2265 Connectivity Verification (VCCV)", RFC 5885, June 2010. 2267 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 2268 Time Protocol Version 4: Protocol and Algorithms 2269 Specification", RFC 5905, June 2010. 2271 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 2272 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 2273 Acronym in the IETF", BCP 161, RFC 6291, June 2011. 2275 [RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., 2276 Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, 2277 Administration, and Maintenance (OAM) Message Mapping", 2278 RFC 6310, July 2011. 2280 [RFC6371] Busi, I. and D. Allan, "Operations, Administration, and 2281 Maintenance Framework for MPLS-Based Transport Networks", 2282 RFC 6371, September 2011. 2284 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 2285 Measurement for MPLS Networks", RFC 6374, September 2011. 2287 [RFC6375] Frost, D. and S. Bryant, "A Packet Loss and Delay 2288 Measurement Profile for MPLS-Based Transport Networks", 2289 RFC 6375, September 2011. 2291 [RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, 2292 "Label Distribution Protocol Extensions for Point-to- 2293 Multipoint and Multipoint-to-Multipoint Label Switched 2294 Paths", RFC 6388, November 2011. 2296 [RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for 2297 Performing Label Switched Path Ping (LSP Ping) over MPLS 2298 Tunnels", RFC 6424, November 2011. 2300 [RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, 2301 S., and T. Nadeau, "Detecting Data-Plane Failures in 2302 Point-to-Multipoint MPLS - Extensions to LSP Ping", RFC 2303 6425, November 2011. 2305 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2306 On-Demand Connectivity Verification and Route Tracing", 2307 RFC 6426, November 2011. 2309 [RFC6427] Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S., 2310 and D. Ward, "MPLS Fault Management Operations, 2311 Administration, and Maintenance (OAM)", RFC 6427, November 2312 2011. 2314 [RFC6428] Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive 2315 Connectivity Verification, Continuity Check, and Remote 2316 Defect Indication for the MPLS Transport Profile", RFC 2317 6428, November 2011. 2319 [RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., 2320 and X. Dai, "MPLS Transport Profile Lock Instruct and 2321 Loopback Functions", RFC 6435, November 2011. 2323 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 2324 for Equal Cost Multipath Routing and Link Aggregation in 2325 Tunnels", RFC 6438, November 2011. 2327 [RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, 2328 "Pseudowire Status for Static Pseudowires", RFC 6478, May 2329 2012. 2331 [RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching 2332 Transport Profile (MPLS-TP) MIB-Based Management 2333 Overview", RFC 6639, June 2012. 2335 [RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, 2336 Administration, and Maintenance (OAM) Toolset for MPLS- 2337 Based Transport Networks", RFC 6669, July 2012. 2339 [RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a 2340 Single Solution for MPLS Transport Profile (MPLS-TP) 2341 Operations, Administration, and Maintenance (OAM)", RFC 2342 6670, July 2012. 2344 [RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security 2345 Mechanism (GTSM) for the Label Distribution Protocol 2346 (LDP)", RFC 6720, August 2012. 2348 [RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label 2349 Switched Path (LSP) Ping for Pseudowire Forwarding 2350 Equivalence Classes (FECs) Advertised over IPv6", RFC 2351 6829, January 2013. 2353 [RFC7023] Mohan, D., Bitar, N., Sajassi, A., DeLord, S., Niger, P., 2354 and R. Qiu, "MPLS and Ethernet Operations, Administration, 2355 and Maintenance (OAM) Interworking", RFC 7023, October 2356 2013. 2358 Appendix A. Organization of References Section 2360 The References section is split into Normative and Informative 2361 subsections. References that directly specify forwarding 2362 encapsulations or behaviors are listed as normative. References 2363 which describe signaling only, though normative with respect to 2364 signaling, are listed as informative. They are informative with 2365 respect to MPLS forwarding. 2367 Authors' Addresses 2369 Curtis Villamizar (editor) 2370 Outer Cape Cod Network Consulting, LLC 2372 Email: curtis@occnc.com 2374 Kireeti Kompella 2375 Juniper Networks 2377 Email: kireeti@juniper.net 2379 Shane Amante 2380 Apple Inc. 2381 1 Infinite Loop 2382 Cupertino, California 95014 2384 Email: samante@apple.com 2386 Andrew Malis 2387 Consultant 2389 Email: agmalis@gmail.com 2390 Carlos Pignataro 2391 Cisco Systems 2392 7200-12 Kit Creek Road 2393 Research Triangle Park, NC 27709 2394 US 2396 Email: cpignata@cisco.com