idnits 2.17.00 (12 Aug 2021) /tmp/idnits47039/draft-ietf-mpls-forwarding-09.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords. (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (March 4, 2014) is 2993 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '1' on line 2744 == Outdated reference: draft-ietf-mpls-psc-updates has been published as RFC 7324 == Outdated reference: draft-ietf-mpls-in-udp has been published as RFC 7510 == Outdated reference: draft-ietf-mpls-special-purpose-labels has been published as RFC 7274 == Outdated reference: draft-ietf-rtgwg-mrt-frr-architecture has been published as RFC 7812 == Outdated reference: draft-ietf-rtgwg-remote-lfa has been published as RFC 7490 == Outdated reference: A later version (-07) exists of draft-ietf-tictoc-1588overmpls-05 -- Obsolete informational reference (is this intentional?): RFC 4379 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 5102 (Obsoleted by RFC 7012) -- Obsolete informational reference (is this intentional?): RFC 6424 (Obsoleted by RFC 8029) -- Obsolete informational reference (is this intentional?): RFC 6829 (Obsoleted by RFC 8029) Summary: 0 errors (**), 0 flaws (~~), 8 warnings (==), 6 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 MPLS C. Villamizar, Ed. 3 Internet-Draft OCCNC 4 Intended status: Informational K. Kompella 5 Expires: September 5, 2014 Juniper Networks 6 S. Amante 7 Apple Inc. 8 A. Malis 9 Huawei 10 C. Pignataro 11 Cisco 12 March 4, 2014 14 MPLS Forwarding Compliance and Performance Requirements 15 draft-ietf-mpls-forwarding-09 17 Abstract 19 This document provides guidelines for implementers regarding MPLS 20 forwarding and a basis for evaluations of forwarding implementations. 21 Guidelines cover many aspects of MPLS forwarding. Topics are 22 highlighted where implementers might otherwise overlook practical 23 requirements which are unstated or under emphasized or are optional 24 for conformance to RFCs but are often considered mandatory by 25 providers. 27 Status of This Memo 29 This Internet-Draft is submitted in full conformance with the 30 provisions of BCP 78 and BCP 79. 32 Internet-Drafts are working documents of the Internet Engineering 33 Task Force (IETF). Note that other groups may also distribute 34 working documents as Internet-Drafts. The list of current Internet- 35 Drafts is at http://datatracker.ietf.org/drafts/current/. 37 Internet-Drafts are draft documents valid for a maximum of six months 38 and may be updated, replaced, or obsoleted by other documents at any 39 time. It is inappropriate to use Internet-Drafts as reference 40 material or to cite them other than as "work in progress." 42 This Internet-Draft will expire on September 5, 2014. 44 Copyright Notice 46 Copyright (c) 2014 IETF Trust and the persons identified as the 47 document authors. All rights reserved. 49 This document is subject to BCP 78 and the IETF Trust's Legal 50 Provisions Relating to IETF Documents 51 (http://trustee.ietf.org/license-info) in effect on the date of 52 publication of this document. Please review these documents 53 carefully, as they describe your rights and restrictions with respect 54 to this document. Code Components extracted from this document must 55 include Simplified BSD License text as described in Section 4.e of 56 the Trust Legal Provisions and are provided without warranty as 57 described in the Simplified BSD License. 59 Table of Contents 61 1. Introduction and Document Scope . . . . . . . . . . . . . . . 3 62 1.1. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 63 1.2. Use of Requirements Language . . . . . . . . . . . . . . 8 64 1.3. Apparent Misconceptions . . . . . . . . . . . . . . . . . 8 65 1.4. Target Audience . . . . . . . . . . . . . . . . . . . . . 10 66 2. Forwarding Issues . . . . . . . . . . . . . . . . . . . . . . 10 67 2.1. Forwarding Basics . . . . . . . . . . . . . . . . . . . . 10 68 2.1.1. MPLS Special Purpose Labels . . . . . . . . . . . . . 11 69 2.1.2. MPLS Differentiated Services . . . . . . . . . . . . 12 70 2.1.3. Time Synchronization . . . . . . . . . . . . . . . . 13 71 2.1.4. Uses of Multiple Label Stack Entries . . . . . . . . 14 72 2.1.5. MPLS Link Bundling . . . . . . . . . . . . . . . . . 15 73 2.1.6. MPLS Hierarchy . . . . . . . . . . . . . . . . . . . 15 74 2.1.7. MPLS Fast Reroute (FRR) . . . . . . . . . . . . . . . 15 75 2.1.8. Pseudowire Encapsulation . . . . . . . . . . . . . . 17 76 2.1.8.1. Pseudowire Sequence Number . . . . . . . . . . . 17 77 2.1.9. Layer-2 and Layer-3 VPN . . . . . . . . . . . . . . . 19 78 2.2. MPLS Multicast . . . . . . . . . . . . . . . . . . . . . 19 79 2.3. Packet Rates . . . . . . . . . . . . . . . . . . . . . . 20 80 2.4. MPLS Multipath Techniques . . . . . . . . . . . . . . . . 22 81 2.4.1. Pseudowire Control Word . . . . . . . . . . . . . . . 23 82 2.4.2. Large Microflows . . . . . . . . . . . . . . . . . . 23 83 2.4.3. Pseudowire Flow Label . . . . . . . . . . . . . . . . 24 84 2.4.4. MPLS Entropy Label . . . . . . . . . . . . . . . . . 24 85 2.4.5. Fields Used for Multipath Load Balance . . . . . . . 25 86 2.4.5.1. MPLS Fields in Multipath . . . . . . . . . . . . 25 87 2.4.5.2. IP Fields in Multipath . . . . . . . . . . . . . 27 88 2.4.5.3. Fields Used in Flow Label . . . . . . . . . . . . 28 89 2.4.5.4. Fields Used in Entropy Label . . . . . . . . . . 29 90 2.5. MPLS-TP and UHP . . . . . . . . . . . . . . . . . . . . . 29 91 2.6. Local Delivery of Packets . . . . . . . . . . . . . . . . 29 92 2.6.1. DoS Protection . . . . . . . . . . . . . . . . . . . 30 93 2.6.2. MPLS OAM . . . . . . . . . . . . . . . . . . . . . . 32 94 2.6.3. Pseudowire OAM . . . . . . . . . . . . . . . . . . . 33 95 2.6.4. MPLS-TP OAM . . . . . . . . . . . . . . . . . . . . . 33 96 2.6.5. MPLS OAM and Layer-2 OAM Interworking . . . . . . . . 35 97 2.6.6. Extent of OAM Support by Hardware . . . . . . . . . . 35 98 2.6.7. Support for IPFIX in Hardware . . . . . . . . . . . . 36 99 2.7. Number and Size of Flows . . . . . . . . . . . . . . . . 36 100 3. Questions for Suppliers . . . . . . . . . . . . . . . . . . . 37 101 3.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . 37 102 3.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 39 103 3.3. Multipath Capabilities and Performance . . . . . . . . . 40 104 3.4. Pseudowire Capabilities and Performance . . . . . . . . . 40 105 3.5. Entropy Label Support and Performance . . . . . . . . . . 41 106 3.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . 41 107 3.7. OAM Capabilities and Performance . . . . . . . . . . . . 41 108 4. Forwarding Compliance and Performance Testing . . . . . . . . 42 109 4.1. Basic Compliance . . . . . . . . . . . . . . . . . . . . 42 110 4.2. Basic Performance . . . . . . . . . . . . . . . . . . . . 43 111 4.3. Multipath Capabilities and Performance . . . . . . . . . 44 112 4.4. Pseudowire Capabilities and Performance . . . . . . . . . 44 113 4.5. Entropy Label Support and Performance . . . . . . . . . . 45 114 4.6. DoS Protection . . . . . . . . . . . . . . . . . . . . . 46 115 4.7. OAM Capabilities and Performance . . . . . . . . . . . . 46 116 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 47 117 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 48 118 7. Security Considerations . . . . . . . . . . . . . . . . . . . 48 119 8. Organization of References Section . . . . . . . . . . . . . 50 120 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 50 121 9.1. Normative References . . . . . . . . . . . . . . . . . . 50 122 9.2. Informative References . . . . . . . . . . . . . . . . . 52 123 9.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 58 124 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 58 126 1. Introduction and Document Scope 128 The initial purpose of this document was to address concerns raised 129 on the MPLS WG mailing list about shortcomings in implementations of 130 MPLS forwarding. Documenting existing misconceptions and potential 131 pitfalls might potentially avoid repeating past mistakes. The 132 document has grown to address a broad set of forwarding requirements. 134 The focus of this document is MPLS forwarding, base pseudowire 135 forwarding, and MPLS Operations, Administration, and Maintenance 136 (OAM). The use of pseudowire control word, and sequence number are 137 discussed. Specific pseudowire Attachment Circuit (AC) and Native 138 Service Processing (NSP) are out of scope. Specific pseudowire 139 applications, such as various forms of Virtual Private Network (VPN), 140 are out of scope. 142 MPLS support for multipath techniques is considered essential by many 143 service providers and is useful for other high capacity networks. In 144 order to obtain sufficient entropy from MPLS traffic service 145 providers and others find it essential for the MPLS implementation to 146 interpret the MPLS payload as IPv4 or IPv6 based on the contents of 147 the first nibble of payload. The use of IP addresses, the IP 148 protocol field, and UDP and TCP port number fields in multipath load 149 balancing are considered within scope. The use of any other IP 150 protocol fields, such as tunneling protocols carried within IP, are 151 out of scope. 153 Implementation details are a local matter and are out of scope. Most 154 interfaces today operate at 1 Gb/s or greater. It is assumed that 155 all forwarding operations are implemented in specialized forwarding 156 hardware rather than on a general purpose processor. This is often 157 referred to as "fast path" and "slow path" processing. Some 158 recommendations are made regarding implementing control or management 159 plane functionality in specialized hardware or with limited 160 assistance from specialized hardware. This advice is based on 161 expected control or management protocol loads and on the need for 162 denial of service (DoS) protection. 164 1.1. Abbreviations 166 The following abbreviations are used. 168 AC Attachment Circuit ([RFC3985]) 170 ACH Associated Channel Header (pseudowires) 172 ACK Acknowledgement (TCP flag and type of TCP packet) 174 AIS Alarm Indication Signal (MPLS-TP OAM) 176 ATM Asynchronous Transfer Mode (legacy switched circuits) 178 BFD Bidirectional Forwarding Detection 180 BGP Border Gateway Protocol 182 CC-CV Connectivity Check and Connectivity Verification 184 CE Customer Edge (LDP, RSVP-TE, other protocols) 186 CPU Central Processing Unit (computer or microprocessor) 188 CT Class Type ([RFC4124]) 190 CW Control Word ([RFC4385]) 192 DCCP Datagram Congestion Control Protocol 193 DDoS Distributed Denial of Service 195 DM Delay Measurement (MPLS-TP OAM) 197 DSCP Differentiated Services Code Point ([RFC2474]) 199 DWDM Dense Wave Division Multiplexing 201 DoS Denial of Service 203 E-LSP EXP-Inferred-PSC LSP ([RFC3270]) 205 EBGP External BGP 207 ECMP Equal Cost Multi-Path 209 ECN Explicit Congestion Notification ([RFC3168] and [RFC5129]) 211 EL Entropy Label ([RFC6790]) 213 ELI Entropy Label Indicator ([RFC6790]) 215 EXP Experimental (field in MPLS renamed to TC in [RFC5462]) 217 FEC Forwarding Equivalence Classes (LDP), also Forward Error 218 Correction in other context 220 FR Frame Relay (legacy switched circuits) 222 FRR Fast Reroute ([RFC4090]) 224 G-ACh Generic Associated Channel ([RFC5586]) 226 GAL Generic Associated Channel Label ([RFC5586]) 228 GFP Generic Framing Protocol (used in OTN) 230 GMPLS Generalized MPLS ([RFC3471]) 232 GTSM Generalized TTL Security Mechanism ([RFC5082]) 234 Gb/s Gigabits per second (billion bits per second) 236 IANA Internet Assigned Numbers Authority 238 ILM Incoming Label Map ([RFC3031]) 240 IP Internet Protocol 241 IPVPN Internet Protocol VPN 243 IPv4 Internet Protocol version 4 245 IPv6 Internet Protocol version 6 247 L-LSP Label-Only-Inferred-PSC LSP ([RFC3270]) 249 L2VPN Layer 2 VPN 251 LDP Label Distribution Protocol ([RFC5036]) 253 LER Label Edge Router ([RFC3031]) 255 LM Loss Measurement (MPLS-TP OAM) 257 LSP Label Switched Path ([RFC3031]) 259 LSR Label Switching Router ([RFC3031]) 261 MP2MP Multipoint to Multipoint 263 MPLS MultiProtocol Label Switching ([RFC3031]) 265 MPLS-TP MPLS Transport Profile ([RFC5317]) 267 Mb/s Megabits per second (million bits per second) 269 NSP Native Service Processing ([RFC3985]) 271 NTP Network Time Protocol 273 OAM Operations, Administration, and Maintenance ([RFC6291]) 275 OOB Out-of-band (not carried within a data channel) 277 OTN Optical Transport Network 279 P Provider router (LDP, RSVP-TE, other protocols) 281 P2MP Point to Multi-Point 283 PE Provider Edge router (LDP, RSVP-TE, other protocols) 285 PHB Per-Hop-Behavior ([RFC2475]) 287 PHP Penultimate Hop Popping ([RFC3443]) 288 POS Packet over SONET 290 PSC This abbreviation has multiple interpretations. 292 1. Packet Switch Capable ([RFC3471] 294 2. PHB Scheduling Class ([RFC3270]) 296 3. Protection State Coordination ([RFC6378]) 298 PTP Precision Time Protocol 300 PW Pseudowire 302 QoS Quality of Service 304 RA Router Alert ([RFC3032]) 306 RDI Remote Defect Indication (MPLS-TP OAM) 308 RSVP-TE RSVP Traffic Engineering 310 RTP Real-Time Transport Protocol 312 SCTP Stream Control Transmission Protocol 314 SDH Synchronous Data Hierarchy (European SONET, a form of TDM) 316 SONET Synchronous Optical Network (US SDH, a form of TDM) 318 T-LDP Targeted LDP (LDP sessions over more than one hop) 320 TC Traffic Class ([RFC5462]) 322 TCP Transmission Control Protocol 324 TDM Time-Division Multiplexing (legacy encapsulations) 326 TOS Type of Service (see [RFC2474]) 328 TTL Time-to-live (a field in IP and MPLS headers) 330 UDP User Datagram Protocol 332 UHP Ultimate Hop Popping (opposite of PHP) 334 VCCV Virtual Circuit Connectivity Verification ([RFC5085]) 335 VLAN Virtual Local Area Network (Ethernet) 337 VOQ Virtual Output Queuing (switch fabric design) 339 VPN Virtual Private Network 341 WG Working Group 343 1.2. Use of Requirements Language 345 This document is informational. The upper case [RFC2119] key words 346 "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" are used in 347 this document in the following cases. 349 1. RFC 2119 keywords are used where requirements stated in this 350 document are called for in referenced RFCs. In most cases the 351 RFC containing the requirement is cited within the statement 352 using an RFC 2119 keyword. 354 2. RFC 2119 keywords are used where explicitly noted that the 355 keywords indicate that operator experiences indicate a 356 requirement, but there are no existing RFC requirements. 358 Advice provided by this document may be ignored by implementations. 359 Similarly, implementations not claiming conformance to specific RFCs 360 may ignore the requirements of those RFCs. In both cases, 361 implementers should consider the risk of doing so. 363 1.3. Apparent Misconceptions 365 In early generations of forwarding silicon (which might now be behind 366 us), there apparently were some misconceptions about MPLS. The 367 following statements provide clarifications. 369 1. There are practical reasons to have more than one or two labels 370 in an MPLS label stack. Under some circumstances the label stack 371 can become quite deep. See Section 2.1. 373 2. The label stack MUST be considered to be arbitrarily deep. 374 Section 3.27.4. "Hierarchy: LSP Tunnels within LSPs" of RFC3031 375 states "The label stack mechanism allows LSP tunneling to nest to 376 any depth." [RFC3031] If a bottom of the label stack cannot be 377 found, but sufficient number of labels exist to forward, an LSR 378 MUST forward the packet. An LSR MUST NOT assume the packet is 379 malformed unless the end of packet is found before bottom of 380 stack. See Section 2.1. 382 3. In networks where deep label stacks are encountered, they are not 383 rare. Full packet rate performance is required regardless of 384 label stack depth, except where multiple pop operations are 385 required. See Section 2.1. 387 4. Research has shown that long bursts of short packets with 40 byte 388 or 44 byte IP payload sizes in these bursts are quite common. 389 This is due to TCP ACK compression [ACK-compression]. The 390 following two sub-bullets constitutes advice that reflects very 391 common non-negotiable requirements of providers. Implementers 392 may ignore this advice but should consider the risk of doing so. 394 A. A forwarding engine SHOULD, if practical, be able to sustain 395 an arbitrarily long sequence of small packets arriving at 396 full interface rate. 398 B. If indefinite full packet rate for small packets is not 399 practical, a forwarding engine MUST be able to buffer a long 400 sequence of small packets inbound to the on-chip decision 401 engine and sustain full interface rate for some reasonable 402 average packet rate. Absent this small on-chip buffering, 403 QoS agnostic packet drops can occur. 405 See Section 2.3. 407 5. The implementations and system designs MUST support pseudowire 408 control word (CW) if MPLS-TP is supported or if ACH [RFC5586] is 409 being used on a pseudowire. The implementation and system design 410 SHOULD support pseudowire CW even if MPLS-TP and ACH [RFC5586] 411 are not used, using instead CW and VCCV Type 1 [RFC5085] to allow 412 the use of multipath in the underlying network topology without 413 impacting the PW traffic. [RFC7079] does note that there are 414 still some deployments where the CW is not always used. It also 415 notes that many service providers do enable the CW. See 416 Section 2.4.1 for more discussion on why deployments SHOULD 417 enable the pseudowire CW. 419 The following statements provide clarification regarding more recent 420 requirements that are often missed. 422 1. The implementer and system designer SHOULD support adding a 423 pseudowire Flow Label [RFC6391]. Deployments MAY enable this 424 feature for appropriate pseudowire types. See Section 2.4.3. 426 2. The implementer and system designer SHOULD support adding an MPLS 427 entropy label [RFC6790]. Deployments MAY enable this feature. 428 See Section 2.4.4. 430 Non-IETF definitions of MPLS exist and these should not be used as 431 normative texts in place of the relevant IETF RFCs. [RFC5704] 432 documents incompatibilities between the IETF definition of MPLS and 433 one such alternative MPLS definition which led to significant issues 434 in the resulting non-IETF specification. 436 1.4. Target Audience 438 This document is intended for multiple audiences: implementer 439 (implementing MPLS forwarding in silicon or in software); systems 440 designer (putting together a MPLS forwarding systems); deployer 441 (running an MPLS network). These guidelines are intended to serve 442 the following purposes: 444 1. Explain what to do and what not to do when a deep label stack is 445 encountered. (audience: implementer) 447 2. Highlight pitfalls to look for when implementing an MPLS 448 forwarding chip. (audience: implementer) 450 3. Provide a checklist of features and performance specifications to 451 request. (audience: systems designer, deployer) 453 4. Provide a set of tests to perform. (audience: systems designer, 454 deployer). 456 The implementer, systems designer, and deployer have a transitive 457 supplier customer relationship. It is in the best interest of the 458 supplier to review their product against their customer's checklist 459 and secondary customer's checklist if applicable. 461 This document identifies and explains many details and potential pit- 462 falls of MPLS forwarding. It is likely that the identified set of 463 potential pit-falls will later prove to be an incomplete set. 465 2. Forwarding Issues 467 A brief review of forwarding issues is provided in the subsections 468 that follow. This section provides some background on why some of 469 these requirements exist. The questions to ask of suppliers is 470 covered in Section 3. Some guidelines for testing are provided in 471 Section 4. 473 2.1. Forwarding Basics 475 Basic MPLS architecture and MPLS encapsulation, and therefore packet 476 forwarding are defined in [RFC3031] and [RFC3032]. RFC3031 and 477 RFC3032 are somewhat LDP centric. RSVP-TE supports traffic 478 engineering (TE) and fast reroute, features that LDP lacks. The base 479 document for RSVP-TE based MPLS is [RFC3209]. 481 A few RFCs update RFC3032. Those with impact on forwarding include 482 the following. 484 1. TTL processing is clarified in [RFC3443]. 486 2. The use of MPLS Explicit NULL is modified in [RFC4182]. 488 3. Differentiated Services is supported by [RFC3270] and [RFC4124]. 489 The "EXP" field is renamed to "Traffic Class" in [RFC5462], 490 removing any misconception that it was available for 491 experimentation or could be ignored. 493 4. ECN is supported by [RFC5129]. 495 5. The MPLS G-ACh and GAL are defined in [RFC5586]. 497 6. [RFC5332] redefines the two data link layer codepoints for MPLS 498 packets. 500 Tunneling encapsulations carrying MPLS, such as MPLS in IP [RFC4023], 501 MPLS in GRE [RFC4023], MPLS in L2TPv3 [RFC4817], or MPLS in UDP 502 [I-D.ietf-mpls-in-udp], are out of scope. 504 Other RFCs have implications to MPLS Forwarding and do not update 505 RFC3032 or RFC3209, including: 507 1. The pseudowire (PW) Associated Channel Header (ACH), defined by 508 [RFC5085], later generalized by the MPLS G-ACh [RFC5586]. 510 2. The entropy label indicator (ELI) and entropy label (EL) are 511 defined by [RFC6790]. 513 A few RFCs update RFC3209. Those that are listed as updating RFC3209 514 generally impact only RSVP-TE signaling. Forwarding is modified by 515 major extension built upon RFC3209. 517 RFCs which impact forwarding are discussed in the following 518 subsections. 520 2.1.1. MPLS Special Purpose Labels 522 [RFC3032] specifies that label values 0-15 are special purpose labels 523 with special meanings. [I-D.ietf-mpls-special-purpose-labels] 524 renamed these from the term "reserved labels" used in [RFC3032] to 525 "special purpose labels". Three values of NULL label are defined 526 (two of which are later updated by [RFC4182]) and a router-alert 527 label is defined. The original intent was that special purpose 528 labels, except the NULL labels, could be sent to the routing engine 529 CPU rather than be processed in forwarding hardware. Hardware 530 support is required by new RFCs such as those defining entropy label 531 and OAM processed as a result of receiving a GAL. For new special 532 purpose labels, some accommodation is needed for LSR that will send 533 the labels to a general purpose CPU or other highly programmable 534 hardware. For example, ELI will only be sent to LSR which have 535 signaled support for [RFC6790] and high OAM packet rate must be 536 negotiated among endpoints. 538 [RFC3429] reserves a label for ITU-T Y.1711, however Y.1711 does not 539 work with multipath and its use is strongly discouraged. 541 The current list of special purpose labels can be found on the 542 "Multiprotocol Label Switching Architecture (MPLS) Label Values" 543 registry reachable at IANA's pages at http://www.iana.org. 545 [I-D.ietf-mpls-special-purpose-labels] introduces an IANA "Extended 546 Special Purpose MPLS Label Values" registry and makes use of the 547 "extension" label, label 15, to indicate that the next label is an 548 extended special purpose label and requires special handling. The 549 range of only 16 values for special purpose labels allows a table to 550 be used. The range of extended special purpose labels with 20 bits 551 available for use may have to be handled in some other way in the 552 unlikely event that in the future the range of currently reserved 553 values 256-1048575 are used. If only the standards action range, 554 16-239, and the experimental range, 240-255, are used, then a table 555 of 256 entries can be used. 557 Unknown special purpose labels and unknown extended special purpose 558 labels are handled the same. When an unknown special purpose label 559 is encountered or a special purpose label not directly handled in 560 forwarding hardware is encountered, the packet should be sent to a 561 general purpose CPU by default. If this capability is supported, 562 there must be an option to either drop or rate limit such packets on 563 a per special purpose label value basis. 565 2.1.2. MPLS Differentiated Services 567 [RFC2474] deprecates the IP Type of Service (TOS) and IP Precedence 568 (Prec) fields and replaces them with the Differentiated Services 569 Field more commonly known as the Differentiated Services Code Point 570 (DSCP) field. [RFC2475] defines the Differentiated Services 571 architecture, which in other forums, is often called a Quality of 572 Service (QoS) architecture. 574 MPLS uses the Traffic Class (TC) field to support Differentiated 575 Services [RFC5462]. There are two primary documents describing how 576 DSCP is mapped into TC. 578 1. [RFC3270] defines E-LSP and L-LSP. E-LSP use a static mapping of 579 DSCP into TC. L-LSP uses a per LSP mapping of DSCP into TC, with 580 one PHB Scheduling Class (PSC) per L-LSP. Each PSC can use 581 multiple Per-Hop Behavior (PHB) values. For example, the Assured 582 Forwarding service defines three PSC, each with three PHB 583 [RFC2597]. 585 2. [RFC4124] defines assignment of a class-type (CT) to an LSP, 586 where a per CT static mapping of TC to PHB is used. [RFC4124] 587 provides a means to support up to eight E-LSP-like mappings of 588 DSCP to TC. 590 To meet Differentiated Services requirements specified in [RFC3270], 591 the following forwarding requirements must be met. An ingress LER 592 MUST be able to select an LSP and then apply a per LSP map of DSCP 593 into TC. A midpoint LSR MUST be able to apply a per LSP map of TC to 594 PHB. The number of mappings supported will be far less than the 595 number of LSP supported. 597 To meet Differentiated Services requirements specified in [RFC4124], 598 the following forwarding requirements must be met. An ingress LER 599 MUST be able to select an LSP and then apply a per LSP map of DSCP 600 into TC. A midpoint LSR MUST be able to apply a per LSP map to CT 601 map and then use Class Type (CT) to map TC to PHB. Since there are 602 only eight allowed values of CT, only eight maps of TC to PHB need to 603 be supported. The LSP label can be used directly to find the TC to 604 PHB mapping, as is needed to support [RFC3270] L-LSP. 606 While support for [RFC4124] and not [RFC3270] would allow support for 607 only eight mappings of TC to PHB, it is common to support both and 608 simply state a limit on the number of unique TC to PHB mappings which 609 can be supported. 611 2.1.3. Time Synchronization 613 PTP or NTP may be carried over MPLS [I-D.ietf-tictoc-1588overmpls]. 614 Generally NTP will be carried within IP with IP carried in MPLS 615 [RFC5905]. Both PTP and NTP benefit from accurate time stamping of 616 incoming packets and the ability to insert accurate time stamps in 617 outgoing packets. PTP correction which occurs when forwarding 618 requires updating a timestamp compensation field based on the 619 difference between packet arrival at an LSR and packet transmit time 620 at that same LSR. 622 Since the label stack depth may vary, hardware should allow a 623 timestamp to be placed in an outgoing packet at any specified byte 624 position. It may be necessary to modify layer-2 checksums or frame 625 check sequences after insertion. PTP and NTP timestamp formats 626 differ in such a way as to require different implementations of the 627 timestamp correction. If NTP or PTP is carried over UDP/IP or UDP/IP 628 /MPLS, the UDP checksum will also have to be updated. 630 Accurate time synchronization in addition to being generally useful 631 is required for MPLS-TP delay measurement (DM) OAM. See 632 Section 2.6.4. 634 2.1.4. Uses of Multiple Label Stack Entries 636 MPLS deployments in the early part of the prior decade (circa 2000) 637 tended to support either LDP or RSVP-TE. LDP was favored by some for 638 its ability to scale to a very large number of PE devices at the edge 639 of the network, without adding deployment complexity. RSVP-TE was 640 favored, generally in the network core, where traffic engineering and 641 /or fast reroute were considered important. 643 Both LDP and RSVP-TE are used simultaneously within major Service 644 Provider networks using a technique known as "LDP over RSVP-TE 645 Tunneling". This technique allows service providers to carry LDP 646 tunnels inside RSVP-TE tunnels. This makes it possible to take 647 advantage of the Traffic Engineering and Fast Re-Route on more 648 expensive Inter-City and Inter-Continental transport paths. The 649 ingress RSVP-TE PEs places many LDP tunnels on a single RSVP-TE LSP 650 and carries it to the egress RSVP-TE PE. The LDP PEs are situated 651 further from the core, for example within a metro network. LDP over 652 RSVP-TE tunneling requires a minimum of two MPLS labels: one each for 653 LDP and RSVP-TE. 655 The use of MPLS FRR [RFC4090] might add one more label to MPLS 656 traffic, but only when FRR protection is in use (active). If LDP 657 over RSVP-TE is in use, and FRR protection is in use, then at least 658 three MPLS labels are present on the label stack on the links through 659 which the Bypass LSP traverses. FRR is covered in Section 2.1.7. 661 LDP L2VPN, LDP IPVPN, BGP L2VPN, and BGP IPVPN added support for VPN 662 services that are deployed by the vast majority of service providers. 663 These VPN services added yet another label, bringing the label stack 664 depth (when FRR is active) to four. 666 Pseudowires and VPN are discussed in further detail in Section 2.1.8 667 and Section 2.1.9. 669 MPLS hierarchy as described in [RFC4206] and updated by [RFC7074] can 670 in principle add at least one additional label. MPLS hierarchy is 671 discussed in Section 2.1.6. 673 Other features such as Entropy Label (discussed in Section 2.4.4) and 674 Flow Label (discussed in Section 2.4.3) can add additional labels to 675 the label stack. 677 Although theoretical scenarios can easily result in eight or more 678 labels, such cases are rare if they occur at all today. For the 679 purpose of forwarding, only the top label needs to be examined if PHP 680 is used, a few more if UHP is used (see Section 2.5). For deep label 681 stacks, quite a few labels may have to be examined for the purpose of 682 load balancing across parallel links (see Section 2.4), however this 683 depth can be bounded by a provider through use of Entropy Label. 685 Other creative use of MPLS within the IETF, such as the use of MPLS 686 label stack in source routing, may result in label stacks that are 687 considerably deeper than those encountered today. 689 2.1.5. MPLS Link Bundling 691 MPLS Link Bundling was the first RFC to address the need for multiple 692 parallel links between nodes [RFC4201]. MPLS Link Bundling is 693 notable in that it tried not to change MPLS forwarding, except in 694 specifying the "All-Ones" component link. MPLS Link Bundling is 695 seldom if ever deployed. Instead multipath techniques described in 696 Section 2.4 are used. 698 2.1.6. MPLS Hierarchy 700 MPLS hierarchy is defined in [RFC4206] and updated by [RFC7074]. 701 Although RFC4206 is considered part of GMPLS, the Packet Switching 702 Capable (PSC) portion of the MPLS hierarchy are applicable to MPLS 703 and may be supported in an otherwise GMPLS free implementation. The 704 MPLS PSC hierarchy remains the most likely means of providing further 705 scaling in an RSVP-TE MPLS network, particularly where the network is 706 designed to provide RSVP-TE connectivity to the edges. This is the 707 case for envisioned MPLS-TP networks. The use of the MPLS PSC 708 hierarchy can add at least one additional label to a label stack, 709 though it is likely that only one layer of PSC will be used in the 710 near future. 712 2.1.7. MPLS Fast Reroute (FRR) 714 Fast reroute is defined by [RFC4090]. Two significantly different 715 methods are defined in RFC4090, the "One-to-One Backup" method which 716 uses the "Detour LSP" and the "Facility Backup" which uses a "bypass 717 tunnel". These are commonly referred to as the detour and bypass 718 methods respectively. 720 The detour method makes use of a presignaled LSP. Hardware 721 assistance is needed for detour FRR only if necessary to accomplish 722 local repair of a large number of LSP within the 10s of milliseconds 723 target. For each affected LSP a swap operation must be reprogrammed 724 or otherwise switched over. The use of detour FRR doubles the number 725 of LSP terminating at any given hop and will increase the number of 726 LSP within a network by a factor dependent on the average detour path 727 length. 729 The bypass method makes use of a tunnel that is unused when no fault 730 exists but may carry many LSP when a local repair is required. There 731 is no presignaling indicating which working LSP will be diverted into 732 any specific bypass LSP. If interface label space is used the bypass 733 LSP MUST extend one hop beyond the merge point, except if the merge 734 point is the egress and PHP is used. If the bypass LSP are not 735 extended in this way, then the merge LSR (egress LSR of the bypass 736 LSP) MUST use platform label space (as defined in [RFC3031]) so that 737 an LSP working path on any given interface can be backed up using a 738 bypass LSP terminating on any other interface. Hardware assistance 739 is needed if necessary to accomplish local repair of a large number 740 of LSP within the 10s of milliseconds target. For each affected LSP 741 a swap operation must be reprogrammed or otherwise switched over with 742 an additional push of the bypass LSP label. The use of platform 743 label space impacts the size of the LSR ILM for LSR with a very large 744 number of interfaces. 746 IP/LDP Fast Reroute (IP/LDR FRR) [RFC5714] is also applicable in MPLS 747 networks. ECMP and Loop-Free Alternates (LFA) [RFC5286] are well 748 established IP/LDP FRR techniques and were the first methods to be 749 widely deployed. Work on IP/LDP FRR is ongoing within the IETF 750 RTGWG. Two topics actively discussed in RTGWG are microloops and 751 partial coverage of the established techniques in some network 752 topologies. [RFC5715] covers the topic of IP/LDP Fast Reroute 753 microloops and microloops prevention. RTGWG has developed additional 754 IP/LDP FRR techniques to handle coverage concerns. RTGWG is 755 extending LFA through the use of remote LFA 756 [I-D.ietf-rtgwg-remote-lfa]. Other techniques that require new 757 forwarding paths to be established are also under consideration, 758 including the IPFRR "not-via" technique defined in [RFC6981] and 759 maximally redundant trees (MRT) 760 [I-D.ietf-rtgwg-mrt-frr-architecture]. ECMP, LFA (but not remote 761 LFA) and MRT swap the top label to an alternate MPLS label. The 762 other methods operate in a similar manner to RFC 4090 facility backup 763 and push an additional label. IP/LDP FRR methods which push more 764 than one label have been suggested but are in early discussion. 766 2.1.8. Pseudowire Encapsulation 768 The pseudowire (PW) architecture is defined in [RFC3985]. A 769 pseudowire, when carried over MPLS, adds one or more additional label 770 entries to the MPLS label stack. A PW Control Word is defined in 771 [RFC4385] with motivation for defining the control word in [RFC4928]. 772 The PW Associated Channel defined in [RFC4385] is used for OAM in 773 [RFC5085]. The PW Flow Label is defined in [RFC6391] and is 774 discussed further in this document in Section 2.4.3. 776 There are numerous pseudowire encapsulations, supporting emulation of 777 services such as Frame Relay, ATM, Ethernet, TDM, and SONET/SDH over 778 packet switched networks (PSNs) using IP or MPLS. 780 The pseudowire encapsulation is out of scope for this document. 781 Pseudowire impact on MPLS forwarding at midpoint LSR is within scope. 782 The impact on ingress MPLS push and egress MPLS UHP pop are within 783 scope. While pseudowire encapsulation is out of scope, some advice 784 is given on sequence number support. 786 2.1.8.1. Pseudowire Sequence Number 788 Pseudowire (PW) sequence number support is most important for PW 789 payload types with a high expectation of lossless and/or in-order 790 delivery. Identifying lost PW packets and the exact amount of lost 791 payload is critical for PW services which maintain bit timing, such 792 as Time Division Multiplexing (TDM) services since these services 793 MUST compensate lost payload on a bit-for-bit basis. 795 With PW services which maintain bit timing, packets that have been 796 received out of order also MUST be identified and MAY be either re- 797 ordered or dropped. Resequencing requires, in addition to sequence 798 numbering, a "reorder buffer" in the egress PE, and ability to 799 reorder is limited by the depth of this buffer. The down side of 800 maintaining a large reorder buffer is added end-to-end service delay. 802 For PW services which maintain bit timing or any other service where 803 jitter must be bounded, a jitter buffer is always necessary. The 804 jitter buffer is needed regardless of whether reordering is done. In 805 order to be effective, a reorder buffer must often be larger than a 806 jitter buffer needs to be creating a tradeoff between reducing loss 807 and minimizing delay. 809 PW services which are not timing critical bit streams in nature are 810 cell oriented or frame oriented. Though resequencing support may be 811 beneficial to PW cell and frame oriented payloads such as ATM, FR and 812 Ethernet, this support is desirable but not required. Requirements 813 to handle out of order packets at all vary among services and 814 deployments. For example for Ethernet PW, occasional (very rare) 815 reordering is usually acceptable. If the Ethernet PW is carrying 816 MPLS-TP, then this reordering may be acceptable. 818 Reducing jitter is best done by an end-system, given that the 819 tradeoff of loss vs delay varies among services. For example with 820 interactive real time services low delay is preferred, while with 821 non-interactive (one way) real time services low loss is preferred. 822 The same end-site may be receiving both types of traffic. Regardless 823 of this, bounded jitter is sometimes a requirement for specific 824 deployments. 826 Packet reordering should be rare except in a small number of 827 circumstances, most of which are due to network design or equipment 828 design errors: 830 1. The most common case is where reordering is rare, occurring only 831 when a network or equipment fault forces traffic on a new path 832 with different delay. The packet loss that accompanies a network 833 or equipment fault is generally more disruptive than any 834 reordering which may occur. 836 2. A path change can be caused by reasons other than a network or 837 equipment fault, such as administrative routing change. This may 838 result in packet reordering but generally without any packet 839 loss. 841 3. If the edge is not using pseudowire control word (CW) and the 842 core is using multipath, reordering will be far more common. If 843 this is occurring, using CW on the edge will solve the problem. 844 Without CW, resequencing is not possible since the sequence 845 number is contained in the CW. 847 4. Another avoidable case is where some core equipment has multipath 848 and for some reason insists on periodically installing a new 849 random number as the multipath hash seed. If supporting MPLS-TP, 850 equipment MUST provide a means to disable periodic hash reseeding 851 and deployments MUST disable periodic hash reseeding. Operator 852 experience dictates that even if not supporting MPLS-TP, 853 equipment SHOULD provide a means to disable periodic hash 854 reseeding and deployments SHOULD disable periodic hash reseeding. 856 In provider networks which use multipath techniques and which may 857 occasionally rebalance traffic or which may change PW paths 858 occasionally for other reasons, reordering may be far more common 859 than loss. Where reordering is more common than loss, resequencing 860 packets is beneficial, rather than dropping packets at egress when 861 out of order arrival occurs. Resequencing is most important for PW 862 payload types with a high expectation of lossless delivery since in 863 such cases out of order delivery within the network results in PW 864 loss. 866 2.1.9. Layer-2 and Layer-3 VPN 868 Layer-2 VPN [RFC4664] and Layer-3 VPN [RFC4110] add one or more label 869 entry to the MPLS label stack. VPN encapsulations are out of scope 870 for this document. Its impact on forwarding at midpoint LSR are 871 within scope. 873 Any of these services may be used on an MPLS entropy label enabled 874 ingress and egress (see Section 2.4.4 for discussion of entropy 875 label) which would add an additional two labels to the MPLS label 876 stack. The need to provide a useful entropy label value impacts the 877 requirements of the VPN ingress LER but is out of scope for this 878 document. 880 2.2. MPLS Multicast 882 MPLS Multicast encapsulation is clarified in [RFC5332]. MPLS 883 Multicast may be signaled using RSVP-TE [RFC4875] or LDP [RFC6388]. 885 [RFC4875] defines a root initiated RSVP-TE LSP setup rather than leaf 886 initiated join used in IP multicast. [RFC6388] defines a leaf 887 initiated LDP setup. Both [RFC4875] and [RFC6388] define point to 888 multipoint (P2MP) LSP setup. [RFC6388] also defined multipoint to 889 multipoint (MP2MP) LSP setup. 891 The P2MP LSP have a single source. An LSR may be a leaf node, an 892 intermediate node, or a "bud" node. A bud serves as both a leaf and 893 intermediate. At a leaf an MPLS pop is performed. The payload may 894 be a IP Multicast packet that requires further replication. At an 895 intermediate node a MPLS swap operation is performed. The bud 896 requires that both a pop operation and a swap operation be performed 897 for the same incoming packet. 899 One strategy to support P2MP functionality is to pop at the LSR 900 interface serving as ingress to the P2MP traffic and then optionally 901 push labels at each LSR interface serving as egress to the P2MP 902 traffic at that same LSR. A given LSR egress chip may support 903 multiple egress interfaces, each of which requires a copy, but each 904 with a different set of added labels and layer-2 encapsulation. Some 905 physical interfaces may have multiple sub-interfaces (such as 906 Ethernet VLAN or channelized interfaces) each requiring a copy. 908 If packet replication is performed at LSR ingress, then the ingress 909 interface performance may suffer. If the packet replication is 910 performed within a LSR switching fabric and at LSR egress, congestion 911 of egress interfaces cannot make use of backpressure to ingress 912 interfaces using techniques such as virtual output queuing (VOQ). If 913 buffering is primarily supported at egress, then the need for 914 backpressure is minimized. There may be no good solution for high 915 volumes of multicast traffic if VOQ is used. 917 Careful consideration should be given to the performance 918 characteristics of high fanout multicast for equipment that is 919 intended to be used in such a role. 921 MP2MP LSP differ in that any branch may provide an input, including a 922 leaf. Packets must be replicated onto all other branches. This 923 forwarding is often implemented as multiple P2MP forwarding trees, 924 one for each potential input interface at a given LSR. 926 2.3. Packet Rates 928 While average packet size of Internet traffic may be large, long 929 sequences of small packets have both been predicted in theory and 930 observed in practice. Traffic compression and TCP ACK compression 931 can conspire to create long sequences of packets of 40-44 bytes in 932 payload length. If carried over Ethernet, the 64 byte minimum 933 payload applies, yielding a packet rate of approximately 150 Mpps 934 (million packets per second) for the duration of the burst on a 935 nominal 100 Gb/s link. The peak rate for other encapsulations can be 936 as high as 250 Mpps (for example IP or MPLS encapsulated using GFP 937 over OTN ODU4). 939 It is possible that the packet rates achieved by a specific 940 implementation is acceptable for a minimum payload size, such as 64 941 byte (64B) payload for Ethernet, but the achieved rate declines to an 942 unacceptable level for other packet sizes, such as 65B payload. 943 There are other packet rates of interest besides TCP ACK. For 944 example, a TCP ACK carried over an Ethernet PW over MPLS over 945 Ethernet may occupy 82B or 82B plus an increment of 4B if additional 946 MPLS labels are present. 948 A graph of packet rate vs. packet size often displays a sawtooth. 949 The sawtooth is commonly due to a memory bottleneck and memory 950 widths, sometimes internal cache, but often a very wide external 951 buffer memory interface. In some cases it may be due to a fabric 952 transfer width. A fine packing, rounding up to the nearest 8B or 16B 953 will result in a fine sawtooth with small degradation for 65B, and 954 even less for 82B packets. A course packing, rounding up to 64B can 955 yield a sharper drop in performance for 65B packets, or perhaps more 956 important, a larger drop for 82B packets. 958 The loss of some TCP ACK packets are not the primary concern when 959 such a burst occurs. When a burst occurs, any other packets, 960 regardless of packet length and packet QoS are dropped once on-chip 961 input buffers prior to the decision engine are exceeded. Buffers in 962 front of the packet decision engine are often very small or non- 963 existent (less than one packet of buffer) causing significant QoS 964 agnostic packet drop. 966 Internet service providers and content providers at one time 967 specified full rate forwarding with 40 byte payload packets as a 968 requirement. Today, this requirement often can be waived if the 969 provider can be convinced that when long sequence of short packets 970 occur no packets will be dropped. 972 Many equipment suppliers have pointed out that the extra cost in 973 designing hardware capable of processing the minimum size packets at 974 full line rate is significant for very high speed interfaces. If 975 hardware is not capable of processing the minimum size packets at 976 full line rate, then that hardware MUST be capable of handling large 977 burst of small packets, a condition which is often observed. This 978 level of performance is necessary to meet Differentiated Services 979 [RFC2475] requirements for without it, packets are lost prior to 980 inspection of the IP DSCP field [RFC2474] or MPLS TC field [RFC5462]. 982 With adequate on-chip buffers before the packet decision engine, an 983 LSR can absorb a long sequence of short packets. Even if the output 984 is slowed to the point where light congestion occurs, the packets, 985 having cleared the decision process, can make use of larger VOQ or 986 output side buffers and be dealt with according to configured QoS 987 treatment, rather than dropped completely at random. 989 These on-chip buffers need not contribute significant delay since 990 they are only used when the packet decision engine is unable to keep 991 up, not in response to congestion, plus these buffers are quite 992 small. For example, an on-chip buffer capable of handling 4K packets 993 of 64 bytes in length, or 256KB, corresponds to 200 usec on a 10 Gb/s 994 link and 20 usec on a 100 Gb/s link. If the packet decision engine 995 is capable of handling packets at 90% of the full rate for small 996 packets, then the maximum added delay is 20 usec and 2 usec 997 respectively, and this delay only applies if a 4K burst of short 998 packets occurs. When no burst of short packets was being processed, 999 no delay is added. These buffers are only needed on high speed 1000 interfaces where it is difficult to process small packets at full 1001 line rate. 1003 Packet rate requirements apply regardless of which network tier 1004 equipment is deployed in. Whether deployed in the network core or 1005 near the network edges, one of the two conditions MUST be met if 1006 Differentiated Services requirements are to be met: 1008 1. Packets must be processed at full line rate with minimum sized 1009 packets. -OR- 1011 2. Packets must be processed at a rate well under generally accepted 1012 average packet sizes, with sufficient buffering prior to the 1013 packet decision engine to accommodate long bursts of small 1014 packets. 1016 2.4. MPLS Multipath Techniques 1018 In any large provider, service providers and content providers, hash 1019 based multipath techniques are used in the core and in the edge. In 1020 many of these providers hash based multipath is also used in the 1021 larger metro networks. 1023 The Differentiated Services requirements for good reasons dictate 1024 that packets within a common microflow SHOULD NOT be reordered 1025 [RFC2474]. Service providers generally impose stronger requirements, 1026 commonly requiring that packets within a microflow MUST NOT be 1027 reordered except in rare circumstances such as load balancing across 1028 multiple links or path change for load balancing or path change for 1029 other reason. 1031 The most common multipath techniques are ECMP applied at the IP 1032 forwarding level, Ethernet LAG with inspection of the IP payload, and 1033 multipath on links carrying both IP and MPLS, where the IP header is 1034 inspected below the MPLS label stack. In most core networks, the 1035 vast majority of traffic is MPLS encapsulated. 1037 In order to support an adequately balanced load distribution across 1038 multiple links, IP header information must be used. Common practice 1039 today is to reinspect the IP headers at each LSR and use the label 1040 stack and IP header information in a hash performed at each LSR. 1041 Further details are provided in Section 2.4.5. 1043 The use of this technique is so ubiquitous in provider networks that 1044 lack of support for multipath makes any product unsuitable for use in 1045 large core networks. This will continue to be the case in the near 1046 future, even as deployment of MPLS entropy label begins to relax the 1047 core LSR multipath performance requirements given the existing 1048 deployed base of edge equipment without the ability to add an entropy 1049 label. 1051 A generation of edge equipment supporting the ability to add an MPLS 1052 entropy label is needed before the performance requirements for core 1053 LSR can be relaxed. However, it is likely that two generations of 1054 deployment in the future will allow core LSR to support full packet 1055 rate only when a relatively small number of MPLS labels need to be 1056 inspected before hashing. For now, don't count on it. 1058 Common practice today is to reinspect the packet at each LSR and use 1059 information from the packet combined plus a hash seed that is 1060 selected by each LSR. Where flow labels or entropy labels are used, 1061 a hash seed must be used when creating these labels. 1063 2.4.1. Pseudowire Control Word 1065 Within the core of a network some form of multipath is almost certain 1066 to be used. Multipath techniques deployed today are likely to be 1067 looking beneath the label stack for an opportunity to hash on IP 1068 addresses. 1070 A pseudowire encapsulated at a network edge must have a means to 1071 prevent reordering within the core if the pseudowire will be crossing 1072 a network core, or any part of a network topology where multipath is 1073 used (see [RFC4385] and [RFC4928]). 1075 Not supporting the ability to encapsulate a pseudowire with a control 1076 word may lock a product out from consideration. A pseudowire 1077 capability without control word support might be sufficient for 1078 applications that are strictly both intra-metro and low bandwidth. 1079 However a provider with other applications will very likely not 1080 tolerate having equipment which can only support a subset of their 1081 pseudowire needs. 1083 2.4.2. Large Microflows 1085 Where multipath makes use of a simple hash and simple load balance 1086 such as modulo or other fixed allocation (see Section 2.4) the 1087 presence of large microflows that each consumes 10% of the capacity 1088 of a component link of a potentially congested composite link, one 1089 such microflow can upset the traffic balance and more than one can in 1090 effect reduce the effective capacity of the entire composite link by 1091 more than 10%. 1093 When even a very small number of large microflows are present, there 1094 is a significant probability that more than one of these large 1095 microflows could fall on the same component link. If the traffic 1096 contribution from large microflows is small, the probability for 1097 three or more large microflows on the same component link drops 1098 significantly. Therefore in a network where a significant number of 1099 parallel 10 Gb/s links exists, even a 1 Gb/s pseudowire or other 1100 large microflow that could not otherwise be subdivided into smaller 1101 flows should carry a flow label or entropy label if possible. 1103 Active management of the hash space to better accommodate large 1104 microflows has been implemented and deployed in the past, however 1105 such techniques are out of scope for this document. 1107 2.4.3. Pseudowire Flow Label 1109 Unlike a pseudowire control word, a pseudowire flow label [RFC6391], 1110 is required only for relatively large capacity pseudowires. There 1111 are many cases where a pseudowire flow label makes sense. Any 1112 service such as a VPN which carries IP traffic within a pseudowire 1113 can make use of a pseudowire flow label. 1115 Any pseudowire carried over MPLS which makes use of the pseudowire 1116 control word and does not carry a flow label is in effect a single 1117 microflow (in [RFC2475] terms) and may result in the types of 1118 problems described in Section 2.4.2. 1120 2.4.4. MPLS Entropy Label 1122 The MPLS entropy label simplifies flow group identification [RFC6790] 1123 at midpoint LSRs. Prior to the MPLS entropy label midpoint LSRs 1124 needed to inspect the entire label stack and often the IP headers to 1125 provide an adequate distribution of traffic when using multipath 1126 techniques (see Section 2.4.5). With the use of MPLS entropy label, 1127 a hash can be performed closer to network edges, placed in the label 1128 stack, and used by midpoint LSRs without fully reinspecting the label 1129 stack and inspecting the payload. 1131 The MPLS entropy label is capable of avoiding full label stack and 1132 payload inspection within the core where performance levels are most 1133 difficult to achieve (see Section 2.3). The label stack inspection 1134 can be terminated as soon as the first entropy label is encountered, 1135 which is generally after a small number of labels are inspected. 1137 In order to provide these benefits in the core, LSR closer to the 1138 edge must be capable of adding an entropy label. This support may 1139 not be required in the access tier, the tier closest to the customer, 1140 but is likely to be required in the edge or the border to the network 1141 core. LSR peering with external networks will also need to be able 1142 to add an entropy label on incoming traffic. 1144 2.4.5. Fields Used for Multipath Load Balance 1146 The most common multipath techniques are based on a hash over a set 1147 of fields. Regardless of whether a hash is used or some other method 1148 is used, the there is a limited set of fields which can safely be 1149 used for multipath. 1151 2.4.5.1. MPLS Fields in Multipath 1153 If the "outer" or "first" layer of encapsulation is MPLS, then label 1154 stack entries are used in the hash. Within a finite amount of time 1155 (and for small packets arriving at high speed that time can be quite 1156 limited) only a finite number of label entries can be inspected. 1157 Pipelined or parallel architectures improve this, but the limit is 1158 still finite. 1160 The following guidelines are provided for use of MPLS fields in 1161 multipath load balancing. 1163 1. Only the 20 bit label field SHOULD be used. The TTL field SHOULD 1164 NOT be used. The S bit MUST NOT be used. The TC field (formerly 1165 EXP) MUST NOT be used. See text following this list for reasons. 1167 2. If an ELI label is found, then if the LSR supports entropy label, 1168 the EL label field in the next label entry (the EL) SHOULD be 1169 used and label entries below that label SHOULD NOT be used and 1170 the MPLS payload SHOULD NOT be used. See below this list for 1171 reasons. 1173 3. Special purpose labels (label values 0-15) MUST NOT be used. 1174 Extended special purpose labels (any label following label 15) 1175 MUST NOT be used. In particular, GAL and RA MUST NOT be used so 1176 that OAM traffic follows the same path as payload packets with 1177 the same label stack. 1179 4. If a new special purpose label or extended special purpose label 1180 is defined which requires special load balance processing, then, 1181 as is the case for the ELI label, a special action may be needed 1182 rather than skipping the special purpose label or extended 1183 special purpose label. 1185 5. The most entropy is generally found in the label stack entries 1186 near the bottom of the label stack (innermost label, closest to 1187 S=1 bit). If the entire label stack cannot be used (or entire 1188 stack up to an EL), then it is better to use as many labels as 1189 possible closest to the bottom of stack. 1191 6. If no ELI is encountered, and the first nibble of payload 1192 contains a 4 (IPv4) or 6 (IPv6), an implementation SHOULD support 1193 the ability to interpret the payload as IPv4 or IPv6 and extract 1194 and use appropriate fields from the IP headers. This feature is 1195 considered a non-negotiable requirement by many service 1196 providers. If supported, there MUST be a way to disable it (if, 1197 for example, PW without CW are used). This ability to disable 1198 this feature is considered a non-negotiable requirement by many 1199 service providers. Therefore an implementation has a very strong 1200 incentive to support both options. 1202 7. A label which is popped at egress (UHP pop) SHOULD NOT be used. 1203 A label which is popped at the penultimate hop (PHP pop) SHOULD 1204 be used. 1206 Apparently some chips have made use of the TC (formerly EXP) bits as 1207 a source of entropy. This is very harmful since it will reorder 1208 Assured Forwarding (AF) traffic [RFC2597] when a subset does not 1209 conform to the configured rates and is remarked but not dropped at a 1210 prior LSR. Traffic which uses MPLS ECN [RFC5129] can also be 1211 reordered if TC is used for entropy. Therefore, as stated in the 1212 guidelines above, the TC field (formerly EXP) MUST NOT be used in 1213 multipath load balancing as it violates Differentiated Services 1214 Ordered Aggregate (OA) requirements in these two instances. 1216 Use of the MPLS label entry S bit would result in putting OAM traffic 1217 on a different path if the addition of a GAL at the bottom of stack 1218 removed the S bit from the prior label. 1220 If an ELI label is found, then if the LSR supports entropy label, the 1221 EL label field in the next label entry (the EL) SHOULD be used and 1222 the search for additional entropy within the packet SHOULD be 1223 terminated. Failure to terminate the search will impact client MPLS- 1224 TP LSP carried within server MPLS LSP. A network operator has the 1225 option to use administrative attributes as a means to identify LSR 1226 which do not terminate the entropy search at the first EL. 1227 Administrative attributes are defined in [RFC3209]. Some 1228 configuration is required to support this. 1230 If the label removed by a PHP pop is not used, then for any PW for 1231 which CW is used, there is no basis for multipath load split. In 1232 some networks it is infeasible to put all PW traffic on one component 1233 link. Any PW which does not use CW will be improperly split 1234 regardless of whether the label removed by a PHP pop is used. 1235 Therefore the PHP pop label SHOULD be used as recommended above. 1237 2.4.5.2. IP Fields in Multipath 1239 Inspecting the IP payload provides the most entropy in provider 1240 networks. The practice of looking past the bottom of stack label for 1241 an IP payload is well accepted and documented in [RFC4928] and in 1242 other RFCs. 1244 Where IP is mentioned in the document, both IPv4 and IPv6 apply. All 1245 LSRs MUST fully support IPv6. 1247 When information in the IP header is used, the following guidelines 1248 apply: 1250 1. Both the IP source address and IP destination address SHOULD be 1251 used. There MAY be an option to reverse the order of these 1252 addresses, improving the ability to provide symmetric paths in 1253 some cases. Many service providers require that both addresses 1254 be used. 1256 2. Implementations SHOULD allow inspection of the IP protocol field 1257 and use of the UDP or TCP port numbers. For many service 1258 providers this feature is considered mandatory, particularly for 1259 enterprise, data center, or edge equipment. If this feature is 1260 provided, it SHOULD be possible to disable use of TCP and UDP 1261 ports. Many service providers consider it a non-negotiable 1262 requirement that use of UDP and TCP ports can be disabled. 1263 Therefore there is a strong incentive for implementations to 1264 provide both options. 1266 3. Equipment suppliers MUST NOT make assumptions that because the IP 1267 version field is equal to 4 (an IPv4 packet) that the IP protocol 1268 will either be TCP (IP protocol 6) or UDP (IP protocol 17) and 1269 blindly fetch the data at the offset where the TCP or UDP ports 1270 would be found. With IPv6, TCP and UDP port numbers are not at 1271 fixed offsets. With IPv4 packets carrying IP options, TCP and 1272 UDP port numbers are not at fixed offsets. 1274 4. The IPv6 header flow field SHOULD be used. This is the explicit 1275 purpose of the IPv6 flow field, however observed flow fields 1276 rarely contains a non-zero value. Some uses of the flow field 1277 have been defined such as [RFC6438]. In the absence of MPLS 1278 encapsulation, the IPv6 flow field can serve a role equivalent to 1279 entropy label. 1281 5. Support for other protocols that share a common Layer-4 header 1282 such as RTP [RFC3550], UDP-Lite [RFC3828], SCTP [RFC4960] and 1283 DCCP [RFC4340] SHOULD be provided, particularly for edge or 1284 access equipment where additional entropy may be needed. 1286 Equipment SHOULD also use RTP, UDP-lite, SCTP and DCCP headers 1287 when creating an entropy label. 1289 6. The following IP header fields should not or must not be used: 1291 A. Similar to avoiding TC in MPLS, the IP DSCP, and ECN bits 1292 MUST NOT be used. 1294 B. The IPv4 TTL or IPv6 Hop Count SHOULD NOT be used. 1296 C. Note that the IP TOS field was deprecated ([RFC0791] was 1297 updated by [RFC2474]). No part of the IP DSCP field can be 1298 used (formerly IP PREC and IP TOS bits). 1300 7. Some IP encapsulations support tunneling, such as IP-in-IP, GRE, 1301 L2TPv3, and IPSEC. These provide a greater source of entropy 1302 which some provider networks carrying large amounts of tunneled 1303 traffic may need, for example as used in [RFC5640] for GRE and 1304 L2TPv3. The use of tunneling header information is out of scope 1305 for this document. 1307 This document makes the following recommendations. These 1308 recommendations are not required to claim compliance to any existing 1309 RFC therefore implementers are free to ignore them, but due to 1310 service provider requirements should consider the risk of doing so. 1311 The use of IP addresses MUST be supported and TCP and UDP ports 1312 (conditional on the protocol field and properly located) MUST be 1313 supported. The ability to disable use of UDP and TCP ports MUST be 1314 available. 1316 Though potentially very useful in some networks, it is uncommon to 1317 support using payloads of tunneling protocols carried over IP. 1318 Though the use of tunneling protocol header information is out of 1319 scope for this document, it is not discouraged. 1321 2.4.5.3. Fields Used in Flow Label 1323 The ingress to a pseudowire (PW) can extract information from the 1324 payload being encapsulated to create a flow label. [RFC6391] 1325 references IP carried in Ethernet as an example. The Native Service 1326 Processing (NSP) function defined in [RFC3985] differs with 1327 pseudowire type. It is in the NSP function where information for a 1328 specific type of PW can be extracted for use in a flow label. Which 1329 fields to use for any given PW NSP is out of scope for this document. 1331 2.4.5.4. Fields Used in Entropy Label 1333 An entropy label is added at the ingress to an LSP. The payload 1334 being encapsulated is most often MPLS, a PW, or IP. The payload type 1335 is identified by the layer-2 encapsulation (Ethernet, GFP, POS, etc). 1337 If the payload is MPLS, then the information used to create an 1338 entropy label is the same information used for local load balancing 1339 (see Section 2.4.5.1). This information MUST be extracted for use in 1340 generating an entropy label even if the LSR local egress interface is 1341 not a multipath. 1343 Of the non-MPLS payload types, only payloads that are forwarded are 1344 of interest. For example, ARP is not forwarded and CNLP (used only 1345 for ISIS) is not forwarded. 1347 The non-MPLS payload type of greatest interest are IPv4 and IPv6. 1348 The guidelines in Section 2.4.5.2 apply to fields used to create and 1349 entropy label. 1351 The IP tunneling protocols mentioned in Section 2.4.5.2 may be more 1352 applicable to generation of an entropy label at edge or access where 1353 deep packet inspection is practical due to lower interface speeds 1354 than in the core where deep packet inspection may be impractical. 1356 2.5. MPLS-TP and UHP 1358 MPLS-TP introduces forwarding demands that will be extremely 1359 difficult to meet in a core network. Most troublesome is the 1360 requirement for Ultimate Hop Popping (UHP, the opposite of 1361 Penultimate Hop Popping or PHP). Using UHP opens the possibility of 1362 one or more MPLS pop operation plus an MPLS swap operation for each 1363 packet. The potential for multiple lookups and multiple counter 1364 instances per packet exists. 1366 As networks grow and tunneling of LDP LSPs into RSVP-TE LSPs is used, 1367 and/or RSVP-TE hierarchy is used, the requirement to perform one or 1368 two or more MPLS pop operations plus a MPLS swap operation (and 1369 possibly a push or two) increases. If MPLS-TP LM (link monitoring) 1370 OAM is enabled at each layer, then a packet and byte count MUST be 1371 maintained for each pop and swap operation so as to offer OAM for 1372 each layer. 1374 2.6. Local Delivery of Packets 1376 There are a number of situations in which packets are destined to a 1377 local address or where a return packet must be generated. There is a 1378 need to mitigate the potential for outage as a result of either 1379 attacks on network infrastructure, or in some cases unintentional 1380 misconfiguration resulting in processor overload. Some hardware 1381 assistance is needed for all traffic destined to the general purpose 1382 CPU that is used in MPLS control protocol processing or network 1383 management protocol processing and in most cases to other general 1384 purpose CPUs residing on an LSR. This is due to the ease of 1385 overwhelming such a processor with traffic arriving on LSR high speed 1386 interfaces, whether the traffic is malicious or not. 1388 Denial of service (DoS) protection is an area requiring hardware 1389 support that is often overlooked or inadequately considered. 1390 Hardware assist is also needed for OAM, particularly the more 1391 demanding MPLS-TP OAM. 1393 2.6.1. DoS Protection 1395 Modern equipment supports a number of control plane and management 1396 plane protocols. Generally no single means of protecting network 1397 equipment from denial of service (DoS) attacks is sufficient, 1398 particularly for high speed interfaces. This problem is not specific 1399 to MPLS, but is a topic that cannot be ignored when implementing or 1400 evaluating MPLS implementations. 1402 Two types of protections are often cited as primary means of 1403 protecting against attacks of all kinds. 1405 Isolated Control/Management Traffic 1406 Control and Management traffic can be carried out-of-band (OOB), 1407 meaning not intermixed with payload. For MPLS, use of G-ACh and 1408 GAL to carry control and management traffic provides a means of 1409 isolation from potentially malicious payload. Used alone, the 1410 compromise of a single node, including a small computer at a 1411 network operations center, could compromise an entire network. 1412 Implementations which send all G-ACh/GAL traffic directly to a 1413 routing engine CPU are subject to DoS attack as a result of such 1414 a compromise. 1416 Cryptographic Authentication 1417 Cryptographic authentication can very effectively prevent 1418 malicious injection of control or management traffic. 1419 Cryptographic authentication can in some circumstances be subject 1420 to DoS attack by overwhelming the capacity of the decryption with 1421 a high volume of malicious traffic. For very low speed 1422 interfaces, cryptographic authentication can be performed by the 1423 general purpose CPU used as a routing engine. For all other 1424 cases, cryptographic hardware may be needed. For very high speed 1425 interfaces, even cryptographic hardware can be overwhelmed. 1427 Some control and management protocols are often carried with payload 1428 traffic. This is commonly the case with BGP, T-LDP, and SNMP. It is 1429 often the case with RSVP-TE. Even when carried over G-ACh/GAL 1430 additional measures can reduce the potential for a minor breach to be 1431 leveraged to a full network attack. 1433 Some of the additional protections are supported by hardware packet 1434 filtering. 1436 GTSM 1437 [RFC5082] defines a mechanism that uses the IPv4 TTL or IPv6 Hop 1438 Limit fields to insure control traffic that can only originate 1439 from an immediate neighbor is not forged and originating from a 1440 distant source. GTSM can be applied to many control protocols 1441 which are routable, for example LDP [RFC6720]. 1443 IP Filtering 1444 At the very minimum, packet filtering plus classification and use 1445 of multiple queues supporting rate limiting is needed for traffic 1446 that could potentially be sent to a general purpose CPU used as a 1447 routing engine. The first level of filtering only allows 1448 connections to be initiated from specific IP prefixes to specific 1449 destination ports and then preferably passes traffic directly to 1450 a cryptographic engine and/or rate limits. The second level of 1451 filtering passes connected traffic, such as TCP connections 1452 having received at least one authenticated SYN or having been 1453 locally initiated. The second level of filtering only passes 1454 traffic to specific address and port pairs to be checked for 1455 cryptographic authentication. 1457 The cryptographic authentication is generally the last resort in DoS 1458 attack mitigation. If a packet must be first sent to a general 1459 purpose CPU, then sent to a cryptographic engine, a DoS attack is 1460 possible on high speed interfaces. Only where hardware can fully 1461 process a cryptographic authentication without intervention from a 1462 general purpose CPU to find the authentication field and to identify 1463 the portion of packet to run the cryptographic algorithm over is 1464 cryptographic authentication beneficial in protecting against DoS 1465 attacks. 1467 For chips supporting multiple 100 Gb/s interfaces, only a very large 1468 number of parallel cryptographic engines can provide the processing 1469 capacity to handle a large scale DoS or distributed DoS (DDoS) 1470 attack. For many forwarding chips this much processing power 1471 requires significant chip real estate and power, and therefore 1472 reduces system space and power density. For this reason, 1473 cryptographic authentication is not considered a viable first line of 1474 defense. 1476 For some networks the first line of defense is some means of 1477 supporting OOB control and management traffic. In the past this OOB 1478 channel might make use of overhead bits in SONET or OTN or a 1479 dedicated DWDM wavelength. G-ACh and GAL provide an alternative OOB 1480 mechanism which is independent of underlying layers. In other 1481 networks, including most IP/MPLS networks, perimeter filtering serves 1482 a similar purpose, though less effective without extreme vigilance. 1484 A second line of defense is filtering, including GTSM. For protocols 1485 such as EBGP, GTSM and other filtering is often the first line of 1486 defense. Cryptographic authentication is usually the last line of 1487 defense and insufficient by itself to mitigate DoS or DDoS attacks. 1489 2.6.2. MPLS OAM 1491 [RFC4377] defines requirements for MPLS OAM that predate MPLS-TP. 1492 [RFC4379] defines what is commonly referred to as LSP Ping and LSP 1493 Traceroute. [RFC4379] is updated by [RFC6424] supporting MPLS 1494 tunnels and stitched LSP and P2MP LSP. [RFC4379] is updated by 1495 [RFC6425] supporting P2MP LSP. [RFC4379] is updated by [RFC6426] to 1496 support MPLS-TP connectivity verification (CV) and route tracing. 1498 [RFC4950] extends the ICMP format to support TTL expiration that may 1499 occur when using IP traceroute within an MPLS tunnel. The ICMP 1500 message generation can be implemented in forwarding hardware, but if 1501 sent to a general purpose CPU must be rate limited to avoid a 1502 potential denial or service (DoS) attack. 1504 [RFC5880] defines Bidirectional Forwarding Detection (BFD), a 1505 protocol intended to detect faults in the bidirectional path between 1506 two forwarding engines. [RFC5884] and [RFC5885] define BFD for MPLS. 1507 BFD can provide failure detection on any kind of path between 1508 systems, including direct physical links, virtual circuits, tunnels, 1509 MPLS Label Switched Paths (LSPs), multihop routed paths, and 1510 unidirectional links as long as there is some return path. 1512 The processing requirements for BFD are less than for LSP Ping, 1513 making BFD somewhat better suited for relatively high rate proactive 1514 monitoring. BFD does not verify that the data plane matches the 1515 control plane, where LSP Ping does. LSP Ping is somewhat better 1516 suited for on-demand monitoring including relatively low rate 1517 periodic verification of data plane and as a diagnostic tool. 1519 Hardware assistance is often provided for BFD response where BFD 1520 setup or parameter change is not involved and may be necessary for 1521 relatively high rate proactive monitoring. If both BFD and LSP Ping 1522 are recognized in filtering prior to passing traffic to a general 1523 purpose CPU, appropriate DoS protection can be applied (see 1524 Section 2.6.1). Failure to recognize BFD and LSP Ping and at least 1525 rate limit creates the potential for misconfiguration to cause 1526 outages rather than cause errors in the misconfigured OAM. 1528 2.6.3. Pseudowire OAM 1530 Pseudowire OAM makes use of the control channel provided by Virtual 1531 Circuit Connectivity Verification (VCCV) [RFC5085]. VCCV makes use 1532 of the Pseudowire Control Word. BFD support over VCCV is defined by 1533 [RFC5885]. [RFC5885] is updated by [RFC6478] in support of static 1534 pseudowires. [RFC4379] is updated by [RFC6829] supporting LSP Ping 1535 for Pseudowire FEC advertised over IPv6. 1537 G-ACh/GAL (defined in [RFC5586]) is the preferred MPLS-TP OAM control 1538 channel and applies to any MPLS-TP end points, including Pseudowire. 1539 See Section 2.6.4 for an overview of MPLS-TP OAM. 1541 2.6.4. MPLS-TP OAM 1543 [RFC6669] summarizes the MPLS-TP OAM toolset, the set of protocols 1544 supporting the MPLS-TP OAM requirements specified in [RFC5860] and 1545 supported by the MPLS-TP OAM framework defined in [RFC6371]. 1547 The MPLS-TP OAM toolset includes: 1549 CC-CV 1550 [RFC6428] defines BFD extensions to support proactive 1551 Connectivity Check and Connectivity Verification (CC-CV) 1552 applications. [RFC6426] provides LSP ping extensions that are 1553 used to implement on-demand connectivity verification. 1555 RDI 1556 Remote Defect Indication (RDI) is triggered by failure of 1557 proactive CC-CV, which is BFD based. For fast RDI initiation, 1558 RDI SHOULD be initiated and handled by hardware if BFD is handled 1559 in forwarding hardware. [RFC6428] provides an extension for BFD 1560 that includes the RDI indication in the BFD format and a 1561 specification of how this indication is to be used. 1563 Route Tracing 1564 [RFC6426] specifies that the LSP ping enhancements for MPLS-TP 1565 on-demand connectivity verification include information on the 1566 use of LSP ping for route tracing of an MPLS-TP path. 1568 Alarm Reporting 1569 [RFC6427] describes the details of a new protocol supporting 1570 Alarm Indication Signal (AIS), Link Down Indication, and fault 1571 management. Failure to support this functionality in forwarding 1572 hardware can potentially result in failure to meet protection 1573 recovery time requirements and is therefore strongly recommended. 1575 Lock Instruct 1576 Lock instruct is initiated on-demand and therefore need not be 1577 implemented in forwarding hardware. [RFC6435] defines a lock 1578 instruct protocol. 1580 Lock Reporting 1581 [RFC6427] covers lock reporting. Lock reporting need not be 1582 implemented in forwarding hardware. 1584 Diagnostic 1585 [RFC6435] defines protocol support for loopback. Loopback 1586 initiation is on-demand and therefore need not be implemented in 1587 forwarding hardware. Loopback of packet traffic SHOULD be 1588 implemented in forwarding hardware on high speed interfaces. 1590 Packet Loss and Delay Measurement 1591 [RFC6374] and [RFC6375] define a protocol and profile for packet 1592 loss measurement (LM) and delay measurement (DM). LM requires a 1593 very accurate capture and insertion of packet and byte counters 1594 when a packet is transmitted and capture of packet and byte 1595 counters when a packet is received. This capture and insertion 1596 MUST be implemented in forwarding hardware for LM OAM if high 1597 accuracy is needed. DM requires very accurate capture and 1598 insertion of a timestamp on transmission and capture of timestamp 1599 when a packet is received. This timestamp capture and insertion 1600 MUST be implemented in forwarding hardware for DM OAM if high 1601 accuracy is needed. 1603 See Section 2.6.2 for discussion of hardware support necessary for 1604 BFD and LSP Ping. 1606 CC-CV and alarm reporting is tied to protection and therefore SHOULD 1607 be supported in forwarding hardware in order to provide protection 1608 for a large number of affected LSP within target response intervals. 1609 Since CC-CV is supported by BFD, for MPLS-TP providing hardware 1610 assistance for BFD processing helps insure that protection recovery 1611 time requirements can be met even for faults affecting a large number 1612 of LSP. 1614 MPLS-TP Protection State Coordination (PSC) is defined by [RFC6378] 1615 and updated by [I-D.ietf-mpls-psc-updates], correcting some errors in 1616 [RFC6378]. 1618 2.6.5. MPLS OAM and Layer-2 OAM Interworking 1620 [RFC6670] provides the reasons for selecting a single MPLS-TP OAM 1621 solution and examines the consequences were ITU-T to develop a second 1622 OAM solution that is based on Ethernet encodings and mechanisms. 1624 [RFC6310] and [RFC7023] specifies the mapping of defect states 1625 between many types of hardware Attachment Circuits (ACs) and 1626 associated Pseudowires (PWs). This functionality SHOULD be supported 1627 in forwarding hardware. 1629 It is beneficial if an MPLS OAM implementation can interwork with the 1630 underlying server layer and provide a means to interwork with a 1631 client layer. For example, [RFC6427] specifies an inter-layer 1632 propagation of AIS and LDI from MPLS server layer to client MPLS 1633 layers. Where the server layer is a Layer-2, such as Ethernet, PPP 1634 over SONET/SDH, or GFP over OTN, interwork among layers is also 1635 beneficial. For high speed interfaces, supporting this interworking 1636 in forwarding hardware helps insure that protection based on this 1637 interworking can meet recovery time requirements even for faults 1638 affecting a large number of LSP. 1640 2.6.6. Extent of OAM Support by Hardware 1642 Where certain requirements must be met, such as relatively high CC-CV 1643 rates and a large number of interfaces, or strict protection recovery 1644 time requirements and a moderate number of affected LSP, some OAM 1645 functionality must be supported by forwarding hardware. In other 1646 cases, such as highly accurate LM and DM OAM or strict protection 1647 recovery time requirements with a large number of affected LSP, OAM 1648 functionality must be entirely implemented in forwarding hardware. 1650 Where possible, implementation in forwarding hardware should be in 1651 programmable hardware such that if standards are later changed or 1652 extended these changes are likely to be accommodated with hardware 1653 reprogramming rather than replacement. 1655 For some functionality there is a strong case for an implementation 1656 in dedicated forwarding hardware. Examples include packet and byte 1657 counters needed for LM OAM as well as needed for management 1658 protocols. Similarly the capture and insertion of packet and byte 1659 counts or timestamps needed for transmitted LM or DM or time 1660 synchronization packets MUST be implemented in forwarding hardware if 1661 high accuracy is required. 1663 For some functions there is a strong case to provide limited support 1664 in forwarding hardware but may make use of an external general 1665 purpose processor if performance criteria can be met. For example 1666 origination of RDI triggered by CC-CV, response to RDI, and 1667 Protection State Coordination (PSC) functionality may be supported by 1668 hardware, but expansion to a large number of client LSP and 1669 transmission of AIS or RDI to the client LSP may occur in a general 1670 purpose processor. Some forwarding hardware supports one or more on- 1671 chip general purpose processors which may be well suited for such a 1672 role. [I-D.ietf-mpls-psc-updates], being a very recent document that 1673 affects a protection state machine that requires hardware support, 1674 underscores the importance of having a degree of programmability in 1675 forwarding hardware. 1677 The customer (system supplier or provider) should not dictate design, 1678 but should independently validate target functionality and 1679 performance. However, it is not uncommon for service providers and 1680 system implementers to insist on reviewing design details (under NDA) 1681 due to past experiences with suppliers and to reject suppliers who 1682 are unwilling to provide details. 1684 2.6.7. Support for IPFIX in Hardware 1686 The IPFIX architecture is defined by [RFC5470]. IPFIX supports per 1687 flow statistics. IPFIX infomation elements (IEs) are defined in 1688 [RFC5102] and include IEs for MPLS. 1690 The forwarding chips used in core routers are not optimized for high 1691 touch applications like IPFIX. Often support for IPFIX in core 1692 routers is limited to optional IPFIX metering, which involves a 1693 1-in-N packet sampling, limited filtering support, and redirection to 1694 either an internal CPU or an external interface. The CPU or device 1695 at the other end of the external interface then implements the full 1696 IPFIX filtering and IPFIX collector functionality. 1698 LSR which are intended to be deployed further from the core may 1699 support lower capacity interfaces but support higher touch 1700 applications on the forwarding hardware and may provide dedicated 1701 hardware to support a greater subset IPFIX functionality before 1702 handing off to a general purpose CPU. In some cases, far from the 1703 core the entire IPFIX functionality up to and including the collector 1704 may be implemented in hardware and firmware in the forwarding 1705 silicon. It is also worth noting that at lower speeds a general 1706 purpose CPU may become adequate to implement IPFIX, particularly if 1707 metering is used. 1709 2.7. Number and Size of Flows 1711 Service provider networks may carry up to hundreds of millions of 1712 flows on 10 Gb/s links. Most flows are very short lived, many under 1713 a second. A subset of the flows are low capacity and somewhat long 1714 lived. When Internet traffic dominates capacity a very small subset 1715 of flows are high capacity and/or very long lived. 1717 Two types of limitations with regard to number and size of flows have 1718 been observed. 1720 1. Some hardware cannot handle some high capacity flows because of 1721 internal paths which are limited, such as per packet backplane 1722 paths or paths internal or external to chips such as buffer 1723 memory paths. Such designs can handle aggregates of smaller 1724 flows. Some hardware with acknowledged limitations has been 1725 successfully deployed but may be increasingly problematic if the 1726 capacity of large microflows in deployed networks continues to 1727 grow. 1729 2. Some hardware approaches cannot handle a large number of flows, 1730 or a large number of large flows due to attempting to count per 1731 flow, rather than deal with aggregates of flows. Hash techniques 1732 scale with regard to number of flows due to a fixed hash size 1733 with many flows falling into the same hash bucket. Techniques 1734 that identify individual flows have been implemented but have 1735 never successfully deployed for Internet traffic. 1737 3. Questions for Suppliers 1739 The following questions should be asked of a supplier. These 1740 questions are grouped into broad categories. The questions 1741 themselves are intended to be an open ended question to the supplier. 1742 The tests in Section 4 are intended to verify whether the supplier 1743 disclosed any compliance or performance limitations completely and 1744 accurately. 1746 3.1. Basic Compliance 1748 Q#1 Can the implementation forward packets with an arbitrarily 1749 large stack depth? What limitations exist, and under what 1750 circumstances do further limitations come into play (such as high 1751 packet rate or specific features enabled or specific types of 1752 packet processing)? See Section 2.1. 1754 Q#2 Is the entire set of basic MPLS functionality described in 1755 Section 2.1 supported? 1757 Q#3 Are the set of MPLS special purpose labels handled correctly 1758 and with adequate performance? Are extended special purpose 1759 labels handled correctly and with adequate performance? See 1760 Section 2.1.1. 1762 Q#4 Are mappings of label value and TC to PHB handled correctly, 1763 including RFC3270 L-LSP mappings and RFC4124 CT mappings to PHB? 1764 See Section 2.1.2. 1766 Q#5 Is time synchronization adequately supported in forwarding 1767 hardware? 1769 A. Are both PTP and NTP formats supported? 1771 B. Is the accuracy of timestamp insertion and incoming stamping 1772 sufficient? 1774 See Section 2.1.3. 1776 Q#6 Is link bundling supported? 1778 A. Can LSP be pinned to specific components? 1780 B. Is the "all-ones" component link supported? 1782 See Section 2.1.5. 1784 Q#7 Is MPLS hierarchy supported? 1786 A. Are both PHP and UHP supported? What limitations exist on 1787 the number of pop operations with UHP? 1789 B. Are the pipe, short-pipe, and uniform models supported? Are 1790 TTL and TC values updated correctly at egress where 1791 applicable? 1793 See Section 2.1.6 regarding MPLS hierarchy. See [RFC3443] 1794 regarding PHP, UHP, and pipe, short-pipe, and uniform models. 1796 Q#8 Is FRR supported? 1798 A. Are both "One-to-One Backup" and "Facility Backup" supported? 1800 B. What forms of IPFRR/LDPFRR are supported? 1802 C. How quickly does protection recovery occur? 1804 D. Does protection recovery speed increase when a fault affects 1805 a large numbers of protected LSP, and if so by how much? 1807 See Section 2.1.7. 1809 Q#9 Are pseudowire sequence numbers handled correctly? See 1810 Section 2.1.8.1. 1812 Q#10 Is VPN LER functionality handled correctly and without 1813 performance issues? See Section 2.1.9. 1815 Q#11 Is MPLS multicast (P2MP and MP2MP) handled correctly? 1817 A. Are packets dropped on uncongested outputs if some outputs 1818 are congested? 1820 B. Is performance limited in high fanout situations? 1822 See Section 2.2. 1824 3.2. Basic Performance 1826 Q#12 Can very small packets be forwarded at full line rate on all 1827 interfaces indefinitely? What limitations exist, and under what 1828 circumstances do further limitations come into play (such as 1829 specific features enabled or specific types of packet 1830 processing)? 1832 Q#13 Customers must decide whether to relax the prior requirement and 1833 to what extent. If the answer to the prior question indicates 1834 that limitations exist, then: 1836 A. What is the smallest packet size where full line rate 1837 forwarding can be supported? 1839 B. What is the longest burst of full rate small packets that can 1840 be supported? 1842 Specify circumstances (such as specific features enabled or 1843 specific types of packet processing) often impact these rates and 1844 burst sizes. 1846 Q#14 How many pop operations can be supported along with a swap 1847 operation at full line rate while maintaining per LSP packet and 1848 byte counts for each pop and swap? This requirement is 1849 particularly relevant for MPLS-TP. 1851 Q#15 How many label push operations can be supported. While this 1852 limitation is rarely an issue, it applies to both PHP and UHP, 1853 unlike the pop limit which applies to UHP. 1855 Q#16 For a worst case where all packets arrive on one LSP, what is 1856 the counter overflow time? Are any means provided to avoid 1857 polling all counters at short intervals? This applies to both 1858 MPLS and MPLS-TP. 1860 3.3. Multipath Capabilities and Performance 1862 Multipath capabilities and performance do not apply to MPLS-TP but 1863 apply to MPLS and apply if MPLS-TP is carried in MPLS. 1865 Q#17 How are large microflows accommodated? Is there active 1866 management of the hash space mapping to output ports? See 1867 Section 2.4.2. 1869 Q#18 How many MPLS labels can be included in a hash based on the MPLS 1870 label stack? 1872 Q#19 Is packet rate performance decreased beyond some number of 1873 labels? 1875 Q#20 Can the IP header and payload information below the MPLS stack 1876 be used in the hash? If so, which IP fields, payload types and 1877 payload fields are supported? 1879 Q#21 At what maximum MPLS label stack depth can Bottom of Stack and 1880 an IP header appear without impacting packet rate performance? 1882 Q#22 Are special purpose labels excluded from the label stack hash? 1883 Are extended purpose labels excluded from the label stack hash? 1884 See Section 2.4.5.1. 1886 Q#23 How is multipath performance affected by high capacity flows or 1887 an extremely large number of flows, or by very short lived flows? 1888 See Section 2.7. 1890 3.4. Pseudowire Capabilities and Performance 1892 Q#24 Is the pseudowire control word supported? 1894 Q#25 What is the maximum rate of pseudowire encapsulation and 1895 decapsulation? Apply the same questions as in Base Performance 1896 for any packet based pseudowire such as IP VPN or Ethernet. 1898 Q#26 Does inclusion of a pseudowire control word impact performance? 1900 Q#27 Are flow labels supported? 1902 Q#28 If so, what fields are hashed on for the flow label for 1903 different types of pseudowires? 1905 Q#29 Does inclusion of a flow label impact performance? 1907 3.5. Entropy Label Support and Performance 1909 Q#30 Can an entropy label be added when acting as in ingress LER and 1910 can it be removed when acting as an egress LER? 1912 Q#31 If so, what fields are hashed on for the entropy label? 1914 Q#32 Does adding or removing an entropy label impact packet rate 1915 performance? 1917 Q#33 Can an entropy label be detected in the label stack, used in the 1918 hash, and properly terminate the search for further information 1919 to hash on? 1921 Q#34 Does using an entropy label have any negative impact on 1922 performance? It should have no impact or a positive impact. 1924 3.6. DoS Protection 1926 Q#35 For each control and management plane protocol in use, what 1927 measures are taken to provide DoS attack hardening? 1929 Q#36 Have DoS attack tests been performed? 1931 Q#37 Can compromise of an internal computer on a management subnet be 1932 leveraged for any form of attack including DoS attack? 1934 3.7. OAM Capabilities and Performance 1936 Q#38 What OAM proactive and on-demand mechanisms are supported? 1938 Q#39 What performance limits exist under high proactive monitoring 1939 rates? 1941 Q#40 Can excessively high proactive monitoring rates impact control 1942 plane performance or cause control plane instability? 1944 Q#41 Ask the prior questions for each of the following. 1946 A. MPLS OAM 1948 B. Pseudowire OAM 1950 C. MPLS-TP OAM 1952 D. Layer-2 OAM Interworking 1953 See Section 2.6. 1955 4. Forwarding Compliance and Performance Testing 1957 Packet rate performance of equipment supporting a large number of 10 1958 Gb/s or 100 Gb/s links is not possible using desktop computers or 1959 workstations. The use of high end workstations as a source of test 1960 traffic was barely viable 20 years ago, but is no longer at all 1961 viable. Though custom microcode has been used on specialized router 1962 forwarding cards to serve the purpose of generating test traffic and 1963 measuring it, for the most part performance testing will require 1964 specialized test equipment. There are multiple sources of suitable 1965 equipment. 1967 The set of tests listed here do not correspond one-to-one to the set 1968 of questions in Section 3. The same categorization is used and these 1969 tests largely serve to validate answers provided to the prior 1970 questions, and can also provide answers where a supplier is unwilling 1971 to disclose compliance or performance. 1973 Performance testing is the domain of the IETF Benchmark Methodology 1974 Working Group (BMWG). Below are brief descriptions of conformance 1975 and performance tests. Some very basic tests are specified in 1976 [RFC5695] which partially cover only the basic performance test T#3. 1978 The following tests should be performed by the systems designer, or 1979 deployer, or performed by the supplier on their behalf if it is not 1980 practical for the potential customer to perform the tests directly. 1981 These tests are grouped into broad categories. 1983 The tests in Section 4.1 should be repeated under various conditions 1984 to retest basic performance when critical capabilities are enabled. 1985 Complete repetition of the performance tests enabling each capability 1986 and combinations of capabilities would be very time intensive, 1987 therefore a reduced set of performance tests can be used to gauge the 1988 impact of enabling specific capabilities. 1990 4.1. Basic Compliance 1992 T#1 Test forwarding at a high rate for packets with varying number 1993 of label entries. While packets with more than a dozen label 1994 entries are unlikely to be used in any practical scenario today, 1995 it is useful to know if limitations exists. 1997 T#2 For each of the questions listed under "Basic Compliance" in 1998 Section 3, verify the claimed compliance. For any functionality 1999 considered critical to a deployment, where applicable performance 2000 using each capability under load should be verified in addition 2001 to basic compliance. 2003 4.2. Basic Performance 2005 T#3 Test packet forwarding at full line rate with small packets. 2006 See [RFC5695]. The most likely case to fail is the smallest 2007 packet size. Also test with packet sizes in four byte increments 2008 ranging from payload sizes or 40 to 128 bytes. 2010 T#4 If the prior tests did not succeed for all packet sizes, then 2011 perform the following tests. 2013 A. Increase the packet size by 4 bytes until a size is found 2014 that can be forwarded at full rate. 2016 B. Inject bursts of consecutive small packets into a stream of 2017 larger packets. Allow some time for recovery between bursts. 2018 Increase the number of packets in the burst until packets are 2019 dropped. 2021 T#5 Send test traffic where a swap operation is required. Also set 2022 up multiple LSP carried over other LSP where the device under 2023 test (DUT) is the egress of these LSP. Create test packets such 2024 that the swap operation is performed after pop operations, 2025 increasing the number of pop operations until forwarding of small 2026 packets at full line rate can no longer be supported. Also check 2027 to see how many pop operations can be supported before the full 2028 set of counters can no longer be maintained. This requirement is 2029 particularly relevant for MPLS-TP. 2031 T#6 Send all traffic on one LSP and see if the counters become 2032 inaccurate. Often counters on silicon are much smaller than the 2033 64 bit packet and byte counters in various IETF MIBs. System 2034 developers should consider what counter polling rate is necessary 2035 to maintain accurate counters and whether those polling rates are 2036 practical. Relevant MIBs for MPLS are discussed in [RFC4221] and 2037 [RFC6639]. 2039 T#7 [RFC6894] provides a good basis for MPLS FRR testing. Similar 2040 testing should be performed to determine restoration times, 2041 however this testing is far more difficult to perform due to the 2042 need for a simulated test topology that is capable of simulating 2043 the signaling used in restoration. The simulated topology should 2044 be comparable with the target deployment in the number of nodes 2045 and links and in resource usage flooding and setup delays. Some 2046 commercial test equipment can support this type of testing. 2048 4.3. Multipath Capabilities and Performance 2050 Multipath capabilities do not apply to MPLS-TP but apply to MPLS and 2051 apply if MPLS-TP is carried in MPLS. 2053 T#8 Send traffic at a rate well exceeding the capacity of a single 2054 multipath component link, and where entropy exists only below the 2055 top of stack. If only the top label is used this test will fail 2056 immediately. 2058 T#9 Move the labels with entropy down in the stack until either the 2059 full forwarding rate can no longer be supported or most or all 2060 packets try to use the same component link. 2062 T#10 Repeat the two tests above with the entropy contained in IP 2063 headers or IP payload fields below the label stack rather than in 2064 the label stack. Test with the set of IP headers or IP payload 2065 fields considered relevant to the deployment or to the target 2066 market. 2068 T#11 Determine whether traffic that contains a pseudowire control 2069 word is interpreted as IP traffic. Information in the payload 2070 MUST NOT be used in the load balancing if the first nibble of the 2071 packet is not 4 or 6 (IPv4 or IPv6). 2073 T#12 Determine whether special purpose labels and extended special 2074 purpose labels are excluded from the label stack hash. They MUST 2075 be excluded. 2077 T#13 Perform testing in the presence of combinations of: 2079 A. Very large microflows. 2081 B. Relatively short lived high capacity flows. 2083 C. Extremely large numbers of flows. 2085 D. Very short lived small flows. 2087 4.4. Pseudowire Capabilities and Performance 2089 T#14 Ensure that pseudowire can be set up with a pseudowire label and 2090 pseudowire control word added at ingress and the pseudowire label 2091 and pseudowire control word removed at egress. 2093 T#15 For pseudowire that contains variable length payload packets, 2094 repeat performance tests listed under "Basic Performance" for 2095 pseudowire ingress and egress functions. 2097 T#16 Repeat pseudowire performance tests with and without a 2098 pseudowire control word. 2100 T#17 Determine whether pseudowire can be set up with a pseudowire 2101 label, flow label, and pseudowire control word added at ingress 2102 and the pseudowire label, flow label, and pseudowire control word 2103 removed at egress. 2105 T#18 Determine which payload fields are used to create the flow label 2106 and whether the set of fields and algorithm provide sufficient 2107 entropy for load balancing. 2109 T#19 Repeat pseudowire performance tests with flow labels included. 2111 4.5. Entropy Label Support and Performance 2113 T#20 Determine whether entropy labels can be added at ingress and 2114 removed at egress. 2116 T#21 Determine which fields are used to create an entropy label. 2117 Labels further down in the stack, including entropy labels 2118 further down and IP headers or IP payload fields where applicable 2119 should be used. Determine whether the set of fields and 2120 algorithm provide sufficient entropy for load balancing. 2122 T#22 Repeat performance tests under "Basic Performance" when entropy 2123 labels are used, where ingress or egress is the device under test 2124 (DUT). 2126 T#23 Determine whether an ELI is detected when acting as a midpoint 2127 LSR and whether the search for further information on which to 2128 base the load balancing is used. Information below the entropy 2129 label SHOULD NOT be used. 2131 T#24 Ensure that the entropy label indicator and entropy label (ELI 2132 and EL) are removed from the label stack during UHP and PHP 2133 operations. 2135 T#25 Insure that operations on the TC field when adding and removing 2136 entropy label are correctly carried out. If TC is changed during 2137 a swap operation, the ability to transfer that change MUST be 2138 provided. The ability to suppress the transfer of TC MUST also 2139 be provided. See "pipe", "short pipe", and "uniform" models in 2140 [RFC3443]. 2142 T#26 Repeat performance tests for a midpoint LSR with entropy labels 2143 found at various label stack depths. 2145 4.6. DoS Protection 2147 T#27 Actively attack LSR under high protocol churn load and determine 2148 control plane performance impact or successful DoS under test 2149 conditions. Specifically test for the following. 2151 A. TCP SYN attack against control plane and management plane 2152 protocols using TCP, including CLI access (typically SSH 2153 protected login), NETCONF, etc. 2155 B. High traffic volume attack against control plane and 2156 management plane protocols not using TCP. 2158 C. Attacks which can be performed from a compromised management 2159 subnet computer, but not one with authentication keys. 2161 D. Attacks which can be performed from a compromised peer within 2162 the control plane (internal domain and external domain). 2163 Assume that per peering keys and per router ID keys rather 2164 than network wide keys are in use. 2166 See Section 2.6.1. 2168 4.7. OAM Capabilities and Performance 2170 T#28 Determine maximum sustainable rates of BFD traffic. If BFD 2171 requires CPU intervention, determine both maximum rates and CPU 2172 loading when multiple interfaces are active. 2174 T#29 Verify LSP Ping and LSP Traceroute capability. 2176 T#30 Determine maximum rates of MPLS-TP CC-CV traffic. If CC-CV 2177 requires CPU intervention, determine both maximum rates and CPU 2178 loading when multiple interfaces are active. 2180 T#31 Determine MPLS-TP DM precision. 2182 T#32 Determine MPLS-TP LM accuracy. 2184 T#33 Verify MPLS-TP AIS/RDI and Protection State Coordination (PSC) 2185 functionality, protection speed, and AIS/RDI notification speed 2186 when a large number of Management Entities (ME) must be notified 2187 with AIS/RDI. 2189 5. Acknowledgements 2191 Numerous very useful comments have been received in private email. 2192 Some of these contributions are acknowledged here, approximately in 2193 chronologic order. 2195 Paul Doolan provided a brief review resulting in a number of 2196 clarifications, most notably regarding on-chip vs. system buffering, 2197 100 Gb/s link speed assumptions in the 150 Mpps figure, and handling 2198 of large microflows. Pablo Frank reminded us of the sawtooth effect 2199 in PPS vs. packet size graphs, prompting the addition of a few 2200 paragraphs on this. Comments from Lou Berger at IETF-85 prompted the 2201 addition of Section 2.7. 2203 Valuable comments were received on the BMWG mailing list. Jay 2204 Karthik pointed out testing methodology hints that after discussion 2205 were deemed out of scope and were removed but may benefit later work 2206 in BMWG. 2208 Nabil Bitar pointed out the need to cover QoS (Differentiated 2209 Services), MPLS multicast (P2MP and MP2MP), and MPLS-TP OAM. Nabil 2210 also provided a number of clarifications to the questions and tests 2211 in Section 3 and Section 4. 2213 Mark Szczesniak provided a thorough review and a number of useful 2214 comments and suggestions that improved the document. 2216 Gregory Mirsky and Thomas Beckhaus provided useful comments during 2217 the MPLS RT review. 2219 Tal Mizrahi provided comments that prompted clarifications regarding 2220 timestamp processing, local delivery of packets, and the need for 2221 hardware assistance in processing OAM traffic. 2223 Alexander (Sasha) Vainshtein pointed out errors in Section 2.1.8.1 2224 and suggested new text which after lengthy discussion resulted in 2225 restating the summarization of requirements from PWE3 RFCs and more 2226 clearly stating the benefits and drawbacks of packet resequencing 2227 based on PW sequence number. 2229 Loa Anderson provided useful comments and corrections prior to WGLC. 2230 Adrian Farrel provided useful comments and corrections prior as part 2231 of the AD review. 2233 Discussion with Steve Kent during SecDir review resulted in expansion 2234 of Section 7, briefly summarizing security considerations related to 2235 forwarding in normative references. Tom Petch pointed out some 2236 editorial errors in private email plus an important math error. Al 2237 Morton during OpsDir review prompted clarification in the target 2238 audience section, suggested more clear wording in places, and found 2239 numerous editorial errors. 2241 Discussion with Steward Bryant and Alia Atlas as part of IESG review 2242 resulted in coverage of IPFIX and improvements to document coverage 2243 of MPLS FRR, and IP/LDP FRR, plus some corrections to the text 2244 elsewhere. 2246 6. IANA Considerations 2248 This memo includes no request to IANA. 2250 7. Security Considerations 2252 This document reviews forwarding behavior specified elsewhere and 2253 points out compliance and performance requirements. As such it 2254 introduces no new security requirements or concerns. 2256 Discussion of hardware support and other equipment hardening against 2257 DoS attack can be found in Section 2.6.1. Section 3.6 provides a 2258 list of question regarding DoS to be asked of suppliers. Section 4.6 2259 suggests types of testing that can provide some assurance of the 2260 effectiveness of supplier DoS hardening claims. 2262 Knowledge of potential performance shortcomings may serve to help new 2263 implementations avoid pitfalls. It is unlikely that such knowledge 2264 could be the basis of new denial of service as these pitfalls are 2265 already widely known in the service provider community and among 2266 leading equipment suppliers. In practice extreme data and packet 2267 rate are needed to affect existing equipment and to affect networks 2268 that may be still vulnerable due to failure to implement adequate 2269 protection. The extreme data and packet rates make this type of 2270 denial of service unlikely and make undetectable denial of service of 2271 this type impossible. 2273 The set of normative references each contain security considerations. 2274 A brief summarization of MPLS security considerations applicable to 2275 forwarding follows: 2277 1. MPLS encapsulation does not support an authentication extension. 2278 This is reflected in the security section of [RFC3032]. 2279 Documents which clarify MPLS header fields such as TTL 2280 [RFC3443], the explicit null label [RFC4182], renaming EXP to TC 2281 [RFC5462], ECN for MPLS [RFC5129], and MPLS Ethernet 2282 encapsulation [RFC5332] make no changes to security 2283 considerations in [RFC3032]. 2285 2. Some cited RFCs are related to Diffserv forwarding. [RFC3270] 2286 refers to MPLS and Diffserv security. [RFC2474] mentions theft 2287 of service and denial of service due to mismarking. [RFC2474] 2288 mentions IPsec interaction, but with MPLS, not being carried by 2289 IP, this type of interaction in [RFC2474] is not relevant. 2291 3. [RFC3209] is cited here due only to make-before-break forwarding 2292 requirements. This is related to resource sharing and the theft 2293 of service and denial of service concerns in [RFC2474] apply. 2295 4. [RFC4090] defines FRR which provides protection but does not add 2296 security concerns. RFC4201 defines link bundling but raises no 2297 additional security concerns. 2299 5. Various OAM control channels are defined in [RFC4385] (PW CW), 2300 [RFC5085] (VCCV), [RFC5586] (G-Ach and GAL). These documents 2301 describe potential abuse of these OAM control channels. 2303 6. [RFC4950] defines ICMP extensions when MPLS TTL expires and 2304 payload is IP. This provides MPLS header information which is 2305 of no use to an IP attacker, but sending this information can be 2306 suppressed through configuration. 2308 7. GTSM [RFC5082] provides a means to improve protection against 2309 high traffic volume spoofing as a form of DoS attack. 2311 8. BFD [RFC5880] [RFC5884] [RFC5885] provides a form of OAM used in 2312 MPLS and MPLS-TP. The security considerations related to the 2313 OAM control channel are relevant. The BFD payload supports 2314 authentication unlike the MPLS encapsulation or MPLS or PW 2315 control channel encapsulation is carried in. Where an IP return 2316 OAM path is used IPsec is suggested as a means of securing the 2317 return path. 2319 9. Other forms of OAM are supported by [RFC6374] [RFC6375] (Loss 2320 and Delay Measurement), [RFC6428] (Connectivity Check/ 2321 Verification based on BFD), and [RFC6427] (Fault Management). 2322 The security considerations related to the OAM control channel 2323 are relevant. IP return paths, where used, can be secured with 2324 IPsec. 2326 10. Linear protection is defined by [RFC6378] and updated by 2327 [I-D.ietf-mpls-psc-updates]. Security concerns related to MPLS 2328 encapsulation and OAM control channels apply. Security concerns 2329 reiterate [RFC5920] as applied to protection switching. 2331 11. The PW Flow Label [RFC6391] and MPLS Entropy Label [RFC6790] 2332 affect multipath load balancing. Security concerns reiterate 2334 [RFC5920]. Security impacts would be limited to load 2335 distribution. 2337 MPLS security including data plane security is discussed in greater 2338 detail in [RFC5920] (MPLS/GMPLS Security Framework). The MPLS-TP 2339 security framework [RFC6941] build upon this, focusing largely on the 2340 MPLS-TP OAM additions and OAM channels with some attention given to 2341 using network management in place of control plane setup. In both 2342 security framework documents MPLS is assumed to run within a "trusted 2343 zone", defined as being where a single service provider (SP) has 2344 total operational control over that part of the network. 2346 If control plane security and management plane security are 2347 sufficiently robust, compromise of a single network element may 2348 result in chaos in the data plane anywhere in the network through 2349 denial of service attacks, but not a Byzantine security failure in 2350 which other network elements are fully compromised. 2352 MPLS security, or lack of, can affect whether traffic can be 2353 misrouted and lost, or intercepted, or intercepted and reinserted (a 2354 man-in-the-middle attack) or spoofed. End user applications, 2355 including control plane and management plane protocols used by the 2356 SP, are expected to make use of appropriate end-to-end authentication 2357 and where appropriate end-to-end encryption. 2359 8. Organization of References Section 2361 The References section is split into Normative and Informative 2362 subsections. References that directly specify forwarding 2363 encapsulations or behaviors are listed as normative. References 2364 which describe signaling only, though normative with respect to 2365 signaling, are listed as informative. They are informative with 2366 respect to MPLS forwarding. 2368 9. References 2370 9.1. Normative References 2372 [I-D.ietf-mpls-psc-updates] 2373 Osborne, E., "Updates to PSC", draft-ietf-mpls-psc- 2374 updates-01 (work in progress), January 2014. 2376 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 2377 Requirement Levels", BCP 14, RFC 2119, March 1997. 2379 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2380 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2381 Encoding", RFC 3032, January 2001. 2383 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2384 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2385 Tunnels", RFC 3209, December 2001. 2387 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2388 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2389 Protocol Label Switching (MPLS) Support of Differentiated 2390 Services", RFC 3270, May 2002. 2392 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2393 in Multi-Protocol Label Switching (MPLS) Networks", RFC 2394 3443, January 2003. 2396 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 2397 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, May 2398 2005. 2400 [RFC4182] Rosen, E., "Removing a Restriction on the use of MPLS 2401 Explicit NULL", RFC 4182, September 2005. 2403 [RFC4201] Kompella, K., Rekhter, Y., and L. Berger, "Link Bundling 2404 in MPLS Traffic Engineering (TE)", RFC 4201, October 2005. 2406 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2407 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2408 Use over an MPLS PSN", RFC 4385, February 2006. 2410 [RFC4950] Bonica, R., Gan, D., Tappan, D., and C. Pignataro, "ICMP 2411 Extensions for Multiprotocol Label Switching", RFC 4950, 2412 August 2007. 2414 [RFC5082] Gill, V., Heasley, J., Meyer, D., Savola, P., and C. 2415 Pignataro, "The Generalized TTL Security Mechanism 2416 (GTSM)", RFC 5082, October 2007. 2418 [RFC5085] Nadeau, T. and C. Pignataro, "Pseudowire Virtual Circuit 2419 Connectivity Verification (VCCV): A Control Channel for 2420 Pseudowires", RFC 5085, December 2007. 2422 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2423 Marking in MPLS", RFC 5129, January 2008. 2425 [RFC5332] Eckert, T., Rosen, E., Aggarwal, R., and Y. Rekhter, "MPLS 2426 Multicast Encapsulations", RFC 5332, August 2008. 2428 [RFC5586] Bocci, M., Vigoureux, M., and S. Bryant, "MPLS Generic 2429 Associated Channel", RFC 5586, June 2009. 2431 [RFC5880] Katz, D. and D. Ward, "Bidirectional Forwarding Detection 2432 (BFD)", RFC 5880, June 2010. 2434 [RFC5884] Aggarwal, R., Kompella, K., Nadeau, T., and G. Swallow, 2435 "Bidirectional Forwarding Detection (BFD) for MPLS Label 2436 Switched Paths (LSPs)", RFC 5884, June 2010. 2438 [RFC5885] Nadeau, T. and C. Pignataro, "Bidirectional Forwarding 2439 Detection (BFD) for the Pseudowire Virtual Circuit 2440 Connectivity Verification (VCCV)", RFC 5885, June 2010. 2442 [RFC6374] Frost, D. and S. Bryant, "Packet Loss and Delay 2443 Measurement for MPLS Networks", RFC 6374, September 2011. 2445 [RFC6375] Frost, D. and S. Bryant, "A Packet Loss and Delay 2446 Measurement Profile for MPLS-Based Transport Networks", 2447 RFC 6375, September 2011. 2449 [RFC6378] Weingarten, Y., Bryant, S., Osborne, E., Sprecher, N., and 2450 A. Fulignoli, "MPLS Transport Profile (MPLS-TP) Linear 2451 Protection", RFC 6378, October 2011. 2453 [RFC6391] Bryant, S., Filsfils, C., Drafz, U., Kompella, V., Regan, 2454 J., and S. Amante, "Flow-Aware Transport of Pseudowires 2455 over an MPLS Packet Switched Network", RFC 6391, November 2456 2011. 2458 [RFC6427] Swallow, G., Fulignoli, A., Vigoureux, M., Boutros, S., 2459 and D. Ward, "MPLS Fault Management Operations, 2460 Administration, and Maintenance (OAM)", RFC 6427, November 2461 2011. 2463 [RFC6428] Allan, D., Swallow Ed. , G., and J. Drake Ed. , "Proactive 2464 Connectivity Verification, Continuity Check, and Remote 2465 Defect Indication for the MPLS Transport Profile", RFC 2466 6428, November 2011. 2468 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2469 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2470 RFC 6790, November 2012. 2472 9.2. Informative References 2474 [ACK-compression] 2475 , , and , "Observations and Dynamics of a Congestion 2476 Control Algorithm: The Effects of Two-Way Traffic", Proc. 2477 ACM SIGCOMM, ACM Computer Communications Review (CCR) Vol 2478 21, No 4, 1991, pp.133-147., 1991. 2480 [I-D.ietf-mpls-in-udp] 2481 Xu, X., Sheth, N., Yong, L., Pignataro, C., and F. 2482 Yongbing, "Encapsulating MPLS in UDP", draft-ietf-mpls-in- 2483 udp-05 (work in progress), January 2014. 2485 [I-D.ietf-mpls-special-purpose-labels] 2486 Kompella, K., Andersson, L., and A. Farrel, "Allocating 2487 and Retiring Special Purpose MPLS Labels", draft-ietf- 2488 mpls-special-purpose-labels-03 (work in progress), July 2489 2013. 2491 [I-D.ietf-rtgwg-mrt-frr-architecture] 2492 Atlas, A., Kebler, R., Envedi, G., Csaszar, A., Tantsura, 2493 J., Konstantynowicz, M., and R. White, "An Architecture 2494 for IP/LDP Fast-Reroute Using Maximally Redundant Trees", 2495 draft-ietf-rtgwg-mrt-frr-architecture-03 (work in 2496 progress), July 2013. 2498 [I-D.ietf-rtgwg-remote-lfa] 2499 Bryant, S., Filsfils, C., Previdi, S., Shand, M., and S. 2500 Ning, "Remote LFA FRR", draft-ietf-rtgwg-remote-lfa-04 2501 (work in progress), November 2013. 2503 [I-D.ietf-tictoc-1588overmpls] 2504 Davari, S., Oren, A., Bhatia, M., Roberts, P., and L. 2505 Montini, "Transporting Timing messages over MPLS 2506 Networks", draft-ietf-tictoc-1588overmpls-05 (work in 2507 progress), June 2013. 2509 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, September 2510 1981. 2512 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 2513 "Definition of the Differentiated Services Field (DS 2514 Field) in the IPv4 and IPv6 Headers", RFC 2474, December 2515 1998. 2517 [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., 2518 and W. Weiss, "An Architecture for Differentiated 2519 Services", RFC 2475, December 1998. 2521 [RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, 2522 "Assured Forwarding PHB Group", RFC 2597, June 1999. 2524 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2525 Label Switching Architecture", RFC 3031, January 2001. 2527 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2528 of Explicit Congestion Notification (ECN) to IP", RFC 2529 3168, September 2001. 2531 [RFC3429] Ohta, H., "Assignment of the 'OAM Alert Label' for 2532 Multiprotocol Label Switching Architecture (MPLS) 2533 Operation and Maintenance (OAM) Functions", RFC 3429, 2534 November 2002. 2536 [RFC3471] Berger, L., "Generalized Multi-Protocol Label Switching 2537 (GMPLS) Signaling Functional Description", RFC 3471, 2538 January 2003. 2540 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 2541 Jacobson, "RTP: A Transport Protocol for Real-Time 2542 Applications", STD 64, RFC 3550, July 2003. 2544 [RFC3828] Larzon, L-A., Degermark, M., Pink, S., Jonsson, L-E., and 2545 G. Fairhurst, "The Lightweight User Datagram Protocol 2546 (UDP-Lite)", RFC 3828, July 2004. 2548 [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- 2549 Edge (PWE3) Architecture", RFC 3985, March 2005. 2551 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, "Encapsulating 2552 MPLS in IP or Generic Routing Encapsulation (GRE)", RFC 2553 4023, March 2005. 2555 [RFC4110] Callon, R. and M. Suzuki, "A Framework for Layer 3 2556 Provider-Provisioned Virtual Private Networks (PPVPNs)", 2557 RFC 4110, July 2005. 2559 [RFC4124] Le Faucheur, F., "Protocol Extensions for Support of 2560 Diffserv-aware MPLS Traffic Engineering", RFC 4124, June 2561 2005. 2563 [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) 2564 Hierarchy with Generalized Multi-Protocol Label Switching 2565 (GMPLS) Traffic Engineering (TE)", RFC 4206, October 2005. 2567 [RFC4221] Nadeau, T., Srinivasan, C., and A. Farrel, "Multiprotocol 2568 Label Switching (MPLS) Management Overview", RFC 4221, 2569 November 2005. 2571 [RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram 2572 Congestion Control Protocol (DCCP)", RFC 4340, March 2006. 2574 [RFC4377] Nadeau, T., Morrow, M., Swallow, G., Allan, D., and S. 2575 Matsushima, "Operations and Management (OAM) Requirements 2576 for Multi-Protocol Label Switched (MPLS) Networks", RFC 2577 4377, February 2006. 2579 [RFC4379] Kompella, K. and G. Swallow, "Detecting Multi-Protocol 2580 Label Switched (MPLS) Data Plane Failures", RFC 4379, 2581 February 2006. 2583 [RFC4664] Andersson, L. and E. Rosen, "Framework for Layer 2 Virtual 2584 Private Networks (L2VPNs)", RFC 4664, September 2006. 2586 [RFC4817] Townsley, M., Pignataro, C., Wainner, S., Seely, T., and 2587 J. Young, "Encapsulation of MPLS over Layer 2 Tunneling 2588 Protocol Version 3", RFC 4817, March 2007. 2590 [RFC4875] Aggarwal, R., Papadimitriou, D., and S. Yasukawa, 2591 "Extensions to Resource Reservation Protocol - Traffic 2592 Engineering (RSVP-TE) for Point-to-Multipoint TE Label 2593 Switched Paths (LSPs)", RFC 4875, May 2007. 2595 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 2596 Cost Multipath Treatment in MPLS Networks", BCP 128, RFC 2597 4928, June 2007. 2599 [RFC4960] Stewart, R., "Stream Control Transmission Protocol", RFC 2600 4960, September 2007. 2602 [RFC5036] Andersson, L., Minei, I., and B. Thomas, "LDP 2603 Specification", RFC 5036, October 2007. 2605 [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. 2606 Meyer, "Information Model for IP Flow Information Export", 2607 RFC 5102, January 2008. 2609 [RFC5286] Atlas, A. and A. Zinin, "Basic Specification for IP Fast 2610 Reroute: Loop-Free Alternates", RFC 5286, September 2008. 2612 [RFC5317] Bryant, S. and L. Andersson, "Joint Working Team (JWT) 2613 Report on MPLS Architectural Considerations for a 2614 Transport Profile", RFC 5317, February 2009. 2616 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2617 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2618 Class" Field", RFC 5462, February 2009. 2620 [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, 2621 "Architecture for IP Flow Information Export", RFC 5470, 2622 March 2009. 2624 [RFC5640] Filsfils, C., Mohapatra, P., and C. Pignataro, "Load- 2625 Balancing for Mesh Softwires", RFC 5640, August 2009. 2627 [RFC5695] Akhter, A., Asati, R., and C. Pignataro, "MPLS Forwarding 2628 Benchmarking Methodology for IP Flows", RFC 5695, November 2629 2009. 2631 [RFC5704] Bryant, S., Morrow, M., and IAB, "Uncoordinated Protocol 2632 Development Considered Harmful", RFC 5704, November 2009. 2634 [RFC5714] Shand, M. and S. Bryant, "IP Fast Reroute Framework", RFC 2635 5714, January 2010. 2637 [RFC5715] Shand, M. and S. Bryant, "A Framework for Loop-Free 2638 Convergence", RFC 5715, January 2010. 2640 [RFC5860] Vigoureux, M., Ward, D., and M. Betts, "Requirements for 2641 Operations, Administration, and Maintenance (OAM) in MPLS 2642 Transport Networks", RFC 5860, May 2010. 2644 [RFC5905] Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network 2645 Time Protocol Version 4: Protocol and Algorithms 2646 Specification", RFC 5905, June 2010. 2648 [RFC5920] Fang, L., "Security Framework for MPLS and GMPLS 2649 Networks", RFC 5920, July 2010. 2651 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 2652 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 2653 Acronym in the IETF", BCP 161, RFC 6291, June 2011. 2655 [RFC6310] Aissaoui, M., Busschbach, P., Martini, L., Morrow, M., 2656 Nadeau, T., and Y(J). Stein, "Pseudowire (PW) Operations, 2657 Administration, and Maintenance (OAM) Message Mapping", 2658 RFC 6310, July 2011. 2660 [RFC6371] Busi, I. and D. Allan, "Operations, Administration, and 2661 Maintenance Framework for MPLS-Based Transport Networks", 2662 RFC 6371, September 2011. 2664 [RFC6388] Wijnands, IJ., Minei, I., Kompella, K., and B. Thomas, 2665 "Label Distribution Protocol Extensions for Point-to- 2666 Multipoint and Multipoint-to-Multipoint Label Switched 2667 Paths", RFC 6388, November 2011. 2669 [RFC6424] Bahadur, N., Kompella, K., and G. Swallow, "Mechanism for 2670 Performing Label Switched Path Ping (LSP Ping) over MPLS 2671 Tunnels", RFC 6424, November 2011. 2673 [RFC6425] Saxena, S., Swallow, G., Ali, Z., Farrel, A., Yasukawa, 2674 S., and T. Nadeau, "Detecting Data-Plane Failures in 2675 Point-to-Multipoint MPLS - Extensions to LSP Ping", RFC 2676 6425, November 2011. 2678 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2679 On-Demand Connectivity Verification and Route Tracing", 2680 RFC 6426, November 2011. 2682 [RFC6435] Boutros, S., Sivabalan, S., Aggarwal, R., Vigoureux, M., 2683 and X. Dai, "MPLS Transport Profile Lock Instruct and 2684 Loopback Functions", RFC 6435, November 2011. 2686 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label 2687 for Equal Cost Multipath Routing and Link Aggregation in 2688 Tunnels", RFC 6438, November 2011. 2690 [RFC6478] Martini, L., Swallow, G., Heron, G., and M. Bocci, 2691 "Pseudowire Status for Static Pseudowires", RFC 6478, May 2692 2012. 2694 [RFC6639] King, D. and M. Venkatesan, "Multiprotocol Label Switching 2695 Transport Profile (MPLS-TP) MIB-Based Management 2696 Overview", RFC 6639, June 2012. 2698 [RFC6669] Sprecher, N. and L. Fang, "An Overview of the Operations, 2699 Administration, and Maintenance (OAM) Toolset for MPLS- 2700 Based Transport Networks", RFC 6669, July 2012. 2702 [RFC6670] Sprecher, N. and KY. Hong, "The Reasons for Selecting a 2703 Single Solution for MPLS Transport Profile (MPLS-TP) 2704 Operations, Administration, and Maintenance (OAM)", RFC 2705 6670, July 2012. 2707 [RFC6720] Pignataro, C. and R. Asati, "The Generalized TTL Security 2708 Mechanism (GTSM) for the Label Distribution Protocol 2709 (LDP)", RFC 6720, August 2012. 2711 [RFC6829] Chen, M., Pan, P., Pignataro, C., and R. Asati, "Label 2712 Switched Path (LSP) Ping for Pseudowire Forwarding 2713 Equivalence Classes (FECs) Advertised over IPv6", RFC 2714 6829, January 2013. 2716 [RFC6894] Papneja, R., Vapiwala, S., Karthik, J., Poretsky, S., Rao, 2717 S., and JL. Le Roux, "Methodology for Benchmarking MPLS 2718 Traffic Engineered (MPLS-TE) Fast Reroute Protection", RFC 2719 6894, March 2013. 2721 [RFC6941] Fang, L., Niven-Jenkins, B., Mansfield, S., and R. 2722 Graveman, "MPLS Transport Profile (MPLS-TP) Security 2723 Framework", RFC 6941, April 2013. 2725 [RFC6981] Bryant, S., Previdi, S., and M. Shand, "A Framework for IP 2726 and MPLS Fast Reroute Using Not-Via Addresses", RFC 6981, 2727 August 2013. 2729 [RFC7023] Mohan, D., Bitar, N., Sajassi, A., DeLord, S., Niger, P., 2730 and R. Qiu, "MPLS and Ethernet Operations, Administration, 2731 and Maintenance (OAM) Interworking", RFC 7023, October 2732 2013. 2734 [RFC7074] Berger, L. and J. Meuric, "Revised Definition of the GMPLS 2735 Switching Capability and Type Fields", RFC 7074, November 2736 2013. 2738 [RFC7079] Del Regno, N. and A. Malis, "The Pseudowire (PW) and 2739 Virtual Circuit Connectivity Verification (VCCV) 2740 Implementation Survey Results", RFC 7079, November 2013. 2742 9.3. URIs 2744 [1] http://www.iana.org 2746 Authors' Addresses 2748 Curtis Villamizar (editor) 2749 Outer Cape Cod Network Consulting, LLC 2751 Email: curtis@occnc.com 2753 Kireeti Kompella 2754 Juniper Networks 2756 Email: kireeti@juniper.net 2757 Shane Amante 2758 Apple Inc. 2759 1 Infinite Loop 2760 Cupertino, California 95014 2762 Email: samante@apple.com 2764 Andrew Malis 2765 Huawei Technologies 2767 Email: agmalis@gmail.com 2769 Carlos Pignataro 2770 Cisco Systems 2771 7200-12 Kit Creek Road 2772 Research Triangle Park, NC 27709 2773 US 2775 Email: cpignata@cisco.com