idnits 2.17.00 (12 Aug 2021) /tmp/idnits33874/draft-ietf-detnet-dp-alt-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (September 20, 2016) is 2062 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'Network' is mentioned on line 165, but not defined -- Looks like a reference, but probably isn't: '1' on line 2306 -- Looks like a reference, but probably isn't: '2' on line 2308 -- Looks like a reference, but probably isn't: '3' on line 2310 -- Looks like a reference, but probably isn't: '4' on line 2312 -- Looks like a reference, but probably isn't: '5' on line 2315 -- Looks like a reference, but probably isn't: '6' on line 2317 -- Looks like a reference, but probably isn't: '46' on line 1848 == Outdated reference: draft-ietf-6man-rfc2460bis has been published as RFC 8200 == Outdated reference: draft-ietf-6man-segment-routing-header has been published as RFC 8754 == Outdated reference: draft-ietf-bier-architecture has been published as RFC 8279 == Outdated reference: draft-ietf-bier-mpls-encapsulation has been published as RFC 8296 == Outdated reference: draft-ietf-detnet-problem-statement has been published as RFC 8557 == Outdated reference: draft-ietf-mpls-residence-time has been published as RFC 8169 == Outdated reference: draft-ietf-spring-segment-routing has been published as RFC 8402 == Outdated reference: A later version (-02) exists of draft-ooamdt-rtgwg-ooam-requirement-01 -- Obsolete informational reference (is this intentional?): RFC 2460 (Obsoleted by RFC 8200) -- Obsolete informational reference (is this intentional?): RFC 4447 (Obsoleted by RFC 8077) Summary: 0 errors (**), 0 flaws (~~), 10 warnings (==), 11 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 DetNet J. Korhonen, Ed. 3 Internet-Draft Broadcom 4 Intended status: Informational J. Farkas 5 Expires: March 24, 2017 G. Mirsky 6 Ericsson 7 P. Thubert 8 Cisco 9 Y. Zhuang 10 Huawei 11 L. Berger 12 LabN 13 September 20, 2016 15 DetNet Data Plane Protocol and Solution Alternatives 16 draft-ietf-detnet-dp-alt-00 18 Abstract 20 This document identifies existing IP and MPLS, and other 21 encapsulations that run over IP and/or MPLS data plane technologies 22 that can be considered as the base line solution for deterministic 23 networking data plane definition. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at http://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on March 24, 2017. 42 Copyright Notice 44 Copyright (c) 2016 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents 49 (http://trustee.ietf.org/license-info) in effect on the date of 50 publication of this document. Please review these documents 51 carefully, as they describe your rights and restrictions with respect 52 to this document. Code Components extracted from this document must 53 include Simplified BSD License text as described in Section 4.e of 54 the Trust Legal Provisions and are provided without warranty as 55 described in the Simplified BSD License. 57 Table of Contents 59 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 60 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 61 3. DetNet Data Plane Overview . . . . . . . . . . . . . . . . . 3 62 3.1. Example DetNet Service Scenarios . . . . . . . . . . . . 6 63 4. Criteria for data plane solution alternatives . . . . . . . . 8 64 4.1. #1 Encapsulation and overhead . . . . . . . . . . . . . . 9 65 4.2. #2 Flow identification . . . . . . . . . . . . . . . . . 9 66 4.3. #3 Packet sequencing and duplicate elimination . . . . . 9 67 4.4. #4 Explicit routes . . . . . . . . . . . . . . . . . . . 10 68 4.5. #5 Flow duplication and merging . . . . . . . . . . . . . 10 69 4.6. #6 Operations, Administration and Maintenance . . . . . . 11 70 4.7. #8 Class and quality of service capabilities . . . . . . 11 71 4.8. #9 Packet traceability . . . . . . . . . . . . . . . . . 12 72 4.9. #10 Technical maturity . . . . . . . . . . . . . . . . . 12 73 5. Data plane solution alternatives . . . . . . . . . . . . . . 12 74 5.1. DetNet Transport layer technologies . . . . . . . . . . . 13 75 5.1.1. Native IPv6 transport . . . . . . . . . . . . . . . . 13 76 5.1.2. Native IPv4 transport . . . . . . . . . . . . . . . . 16 77 5.1.3. Multiprotocol Label Switching (MPLS) . . . . . . . . 19 78 5.1.4. Bit Indexed Explicit Replication (BIER) . . . . . . . 23 79 5.2. DetNet Service layer technologies . . . . . . . . . . . . 27 80 5.2.1. Generic Routing Encapsulation (GRE) . . . . . . . . . 27 81 5.2.2. MPLS-based Services for DetNet . . . . . . . . . . . 29 82 5.2.3. Pseudo Wire Emulation Edge-to-Edge (PWE3) . . . . . . 30 83 5.2.4. MPLS-Based Ethernet VPN (EVPN) . . . . . . . . . . . 34 84 5.2.5. Higher layer header fields . . . . . . . . . . . . . 36 85 6. Summary of data plane alternatives . . . . . . . . . . . . . 38 86 7. Security considerations . . . . . . . . . . . . . . . . . . . 40 87 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 40 88 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 41 89 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 41 90 10.1. Informative References . . . . . . . . . . . . . . . . . 41 91 10.2. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 50 92 Appendix A. Examples of combined DetNet Service and Transport 93 layers . . . . . . . . . . . . . . . . . . . . . . . 50 94 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 50 96 1. Introduction 98 Deterministic Networking (DetNet) [I-D.ietf-detnet-problem-statement] 99 provides a capability to carry unicast or multicast data flows for 100 real-time applications with extremely low data loss rates, timely 101 delivery and bounded packet delay variation 102 [I-D.finn-detnet-architecture]. The deterministic networking Quality 103 of Service (QoS) is expressed as 1) the minimum and the maximum end- 104 to-end latency from source (talker) to destination (listener), and 2) 105 probability of loss of a packet. Only the worst-case values for the 106 mentioned parameters are concerned. 108 There are three techniques to achieve the QoS required by 109 deterministic networks: 111 o Congestion protection, 112 o explicit routes, 113 o service protection. 115 This document identifies existing IP and Multiprotocol Label 116 Switching (MPLS) [RFC3031], layer-2 or layer-3 encapsulations and 117 transport protocols that could be considered as foundations for a 118 deterministic networking data plane. The full scope of the 119 deterministic networking data plane solution is considered including, 120 as appropriate: quality of service (QoS); Operations, Administration 121 and Maintenance (OAM); and time synchronization among other criteria 122 described in Section 4. 124 This document does not select a deterministic networking data plane 125 protocol. It does, however, elaborate what it would require to adapt 126 and use a specific protocol as the deterministic networking data 127 plane solution. This document is only concerned with data plane 128 considerations and, specifically, with topics that potentially impact 129 potential deterministic networking aware data plane hardware. 130 Control plane considerations are out of scope of this document. 132 2. Terminology 134 This document uses the terminology established in the DetNet 135 architecture [I-D.finn-detnet-architecture]. 137 3. DetNet Data Plane Overview 139 A "Deterministic Network" will be composed of DetNet enabled nodes 140 i.e., End Systems, Edge Nodes, Relay Nodes and collectively deliver 141 DetNet services. DetNet enabled nodes are interconnected via Transit 142 Nodes (i.e., routers) which support DetNet, but are not DetNet 143 service aware. Transit nodes see DetNet nodes as end points. All 144 DetNet enabled nodes are connect to sub-networks, where a point-to- 145 point link is also considered as a simple sub-network. These sub- 146 networks will provide DetNet compatible service for support of DetNet 147 traffic. Examples of sub-networks include IEEE 802.1TSN and OTN. Of 148 course, multi-layer DetNet systems may also be possible, where one 149 DetNet appears as a sub-network, and provides service to, a higher 150 layer DetNet system. A simple DetNet concept network is shown in 151 Figure 1. 153 TSN Edge Transit Relay DetNet 154 End System Node Node Node End System 156 +---------+ +.........+ +---------+ 157 | Appl. |<---:Svc Proxy:-- End to End Service ---------->| Appl. | 158 +---------+ +---------+ +---------+ +---------+ 159 | TSN | |TSN| |Svc|<-- DetNet flow ---: Service :-->| Service | 160 +---------+ +---+ +---+ +---------+ +---------+ +---------+ 161 |Transport| |Trp| |Trp| |Transport| |Trp| |Trp| |Transport| 162 +-------.-+ +-.-+ +-.-+ +--.----.-+ +-.-+ +-.-+ +---.-----+ 163 : Link : / ,-----. \ : Link : / ,-----. \ 164 +........+ +-[ Sub ]-+ +........+ +-[ Sub ]-+ 165 [Network] [Network] 166 `-----' `-----' 168 Figure 1: A Simple DetNet Enabled Network 170 The DetNet data plane is logically divided into two layers (also see 171 Figure 2): 173 DetNet Service Layer 175 The DetNet service layer provides adaptation of DetNet services. 176 It is composed of a shim layer to carry deterministic flow 177 specific attributes, which are needed during forwarding and for 178 service protection. DetNet enabled end systems originate and 179 terminate the DetNet Service layer and are peers at the DetNet 180 Service layer. DetNet relay and edge nodes also implement DetNet 181 Service layer functions. The DetNet service layer is used to 182 deliver traffic end to end across a DetNet domain. 184 DetNet Transport Layer 186 The DetNet transport layer is required on all DetNet nodes. All 187 DetNet nodes are end points at the transport layer. Non-DetNet 188 service aware transit nodes deliver traffic between DetNet nodes. 189 The DetNet transport layer operates below and supports the DetNet 190 Service layer and optionally provides congestion protection for 191 DetNet flows. 193 Distinguishing the function of these two DetNet data plane layers 194 helps to explore and evaluate various combinations of the data plane 195 solutions available. This separation of DetNet layers, while 196 helpful, should not be considered as formal requirement. For 197 example, some technologies may violate these strict layers and still 198 be able to deliver a DetNet service. 200 . 201 . 202 +-----------+ 203 | Service | PW, RTP/(UDP), GRE 204 +-----------+ 205 | Transport | (UDP)/IPv6, (UDP)/IPv4, MPLS LSPs, BIER 206 +-----------+ 207 . 208 . 210 Figure 2: DetNet adaptation to data plane 212 The two logical layers defined here aim to help to identify which 213 data plane technology can be used for what purposes in the DetNet 214 context. This layering is similar to the data plane concept of MPLS, 215 where some part of the label stack is "Service" specific (e.g., PW 216 labels, VPN labels) and an other part is "Transport" specific (e.g, 217 LSP label, TE label(s)). 219 In some networking scenarios, the end system initially provides a 220 DetNet flow encapsulation, which contains all information needed by 221 DetNet nodes (e.g., Real-time Transport Protocol (RTP) [RFC3550] 222 based DetNet flow transported over a native UDP/IP network or 223 PseudoWire). In other scenarios, the encapsulation formats might 224 differ significantly. As an example, a CPRI "application's" I/Q data 225 mapped directly to Ethernet frames may have to be transported over an 226 MPLS-based packet switched network (PSN). 228 There are many valid options to create a data plane solution for 229 DetNet traffic by selecting a technology approach for the DetNet 230 Service layer and also selecting a technology approach for the DetNet 231 Transport layer. There are a high number of valid combinations. 232 Therefore, not the combinations but the different technologies are 233 evaluated along the criteria collected in Section 4. Different 234 criteria apply for the DetNet Service layer and the DetNet Transport 235 layer, however, some of the criteria are valid for both layers. 237 One of the most fundamental differences between different potential 238 data plane options is the basic addressing and headers used by DetNet 239 end systems. For example, is the basic service a Layer 2 (e.g., 240 Ethernet) or Layer 3 (i.e., IP) service. This decision impacts how 241 DetNet end systems are addressed, and the basic forwarding logic for 242 the DetNet Service layer. 244 3.1. Example DetNet Service Scenarios 246 In an attempt to illustrate a DetNet date plane, this document uses 247 the Multi-Segment Pseudowire Emulation Edge-to-Edge (PWE3) [RFC5254] 248 reference model shown in Figure 3 as the foundation for different 249 DetNet data plane deployment options and how layering could work. 250 Other reference models are possible but not covered in this document. 251 Note that other technologies can be also used to implement DetNet, 252 Multi-Segment PW is only used here to illustrate functions, features 253 and layering from the perspective of the architecture. 255 Native |<--------Multi-Segment Pseudowire----->| Native 256 Service | PSN PSN | Service 257 (AC) | |<-Tunnel->| |<-Tunnel->| | (AC) 258 | V V 1 V V 2 V V | 259 | +-----+ +-----+ +---- + | 260 +---+ | |T-PE1|==========|S-PE1|==========|T-PE2| | +---+ 261 | |---|-----|........PW1...........|...PW3..........|---|----| | 262 |CE1| | | | | | | | | |CE2| 263 | |---------|........PW2...........|...PW4..........|--------| | 264 +---+ | | |==========| |==========| | | +---+ 265 ^ +-----+ +-----+ +-----+ ^ 266 | Provider Edge 1 ^ Provider Edge 3 | 267 | | | 268 | PW switching point | 269 | | 270 |<------------------- Emulated Service ------------------->| 272 Figure 3: Pseudo Wire switching reference model 274 Figure 4 illustrates how DetNet can provide services for IEEE 275 802.1TSN end systems over a DetNet enabled network. The edge nodes 276 insert and remove required DetNet data plane encapsulation. The 'X' 277 in the edge and relay nodes represents a potential DetNet flow packet 278 replication and elimination point. This conceptually parallels L2VPN 279 services, and could leverage existing related solutions as discussed 280 below. 282 TSN |<----- End to End DetNet Service ----->| TSN 283 Service | Transit Transit | Service 284 TSN (AC) | |<-Tunnel->| |<-Tunnel->| | (AC) TSN 285 End | V V 1 V V 2 V V | End 286 System | +-----+ +-----+ +---- + | System 287 +---+ | |T-PE1|==========|S-PE1|==========|T-PE2| | +---+ 288 | |---|-----|.X_..DetNet Flow1..X..|...DF3........X.|---|----| | 289 |CE1| | | \ | | | | / | | |CE2| 290 | | |...X_...DF2........X..|...DF4......X_..| | | 291 +---+ | |==========| |==========| | +---+ 292 ^ +-----+ +-----+ +-----+ ^ 293 | Edge Node Relay Node Edge Node | 294 | | 295 |<--------------- Emulated TSN Service ------------------->| 297 Figure 4: IEEE 802.1TSN over DetNet 299 Figure 5 illustrates how end to end native DetNet service can be 300 provided. In this case, the end systems are able to send and receive 301 native DetNet flows. For example, as PseudoWire (PW) encapsulated 302 IP. Like earlier the 'X' in the end systems, edge and relay nodes 303 represents potential DetNet flow packet replication and elimination 304 points. Here the relay nodes may change the underlying transport, 305 for example replacing IP with MPLS or tunneling IP over MPLS (e.g., 306 via L3VPNs), or simply interconnect network domains. 308 DetNet DetNet 309 Service Transit Transit Service 310 DetNet | |<-Tunnel->| |<-Tunnel->| | DetNet 311 End | V 1 V V 2 V | End 312 System | +-----+ +-----+ +-----+ | System 313 +---+ | |S-PE1|==========|S-PE2|==========|S-PE3| | +---+ 314 | X....DFa.....X_.......DF1.......X_....DF3........X.....DFa...X | 315 |CE1|=========| \ | | / | | / |========|CE2| 316 | | | | \......DF2.....X_......DF4....../ | | | | 317 +---+ | |==========| |==========| | +---+ 318 ^ +-----+ +-----+ +-----+ ^ 319 | Relay Node Relay Node Relay Node | 320 | | 321 |<------------- End to End DetNet Service ---------------->| 323 Figure 5: Native DetNet 325 Figure 6 illustrates how a IEEE 802.1TSN end system could communicate 326 with a native DetNet end system through an edge node which provides a 327 TSN to DetNet inter-working capability. The edge node would add and 328 remove required DetNet data plane encapsulation as well as provide 329 any needed address mapping. As in previous figures, the 'X' in the 330 end systems, edge and relay nodes represents potential DetNet flow 331 packet duplication and elimination points. 333 TSN |<----- End to End DetNet Service -------------->| 334 Service | Transit Transit | 335 TSN (AC) | |<-Tunnel->| |<-Tunnel->| DetNet | DetNet 336 End | V V 1 V V 2 V Service | End 337 System | +-----+ +-----+ +-----+ | V System 338 +---+ | |T-PE1|==========|S-PE1|==========|S-PE2| | +---+ 339 | |---|-----|.X_.......DF1......X..|...DF3........X.|...DFa...X | 340 |CE1| | | \ | | | | / |========|CE2| 341 | | | \.....DF2.......X..|...DF4....../ | | | | 342 +---+ | |==========| |==========| | +---+ 343 ^ +-----+ +-----+ +-----+ ^ 344 | Edge Node Relay Node Relay Node | 345 | | 346 |<----------------- End to End Service ------------------->| 348 Figure 6: IEEE 802.1TSN to native DetNet 350 4. Criteria for data plane solution alternatives 352 This section provides criteria to help to evaluate potential options. 353 Each deterministic networking data plane solution alternative is 354 described and evaluated using the criteria described in this section. 355 The used criteria enumerated in this section are selected so that 356 they highlight the existence or lack of features that are expected or 357 seen important to a solution alternative for the data plane solution. 359 The criteria for the DetNet Service layer: 361 #1 Encapsulation and overhead 362 #2 Flow identification (Service ID part of the DetNet flows) 363 #3 Packet sequencing and duplicate elimination 364 #5 Flow duplication and merging 365 #6 Operations, Administration and Maintenance (capabilities) 366 #8 Class and quality of service capabilities (DetNet Service 367 specific) 368 #10 Technical maturity 370 The criteria for the DetNet Transport layer: 372 #1 Encapsulation and overhead 373 #2 Flow identification 374 #4 Explicit routes (network path) 375 #5 Flow duplication and merging (sometimes, flow duplication and 376 merging is also doable at the transport layer, not just at the 377 service layer) 379 #6 Operations, Administration and Maintenance (capabilities, 380 performance management, packet traceability) 381 #8 Class and quality of service capabilities (DetNet Transport 382 specific) 383 #9 Packet traceability (can be part of OAM) 384 #10 Technical maturity 386 [Editor's Note: numbering is off because #7 is removed.] 388 [Editor's Note: #9 should(?) be integrated into #6.] 390 Most of the criteria is relevant for both the DetNet Service and 391 DetNet Transport layers. However, different aspects of the same 392 criteria may relevant for different layers, for example, as it is the 393 case with criteria #5 Packet replication and elimination. 395 4.1. #1 Encapsulation and overhead 397 Encapsulation and overhead is related to how the DetNet data plane 398 carries DetNet flow. In several cases a DetNet flow has to be 399 encapsulated inside other protocols, for example, when transporting a 400 layer-2 Ethernet frame over an IP transport network. In some cases a 401 tunneling like encapsulation can be avoided by underlying transport 402 protocol translation, for example, translating layer-2 Ethernet frame 403 including addressing and flow identification into native IP traffic. 404 Last it is possible that sources and destinations handle 405 deterministic flows natively in layer-3. This criteria concerns what 406 is the encapsulation method the solution alternative support: 407 tunneling like encapsulation, protocol translation or native layer-3 408 transport. In addition to the encapsulation mechanism this criteria 409 is also concerned with the processing and specifically the 410 encapsulation header overhead. 412 4.2. #2 Flow identification 414 The solution alternative has to provide means to identify specific 415 deterministic flows. The flow identification can, for example, be an 416 explicit field in the data plane encapsulation header or implicitly 417 encoded into the addressing scheme of the used data plane protocol or 418 their combination. This criteria concerns the availability and 419 details of deterministic flow identification the data plane protocol 420 alternative has. 422 4.3. #3 Packet sequencing and duplicate elimination 424 The solution alternative has to provide means for end systems to 425 number packets sequentially and transport that sequencing information 426 along with the sent packets. In addition to possible reordering of 427 packets other important uses for sequencing are detecting duplicates 428 and lost packets. 430 In a case of intentional packet duplication a combination of flow 431 identification and packet sequencing allows for detecting and 432 eliminating duplicates at the destination (see Section 4.5 for more 433 details). 435 4.4. #4 Explicit routes 437 The solution alternative has to provide a mechanism(s) for 438 establishing explicit routes that all packets belonging to a 439 deterministic flow will follow. The explicit route can be seen as a 440 form of source routing or a pre-reserved path e.g., using some 441 network management procedure. It should be noted that the explicit 442 route does not need to be detailed to a level where every possible 443 intermediate node along the path is part of the named explicit route. 444 RSVP-TE [RFC3209] supports explicit routes, and typically provides 445 pinned data paths for established LSPs. At Layer-2, the IEEE 446 802.1Qca [IEEE802.1Qca] specification defines how to do explicit path 447 control in a bridged network and its IETF counter part is defined in 448 [RFC7813]. This criteria concerns the available mechanisms for 449 explicit routes for the data plane protocol alternative. 451 4.5. #5 Flow duplication and merging 453 Flow duplication and flow merging are methods being considered to 454 provide DetNet service protection. The objective for supporting flow 455 duplication and flow merging is to enable hitless (or lossless) 1+1 456 protection. Other methods, if so identified, are also permissible. 458 The solution alternative has to provide means for end systems, relay 459 and edge nodes to be able to duplicate packets into duplicate flows, 460 and later merge the flows into one for duplicate elimination. The 461 duplication and merging may take place at multiple points in the 462 network in order to ensure that one (or more) equipment failure 463 event(s) still leave at least one path intact for a deterministic 464 networking flow. The goal is again to enable hitless 1+1 protection 465 in a way that no packet gets lost or there is no ramp up time when 466 either one of the paths fails for one reason or another. 468 Another concern regarding packet duplication is how to enforce 469 duplicated packets to take different route or path while the final 470 destination still remains the same. With strict source routing, all 471 the intermediate hops are listed and paths can be guaranteed to be 472 non-overlapping. Loose source routing only signals some of the 473 intermediate hops and it takes additional knowledge to ensure that 474 there is no single point of failure. 476 The IEEE 802.1CB (seamless redundancy) [IEEE8021CB] is an example of 477 Ethernet-based solution that defines packet sequence numbering, flow 478 duplication, flow merging, duplicate packet identification and 479 elimination. The deterministic networking data plane solution 480 alternative at layer-3 has to provide equivalent functionality. This 481 criteria concerns the available mechanisms for packet replication and 482 duplicate deletion the data plane protocol alternative has. 484 4.6. #6 Operations, Administration and Maintenance 486 The solution alternative should demonstrate availability of 487 appropriate standardized OAM tools that can be extended for 488 deterministic networking purposes with a reasonable effort, when 489 required. The OAM tools do not necessarily need to be specific to 490 the data plane protocol as it could be the case, for example, with 491 MPLS-based data planes. But any OAM-related implications or 492 requirements on data plane hardware must be considered. 494 The OAM includes but is not limited to tools listed in the 495 requirements for overlay networks 496 [I-D.ooamdt-rtgwg-ooam-requirement]. Specifically, the performance 497 management requirements are of interest at both service and transport 498 layers. 500 4.7. #8 Class and quality of service capabilities 502 Class and quality of service, i.e., CoS and QoS, are terms that are 503 often used interchangeably and confused. In the context of DetNet, 504 CoS is used to refer to mechanisms that provide traffic forwarding 505 treatment based on aggregate group basis and QoS is used to refer to 506 mechanisms that provide traffic forwarding treatment based on a 507 specific DetNet flow basis. Examples of CoS mechanisms include 508 DiffServ which is enabled by IP header differentiated services code 509 point (DSCP) field [RFC2474] and MPLS label traffic class field 510 [RFC5462], and at Layer-2, by IEEE 802.1p priority code point (PCP). 512 Quality of Service (QoS) mechanisms for flow specific traffic 513 treatment typically includes a guarantee/agreement for the service, 514 and allocation of resources to support the service. Example QoS 515 mechanisms include discrete resource allocation, admission control, 516 flow identification and isolation, and sometimes path control, 517 traffic protection, shaping, policing and remarking. Example 518 protocols that support QoS control include Resource ReSerVation 519 Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473]. 521 A critical DetNet service enabled by QoS (and perhaps CoS) is 522 delivering zero congestion loss. There are different mechanisms that 523 maybe used separately or in combination to deliver a zero congestion 524 loss service. The key aspect of this objective is that DetNet 525 packets are not discarded due to congestion at any point in a DetNet 526 aware network. 528 In the context of the data plane solution there should be means for 529 flow identification, which then can be used to map a flow against 530 specific resources and treatment in a node enforcing the QoS. For 531 DetNet, certain aspects of CoS and QoS may be provided by the 532 underlying sub-net technology, e.g., actual queuing or IEEE 802.3x 533 priority flow control (PFC). 535 4.8. #9 Packet traceability 537 For the network management and specifically for tracing 538 implementation or network configuration errors any means to find out 539 whether a packet is a replica, which node performed replication, and 540 which path was intended for the replica, can be very useful. This 541 criteria concerns the availability of solutions for tracing packets 542 in the context of data plane protocol alternative. Packet 543 traceability can also be part of OAM. 545 4.9. #10 Technical maturity 547 The technical maturity of the data plane solution alternative is 548 crucial, since it basically defines the effort, time line and risks 549 involved for the use of the solution in deployments. For example, 550 the maturity level can be categorized as available immediately, 551 available with small extensions, available with re-purposing/ 552 redefining portions of the protocol or its header fields. Yet 553 another important measure for maturity is the deployment experience. 554 This criteria concerns the maturity of the data plane protocol 555 alternative as the solution alternative. This criteria is 556 particularly important given, as previously noted, that the DetNet 557 data plane solution is expected to impact, i.e., be supported in, 558 hardware. 560 5. Data plane solution alternatives 562 The following sections describe and rate deterministic data plane 563 solution alternatives. In "Analysis and Discussion" section each 564 alternative is evaluated against the criteria given in Section 4 and 565 rated using the following: (M)eets the criteria, (W)ork needed, and 566 (N)ot suitable or too much work envisioned. 568 5.1. DetNet Transport layer technologies 570 5.1.1. Native IPv6 transport 572 5.1.1.1. Solution description 574 This section investigates the application of native IPv6 [RFC2460] as 575 the data plane for deterministic networking along the criteria 576 collected in Section 4. 578 The application of higher OSI layer headers, i.e., headers deeper in 579 the packet, can be considered. Two aspects have to be taken into 580 account for such solutions. (i) Those header fields can be encrypted. 581 (ii) Those header fields are deeper in the packet, therefore, routers 582 have to apply deep packet inspection. See further details in 583 Section 5.2.5. 585 5.1.1.2. Analysis and Discussion 587 #1 Encapsulation and overhead (M) 589 IPv6 can encapsulate DetNet Service layer headers (and associated 590 DetNet flow payload) like any other upper-layer header indicated 591 by the Next Header. The fixed header of an IPv6 packet is 40 592 bytes [RFC2460]. This overhead is bigger if any Extension Header 593 is used, and a generic behaviour for host and forwarding nodes is 594 specified in [RFC7045]. However, the exact overhead (Section 4.1) 595 depends on what solution is actually used to provide DetNet 596 features, e.g., explicit routing or DetNet service protection if 597 any of these is applied. 599 IPv6 has two types of Extension Headers that are processed by 600 intermediate routers between the source and the final destination 601 and may be of interest for the data plane signaling, the Routing 602 Header that is used to direct the traffic via intermediate routers 603 in a strict or loose source routing way, and the Hop-by-Hop 604 Options Header that carries optional information that must be 605 examined by every node along a packet's delivery path. The Hop- 606 by-Hop Options Header, when present, must immediately follow the 607 IPv6 Header and it is not possible to limit its processing to the 608 end points of Source Routed segments. 610 IPv6 also provides a Destination Options Header that is used to 611 carry optional information to be examined only by a packet's 612 destination node(s). The encoding of the options used in the Hop- 613 by-Hop and in the Destination Options Header indicates the 614 expected behavior when a processing IPv6 node does not recognize 615 the Option Type, e.g. skip or drop; it should be noted that due to 616 performance restrictions nodes may ignore the Hop-by-Hop Option 617 Header, drop packets containing a Hop-by-Hop Option Header, or 618 assign packets containing a Hop-by-Hop Option Header to a slow 619 processing path [I-D.ietf-6man-rfc2460bis] (e.g. punt packets from 620 hardware to software forwarding which is highly detrimental to the 621 performance). 623 The creation of new Extension Headers that would need to be 624 processed by intermediate nodes is strongly discouraged. In 625 particular, new Extension Header(s) having hop-by-hop behavior 626 must not be created or specified. New options for the existing 627 Hop-by-Hop Header should not be created or specified unless no 628 alternative solution is feasible [RFC6564]. 630 #2 Flow identification (W) 632 The 20-bit flow label field of the fixed IPv6 header is suitable 633 to distinguish different deterministic flows. But guidance on the 634 use of the flow label provided by [RFC6437] places restrictions on 635 how the flow label can be used. In particular, labels should be 636 chosen from an approximation to a discrete uniform distribution. 637 Additionally, existing implementations generally do not open APIs 638 to control the flow label from the upper layers. 640 Alternatively, the Flow identification could be transported in a 641 new option in the Hop-by-Hop Options Header. 643 #4 Explicit routes (W) 645 One possibility is for a Software-Defined Networking (SDN) 646 [RFC7426] based approach to be applied to compute, establish and 647 manage the explicit routes, leveraging Traffic Engineering (TE) 648 extensions to routing protocols [RFC5305] [RFC7752] and evolving 649 to the Path Computation Element (PCE) Architecture [RFC5440], 650 though a number of issues remain to be solved [RFC7399]. 652 Segment Routing (SR) [I-D.ietf-spring-segment-routing] is a new 653 initiative to equip IPv6 with explicit routing capabilities. The 654 idea for the DetNet data plane would be to apply SR to IPv6 with 655 the addition of a new type of routing extension header 656 [I-D.ietf-6man-segment-routing-header] to explicitly signal the 657 path in the data plane between the source and the destination, 658 and/or between replication points and elimination points if this 659 functionality is used. 661 #5 Flow duplication and merging (W) 662 The functionality of replicating a packet exists in IPv6 but is 663 limited to multicast flows. In order to enforce replicated 664 packets to take different routes and eventually again merge flow 665 (bring them to a specific merging point), IP-in-IP encapsulation 666 and Segment Routing could be leveraged to signal a segment in a 667 packet. A replication point would insert a different routing 668 header in each copy it makes, the routing header providing 669 explicitly the hops to the merging point for that particular 670 replica of the packet, in a strict or in a loose source routing 671 fashion. A flow merging point would pop the routing headers from 672 the various copies it gets and do the rest of the required 673 processing for merging the two flows into one flow. 675 #6 Operations, Administration and Maintenance (M/W) 677 IPv6 enjoys the existing toolbox for generic IP network 678 management. However, IPv6 specific management features are still 679 not at the level comparable to that of IPv4. Particular areas of 680 concerns are those that are IPv6 specific, for example, related to 681 neighbor discovery protocol (ND), stateless address 682 autoconfiguration (SLAAC), subscriber identification, and 683 security. While the standards are already mostly in place the 684 implementations in deployed equipment can be lacking or inadequate 685 for commercial deployments. This is larger issue with older 686 existing equipment. 688 #8 Class and quality of service capabilities (W) 690 IPv6 provides support for CoS and QoS. CoS is provided by 691 DiffServ which is enabled by IP header differentiated services 692 code point (DSCP) and QoS is defined as part of RSVP [RFC2205]. 693 DiffServ support is widely available, while RSVP for IP packets is 694 generally not supported. 696 #9 Packet traceability (W) 698 The traceability of replicated packets involves the capability to 699 resolve which replication point issued a particular copy of a 700 packet, which segment was intended for that replica, and which 701 particular packet of which particular flow this is. Sequence also 702 depends on the sequencing mechanism. As an example, the 703 replication point may be indicated as the source of the packet if 704 IP-in-IP encapsulation is used to forward along segments. Another 705 alternate to IP-in-IP tunneling along segments would be to protect 706 the original source address in a destination option similar to the 707 Home Address option [RFC6275] and then use the address of the 708 replication point as source in the IP header. 710 The traceability also involves the capability to determine if a 711 particular segment is operational. While IPv6 as such has no 712 support for reversing a path, it appears source route extensions 713 such as the one defined for segment routing could be used for 714 tracing purposes. Though it is not a usual practice, IPv6 715 [RFC2460] expects that a Source Route path may be reversed, and 716 the standard insists that a node must not include the reverse of a 717 Routing Header in the response unless the received Routing Header 718 was authenticated. 720 #10 Technical maturity (M/W) 722 IPv6 has been around about 20 years. However, large scale global 723 and commercial IPv6 deployments are rather new dating only few 724 years back to around 2012. While IPv6 has proven itself for best 725 effort traffic, DiffServ usage is less common and QoS capabilities 726 are not currently present. Additional, there are number of small 727 issues to work on as they show up once operations experience 728 grows. 730 The Cisco 6Lab site [1] provides information on IPv6 deployment 731 per country, indicating figures for prefixes, transit AS, content 732 and users. Per this site, many countries, including Canada, 733 Brazil, the USA, Germany, France, Japan, Portugal, Sweden, 734 Finland, Norway, Greece, and Ecuador, achieve a deployment ratio 735 above 30 percent, and the overall adoption reported by Google 736 Statistics [2] is now above 10 percent. 738 5.1.1.3. Summary 740 IPv6 supports a significant portion of the identified DetNet data 741 plane criteria today. There are aspects of the DetNet data plane 742 that are not fully supported, notably QoS, but these can be 743 incrementally added or supplemented by the underlying sub-network 744 layer. IPv6 may be a choice as the DetNet Transport layer in 745 networks where other technologies such as MPLS are not deployed. 747 5.1.2. Native IPv4 transport 749 5.1.2.1. Solution description 751 IPv4 [RFC0791] is in principle the same as IPv6, except that it has a 752 smaller address space. However, IPv6 was designed around the fact 753 that extension headers are an integral part of the protocol and 754 operation from the beginning, although the practice may some times 755 prove differently [RFC7872]. IPv4 does support header options, but 756 these have historically not been supported in hardware-based 757 forwarding so are generally blocked or handled at a much slower rate. 758 In either case, the use of IP header options is generally avoided. 759 In the context of deterministic networking data plane solutions the 760 major difference between IPv4 and IPv6 seems to be the practical 761 support for header extensibility. Anything below and above the IP 762 header independent of the version is practically the same. 764 5.1.2.2. Analysis and Discussion 766 #1 Encapsulation and overhead (M) 768 The fixed header of an IPv4 packet is 20 bytes [RFC0791]. IP 769 options add overhead, but are not generally used and are not 770 considered as part of this document. 772 #2 Flow identification (W) 774 The IPv4 header has a 16-bit identification field that was 775 originally intended for assisting fragmentation and reassembly of 776 IPv4 packets as described in [RFC0791]. The identification field 777 has also been proposed to be used for actually identifying flows 778 between two IP addresses and a given protocol for detecting and 779 removing duplicate packets [RFC1122]. However, recent update 780 [RFC6864] to both [RFC0791] and [RFC1122] restricts the use of 781 IPv4 identification field only to fragmentation purposes. 783 The IPv4 also has a stream identifier option [RFC0791], which 784 contains a 16-bit SATNET stream identifier. However, the option 785 has been deprecated [RFC6814]. The conclusion is that stream 786 identification does not work nicely with IPv4 header alone and a 787 traditional 5-tuple identification might not also be enough in a 788 case of a flow duplication or encrypted flows. For a working 789 solution, upper layer protocol headers such as RTP or PWs may be 790 required for unambiguous flow identification. There is also 791 emerging work within the IETF that may provide new flow 792 identification alternatives. 794 #4 Explicit routes (W) 796 IPv4 has two source routing option specified: the loose source and 797 record route option (LSRR), and the strict source and record route 798 option (SSRR) [RFC0791]. The support of these options in the 799 Internet is questionable but within a closed network the support 800 may be assumed. But as both these options use IP header options, 801 which are generally not supported in hardware, use of these 802 options are questionable. Of course, the same options of SDN and 803 SR approaches discussed above for IPv6 may be equally applicable 804 to IPv4. 806 #5 Flow duplication and merging (W/N) 808 The functionality of replicating a packet exists in IPv4 but is 809 limited to multicast flows. In general the issue regarding the 810 IPv6 packet replication also applies to IPv4. Duplicate packet 811 detection for IPv4 is studied in [RFC6621] to a great detail in 812 the context of simplified multicast forwarding. In general there 813 is no good way to detect duplicated packets for IPv4 without 814 additional upper layer protocol support. 816 #6 Operations, Administration and Maintenance (M) 818 IPv4 enjoys the extensive and "complete" existing toolbox for 819 generic IP network management. 821 #8 Class and quality of service capabilities (M/W) 823 IPv4 provides support for CoS and QoS. CoS is provided by 824 DiffServ which is enabled by IP header differentiated services 825 code point (DSCP) and QoS is defined as part of RSVP [RFC2205]. 826 DiffServ support is widely available, while RSVP for IP packets is 827 generally not supported. 829 #9 Packet traceability (W) 831 The IPv4 has similar needs and requirements for traceability as 832 IPv6 (see Section 5.1.1.2). The IPv4 has a traceroute option 833 [RFC6814] that could be used to record the route the packet took. 834 However, the option has been deprecated [RFC6814]. 836 #10 Technical maturity (M/W) 838 IPv4 can be considered mature technology with over 30 years of 839 implementation, deployment and operations experience. As with 840 IPv6, today's commercial implementations and deployments of IPv4 841 generally lack any support for QoS. 843 5.1.2.3. Summary 845 The IPv4 has specifications to support most of the identified DetNet 846 data plane criteria today. However, several of those have already 847 been deprecated or their wide support is not guaranteed. The DetNet 848 data plane criteria that are not fully supported could be 849 incrementally added or supplemented by the underlying sub-network 850 layer. Unfortunately, the IPv4 has had limited success getting its 851 extensions deployed at large. However, introducing new extensions 852 might have a better success in closed networks (like DetNet) than in 853 Internet. Due to the popularity of the IPv4, it should be considered 854 as a potential choice for the DetNet Transport layer. 856 5.1.3. Multiprotocol Label Switching (MPLS) 858 Multiprotocol Label Switching Architecture (MPLS) [RFC3031] and its 859 variants, MPLS with Traffic Engineering (MPLS-TE) [RFC3209] and 860 [RFC3473], and MPLS Transport Profile (MPLS-TP) [RFC5921] is a widely 861 deployed technology that switches traffic based on MPLS label stacks 862 [RFC3032] and [RFC5960]. MPLS is the foundation for Pseudowire-based 863 services Section 5.2.3 and emerging technologies such as Bit-Indexed 864 Explicit Replication (BIER) Section 5.1.4 and Source Packet Routing 865 [3]. 867 MPLS supports the equivalent of both the DetNet Service and DetNet 868 Transport layers, and provides a very rich set of mechanisms that can 869 be reused directly, and perhaps augmented in certain cases, to 870 deliver DetNet services. At the DetNet Transport layer, MPLS 871 provides forwarding, protection and OAM services. At the DetNet 872 Service Layer it provides client service adaption, directly, via 873 Pseudowires Section 5.2.3 and via other label-like mechanisms such as 874 EPVN Section 5.2.4. A representation of these options are shown in 875 Figure 7. 877 PW-Based EVPN Labeled IP 878 Services Services Transport 879 |------------| |-----------------------------| |------------| 881 Emulated EVPN over LSP EVPN w/ ESI ID IP 882 Service 883 +------------+ 884 | Payload | 885 +------------+ +------------+ +------------+ (Service) 886 | PW Payload | | Payload | |ESI Lbl(S=1)| 887 +------------+ +------------+ +------------+ +------------+ 888 |PW Lbl(S=1) | |VPN Lbl(S=1)| |VPN Lbl(S=0)| | IP | 889 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 890 |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=0)| |LSP Lbl(S=1)| 891 +------------+ +------------+ +------------+ +------------+ 892 . . . . 893 . . . . (Transport) 894 . . . . 896 ~~~~~~~~~~~ denotes DetNet Service <-> DetNet Transport layer boundary 898 Figure 7: MPLS-based Services 900 MPLS can be controlled in a number of ways including via a control 901 plane, via the management plane, or via centralized controller (SDN) 902 based approaches. MPLS also provides standard control plane 903 reference points. Additional information on MPLS architecture and 904 control can be found in [RFC5921]. A summary of MPLS control plane 905 related functions can be found in [RFC6373]. The remainder of this 906 section will focus on the MPLS transport data plane, additional 907 information on the MPLS service data plane can be found below in 908 Section 5.2.2. 910 5.1.3.1. Solution description 912 The following draws heavily from [RFC5960]. 914 Encapsulation and forwarding of packets traversing MPLS LSPs follows 915 standard MPLS packet encapsulation and forwarding as defined in 916 [RFC3031], [RFC3032], [RFC5331], and [RFC5332]. 918 Data plane Quality of Service capabilities are included in the MPLS 919 in the form of Traffic Engineered (TE) LSPs [RFC3209] and the MPLS 920 Differentiated Services (DiffServ) architecture [RFC3270]. Both 921 E-LSP and L-LSP MPLS DiffServ modes are defined. The Traffic Class 922 field (formerly the EXP field) of an MPLS label follows the 923 definition of [RFC5462] and [RFC3270]. 925 Except for transient packet reordering that may occur, for example, 926 during fault conditions, packets are delivered in order on L-LSPs, 927 and on E-LSPs within a specific ordered aggregate. 929 The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL 930 processing models are described in [RFC3270] and [RFC3443] and may be 931 used for MPLS LSPs. 933 Equal-Cost Multi-Path (ECMP) load-balancing is possible with MPLS 934 LSPs and can be avoided using a number of techniques. The same holds 935 for Penultimate Hop Popping (PHP). 937 MPLS includes the following LSP types: 939 o Point-to-point unidirectional 940 o Point-to-point associated bidirectional 941 o Point-to-point co-routed bidirectional 942 o Point-to-multipoint unidirectional 944 Point-to-point unidirectional LSPs are supported by the basic MPLS 945 architecture [RFC3031]. 947 A point-to-point associated bidirectional LSP between LSRs A and B 948 consists of two unidirectional point-to-point LSPs, one from A to B 949 and the other from B to A, which are regarded as a pair providing a 950 single logical bidirectional transport path. 952 A point-to-point co-routed bidirectional LSP is a point-to-point 953 associated bidirectional LSP with the additional constraint that its 954 two unidirectional component LSPs in each direction follow the same 955 path (in terms of both nodes and links). An important property of 956 co-routed bidirectional LSPs is that their unidirectional component 957 LSPs share fate. 959 A point-to-multipoint unidirectional LSP functions in the same manner 960 in the data plane, with respect to basic label processing and packet- 961 switching operations, as a point-to-point unidirectional LSP, with 962 one difference: an LSR may have more than one (egress interface, 963 outgoing label) pair associated with the LSP, and any packet it 964 transmits on the LSP is transmitted out all associated egress 965 interfaces. Point-to-multipoint LSPs are described in [RFC4875] and 966 [RFC5332]. TTL processing and exception handling for point-to- 967 multipoint LSPs is the same as for point-to-point LSPs. 969 Additional data plane capabilities include Linear Protection, 970 [RFC6378] and [RFC7271]. And the in progress work on MPLS support 971 for time synchronization [I-D.ietf-mpls-residence-time]. 973 5.1.3.2. Analysis and Discussion 975 #1 Encapsulation and overhead (M) 977 There are two perspectives to consider when looking at 978 encapsulation. The first is encapsulation to support services. 979 These considerations are part of the DetNet service layer and are 980 covered below, see Sections 5.2.3 and 5.2.4. 982 The second perspective relates to encapsulation, if any, which is 983 needed to transport packets across network. In this case, the 984 MPLS label stack, [RFC3032] is used to identify flows across a 985 network. MPLS labels are compact and highly flexible. They can 986 be stacked to support client adaptation, protection, network 987 layering, source routing, etc. 989 The number of DetNet Transport layer specific labels is flexible 990 and support a wide range of applicable functions and MPLS domain 991 characteristics (e.g., TE-tunnels, Hierarchical-LSPs, etc.). 993 #2 Flow identification (M) 994 MPLS label stacks provide highly flexible ways to identify flows. 995 Basically, they enable the complete separation of traffic 996 classification from traffic treatment and thereby enable arbitrary 997 combinations of both. 999 For the DetNet flow identification the MPLS label stack can be 1000 used to support n-layers of DetNet flow identification. For 1001 example, using dedicated LSP per DetNet flow would simplify flow 1002 identification for intermediate transport nodes, and additional 1003 hierarchical LSPs could be used to facilitate scaling. 1005 #4 Explicit routes (M) 1007 MPLS supports explicit routes based on how LSPs are established, 1008 e.g., via TE explicit routes [RFC3209]. Additional, but not 1009 required, capabilities are being defined as part of Segment 1010 Routing (SR) [I-D.ietf-spring-segment-routing]. 1012 #5 Flow duplication and merging (M/W) 1014 MPLS as DetNet Transport layer supports the replication via point- 1015 to-multipoint LSPs. At the MPLS LSP level, there are mechanisms 1016 defined to provide 1+1 protection, which could help in realizing 1017 the flow merging function. The current definitions [RFC6378] and 1018 [RFC7271] use OAM mechanisms to support and coordinate protection 1019 switching and packet loss is possible during a switch. While such 1020 this level of protection may be sufficient for many DetNet 1021 applications, when truly hitless (i.e., zero loss) switching is 1022 required, additional mechanisms will be needed. It is expected 1023 that these additional mechanisms will be defined at a DetNet 1024 layer. 1026 #6 Operations, Administration and Maintenance (M) 1028 MPLS already includes a rich set of OAM functions at both the 1029 Service and Transport Layers. This includes LSP ping [ref] and 1030 those enabled via the MPLS Generic Associated Channel [RFC5586] 1031 and registered by IANA [4]. 1033 #8 Class and quality of service capabilities (M/W) 1035 As previously mentioned, Data plane Quality of Service 1036 capabilities are included in the MPLS in the form of Traffic 1037 Engineered (TE) LSPs [RFC3209] and the MPLS Differentiated 1038 Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP 1039 MPLS DiffServ modes are defined. The Traffic Class field 1040 (formerly the EXP field) of an MPLS label follows the definition 1041 of [RFC5462] and [RFC3270]. One potential open area of work is 1042 synchronized, time based scheduling. Another is shaping, which is 1043 generally not supported in shipping MPLS hardware. 1045 #9 Packet traceability (M) 1047 MPLS supports multiple tracing mechanisms. A control based one is 1048 defined in [RFC3209]. An OAM based mechanism is defined in MPLS 1049 On-Demand Connectivity Verification and Route Tracing [RFC6426]. 1051 #10 Technical maturity (M) 1053 MPLS as a mature technology that has been widely deployed in many 1054 networks for many years. Numerous vendor products and multiple 1055 generations of MPLS hardware have been built and deployed. 1057 5.1.3.3. Summary 1059 MPLS is a mature technology that has been widely deployed. Numerous 1060 vendor products and multiple generations of MPLS hardware have been 1061 built and deployed. MPLS LSPs support a significant portion of the 1062 identified DetNet data plane criteria today. Aspects of the DetNet 1063 data plane that are not fully supported can be incrementally added. 1064 It's worth noting that a number of limitations are in shipping 1065 hardware, versus at the protocol specification level, e.g., shaping. 1067 5.1.4. Bit Indexed Explicit Replication (BIER) 1069 Bit Indexed Explicit Replication [I-D.ietf-bier-architecture] (BIER) 1070 is a network plane replication technique that was initially intended 1071 as a new method for multicast distribution. In a nutshell, a BIER 1072 header includes a bitmap that explicitly signals the destinations 1073 that are intended for a particular packet, which means that 1) the 1074 source is aware of the individual destinations and 2) the BIER 1075 control plane is a simple extension of the unicast routing as opposed 1076 to a dedicated multicast data plane, which represents a considerable 1077 reduction in OPEX. For this reason, the technology is getting a lot 1078 of traction from Service Providers. In the context of DetNet, BIER 1079 may be applicable for implementing packet replication, as described 1080 in Section 5.1.4. 1082 The encapsulation of a BIER packet in an MPLS network is shown in 1083 Figure 8 1084 0 1 2 3 1085 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1086 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1087 | Label Stack Element | 1088 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1089 | Label Stack Element | 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1091 | BIER-MPLS label | |1| | 1092 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1093 |0 1 0 1| Ver | Len | Entropy | 1094 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1095 | BitString (first 32 bits) ~ 1096 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1097 ~ ~ 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 ~ BitString (last 32 bits) | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1101 |OAM| Reserved | Proto | BFIR-id | 1102 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 Figure 8: BIER packet in MPLS encapsulation 1106 5.1.4.1. Solution description 1108 A distinctive BIER payload type (with its own Proto(col) ID) would be 1109 created for DetNet, providing a header that would identify: 1111 o Version; 1112 o Sequence Number; 1113 o Timestamp; 1114 o Payload type, e.g. data vs. OAM. 1116 A DetNet node, collocated with a BFIR (Bit-Forwarding Ingress Router) 1117 may use multiple BIER sub-domains to create replicated flows. 1118 Downstream DetNet nodes, collocated with BFER (Bit-Forwarding Ingress 1119 Router) would terminate redundant flows based on Sequence Number and/ 1120 or Timestamp information. Thus a DetNet node may be collocated with 1121 a BFER in one BIER sub- domain and with a BFIR in another, and a 1122 DetNet flow could traverse several BIER sub-domains. 1124 +-----+ 1125 | A | 1126 +-----+ 1127 / \ 1128 . . 1129 / . 1130 . \ 1131 / . 1132 . . 1133 / \ 1134 +-----+ +-----+ 1135 | B | | C | 1136 +-----+ +-----+ 1137 / \ / \ 1138 . . . . 1139 / \ . . 1140 . . / \ 1141 / \ . . 1142 . . . . 1143 / \ / \ 1144 +-----+ +-----+ +-----+ 1145 | D | | E | | F | 1146 +-----+ +-----+ +-----+ 1147 \ . . / 1148 . . . . 1149 \ . . . 1150 . . . / 1151 \ . . . 1152 . . . . 1153 \ . . / 1154 +-----+ +-----+ 1155 | G | | H | 1156 +-----+ +-----+ 1158 Figure 9: DetNet in BIER domain 1160 Consider a DetNet flow that must traverse BIER enabled domain from A 1161 to G and H. DetNet may use three BIER subdomains: 1163 o A-B-D-E-G (dash-dot): A is BFIR, E and G are BFERs, 1164 o A-C-E-F-H (dash-double-dot): A is BFIR, E and H are BFERs, 1165 o E-G-H (dotted): E is BFIR, G and H are BFERs. 1167 DetNet node A sends DetNet into red and purple BIER sub-domains. 1168 DetNet node E receives DetNet packet and sends into green sub-domain 1169 while terminating duplicates and those that are deemed "too-late". 1171 DetNet nodes G and H receive DetNet flows, terminate duplicates and 1172 those that are too-late. 1174 5.1.4.2. Analysis and Discussion 1176 #1 Encapsulation and overhead (M) 1178 BIER over MPLS network encapsulation ("BIER over MPLS"), Figure 8, 1179 is being defined [I-D.ietf-bier-mpls-encapsulation] within the 1180 BIER working group. 1182 #2 Flow identification (M) 1184 Flow identification and separation can be achieved through use of 1185 BIER domains and/or Entropy value in the BIER over MPLS, Figure 8. 1187 #4 Explicit routes (M) 1189 Explicit routes may be used as underlay for BIER domain. BIER 1190 underlay may be calculated using PCE and instantiated using any 1191 southbound mechanism. 1193 #5 Flow duplication and merging (M/W) 1195 Packet replication, as indicated by its name, is a core function 1196 of the Bit-Indexed Explicit Replication. Elimination of the 1197 duplicates and/or too-late packets cannot be done within BIER sub- 1198 domain but may be done at DetNet overlay at the edge of the BIER 1199 sub-domain. 1201 [Editor's note: how about the flow merging?] 1203 #6 Operations, Administration and Maintenance (M/W) 1205 BIER over MPLS guarantees that OAM is fate-sharing, i.e. in-band 1206 with a data flow being monitored or measured. Additionally, BIER 1207 over MPLS enables passive performance measurement, e.g. with the 1208 marking method [I-D.mirsky-bier-pmmm-oam]. Existing OAM protocols 1209 can be applied and used in BIER over MPLS as demonstrated in 1210 [I-D.ooamdt-rtgwg-oam-gap-analysis], while new protocols are being 1211 worked on, e.g. ping/traceroute [I-D.kumarzheng-bier-ping] or Path 1212 MTU Discovery [I-D.mirsky-bier-path-mtu-discovery]. 1214 #8 Class and quality of service capabilities (M/W) 1216 Class of Service can be inherited from the underlay of the 1217 particular BIER sub-domain. Quality of Service, i.e. scheduling 1218 and bandwidth reservations can be used among other constraints in 1219 calculating explicit paths for the BIER sub-domain's underlay. 1221 #9 Packet traceability (W) 1223 Ability to do passive performance measurement by using OAM field 1224 of the BIER over MPLS, Figure 8, is unmatched and significantly 1225 simplifies truly passive tracing of selected flows and packets 1226 within them. 1228 #10 Technical maturity (W) 1230 The BIER over MPLS is nearing finalization within the BIER WG and 1231 several experimental implementations are expected soon. 1233 5.1.4.3. Summary 1235 BIER over MPLS supports a significant portion of the identified 1236 DetNet data plane requirements, including controlled packet 1237 replication, traffic engineering, while some requirements, e.g. 1238 duplicate and too-late packet elimination may be realized as function 1239 of the DetNet overlay. BIER over MPLS is a viable candidate as the 1240 DetNet Transport layer in MPLS networks. 1242 5.2. DetNet Service layer technologies 1244 5.2.1. Generic Routing Encapsulation (GRE) 1246 5.2.1.1. Solution description 1248 Generic Routing Encapsulation (GRE) [RFC2784] provides an 1249 encapsulation of an arbitrary network layer protocol over another 1250 arbitrary network layer protocol. The encapsulation of a GRE packet 1251 can be found in Figure 10. 1253 +-------------------------------+ 1254 | Delivery Header | 1255 +-------------------------------+ 1256 | GRE Header | 1257 +-------------------------------+ 1258 | Payload packet | 1259 +-------------------------------+ 1261 Figure 10: Encapsulation of a GRE packet 1263 Based on RFC2784, [RFC2890] further includes sequencing number and 1264 Key in optional fields of the GRE header, which may help to transport 1265 DetNet traffic flows over IP networks. The format of a GRE header is 1266 presented in Figure 11. 1268 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 1269 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1270 |C| |K|S| Reserved0 | Ver | Protocol Type | 1271 +-----------------------------------------------------------------+ 1272 | Checksum (optional) | Reserved1 (Optional) | 1273 +-----------------------------------------------------------------+ 1274 | Key (optional) | 1275 +-----------------------------------------------------------------+ 1276 | Sequence Number (optional) | 1277 +-----------------------------------------------------------------+ 1279 Figure 11: Format of a GRE header 1281 5.2.1.2. Analysis and Discussion 1283 #1 Encapsulation and overhead (M) 1285 GRE can provide encapsulation at the service layer over the 1286 transport layer. A new protocol type for DetNet traffic should be 1287 allocated as an "Ether Type" in [RFC3232] and in IANA Ethernet 1288 Numbers [5]. The fixed header of a GRE packet is 4 octets while 1289 the maximum header is 16 octets with optional fields in Figure 11. 1291 #2 Flow identification (W) 1293 There is no flow identification field in GRE header. However, it 1294 can rely on the flow identification mechanism applied in the 1295 delivery protocols, such as flow identification stated in IP 1296 Sections 5.1.1 and 5.1.2 when the delivery protocols are IPv6 and 1297 IPv4 respectively. Alternatively, the Key field can also be 1298 extended to carry the flow identification. The size of Key field 1299 is 4 octets. 1301 #3 Packet sequencing and duplicate elimination (M/W) 1303 As stated in Section 5.2.1, GRE provides an optional sequencing 1304 number in its header to provide sequencing services for packets. 1305 The size of the sequencing number is 32 bits. The GRE header 1306 could be extended to indicate the duplicated packets by defining a 1307 flag in reserved fields or using the sequencing number of a flow. 1309 #5 Flow duplication and merging (W/N) 1311 GRE has no flow/packet replication and merging support in its 1312 header. It can use the transport IPv4/IPv6 protocols at the 1313 transport layer to replicate the packets and take the different 1314 routes as discussed in Section 5.1.1 and Section 5.1.2. 1316 #6 Operations, Administration and Maintenance (M) 1318 GRE uses the network management provided by the IP protocols as 1319 transport layer. 1321 #8 Class and quality of service capabilities (W) 1323 For the class of service capability, an optional code point field 1324 to indicate CoS of the traffic could be added into the GRE header. 1325 Otherwise, GRE can reuse the class and quality of service of 1326 delivery protocols at transport layer such as IPv6 and IPv4 stated 1327 in Section 5.1.1 and Section 5.1.2. 1329 #10 Technical maturity (M) 1331 GRE has been developed over 20 years. The delivery protocol 1332 mostly used is IPv4, while the IPv6 support for GRE is to be 1333 standardized now in IETF as [RFC7676]. Due to its good 1334 extensibility, GRE has also been extended to support network 1335 virtualization in Data Center, which is NVGRE [RFC7637]. 1337 5.2.1.3. Summary 1339 As a tunneling protocol, GRE can encapsulate a wide variety of 1340 network layer protocols over another network layer, which can 1341 naturally serve as the service layer protocol for DetNet. Currently, 1342 it supports a portion of the Detnet service layer criteria, and still 1343 some are not fully supported but can be incrementally added or 1344 supported by delivery protocols at as the transport layer. In 1345 general, GRE can be a choice as the DetNet service layer and can work 1346 with IPv6 and IPv4 as the DetNet Transport layer. 1348 5.2.2. MPLS-based Services for DetNet 1350 MPLS based technologies support both the DetNet Service and DetNet 1351 Transport layers. This, as well as a general overview of MPLS, is 1352 covered above in Section 5.1.3. Following sections focus on the 1353 DetNet Service Layer which provides client service adaption, via 1354 Pseudowires Section 5.2.3 and via native and other label-like 1355 mechanisms such as EPVN in Section 5.2.4. A representation of these 1356 options was previously discussed and is shown in Figure 7. 1358 The following text is adapted from [RFC5921]: 1360 The MPLS native service adaptation functions interface the client 1361 layer network service to MPLS. For Pseudowires, these adaptation 1362 functions are the payload encapsulation described in Section 4.4 1363 of [RFC3985] and Section 6 of [RFC5659]. For network layer client 1364 services, the adaptation function uses the MPLS encapsulation 1365 format as defined in [RFC3032]. 1367 The purpose of this encapsulation is to abstract the data plane of 1368 the client layer network from the MPLS data plane, thus 1369 contributing to the independent operation of the MPLS network. 1371 MPLS may itself be a client of an underlying server layer. MPLS 1372 can thus also be bounded by a set of adaptation functions to this 1373 server layer network, which may itself be MPLS. These adaptation 1374 functions provide encapsulation of the MPLS frames and for the 1375 transparent transport of those frames over the server layer 1376 network. 1378 While MPLS service can provided on a true end-system to end-system 1379 basis, it's more likely that DetNet service will be provided over 1380 Pseudowires as described in Section 5.2.3 or via an EPVN-based 1381 service described in Section 5.2.4 . 1383 MPLS labels in the label stack may be used to identify transport 1384 paths, see Section 5.1.3, or as service identifiers. Typically a 1385 single label is used for service identification. 1387 Packet sequencing mechanisms are added in client-related 1388 adaptation processing, see Sections 5.2.3 and 5.2.4. 1390 The MPLS client inherits its Quality of Service (QoS) from the 1391 MPLS transport layer, which in turn inherits its QoS from the 1392 server (sub-network) layer. The server layer therefore needs to 1393 provide the necessary QoS to ensure that the MPLS client QoS 1394 commitments can be satisfied. 1396 5.2.3. Pseudo Wire Emulation Edge-to-Edge (PWE3) 1398 5.2.3.1. Solution description 1400 Pseudo Wire Emulation Edge-to-Edge (PWE3) [RFC3985] or simply 1401 PseudoWires (PW) provide means of emulating the essential attributes 1402 and behaviour of a telecommunications service over a packet switched 1403 network (PSN) using IP or MPLS transport. In addition to traditional 1404 telecommunications services such as T1 line or Frame Relay, PWs also 1405 provide transport for Ethernet service [RFC4448] and for generic 1406 packet service [RFC6658]. Figure 12 illustrate the reference PWE3 1407 stack model. 1409 +----------------+ +----------------+ 1410 |Emulated Service| |Emulated Service| 1411 |(e.g., Eth, ...)|<= Emulated Service =>|(e.g., Eth, ...)| 1412 +----------------+ +----------------+ 1413 | Payload | | Payload | CW, 1414 | Encapsulation |<=== Pseudo Wire ====>| Encapsulation | Timing, 1415 | | | | Seq., .. 1416 +----------------+ +----------------+ 1417 |PW Demultiplexer| |PW Demultiplexer| 1418 | PSN Tunnel, |<==== PSN Tunnel ====>| PSN Tunnel, | MPLS. 1419 | PSN & Physical | | PSN & Physical | L2TP, 1420 | Layers | | Layers | IP, .. 1421 +-------+--------+ ___________ +---------+------+ 1422 | / \ | 1423 +============/ PSN \==============+ 1424 \ / 1425 \___________/ 1427 Figure 12: PWE3 protocol stack reference model 1429 PWs appear as a good data plane solution alternative for a number of 1430 reasons. PWs are a proven and deployed technology with a rich OAM 1431 control plane [RFC4447], and enjoy the toolbox developed for MPLS 1432 networks in a case of MPLS-based PSN. Furthermore, PWs may have an 1433 optional Control Word (CW) as part of the payload encapsulation 1434 between the PSN and the emulated service that is, for example, 1435 capable of frame sequencing and duplicate detection. The 1436 encapsulation layer may also provide timing using RTP as described in 1437 Sections 5.2.2, 5.4.1 and 5.4.2 of [RFC3985] and utilized by 1438 [RFC4553][RFC5087]. Furthermore, advanced DetNet node functions are 1439 conceptually already supported by PW framework (with some added 1440 functional required), such as the DetNet Relay node modeled after the 1441 Multi-Segment PWE3 [RFC5254]. 1443 PWs can be also used if the PSN is IP, which enables the application 1444 of PWs in networks that do not have MPLS enabled in their core 1445 routers. One approach to provide PWs over IP is to provide MPLS over 1446 IP in some way and then leverage what is available for PWs over MPLS. 1447 The following standard solutions are available both for IPv4 and IPv6 1448 to follow this approach. The different solutions have different 1449 overhead as discussed in the following subsection. The MPLS-in-IP 1450 encapsulation is specified by [RFC4023]. The IPv4 Protocol Number 1451 field or the IPv6 Next Header field is set to 137, which indicates an 1452 MPLS unicast packet. (The use of the MPLS-in-IP encapsulation for 1453 MPLS multicast packets is not supported.) The MPLS-in-GRE 1454 encapsulation is specified in [RFC4023], where the IP header (either 1455 IPv4 or IPv6) is followed by a GRE header, which is followed by an 1456 MPLS label stack. The protocol type field in the GRE header is set 1457 to MPLS Unicast (0x8847) or Multicast (0x8848). MPLS over L2TPv3 1458 over IP encapsulation is specified by [RFC4817]. The MPLS-in-UDP 1459 encapsulation is specified by [RFC7510], where the UDP Destination 1460 Port indicates tunneled MPLS packet and the UDP Source Port is an 1461 entropy value that is generated by the encapsulator to uniquely 1462 identify a flow. MPLS-in-UDP encapsulation can be applied to enable 1463 UDP-based ECMP (Equal-Cost Multipath) or Link Aggregation. All these 1464 solutions can be secured with IPsec [RFC4303]. 1466 5.2.3.2. Analysis and Discussion 1468 #1 Encapsulation and overhead (M) 1470 PWs offer encapsulation services practically for practically any 1471 type of payload over any PSN. New PW types need a code point 1472 allocation [RFC4446] and in some cases an emulated service 1473 specific document. 1475 Specifically in the case of the MPLS PSN the PW encapsulation 1476 overhead is minimal. Typically minimum two labels and a CW is 1477 needed, which totals to 12 octets. PW type specific handling 1478 might, however, allow optimizations on the emulated service in the 1479 provider edge (PE) device's native service processing (NSP) / 1480 forwarder function. These optimizations could be used, for 1481 example, to reduce header overhead. Ethernet PWs already have 1482 rather low overhead [RFC4448]. Without a CW and VLAN tags the 1483 Ethernet header gets reduced to 14 octets (minimum Ethernet header 1484 overhead is 26). 1486 The overhead is somewhat bigger in case of IP PSN if an MPLS over 1487 IP solution is applied to provide PWs. IP adds at least 20 (IPv4) 1488 or 40 (IPv6) bytes overhead to the PW over MPLS overhead; 1489 furthermore, the GRE, L2TPv3, or UDP header has to be taken into 1490 account if any of these further encapsulations is used. 1492 #2 Flow identification (M) 1494 PWs provide multiple layers of flow identification, especially in 1495 the case of the MPLS PSN. The PWs are typically prepended with an 1496 endpoint specific PW label that can be used to identify a specific 1497 PW per endpoint. Furthermore, the MPLS PSN also uses one or more 1498 labels to transport packets over specific label switched paths 1499 (that then would carry PWs). So, a DetNet flow can be identified 1500 in this example by the service and transport layer labels. IP 1501 (and other) PSNs may need other mechanisms, such as, UDP port 1502 numbers, upper layer protocol header (like RTP) or some IP 1503 extension header to provide required flow identification. 1505 #3 Packet sequencing and duplicate elimination (M) 1507 As mentioned earlier PWs may contain an optional CW that is able 1508 to provide sequencing services. The size of the sequence number 1509 in the generic CW is 16 bits, which might be, depending on the 1510 used link and DetNet flow speed be too little. The PW duplicate 1511 detection mechanism is already conceptually specified [RFC3985] 1512 but no emulated service makes use of it currently. 1514 #5 Flow duplication and merging (W) 1516 PWs could use a (extended) version of existing transport layer 1517 provided protection mechanisms (e.g., hitless 1+1 protection) for 1518 both flow duplication and flow merging. The service layer has to 1519 provide the functionality to map DetNet flows into appropriate 1520 transport layer connection. 1522 #6 Operations, Administration and Maintenance (M/W) 1524 PWs have rich control plane for OAM and in a case of the MPLS PSN 1525 enjoy the full control plane toolbox developed for MPLS network 1526 OAM likewise IP PSN has the full toolbox of IP network OAM tools. 1527 There could be, however, need for deterministic networking 1528 specific extensions for the mentioned control planes. 1530 #8 Class and quality of service capabilities (M/W) 1532 In a case of IP PSN the 6-bit differentiated services code point 1533 (DSCP) field can be used for indicating the class of service 1534 [RFC2474] and 2-bit field reserved for the explicit congestion 1535 notification (ECN) [RFC3168]. Similarly, in a case of MPLS PSN, 1536 there are 3-bit traffic class field (TC) [RFC5462] in the label 1537 reserved for for both Explicitly TC-encoded-PSC LSPs (E-LSP) 1538 [RFC3270] and ECN [RFC5129]. Due to the limited number of bits in 1539 the TC field, their use for QoS and ECN functions is restricted 1540 and intended to be flexible. Although the QoS/CoS mechanism is 1541 already in place some clarifications may be required in the 1542 context of deterministic networking flows, for example, if some 1543 specific mapping between bit fields have to be done. 1545 When PWs are used over MPLS, MPLS LSPs can be used to provide both 1546 CoS (E-LSPs and L-LSPs) and QoS (dedicated TE LSPS). 1548 #10 Technical maturity (M) 1550 PWs, IP and MPLS are proven technologies with wide variety of 1551 deployments and years of operational experience. Furthermore, the 1552 estimated work for missing functionality (packet replication and 1553 elimination) does not appear to be extensive, since the existing 1554 protection mechanism already gets close to what is needed from the 1555 deterministic networking data plane solution. 1557 5.2.3.3. Summary 1559 PseudoWires appear to be a strong candidate as the deterministic 1560 networking data plane solution alternative for the DetNet Service 1561 layer. The strong points are the technical maturity and the 1562 extensive control plane for OAM. This holds specifically for MPLS- 1563 based PSN. 1565 Extensions are required to realize the packet replication and 1566 duplicate detection features of the deterministic networking data 1567 plane. 1569 5.2.4. MPLS-Based Ethernet VPN (EVPN) 1571 5.2.4.1. Solution description 1573 MPLS-Based Ethernet VPN (EVPN), in the form documented in [RFC7432] 1574 and [RFC7209], is an increasingly popular approach to delivering 1575 MPLS-based Ethernet services and is designed to be the successor to 1576 Virtual Private LAN Service (VPLS), [RFC4664]. 1578 EVPN provides client adaptation and reuses the MPLS data plane 1579 discussed above in Section 5.2.2. While not required, the PW Control 1580 Word is also used. EVPN control is via BGP, [RFC7432], and may use 1581 TE-LSPs, e.g., controlled via [RFC3209] for MPLS transport. 1582 Additional EVPN related RFCs and in progress drafts are being 1583 developed by the BGP Enabled Services Working Group [6]. 1585 5.2.4.2. Analysis and Discussion 1587 #1 Encapsulation and overhead (M) 1589 EVPN generally uses a single MPLS label stack entry to support its 1590 client adaptation service. The optional addition of a second 1591 label is also supported. In certain cases PW Control Word may 1592 also be used. 1594 #2 Flow identification (W) 1596 EVPN currently uses labels to identify flows per {Ethernet Segment 1597 Identifier, VLAN} or per MAC level. Additional definition will be 1598 needed to standardize identification of finer granularity DetNet 1599 flows as well as mapping of TSN services to DetNet Services. 1601 #3 Packet sequencing and duplicate elimination (M) 1603 Like MPLS, EVPN generally orders packets similar to Ethernet. 1604 Reordering is possible primarily during path changes and 1605 protection switching. In order to avoid misordering due to ECMP, 1606 EVPN uses the "Preferred PW MPLS Control Word" [RFC4385] (in which 1607 case EVPN inherits this function from PWs) or the entropy labels 1608 [RFC6790]. 1610 If additional ordering mechanisms are required, such mechanisms 1611 will need to be defined. 1613 #5 Flow duplication and merging (M/W) 1615 EVPN relies on the MPLS layer for all protection functions. See 1616 Section 5.1.3 and Section 5.2.2. Some extensions, either at the 1617 EVPN or MPLS levels, will be need to support those DetNet 1618 applications which require true hitless (i.e., zero loss) 1+1 1619 protection switching. (Network coding may be an interesting 1620 alternative to investigate to delivering such hitless loss 1621 protection capability.) 1623 #6 Operations, Administration and Maintenance (M/W) 1625 Nodes supporting EVPN may participate in either or both Ethernet 1626 level and MPLS level OAM. It is likely that it may make sense to 1627 map or adapt the OAM functions at the different levels, but such 1628 has yet to be defined. [RFC6371] provides some useful background 1629 on this topic. 1631 #8 Class and quality of service capabilities (M/W) 1633 EVPN is largely silent on the topics of CoS and QoS, but the 802.1 1634 TSN Ethernet and existing MPLS TE mechanisms can be directly used. 1635 The inter-working of such is new work and within the scope of 1636 DetNet. The existing MPLS mechanisms include both CoS (E-LSPs and 1637 L-LSPs) and QoS (dedicated TE LSPs). 1639 #10 Technical maturity (M) 1640 EVPN is a second (or third) generation MPLS-based L2VPN service 1641 standard. From a data plane standpoint it makes uses of existing 1642 MPLS data plane mechanisms. The mechanisms have been widely 1643 implemented and deployed. 1645 5.2.4.3. Summary 1647 EVPN is the emerging successor to VPLS. EVPN is standardized, 1648 implemented and deployed. It makes use of the mature MPLS data 1649 plane. While offering a mature and very comprehensive set of 1650 features, certain DetNet required features are not fully/directly 1651 supported and additional standardization in these areas are needed. 1652 Examples include: mapping CoS and QoS; use of labels per DetNet flow, 1653 and hitless 1+1 protection. 1655 5.2.5. Higher layer header fields 1657 Fields of headers belonging to higher OSI layers can be used to 1658 implement functionality that is not provided e.g., by the IPv6 or 1659 IPv4 header fields. However, this approach cannot be always applied, 1660 e.g., due to encryption. Furthermore, even if this approach is 1661 applicable, it requires deep packet inspection from the routers and 1662 switches. There are implementation dependent limits how far into the 1663 packet the lookup can be done efficiently in the fast path. When 1664 encryption is not used, a safe bet is generally between 128 and 256 1665 octets for the maximum lookup depth. Various higher layer protocols 1666 can be applied. Some examples are provided here for the sequence 1667 numbering feature (Section 4.3). 1669 5.2.5.1. TCP 1671 The TCP header includes a sequence number parameter, which can be 1672 applied to detect and eliminate duplicate packets if DetNet service 1673 protection is used. As the TCP header is right after the IP header, 1674 it does not require very deep packet inspection; the 4-byte sequence 1675 number is conveyed by bits 32 through 63 of the TCP header. In 1676 addition to sequencing, the TCP header also contain source and 1677 destination port information that can be used for assisting the flow 1678 identification. 1680 5.2.5.2. RTP 1682 5.2.5.2.1. Solution Description 1684 Real-time Transport Protocol (RTP) [RFC3550] is often used to deliver 1685 time critical traffic in IP networks. RTP is typically carried on 1686 top of UDP/IP. However, as noted earlier in Section 5.2.3 1687 PseudoWires also have a well-defined way of embedding and 1688 transposting RTP headers as part of its payload encapsulation 1689 headers/sub-layer. RTP is also augmented by its own control protocol 1690 RTCP, which monitors the data delivery and provides minimal control 1691 and identification functionality. RTCP packets do not carry "media 1692 payload". Although both RTP and RTCP are typically used with UDP/IP 1693 transport they are designed to be independent of the underlying 1694 transport and network layers. 1696 The RTP header includes a 2-byte sequence number, which can be used 1697 to detect and eliminate duplicate packets if DetNet service 1698 protection is used. The sequence number is conveyed by bits 16 1699 through 31 of the RTP header. In addition to the sequence number the 1700 RTP header has also timestamp field (bits 32 through 63) that can be 1701 useful for time synchronization purposes. Furthermore, the RTP 1702 header has also one or more synchronization sources (bits starting 1703 from 64) that can potentially be useful for flow identification 1704 purposes. 1706 5.2.5.2.2. Analysis and Discussion 1708 #1 Encapsulation and overhead (M) 1710 RTP adds minimum 12 octets of header overhead. Typically 8 octets 1711 overhead of UDP header has to be also added, at least in a case 1712 when RTP is transported over IP. Although RTCP packets do not 1713 contribute to the media payload transport they still consume 1714 overall network capacity, since all participants to an RTP session 1715 including sourcess and multicast session destinations are expected 1716 to send RTCP reports. 1718 #2 Flow identification (M) 1720 The RTP header contains a synchronization source (SSRC) 1721 identifier. The intent is that no two synchronization sources 1722 within the same RTP session have the same SSRC identifier. 1724 #3 Packet sequencing and duplicate elimination (M) 1726 The RTP header contains a 16 bit sequence number. The sequence 1727 number can be also used to detect duplicate packets. 1729 #5 Flow duplication and merging (M/W) 1731 RTP has precedence of being used for hitless protection switching 1732 [ST20227], which essentially is equivalent to DetNet service 1733 protection. Furthermore, recent work in IETF for RTP stream 1734 duplication [RFC7198] as a mechanism to protect media flows from 1735 packet loss is again equivalent to Detnet service protection. 1737 #6 Operations, Administration and Maintenance (M) 1739 RTP has its own control protocol RTCP for (minimal) management and 1740 stream monitoring purposes. Existing IP OAM tools can directly 1741 leveraged when RTP is deployed over IP transport. 1743 #8 Class and quality of service capabilities (M/W) 1745 TBD. [Editor's note: relies on lower layers to provide CoS/QoS] 1747 #10 Technical maturity (M) 1749 RTP has been deployed and used in large commercial systems for 1750 over ten years and can be considered a mature technology. 1752 5.2.5.2.3. Summary 1754 RTP appears to be a good candidate as the deterministic networking 1755 data plane solution alternative for the DetNet Service layer. The 1756 strong points are the technical maturity and the fact it was designed 1757 for transporting time-sensitive payload from the beginning. RTP is 1758 specifically well suited to be used with (UDP)/IP transport. 1760 Extensions may be required to realize the packet replication and 1761 duplicate detection features of the deterministic networking data 1762 plane. However, there is already precedence of similar solutions 1763 that could potentially be leveraged [ST20227][RFC7198]. 1765 6. Summary of data plane alternatives 1767 The following table summarizes the criteria (Section 4) used for the 1768 evaluation of data plane options. 1770 Applicability per Alternative 1772 +--------+---------------------------------------------+ 1773 | Item # | Meaning | 1774 +--------+---------------------------------------------+ 1775 | #1 | Encapsulation and overhead | 1776 | #2 | Flow identification | 1777 | #3 | Packet sequencing and duplicate elimination | 1778 | #4 | Explicit routes | 1779 | #5 | Flow duplication and merging | 1780 | #6 | Operations, Administration and Maintenance | 1781 | #8 | Class and quality of service capabilities | 1782 | #9 | Packet traceability | 1783 | #10 | Technical maturity | 1784 +--------+---------------------------------------------+ 1786 Table 1: Evaluation criteria 1788 There is no single technology that could meet all the criteria on its 1789 own. Distinguishing the DetNet Service and the DetNet Transport, as 1790 explained in (Section 3), allows a number of combinations, which can 1791 meet most of the criteria. There is no room here to evaluate all 1792 possible combinations. Therefore, only some combinations are 1793 highlighted here, which are selected based on the number of criteria 1794 that are met and the maturity of the technology (#10). 1796 The following table summarizes the evaluation of the data plane 1797 options that can be used for the DetNet Transport Layer against the 1798 evaluation criteria. Each value in the table is from the 1799 corresponding section. 1801 Applicability per Transport Alternative 1803 +----------+----+----+----+-----+-----+-----+----+-----+ 1804 | Solution | #1 | #2 | #4 | #5 | #6 | #8 | #9 | #10 | 1805 +----------+----+----+----+-----+-----+-----+----+-----+ 1806 | IPv6 | M | W | W | W | M | W | W | M/W | 1807 | IPv4 | M | W | W | W/N | M | M/W | W | M/W | 1808 | MPLS | M | M | M | M/W | M | M/W | M | M | 1809 | BIER | M | M | M | M/W | M/W | M/W | M | W | 1810 +----------+----+----+----+-----+-----+-----+----+-----+ 1812 Summarizing Transport capabilities 1814 Table 2: DetNet Transport Layer 1816 The following table summarizes the evaluation of the data plane 1817 options that can be used for the DetNet Service Layer against the 1818 criteria evaluation criteria. Each value in the table is from the 1819 corresponding section. 1821 Applicability per Service Alternative 1823 +----------+----+----+-----+-----+-----+-----+-----+ 1824 | Solution | #1 | #2 | #3 | #5 | #6 | #8 | #10 | 1825 +----------+----+----+-----+-----+-----+-----+-----+ 1826 | GRE | M | W | M/W | W/N | M | W | M | 1827 | PWE3 | M | M | M | W | M/W | M/W | M | 1828 | EVPN | M | W | M | M/W | M/W | M/W | M | 1829 | RTP | M | M | M | M/W | M | M/W | M | 1830 +----------+----+----+-----+-----+-----+-----+-----+ 1832 Summarizing Service capabilities 1834 Table 3: DetNet Service Layer 1836 PseudoWire (Section 5.2.3) is a technology that is mature and meets 1837 most of the criteria for the DetNet Service layer as shown in the 1838 table above. From upper layer protocols PWs or RTP can be a 1839 candidate for non-MPLS PSNs. The identified work for PWs is to 1840 figure out how to implement duplicate detection for these protocols 1841 (e.g., based on [RFC3985]). In a case of RTP there is precedence of 1842 implementing packet duplication and duplicate elimination 1843 [ST20227][RFC7198]. 1845 PWs can be carried over MPLS or IP. MPLS is the most common 1846 technology that is used as PSN for PseudoWires; furthermore, MPLS is 1847 a mature technology and meets most DetNet Transport layer criteria. 1848 IPv[46] can be also used as PSN and both are mature technologies, 1849 although both generally only support CoS (DiffServ) in deployed 1850 networks. RTP is independent of the underlying transport technology 1851 and network. However, it is well suited for UDP/IP transport or 1852 embedded as a part of the PseudoWire timing sub-layer. 1854 7. Security considerations 1856 This document does not add any new security considerations beyond 1857 what the referenced technologies already have. 1859 8. IANA Considerations 1861 This document has no IANA considerations. 1863 9. Acknowledgements 1865 The author(s) ACK and NACK. 1867 The following people were part of the DetNet Data Plane Design Team: 1869 Jouni Korhonen 1870 Janos Farkas 1871 Norman Finn 1872 Olivier Marce 1873 Gregory Mirsky 1874 Pascal Thubert 1875 Zhuangyan Zhuang 1877 Substantial contributions were received from: 1879 Balazs Varga (service model) 1881 The DetNet chairs serving during the DetNet Data Plane Design Team: 1883 Lou Berger 1884 Pat Thaler 1886 10. References 1888 10.1. Informative References 1890 [I-D.finn-detnet-architecture] 1891 Finn, N. and P. Thubert, "Deterministic Networking 1892 Architecture", draft-finn-detnet-architecture-08 (work in 1893 progress), August 2016. 1895 [I-D.ietf-6man-rfc2460bis] 1896 Hinden, R., "Internet Protocol, Version 6 (IPv6) 1897 Specification", draft-ietf-6man-rfc2460bis-06 (work in 1898 progress), September 2016. 1900 [I-D.ietf-6man-segment-routing-header] 1901 Previdi, S., Filsfils, C., Field, B., Leung, I., Linkova, 1902 J., Aries, E., Kosugi, T., Vyncke, E., and D. Lebrun, 1903 "IPv6 Segment Routing Header (SRH)", draft-ietf-6man- 1904 segment-routing-header-02 (work in progress), September 1905 2016. 1907 [I-D.ietf-bier-architecture] 1908 Wijnands, I., Rosen, E., Dolganow, A., Przygienda, T., and 1909 S. Aldrin, "Multicast using Bit Index Explicit 1910 Replication", draft-ietf-bier-architecture-04 (work in 1911 progress), July 2016. 1913 [I-D.ietf-bier-mpls-encapsulation] 1914 Wijnands, I., Rosen, E., Dolganow, A., Tantsura, J., 1915 Aldrin, S., and I. Meilik, "Encapsulation for Bit Index 1916 Explicit Replication in MPLS Networks", draft-ietf-bier- 1917 mpls-encapsulation-05 (work in progress), July 2016. 1919 [I-D.ietf-detnet-problem-statement] 1920 Finn, N. and P. Thubert, "Deterministic Networking Problem 1921 Statement", draft-ietf-detnet-problem-statement-00 (work 1922 in progress), April 2016. 1924 [I-D.ietf-mpls-residence-time] 1925 Mirsky, G., Ruffini, S., Gray, E., Drake, J., Bryant, S., 1926 and S. Vainshtein, "Residence Time Measurement in MPLS 1927 network", draft-ietf-mpls-residence-time-11 (work in 1928 progress), July 2016. 1930 [I-D.ietf-spring-segment-routing] 1931 Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., 1932 and R. Shakir, "Segment Routing Architecture", draft-ietf- 1933 spring-segment-routing-09 (work in progress), July 2016. 1935 [I-D.kumarzheng-bier-ping] 1936 Kumar, N., Pignataro, C., Akiya, N., Zheng, L., Chen, M., 1937 and G. Mirsky, "BIER Ping and Trace", draft-kumarzheng- 1938 bier-ping-03 (work in progress), July 2016. 1940 [I-D.mirsky-bier-path-mtu-discovery] 1941 Mirsky, G., Przygienda, T., and A. Dolganow, "Path Maximum 1942 Transmission Unit Discovery (PMTUD) for Bit Index Explicit 1943 Replication (BIER) Layer", draft-mirsky-bier-path-mtu- 1944 discovery-01 (work in progress), April 2016. 1946 [I-D.mirsky-bier-pmmm-oam] 1947 Mirsky, G., Zheng, L., Chen, M., and G. Fioccola, 1948 "Performance Measurement (PM) with Marking Method in Bit 1949 Index Explicit Replication (BIER) Layer", draft-mirsky- 1950 bier-pmmm-oam-01 (work in progress), March 2016. 1952 [I-D.ooamdt-rtgwg-oam-gap-analysis] 1953 Mirsky, G., Nordmark, E., Pignataro, C., Kumar, N., Kumar, 1954 D., Chen, M., Yizhou, L., Mozes, D., Networks, J., and I. 1955 Bagdonas, "Operations, Administration and Maintenance 1956 (OAM) for Overlay Networks: Gap Analysis", draft-ooamdt- 1957 rtgwg-oam-gap-analysis-02 (work in progress), July 2016. 1959 [I-D.ooamdt-rtgwg-ooam-requirement] 1960 Kumar, N., Pignataro, C., Kumar, D., Mirsky, G., Chen, M., 1961 Nordmark, E., Networks, J., and D. Mozes, "Overlay OAM 1962 Requirements", draft-ooamdt-rtgwg-ooam-requirement-01 1963 (work in progress), July 2016. 1965 [IEEE802.1Qca] 1966 IEEE 802.1, "IEEE 802.1Qca Bridges and Bridged Networks - 1967 Amendment 24: Path Control and Reservation", IEEE 1968 P802.1Qca/D2.1 P802.1Qca, June 2015, 1969 . 1972 [IEEE8021CB] 1973 Finn, N., "Draft Standard for Local and metropolitan area 1974 networks - Seamless Redundancy", IEEE P802.1CB 1975 /D2.1 P802.1CB, December 2015, 1976 . 1979 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1980 DOI 10.17487/RFC0791, September 1981, 1981 . 1983 [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - 1984 Communication Layers", STD 3, RFC 1122, 1985 DOI 10.17487/RFC1122, October 1989, 1986 . 1988 [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. 1989 Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 1990 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, 1991 September 1997, . 1993 [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1994 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, 1995 December 1998, . 1997 [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, 1998 "Definition of the Differentiated Services Field (DS 1999 Field) in the IPv4 and IPv6 Headers", RFC 2474, 2000 DOI 10.17487/RFC2474, December 1998, 2001 . 2003 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 2004 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 2005 DOI 10.17487/RFC2784, March 2000, 2006 . 2008 [RFC2890] Dommety, G., "Key and Sequence Number Extensions to GRE", 2009 RFC 2890, DOI 10.17487/RFC2890, September 2000, 2010 . 2012 [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol 2013 Label Switching Architecture", RFC 3031, 2014 DOI 10.17487/RFC3031, January 2001, 2015 . 2017 [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., 2018 Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack 2019 Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, 2020 . 2022 [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition 2023 of Explicit Congestion Notification (ECN) to IP", 2024 RFC 3168, DOI 10.17487/RFC3168, September 2001, 2025 . 2027 [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., 2028 and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP 2029 Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, 2030 . 2032 [RFC3232] Reynolds, J., Ed., "Assigned Numbers: RFC 1700 is Replaced 2033 by an On-line Database", RFC 3232, DOI 10.17487/RFC3232, 2034 January 2002, . 2036 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, 2037 P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- 2038 Protocol Label Switching (MPLS) Support of Differentiated 2039 Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, 2040 . 2042 [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing 2043 in Multi-Protocol Label Switching (MPLS) Networks", 2044 RFC 3443, DOI 10.17487/RFC3443, January 2003, 2045 . 2047 [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label 2048 Switching (GMPLS) Signaling Resource ReserVation Protocol- 2049 Traffic Engineering (RSVP-TE) Extensions", RFC 3473, 2050 DOI 10.17487/RFC3473, January 2003, 2051 . 2053 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 2054 Jacobson, "RTP: A Transport Protocol for Real-Time 2055 Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, 2056 July 2003, . 2058 [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation 2059 Edge-to-Edge (PWE3) Architecture", RFC 3985, 2060 DOI 10.17487/RFC3985, March 2005, 2061 . 2063 [RFC4023] Worster, T., Rekhter, Y., and E. Rosen, Ed., 2064 "Encapsulating MPLS in IP or Generic Routing Encapsulation 2065 (GRE)", RFC 4023, DOI 10.17487/RFC4023, March 2005, 2066 . 2068 [RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", 2069 RFC 4303, DOI 10.17487/RFC4303, December 2005, 2070 . 2072 [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, 2073 "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for 2074 Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, 2075 February 2006, . 2077 [RFC4446] Martini, L., "IANA Allocations for Pseudowire Edge to Edge 2078 Emulation (PWE3)", BCP 116, RFC 4446, 2079 DOI 10.17487/RFC4446, April 2006, 2080 . 2082 [RFC4447] Martini, L., Ed., Rosen, E., El-Aawar, N., Smith, T., and 2083 G. Heron, "Pseudowire Setup and Maintenance Using the 2084 Label Distribution Protocol (LDP)", RFC 4447, 2085 DOI 10.17487/RFC4447, April 2006, 2086 . 2088 [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, 2089 "Encapsulation Methods for Transport of Ethernet over MPLS 2090 Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, 2091 . 2093 [RFC4553] Vainshtein, A., Ed. and YJ. Stein, Ed., "Structure- 2094 Agnostic Time Division Multiplexing (TDM) over Packet 2095 (SAToP)", RFC 4553, DOI 10.17487/RFC4553, June 2006, 2096 . 2098 [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 2099 2 Virtual Private Networks (L2VPNs)", RFC 4664, 2100 DOI 10.17487/RFC4664, September 2006, 2101 . 2103 [RFC4817] Townsley, M., Pignataro, C., Wainner, S., Seely, T., and 2104 J. Young, "Encapsulation of MPLS over Layer 2 Tunneling 2105 Protocol Version 3", RFC 4817, DOI 10.17487/RFC4817, March 2106 2007, . 2108 [RFC4875] Aggarwal, R., Ed., Papadimitriou, D., Ed., and S. 2109 Yasukawa, Ed., "Extensions to Resource Reservation 2110 Protocol - Traffic Engineering (RSVP-TE) for Point-to- 2111 Multipoint TE Label Switched Paths (LSPs)", RFC 4875, 2112 DOI 10.17487/RFC4875, May 2007, 2113 . 2115 [RFC5087] Stein, Y(J)., Shashoua, R., Insler, R., and M. Anavi, 2116 "Time Division Multiplexing over IP (TDMoIP)", RFC 5087, 2117 DOI 10.17487/RFC5087, December 2007, 2118 . 2120 [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion 2121 Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2122 2008, . 2124 [RFC5254] Bitar, N., Ed., Bocci, M., Ed., and L. Martini, Ed., 2125 "Requirements for Multi-Segment Pseudowire Emulation Edge- 2126 to-Edge (PWE3)", RFC 5254, DOI 10.17487/RFC5254, October 2127 2008, . 2129 [RFC5305] Li, T. and H. Smit, "IS-IS Extensions for Traffic 2130 Engineering", RFC 5305, DOI 10.17487/RFC5305, October 2131 2008, . 2133 [RFC5331] Aggarwal, R., Rekhter, Y., and E. Rosen, "MPLS Upstream 2134 Label Assignment and Context-Specific Label Space", 2135 RFC 5331, DOI 10.17487/RFC5331, August 2008, 2136 . 2138 [RFC5332] Eckert, T., Rosen, E., Ed., Aggarwal, R., and Y. Rekhter, 2139 "MPLS Multicast Encapsulations", RFC 5332, 2140 DOI 10.17487/RFC5332, August 2008, 2141 . 2143 [RFC5440] Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation 2144 Element (PCE) Communication Protocol (PCEP)", RFC 5440, 2145 DOI 10.17487/RFC5440, March 2009, 2146 . 2148 [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching 2149 (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic 2150 Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2151 2009, . 2153 [RFC5586] Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed., 2154 "MPLS Generic Associated Channel", RFC 5586, 2155 DOI 10.17487/RFC5586, June 2009, 2156 . 2158 [RFC5659] Bocci, M. and S. Bryant, "An Architecture for Multi- 2159 Segment Pseudowire Emulation Edge-to-Edge", RFC 5659, 2160 DOI 10.17487/RFC5659, October 2009, 2161 . 2163 [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, 2164 L., and L. Berger, "A Framework for MPLS in Transport 2165 Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, 2166 . 2168 [RFC5960] Frost, D., Ed., Bryant, S., Ed., and M. Bocci, Ed., "MPLS 2169 Transport Profile Data Plane Architecture", RFC 5960, 2170 DOI 10.17487/RFC5960, August 2010, 2171 . 2173 [RFC6275] Perkins, C., Ed., Johnson, D., and J. Arkko, "Mobility 2174 Support in IPv6", RFC 6275, DOI 10.17487/RFC6275, July 2175 2011, . 2177 [RFC6371] Busi, I., Ed. and D. Allan, Ed., "Operations, 2178 Administration, and Maintenance Framework for MPLS-Based 2179 Transport Networks", RFC 6371, DOI 10.17487/RFC6371, 2180 September 2011, . 2182 [RFC6373] Andersson, L., Ed., Berger, L., Ed., Fang, L., Ed., Bitar, 2183 N., Ed., and E. Gray, Ed., "MPLS Transport Profile (MPLS- 2184 TP) Control Plane Framework", RFC 6373, 2185 DOI 10.17487/RFC6373, September 2011, 2186 . 2188 [RFC6378] Weingarten, Y., Ed., Bryant, S., Osborne, E., Sprecher, 2189 N., and A. Fulignoli, Ed., "MPLS Transport Profile (MPLS- 2190 TP) Linear Protection", RFC 6378, DOI 10.17487/RFC6378, 2191 October 2011, . 2193 [RFC6426] Gray, E., Bahadur, N., Boutros, S., and R. Aggarwal, "MPLS 2194 On-Demand Connectivity Verification and Route Tracing", 2195 RFC 6426, DOI 10.17487/RFC6426, November 2011, 2196 . 2198 [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, 2199 "IPv6 Flow Label Specification", RFC 6437, 2200 DOI 10.17487/RFC6437, November 2011, 2201 . 2203 [RFC6564] Krishnan, S., Woodyatt, J., Kline, E., Hoagland, J., and 2204 M. Bhatia, "A Uniform Format for IPv6 Extension Headers", 2205 RFC 6564, DOI 10.17487/RFC6564, April 2012, 2206 . 2208 [RFC6621] Macker, J., Ed., "Simplified Multicast Forwarding", 2209 RFC 6621, DOI 10.17487/RFC6621, May 2012, 2210 . 2212 [RFC6658] Bryant, S., Ed., Martini, L., Swallow, G., and A. Malis, 2213 "Packet Pseudowire Encapsulation over an MPLS PSN", 2214 RFC 6658, DOI 10.17487/RFC6658, July 2012, 2215 . 2217 [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and 2218 L. Yong, "The Use of Entropy Labels in MPLS Forwarding", 2219 RFC 6790, DOI 10.17487/RFC6790, November 2012, 2220 . 2222 [RFC6814] Pignataro, C. and F. Gont, "Formally Deprecating Some IPv4 2223 Options", RFC 6814, DOI 10.17487/RFC6814, November 2012, 2224 . 2226 [RFC6864] Touch, J., "Updated Specification of the IPv4 ID Field", 2227 RFC 6864, DOI 10.17487/RFC6864, February 2013, 2228 . 2230 [RFC7045] Carpenter, B. and S. Jiang, "Transmission and Processing 2231 of IPv6 Extension Headers", RFC 7045, 2232 DOI 10.17487/RFC7045, December 2013, 2233 . 2235 [RFC7198] Begen, A. and C. Perkins, "Duplicating RTP Streams", 2236 RFC 7198, DOI 10.17487/RFC7198, April 2014, 2237 . 2239 [RFC7209] Sajassi, A., Aggarwal, R., Uttaro, J., Bitar, N., 2240 Henderickx, W., and A. Isaac, "Requirements for Ethernet 2241 VPN (EVPN)", RFC 7209, DOI 10.17487/RFC7209, May 2014, 2242 . 2244 [RFC7271] Ryoo, J., Ed., Gray, E., Ed., van Helvoort, H., 2245 D'Alessandro, A., Cheung, T., and E. Osborne, "MPLS 2246 Transport Profile (MPLS-TP) Linear Protection to Match the 2247 Operational Expectations of Synchronous Digital Hierarchy, 2248 Optical Transport Network, and Ethernet Transport Network 2249 Operators", RFC 7271, DOI 10.17487/RFC7271, June 2014, 2250 . 2252 [RFC7399] Farrel, A. and D. King, "Unanswered Questions in the Path 2253 Computation Element Architecture", RFC 7399, 2254 DOI 10.17487/RFC7399, October 2014, 2255 . 2257 [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., 2258 Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- 2259 Defined Networking (SDN): Layers and Architecture 2260 Terminology", RFC 7426, DOI 10.17487/RFC7426, January 2261 2015, . 2263 [RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A., 2264 Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based 2265 Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February 2266 2015, . 2268 [RFC7510] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, 2269 "Encapsulating MPLS in UDP", RFC 7510, 2270 DOI 10.17487/RFC7510, April 2015, 2271 . 2273 [RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network 2274 Virtualization Using Generic Routing Encapsulation", 2275 RFC 7637, DOI 10.17487/RFC7637, September 2015, 2276 . 2278 [RFC7676] Pignataro, C., Bonica, R., and S. Krishnan, "IPv6 Support 2279 for Generic Routing Encapsulation (GRE)", RFC 7676, 2280 DOI 10.17487/RFC7676, October 2015, 2281 . 2283 [RFC7752] Gredler, H., Ed., Medved, J., Previdi, S., Farrel, A., and 2284 S. Ray, "North-Bound Distribution of Link-State and 2285 Traffic Engineering (TE) Information Using BGP", RFC 7752, 2286 DOI 10.17487/RFC7752, March 2016, 2287 . 2289 [RFC7813] Farkas, J., Ed., Bragg, N., Unbehagen, P., Parsons, G., 2290 Ashwood-Smith, P., and C. Bowers, "IS-IS Path Control and 2291 Reservation", RFC 7813, DOI 10.17487/RFC7813, June 2016, 2292 . 2294 [RFC7872] Gont, F., Linkova, J., Chown, T., and W. Liu, 2295 "Observations on the Dropping of Packets with IPv6 2296 Extension Headers in the Real World", RFC 7872, 2297 DOI 10.17487/RFC7872, June 2016, 2298 . 2300 [ST20227] SMPTE 2022, "Seamless Protection Switching of SMPTE ST 2301 2022 IP Datagrams", ST 2022-7:2013, 2013, 2302 . 2304 10.2. URIs 2306 [1] http://6lab.cisco.com/stats/ 2308 [2] https://www.google.com/intl/en/ipv6/statistics.html 2310 [3] https://datatracker.ietf.org/wg/spring/charter/ 2312 [4] http://www.iana.org/assignments/g-ach-parameters/g-ach- 2313 parameters.xhtml 2315 [5] http://ftp.isi.edu/in-notes/iana/assignments/ethernet-numbers 2317 [6] https://tools.ietf.org/wg/bess/ 2319 Appendix A. Examples of combined DetNet Service and Transport layers 2321 Authors' Addresses 2322 Jouni Korhonen (editor) 2323 Broadcom 2324 3151 Zanker Road 2325 San Jose, CA 95134 2326 USA 2328 Email: jouni.nospam@gmail.com 2330 Janos Farkas 2331 Ericsson 2332 Konyves Kalman krt. 11/B 2333 Budapest 1097 2334 Hungary 2336 Email: janos.farkas@ericsson.com 2338 Gregory Mirsky 2339 Ericsson 2341 Email: gregory.mirsky@ericsson.com 2343 Pascal Thubert 2344 Cisco 2346 Email: pthubert@cisco.com 2348 Yan Zhuang 2349 Huawei 2351 Email: zhuangyan.zhuang@huawei.com 2353 Lou Berger 2354 LabN Consulting, L.L.C. 2356 Email: lberger@labn.net