idnits 2.17.00 (12 Aug 2021) /tmp/idnits44049/draft-thubert-tsvwg-detnet-transport-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == Line 345 has weird spacing: '...e stack v ...' == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document date (October 24, 2017) is 1670 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: draft-ietf-detnet-architecture has been published as RFC 8655 == Outdated reference: draft-ietf-detnet-problem-statement has been published as RFC 8557 == Outdated reference: draft-ietf-detnet-security has been published as RFC 9055 == Outdated reference: draft-ietf-detnet-use-cases has been published as RFC 8578 Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 TSVWG P. Thubert, Ed. 3 Internet-Draft Cisco 4 Intended status: Standards Track October 24, 2017 5 Expires: April 27, 2018 7 A Transport Layer for Deterministic Networks 8 draft-thubert-tsvwg-detnet-transport-00 10 Abstract 12 This document specifies the behavior of a Transport Layer operating 13 over a Deterministic Network and implementing a DetNet Service Layer 14 and a Northbound side of the DetNet User-to-Network Interface. 16 Status of This Memo 18 This Internet-Draft is submitted in full conformance with the 19 provisions of BCP 78 and BCP 79. 21 Internet-Drafts are working documents of the Internet Engineering 22 Task Force (IETF). Note that other groups may also distribute 23 working documents as Internet-Drafts. The list of current Internet- 24 Drafts is at https://datatracker.ietf.org/drafts/current/. 26 Internet-Drafts are draft documents valid for a maximum of six months 27 and may be updated, replaced, or obsoleted by other documents at any 28 time. It is inappropriate to use Internet-Drafts as reference 29 material or to cite them other than as "work in progress." 31 This Internet-Draft will expire on April 27, 2018. 33 Copyright Notice 35 Copyright (c) 2017 IETF Trust and the persons identified as the 36 document authors. All rights reserved. 38 This document is subject to BCP 78 and the IETF Trust's Legal 39 Provisions Relating to IETF Documents 40 (https://trustee.ietf.org/license-info) in effect on the date of 41 publication of this document. Please review these documents 42 carefully, as they describe your rights and restrictions with respect 43 to this document. Code Components extracted from this document must 44 include Simplified BSD License text as described in Section 4.e of 45 the Trust Legal Provisions and are provided without warranty as 46 described in the Simplified BSD License. 48 Table of Contents 50 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 51 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 52 3. On Deterministic Networking . . . . . . . . . . . . . . . . . 5 53 3.1. Applications and Requirements . . . . . . . . . . . . . . 5 54 3.2. The DetNet User-to-Network Interface (UNI) . . . . . . . 7 55 3.3. The DetNet Stack . . . . . . . . . . . . . . . . . . . . 8 56 3.4. The DetNet Service Model . . . . . . . . . . . . . . . . 8 57 4. DetTrans Operations . . . . . . . . . . . . . . . . . . . . . 8 58 4.1. DetTrans Overview . . . . . . . . . . . . . . . . . . . . 8 59 4.2. Application Requirements . . . . . . . . . . . . . . . . 10 60 4.2.1. Packet Normalization . . . . . . . . . . . . . . . . 10 61 4.2.2. Packet Streaming . . . . . . . . . . . . . . . . . . 10 62 4.3. Deterministic Flow Services . . . . . . . . . . . . . . . 10 63 4.3.1. Deterministic Flows . . . . . . . . . . . . . . . . . 10 64 4.3.2. Local Loop Flow Control . . . . . . . . . . . . . . . 11 65 4.3.2.1. Dichotomy of a DetNet End System . . . . . . . . 11 66 4.3.2.2. Local Loop Location . . . . . . . . . . . . . . . 12 67 4.3.2.3. Network Pull vs. Rate Based Flow Control . . . . 14 68 4.3.3. Load Sharing . . . . . . . . . . . . . . . . . . . . 14 69 4.3.4. PRE vs. 1+1 Redundancy . . . . . . . . . . . . . . . 15 70 4.3.5. Network Coding . . . . . . . . . . . . . . . . . . . 15 71 4.3.6. Multipath DetTrans Services . . . . . . . . . . . . . 15 72 5. The DetNet-UNI . . . . . . . . . . . . . . . . . . . . . . . 16 73 5.1. the "More" Message . . . . . . . . . . . . . . . . . . . 17 74 5.2. the "Time-Correction" Message . . . . . . . . . . . . . . 17 75 5.3. Loss of a Control Message . . . . . . . . . . . . . . . . 18 76 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18 77 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 78 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19 79 9. Informative References . . . . . . . . . . . . . . . . . . . 19 80 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 21 82 1. Introduction 84 Over last twenty years, voice, data and video networks have converged 85 to digital over IP. Mail delivery has become quasi-immediate and 86 volumes have multiplied; long distance voice is now mostly free and 87 the videophone is finally a reality; TV is available on-demand and 88 games became interactive and massively multi-player. The convergence 89 of highly heterogeneous networks over IP resulted in significant 90 drops in price for the end-user while adding new distinct value to 91 the related services. Yet, and even though similar benefits can be 92 envisioned when converging new applications over the Internet, there 93 are still many disjoint branches in the networking family tree, many 94 use-cases where mission-specific applications continue to utilize 95 dedicated point-to-point analog and digital technologies for their 96 operations. 98 Forty years ago, Control Information was first encoded as an analog 99 modulation of current (typically 4 to 20 mA) that can be carried 100 virtually instantly and with no loss over a distance. Then came 101 digitization, which enabled to multiplex data with the control signal 102 and manage the devices, but at the same time introduced latency to 103 industrial processes, the necessary delay to encode a series of bits 104 on a link and transport them along, which in turn may limit the 105 amount of transported information. The need to save cable and 106 simplify wiring lead to the Time Division Multiplexing (TDM) of 107 signals from multiple devices over shared digital buses, each signal 108 being granted access to the medium at a fixed period for a fixed 109 duration; with TDM, came more latency, waiting for the next reserved 110 access time. Statistical multiplexing, with Ethernet and IP, was 111 then introduced to achieve higher speeds at lower cost, and with it 112 came jitter and congestion loss. 114 A number of Operational Technology (OT) applications are now 115 migrating to Ethernet and IP, but that comes at the expense of 116 additional latency for the flows, to compensate for the degradation 117 of the transport discussed above. This also comes at the expense of 118 additional complexity in particular, applications may need to 119 transport a sense of time, provide some Forward Error Correction 120 (FEC) and include a jitter absorption buffer. for that reason, many 121 applications were never ported and OT networks are still largely 122 operated on point-to-point serial links and TDM buses. 124 A sense of what Deterministic Networking is has emerged as the 125 capability to make the Application simple again and enable a larger 126 migration of existing applicationsby absorbing the complexity lower 127 in the stack, at the Transport, Network and Link layers. A 128 Deterministic Network should be capable to emulate point-to-point 129 wires over a packet network, sharing the network resources between 130 deterministic and non-deterministic flows in such a fashion that 131 there can no observable influence whatsoever on a deterministic flow 132 from any other flow, regardless of the load of the network. 134 The generalization of the needs for more deterministic networks have 135 led to the IEEE 802.1 AVB Task Group becoming the Time-Sensitive 136 Networking (TSN) [IEEE802.1TSNTG] Task Group (TG), with a much- 137 expanded constituency from the industrial and vehicular markets. In 138 order to address the problem at the network layer, the DetNet Working 139 Group was formed to specify the signaling elements to be used to 140 establish a path and the tagging elements to be used identify the 141 flows that are to be forwarded along that path. 143 The "Deterministic Networking Use Cases" [I-D.ietf-detnet-use-cases] 144 indicates that beyond the classical case of industrial automation and 145 control systems (IACS), there are in fact multiple industries with 146 strong and yet relatively similar needs for deterministic network 147 services such as latency guarantees and ultra-low packet loss. The 148 "Deterministic Networking Problem Statement" 149 [I-D.ietf-detnet-problem-statement] documents the specific 150 requirements for the use of routed networks to support these 151 applications and the "Deterministic Networking Architecture" 152 [I-D.ietf-detnet-architecture] introduces the model that must be 153 proposed to integrate determinism in IT technology. 155 A DetNet network will guarantee a bounded latency and a very low 156 packet loss as long as the incoming flows respect a certain Service 157 Level Agreement (SLA), as typically expressed in the form of a 158 maximum packet size, a time window of observation and a maximum 159 number of packets per time window. 161 Outside the scope of DetNet, the IETF will also need to specify the 162 necessary protocols, or protocol additions, based on relevant IETF 163 technologies, to enable end-to-end deterministic flows. One critical 164 element is the Deterministic Transport Layer (DetTrans) that adapts 165 the flows coming from the Application Layer to the SLA of the DetNet 166 Network and provide end-to-end guarantees such as loss, latency and 167 timeliness. 169 The DetTrans Layer should in particular ensure that: 171 o the Deterministic Network setup matches the needs of the 172 Application 174 o the Application flows are presented to the Deterministic Network 175 in accordance to the SLA regardless of the way the data is passed 176 from the application 178 o the use of the network is optimized so as to ensure that every 179 byte from the application can effectively be transported 181 o the application flow is delivered reliabily and with a bounded 182 latency to the other Transport End Point, which may imply a FEC 183 technique such as Network Coding or Packet Replication and 184 Elimination (PRE) for a basic 1+1 redundancy. 186 o the full of the application flow is served, which may require the 187 use of multiple reservations in parallel, and the reordering of 188 the flows 190 On the one hand, the Deterministic Network will typically guarantee a 191 constant rate, so the classical Transport feature of flow control 192 will not be needed in a Deterministic Transport. On the other hand, 193 the Application and Transport layers may not reside in the same 194 device as the DetNet Router and/or the IEEE Std. 802.1 TSN Bridge 195 that acts as ingress point to the Deterministic Network. It results 196 that a minimum reliability and flow control must take place over the 197 Local Loop between these devices to ensure that the Deterministic 198 Network is kept optimally fed, meaning that packets are received just 199 in time for their scheduled transmission opportunities. 201 2. Terminology 203 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 204 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 205 document are to be interpreted as described in [RFC2119]. 207 3. On Deterministic Networking 209 3.1. Applications and Requirements 211 The Internet is not the only digital network that has grown 212 dramatically over the last 30-40 years. Video and audio 213 entertainment, and control systems for machinery, manufacturing 214 processes, and vehicles are also ubiquitous, and are now based almost 215 entirely on digital technologies. Over the past 10 years, engineers 216 in these fields have come to realize that significant advantages in 217 both cost and in the ability to accelerate growth can be obtained by 218 basing all of these disparate digital technologies on packet 219 networks. 221 The goals of Deterministic Networking are to enable the migration of 222 applications that use special-purpose fieldbus technologies (HDMI, 223 CANbus, ProfiBus, etc... even RS-232!) to packet technologies in 224 general, and the Internet Protocol in particular, and to support both 225 these new applications, and existing packet network applications, 226 over the same physical network. 228 Considerable experience ([ODVA]/[EIP], [AVnu], [Profinet],[HART], 229 [IEC62439], [ISA100.11a] and [WirelessHART], etc...) has shown that 230 these applications need a some or all of a suite of deterministic 231 features. 233 That suite of deterministic features includes: 235 1. Time synchronization of all Host and network nodes (Routers and/ 236 or Bridges), accurate to something between 10 nanoseconds and 10 237 microseconds, depending on the application. 239 2. Support for critical packet flows that: 241 * Can be unicast or multicast; 243 * Need absolute guarantees of minimum and maximum latency end- 244 to-end across the network; sometimes a tight jitter is 245 required as well; 247 * Need a packet loss ratio beyond the classical range for a 248 particular medium, in the range of 10^-9 to 10^-12, or better, 249 on Ethernet, and in the order of 10^-5 in Wireless Sensor Mesh 250 Networks; 252 * Can, in total, absorb more than half of the network's 253 available bandwidth (that is, massive over-provisioning is 254 ruled out as a solution); 256 * Cannot suffer throttling, flow control, or any other network- 257 imposed latency, for flows that can be meaningfully 258 characterized either by a fixed, repeating transmission 259 schedule, or by a maximum bandwidth and packet size; 261 3. Multiple methods to schedule, shape, limit, and otherwise control 262 the transmission of critical packets at each hop through the 263 network data plane; 265 4. Robust defenses against misbehaving Hosts, Routers, or Bridges, 266 both in the data and control planes, with guarantees that a 267 critical flow within its guaranteed resources cannot be affected 268 by other flows whatever the pressures on the network; 270 5. One or more methods to reserve resources in Bridges and Routers 271 to carry these flows. 273 Robustness is a common need for networking protocols, but plays a 274 more important part in real-time control networks, where expensive 275 equipment, and even lives, can be lost due to misbehaving equipment. 276 Reserving resources before packet transmission is the one fundamental 277 shift in the behavior of network applications that is impossible to 278 avoid. In the first place, a network cannot deliver finite latency 279 and practically zero packet loss to an arbitrarily high offered load. 280 Secondly, achieving practically zero packet loss for un-throttled 281 (though bandwidth limited) flows means that Bridges and Routers have 282 to dedicate buffer resources to specific flows or to classes of 283 flows. The requirements of each reservation have to be translated 284 into the parameters that control each Host's, Bridge's, and Router's 285 queuing, shaping, and scheduling functions and delivered to the 286 Hosts, Bridges, and Routers. 288 3.2. The DetNet User-to-Network Interface (UNI) 290 The "Deterministic Networking Architecture" 291 [I-D.ietf-detnet-architecture] presents the end-to-end networking 292 model and the DetNet services; in particular, it depicts the DetNet 293 User-to-Network Interfaces (DetNet-UNIs) ("U" in Figure 1) between 294 the Edge nodes (PE) of the Deterministic Network and the End Systems. 295 These UNIs are assumed to be packet-based reference points and 296 provide connectivity over the packet network. The Architecture also 297 mentions internal reference points between the CPU and the NIC in the 298 End System. 300 DetNet DetNet 301 End System End System 302 _ _ 303 / \ +----DetNet-UNI (U) / \ 304 /App\ | /App\ 305 /-----\ | /-----\ 306 | NIC | v ________ | NIC | 307 +--+--+ _____ / \ DetNet-UNI (U) --+ +--+--+ 308 | / \__/ \ | | 309 | / +----+ +----+ \_____ | | 310 | / | | | | \_______ | | 311 +------U PE +----+ P +----+ \ _ v | 312 | | | | | | | ___/ \ | 313 | +--+-+ +----+ | +----+ | / \_ | 314 \ | | | | | / \ | 315 \ | +----+ +--+-+ +--+PE |-------- U------+ 316 \ | | | | | | | | | \_ _/ 317 \ +---+ P +----+ P +--+ +----+ | \____/ 318 \___ | | | | / 319 \ +----+__ +----+ DetNet-1 DetNet-2 320 | \_____/ \___________/ | 321 | | 322 | | End-to-End-Service | | | | 323 <----------------------------------------------------------------> 324 | | DetNet-Service | | | | 325 | <--------------------------------------------------> | 326 | | | | | | 328 Figure 1: DetNet Service Reference Model (multi-domain) 330 A specific hardware is necessary for the time-sensitive functions of 331 synchronization, shaping and scheduling. This hardware may or may 332 not be fully available on a NIC inside the Host system. This 333 specification makes a distinction between a fully DetNet-Capable NIC, 334 and a DetNet-Aware NIC that participates to the DetNet-UNI, but is 335 not synchronized and scheduled with the Deterministic Network. 337 3.3. The DetNet Stack 339 The "Deterministic Networking Architecture" 340 [I-D.ietf-detnet-architecture] presents a conceptual DetNet data 341 plane layering model. The protocol stack includes a Service Layer 342 and a Transport Layer and is illustrated in Figure 2. 344 | packets going | ^ packets coming ^ 345 v down the stack v | up the stack | 346 +----------------------+ +-----------------------+ 347 | Source | | Destination | 348 +----------------------+ +-----------------------+ 349 | Service layer | | Service layer | 350 | Packet sequencing | | Duplicate elimination | 351 | Flow duplication | | Flow merging | 352 | Packet encoding | | Packet decoding | 353 +----------------------+ +-----------------------+ 354 | Transport layer | | Transport layer | 355 | Congestion prot. | | Congestion prot. | 356 +----------------------+ +-----------------------+ 357 | Lower layers | | Lower layers | 358 +----------------------+ +-----------------------+ 359 DetNet End System DetNet End System 360 v ^ 361 \________DetNet Transport______/ 363 Figure 2: DetNet-Capable End-System Protocol Stack 365 3.4. The DetNet Service Model 367 The "DetNet Service Model" [I-D.varga-detnet-service-model] provides 368 more details on the distribution of DetNet awareness and services. 370 4. DetTrans Operations 372 4.1. DetTrans Overview 374 The DetNet Service Layer mostly operates between the end-points, 375 though it is possible that some operations such as Packet Replication 376 and Elimination are also performed in selected intermediate nodes. 377 The DetNet Transport Layer represents the methods that ensure that a 378 packet is deterministically forwarded hop-by-hop from a Detnet Relay 379 to the next. The term "Transport" in the DetNet terminology must not 380 be confused with the function described in this document. This 381 document defines Detrans as a Layer-4 operation and an IETF Transport 382 Layer; DetTrans provides DetNet End-To-End Services for its 383 Applications, as well as intermediate services in selected points. 385 Following the DetNet Architecture, DetTrans mostly corresponds to the 386 DetNet Service Layer as opposed to the Detnet Transport Layer. 387 Figure 3 illustrates a simple example of classical networked devices 388 implementing the DetNet architecture. In that example, applications 389 reside on Host systems and run on main CPUs; DetTrans is collocated 390 with its applications and provides them with a Deterministic Service 391 through DetTrans APIs. Network Interface Cards (NIC) provides the 392 connectivity to the Deterministic Routers or Bridges acting at DetNet 393 Edge and Relay Nodes - say as an example that they are IEEE Std. 394 802.1 TSN Bridges. 396 Deterministic 397 Host System Routers and Bridges Host System 398 +---+-----------+ | +--------+ +--------+ | +---+-----------+ 399 | C |Application| |---| |----| |---| | C |Application| 400 | P +-----------+ | |<- DetNet Transport ->|-------+ P +-----------+ 401 | U | DetTrans | | +--------+ +--------+ | | U | DetTrans | 402 +---+-----------+ <------------ DetTrans ------------> +---+-----------+ 403 | N | Lower | | +--------+ +--------+ | | N | Lower | 404 | I | Layers |---| |<- DetNet Transport ->| |---| I | Layers | 405 | C | (queues) | |---| |----| |---| | C | | 406 +---+-----------+ | +--------+ +--------+ | +---+-----------+ 408 <-UNI-> <-- DetNet Services -> <-UNI-> 410 <------------- DetNet End-To-End Services ------------------> 412 Figure 3: Example Physical Network 414 Compared to a traditional IETF Transport Layer, DetTrans performs 415 similar operation of end-to-end reliability, flow control and 416 multipath load sharing, but differs on how those functionalities are 417 achieved. 419 Architectural variations are also introduced, for instance: 421 o Multipath operations are not necessarily end-to-end and a DetTrans 422 function may be present inside the network to relay between N 423 parallel paths and M parallel path, and or perform reliability 424 functionality such as Packet Replication and Elimination. 426 o The flow control is only needed between the DetTrans Layer and the 427 first Deterministic Relay, for instance a DetNet Router or an IEEE 428 Std. 802.1 TSN Bridge. From that point on, the flow is strictly 429 controlled by the DetNet operation. Architecturally speaking, the 430 flow control does not belong to the DetNet Service Layer but to 431 the DetNet Transport Layer, which means that DetTrans incorporates 432 a sublayer from the DetNet Transport Layer. 434 4.2. Application Requirements 436 4.2.1. Packet Normalization 438 A typical SLA for DetNet must be simple, for instance a maximum 439 packet size, and a maximum number of packets per window of time. 440 Smaller packets will mean wasted bandwidth, and excess packets within 441 a time window will be destroyed by the ingress shaping at the first 442 DetNet Bridge or Router. 444 The way the application layer feed the DetTrans layer may not 445 necessarily match the SLA with the Deterministic Network and in order 446 to provide the expected service, the DetTrans layer must pack the 447 data in packets that are as close to the maximum packet size as 448 possible, and yet make them available for transmission before 449 scheduled time. 451 4.2.2. Packet Streaming 453 The DetTrans Layer operates on its own sense of time which may be 454 loosely connected to the shared sense of time in the Deterministic 455 Network. 457 The DetTrans layer must shape its transmissions so as to ensure that 458 packets are delivered just in time to be injected along schedule in 459 the Deterministic Network. 461 4.3. Deterministic Flow Services 463 4.3.1. Deterministic Flows 465 Deterministic forwarding can only apply on flows with well-defined 466 characteristics such as periodicity and burstiness. Before a path 467 can be established to serve them, the expression of those 468 characteristics, and how the network can serve them, for instance in 469 shaping and forwarding operations, must be specified. 471 An application flow is established end-to-end between the DetTrans 472 layers and uses one or more lower-layer deterministic flows either in 473 parallel or in serial modes. At the time of this writing, the 474 distinction between application layer flows and lower layer flows is 475 not clearly stated in the "Deterministic Networking Architecture" 476 [I-D.ietf-detnet-architecture]. For the purpose of this document, we 477 use the term Determistic End-to-End Service Flow (DEESF), or DetTrans 478 Flow, to refer to an end-to-end application flow, and the term 479 Determistic Service Flow (DSF), or DetNet Flow, to refer to a lower 480 layer deterministic transport. 482 This is illustrated in Figure 4. 484 +---------------+ +--------+ +--------+ +---------------+ 485 | Application | |<-- DetNet Service -->| | Application | 486 +---------v-----+ +-------------Flow-------------+ +-----^---------+ 487 | | | | | | | | | | | | 488 | Network | | | +--------+ +--------+ | | | Network | 489 | Coding, | | | | | | Coding, | 490 | Flow | <------- DetNet End-to-End Service Flow -------> | Flow | 491 | Distri- | | | | | | Re-or- | 492 | bution, | +-----+ +--------+ +--------+ +-----+ | dering, | 493 | Packet +---+ | |<-- DetNet Service -->| | +---+ Packet | 494 | Replication +-------------------Flow-------------------+ Elimination | 495 | | | | | | | | | | 496 +---------------+ +--------+ +--------+ +---------------+ 497 DetNet Deterministic DetNet 498 End System Routers and Bridges End System 500 Figure 4: DetTrans vs. DetNet Flows 502 At Application and DetTrans Layers, the characteristics of a flow 503 relate to aggregate properties such as throughput, loss, and traffic 504 shape, and the Traffic Specification (TSPec) is expressed as a 505 Constant Bit Rate (CBR) or a Variable Bit Rate (VBR), burstiness 506 (e.g. video I-Frames), reliability (e.g. five nines), worst case 507 latency, amount of data to transfer, and expected duration of the 508 flow. 510 At the DetNet Transport Layer (between Relays), metrics are very 511 different, and relate to immediate actions on a packet as opposed to 512 general characteristics of a flow. DetNet Transport Layer 513 characteristics include time sync precision, time interval between 514 packets, packet size, jitter, and number of packets per window of 515 time. This is how the network SLA is defined, but this is not the 516 native terms for the application and a complex mapping must ensure 517 that the path that is setup and the DetNet Transport Layer 518 effectively matches the requirements from the DetNet Services Layer 519 and above. 521 4.3.2. Local Loop Flow Control 523 4.3.2.1. Dichotomy of a DetNet End System 525 The logical DetNet End System depicted in Figure 2 comprises several 526 elements which may implemented in one or separate physical Systems. 527 The example dichotomy in Figure 4 segregates ingress shaping and 528 DetNet Relay functions, which are performed by IEEE Std. 802.1 TSN 529 Bridges, from a DetNet-Aware Host. 531 Hosts and Edge Bridges are connected over Ethernet and together they 532 form a DetNet End System. As it goes, this example introduces a 533 further dichotomy within the Host, between the CPU and the NIC, 534 across a local bus such as PCI, as illustrated in Figure 5. 536 +-------------+ +-------+ +-----------+ 537 | Application | | MAC | | Ingress | 538 +-------------+ <--PCI--> +-------+ <--- DetNet-UNI ---> | Shaping | 539 | DetTrans | | PHY | | and Relay | 540 +-------------+ +-------+ +-----------+ 541 CPU NIC IEEE Std. 542 802.1 543 <----------- Host System -------------> TSN Bridge 544 <------------------------- DetNet End System ------------------------> 546 Figure 5: Chained Functions 548 The NICs in the Host System may not participate to the network time 549 Synchronization and may not be aware of the DetNet protocols running 550 between the Deterministic Routers and Bridges, and the associated 551 scheduling rules. In that situation, the DetNet-UNI operates on a 552 Local Loop to ensure that a packet that leaves the Transport reaches 553 the Router or Bridge just in time for injection into the 554 Deterministic data plane and to provide a flow control that avoids 555 congestion loss at the interface. 557 It is also possible that the NIC participates to the Deterministic 558 Network but still has asynchronous communication with DetTrans Layer 559 running on the the CPU. Either way, a flow control over a local loop 560 must be implemented to drain the queues from the DetTrans layer and 561 feed the network just in time for the deterministic transmission. 563 Depending on the level of support by the NIC, the loop may be placed 564 on a different interface but remains functionally the same. 566 4.3.2.2. Local Loop Location 568 If the NIC is not aware at all of DetNet, then it is a plain pipe for 569 the Deterministic Traffic. The Local Loop operates between the Edge 570 TSN Bridge and the CPU as illustrated in Figure 6. 572 +---+-----------+ +---+-------+ +-----------+ 573 | C |Application| | N | MAC | | Ingress | 574 | P +-----------+ <--PCI--> | I +-------+ <-- Ethernet --> | Shaping | 575 | U |DetTrans | | C | PHY | (Bridged | and Relay | 576 +---+-----------+ +---+-------+ or P2P) +-----------+ 578 <------------- Local Loop -------------> Edge 802.1 579 <----------- Host System -------------> TSN Bridge 581 Figure 6: DetNet Unaware NIC 583 If the NIC is fully DetNet-Capable and participates to the 584 deterministic Network including time synchronization and scheduling, 585 then the local loop operates between the CPU and the NIC as 586 illustrated in Figure 7. 588 +---+-----------+ +---+-----------+ +----------+ 589 | C |Application| | N | Ingress | | DetNet | 590 | P +-----------+ <--PCI--> | I + Shaping | <--DetNet--> | Relay | 591 | U |DetTrans | | C | and Relay | Transport | | 592 +---+-----------+ +---+-----------+ +----------+ 593 <-Local- 594 -Loop -> 595 <-------------- Host System --------------> TSN Bridge 597 Figure 7: DetNet Capable NIC 599 If the NIC is DetNet-Aware and does not participates to the 600 deterministic Network including time synchronization and scheduling, 601 then there are two local loops, one that operates between the CPU and 602 the NIC and one that operates between the NIC and the Edge TSN Bridge 603 as illustrated in Figure 8. 605 +---+-----------+ +---+-------+ +-----------+ 606 | C |Application| | N | MAC | | Ingress | 607 | P +-----------+ <--PCI--> | I +-------+ <-- Ethernet --> | Shaping | 608 | U |DetTrans | | C | PHY | (Bridged | and Relay | 609 +---+-----------+ +---+-------+ or P2P) +-----------+ 610 <-Local- 611 -Loop -> <-- Local Loop --> Edge 612 <----------- Host System -------------> DetNet Relay 614 Figure 8: DetNet Capable NIC 616 4.3.2.3. Network Pull vs. Rate Based Flow Control 618 The flow control at the DetNet-UNI can take any of two forms: 620 Network Pull In that Model, the DetNet Edge node drains the DetTrans 621 queue by sending a DetNet-UNI "More" command some estimated amount 622 of time ahead of the scheduled time of transmission for each 623 packet; in case of load sharing, multiple DetNet Edge nodes may 624 drain a queue at their own rates; in case of a high jitter on the 625 UNI Local Loop (e.g. there is a non-deterministic Bridge in 626 between, or the NIC is not DetNet-Aware and the flows suffer from 627 the more erratic response time of the CPU), the DetNet Edge node 628 may need to pull a window of packets to maintain its own 629 transmission queues fed at all times 631 Rate Based In that model, the NIC is aware of the rate of the 632 deterministic transmission and is drained by its internal timers. 633 Since the NIC is not synchronized with the Deterministic Network, 634 the Bridge uses a DetNet-UNI "Time-Correction" command 635 asynchronously to move forward or backward the next timeout of the 636 NIC for that flow, in order to keep the Rate-Based transmission by 637 the NIC in rough alignment with the scheduled transmission over 638 the DetNet network. 640 if the NIC is DetNet-Aware, it is expected that it maintains the 641 DetTrans queues in order to provide a deterministic response to the 642 DetNet-UNI, and in that case another control loop between the NIC and 643 the CPU is needed to ensure that the queue in the NIC is always fed 644 in time by the DetTrans Layer; this second loop may be of a different 645 nature than the DetNet-UNI one and may for instance be operated 646 within an interrupt to limit the asynchronism related to message 647 queueing. 649 4.3.3. Load Sharing 651 Multiple DetNet Flows may be used in parallel for Load Sharing. Load 652 Sharing refers to the use of multiple DetNet Flowss to carry a single 653 DetTrans Flow. Packets are sequenced at the DetTrans Layer and 654 distributed over the DetNet Transports paths in accordance to their 655 relative capacities. In case of inconsistent jitter and Latency 656 characteristics, packets may need to be reordered at the receiving 657 DetTrans Layer based on the DSF Sequence. In order to achieve this 658 function, a Load Distribution function is required at the source and 659 a Re-Ordering Function is required at the destination DetTrans End 660 Point. 662 4.3.4. PRE vs. 1+1 Redundancy 664 The DetNet Flows may also be used for Packet Replication and 665 Elimination, in which case an elimination function is required at the 666 DetTrans Termination. 668 1+1 Redundancy refers to injecting identical copies of a packet at 669 the ingress of two non-congruent paths, and eliminating the excess 670 copy when both are received at the egress of the paths. Packet 671 Replication and Elimination extends the concept by enabling more than 672 2 paths, and allowing non-end-to-end redundant path with intermediate 673 Replication and Elimination points. 675 4.3.5. Network Coding 677 Redundancy and Load Sharing may be combined with the use of Network 678 Coding whereby a coded packet may carry redundancy information for 679 previous data packet and cover the loss of one, in which case the 680 recovery function is required at the other DetTrans End Point. 681 Network Coding provides a Forward Error Correction between multiple 682 packets or multiple fragments of a packet. It may be used at the DSF 683 layer to enable an efficient combination of redundancy and load 684 sharing. 686 4.3.6. Multipath DetTrans Services 688 A DetTrans Flow may leverage multiple DetNet Flows in parallel in 689 order to achieve its requirements in terms of reliability and 690 Aggregate throughput. The "Deterministic Networking Architecture" 691 [I-D.ietf-detnet-architecture] clearly states that the capability of 692 Replication and Elimination is not limited to the DetNet End Systems. 693 Intermediate Systems that operate DetTrans but then relay the packets 694 are needed when the DetTrans operations are not end-to-end. 696 It may be that the DetTrans flow may need to traverse different 697 domains where those Services are operated differently, e.g. 698 controlled by different controllers or leveraging different 699 technologies. It may also be that the bandwidth that is required is 700 only available one segemnt at a time, and that for each segment, a 701 different number of DetNet flows must be setup to transport the full 702 amount of the DetTrans flow. 704 Figure 9 illustrates an example of the latter case, whereby The 705 DetTrans Flow is distributed over two DetNet Flows, maybe operating 706 PRE, then over three DetNet Flows, for instance operating Network 707 Coding between them but using a smaller banswidth for each flow, and 708 then two DetNet Flows again. 710 DetTrans is needed at the interconnection points to adapt the flows, 711 recover losses and reinject the appropriate rates in the next 712 segment. 714 +---+-----------+ +--------+ +--------+ 715 | C |Application| +------------------------------+ 716 | P +-----------+ | |<- DetNet Transport ->| | +---------------+ 717 | U |DetTrans | | +--------+ +--------+ | | DetTrans | 718 +---+-----------+ | | | Elimination | 719 | N | Packet +-----+ +--------+ +--------+ | | Re-ordering | 720 | I | Queues -+ | |<- DetNet Transport ->| +--------+---+---+ | 721 | C | +---------------------------------------------+---+---+ | 722 +---+-----------+ +--------+ +--------+ +----|---|---|--+ 723 | | | 724 DetNet | | | 725 Transport | | | 726 | | | 727 +---+-----------+ +--------+ +--------+ | | | 728 | C |Application| +------------------------------+ | | | 729 | P +-----------+ | |<- DetNet Transport ->| | +----|---|---|--+ 730 | U |DetTrans | | +--------+ +--------+ +--------+---+---+ | 731 +---+-----------+ | +--------+---+---+ | 732 | N | +-----+ +--------+ +--------+ | | DetTrans | 733 | I | <-+ | |<- DetNet Transport ->| | | Elimination | 734 | C | +------------------------------------+ | Re-ordering | 735 +---+-----------+ +--------+ +--------+ +---------------+ 736 Deterministic Intermediate 737 Host Systems Routers and Bridges Systems 739 Figure 9: Intermediate Systems 741 5. The DetNet-UNI 743 The DetTrans Layer aggregates the data coming from the application up 744 to a maximum frame size that is part of the SLA with the DetNet 745 Transport. Packets thus formed can be distributed over any of 746 multiple DetNet Transport sessions that are defined to accept that 747 packet size. Packets formed at the DetTrans Layer are queued and 748 ready to be delivered through the DetNet-UNI either with a Rate-Based 749 or a Network-Pull mechanism. 751 If the NIC is DetNet-Aware then the queue can be offboarded to the 752 NIC and it can be drained with a time gate (Rate-Base) or a message- 753 driven gate (Network-Pull). Else, the queue is handled by the CPU 754 and hopefully it can be drained within an interrupt, either for a 755 timer (Rate-Base) or for a message (Network-Pull). 757 The DetNet-UNI protocol enables the DetNet transport ingress point to 758 control when the DetTrans Layer transmits its Data packets. It may 759 happen that the DetTrans Layer has not formed a fully-sized packet 760 when time comes for sending it, in which case the packet will be sent 761 with a size below the maximum. 763 The DetNet UNI uses ICMPv6 to carry its protocol elements. Data 764 Packets across the UNI are encapsulated in order to carry DetNet-UNI 765 control information to identify the reason of a loss or a delay, and 766 determine the action to be taken in case of a packet lost or delayed 767 over the interface. 769 5.1. the "More" Message 771 The "More" message enables a DetNet Transport Edge to pull one packet 772 from the DetTrans Layer in Network-Pull mode. The message is 773 associated with a future transmission opportunity for a packet. The 774 "More" messages are indexed by a wrapping More Sequence Counter 775 (MSC). The Transport Edge also maintains wrapping counters of 776 Successful Packet Transmissions (SPT) and Missed Transmit 777 Opportunities (MTO). The current value of these counters is placed 778 in the "More "message. 780 Upon reception of a "More" message, the DetTrans Layer, or the NIC on 781 behalf of the DetTrans Layer, sends the next available packet for 782 that session. The packet is encapsulated and the encapsulation 783 indicates the MSC. This enables the DetNet Transport Edge to 784 correlate the packet with the transmission opportunity and drop 785 packets that are overly delayed. 787 5.2. the "Time-Correction" Message 789 The "Time-Correction" message enables a DetNet Transport Edge to 790 adjust the timer associated to the DetNet-UNI session in Rate-Based 791 mode. In that mode, the DetTrans Layer sends a packet and restarts a 792 timer at a period that corresponds to the transmission opportunity of 793 the DetNet Transport Edge. If the clock in the CPU drifts, the 794 DetNet Transport Edge will start receiving packets increasingly ahead 795 of expected time or behind expected time. It is expected that the 796 DetNet Transport Edge is protected against a minimum drift by a guard 797 time, but if the drift becomes too important, then the DetNet 798 Transport Edge issues a "Time-Correction" message indicating a number 799 of time units (e.g. microseconds) by which the DetTrans Layer should 800 advance or delay is next time out. 802 5.3. Loss of a Control Message 804 The loss of a packet beween the DetTrans Layer and the DetNet 805 Transport Edge will correspond to a missed Transmission Opportunity 806 but this does not mean that packets are piling up at the DetTrans 807 Layer. OTOH, if a "More" message is lost, then one packet will not 808 be dequeued and the Detrans queue might grow, increasingly augmenting 809 the latency. It is thus important to differentiate these situations, 810 and in the latter case, discard an extraneous packet to restore the 811 normal level in the DetTrans queue for that session. 813 In order to do so, the DetTrans Layer maintains the record of the 814 Number of Packets Sent (NPS), that it compares with the variation of 815 the MTO and SPT counters in the "More" message. Upon a "More" 816 message, the DetTrans Layer computes the variation of NPS 817 (dNPS=NPS2-NPS1) and the variation of SPT (dSPT=SPT2-SPT1) since the 818 previous "More" Message . dNPS is typically 1 if the transport 819 always has data to send. Packets in flight when the "More" message 820 is sent are considered lost since they will be received after their 821 scheduled transmission opportunity, so the Number of Packets Losses 822 (NPL) is (NPL=dNPS-dSPT). The DetTrans Layer also computes the 823 variation of MTO since the previous "More" Message (dMTO=MTO2-MTO1). 824 Since a packet loss implies a missed transmission opportunity, there 825 cannot be more packets losses than missed opportunities, so we have 826 dMTO>=NPL. dMTO-NPL represents the number of missed opportunities 827 that are not due to a packet lost or late arrival, thus this is the 828 sub-count of MTOs due to the loss of a "More" message. 830 For each loss of a "More" message, a packet in the DetTrans queue 831 should be discarded. In order to simplify that operation and 832 outboard it to the NIC, the Transports marks some packets as "Discard 833 Eligible" (DE). A packet can be marked DE if there are enough 834 alternate transmissions of non-De packets to recover this. For 835 instance, in case of Packet Replication and Elimination only one copy 836 can be marked DE, and the marking should alternate between the 837 sessions to cover a loss on either one rapidly. 839 6. Security Considerations 841 The generic threats against Deterministic Networking are discussed in 842 the "Deterministic Networking Security" [I-D.ietf-detnet-security] 843 document. 845 Security in the context of Deterministic Networking has an added 846 dimension; the time of delivery of a packet can be just as important 847 as the contents of the packet, itself. A man-in-the-middle attack, 848 for example, can impose, and then systematically adjust, additional 849 delays into a link, and thus disrupt or subvert a real-time 850 application without having to crack any encryption methods employed. 851 See [RFC7384] for an exploration of this issue in a related context. 853 Packet Replication and Elimination of done right can prevent a man- 854 in-the-middle attack on one leg to actually impact the flow beyond 855 the loss of an individual packet for lack of redundancy. This 856 specification expects that PRE is performed at the transport level 857 and provides specific means to protect one leg against misuse of the 858 other. 860 7. IANA Considerations 862 This document does not require an action from IANA. 864 8. Acknowledgments 866 The authors wish to thank Patrick Wetterwald, Leon Turkevitch, Balazs 867 Varga and Janos Farkas for their various contributions to this work. 868 Special thanks to Norm Finn for being a (if not the) major thought 869 leader to the whole deterministic effort, and for some text that is 870 inlined here from other IETF documents, for the convenience of the 871 reader. 873 9. Informative References 875 [AVnu] http://www.avnu.org/, "The AVnu Alliance tests and 876 certifies devices for interoperability, providing a simple 877 and reliable networking solution for AV network 878 implementation based on the IEEE Audio Video Bridging 879 (AVB) and Time-Sensitive Networking (TSN) standards.". 881 [EIP] http://www.odva.org/, "EtherNet/IP provides users with the 882 network tools to deploy standard Ethernet technology (IEEE 883 802.3 combined with the TCP/IP Suite) for industrial 884 automation applications while enabling Internet and 885 enterprise connectivity data anytime, anywhere.", 886 . 890 [HART] www.hartcomm.org, "Highway Addressable Remote Transducer, 891 a group of specifications for industrial process and 892 control devices administered by the HART Foundation". 894 [I-D.ietf-detnet-architecture] 895 Finn, N., Thubert, P., Varga, B., and J. Farkas, 896 "Deterministic Networking Architecture", draft-ietf- 897 detnet-architecture-03 (work in progress), August 2017. 899 [I-D.ietf-detnet-problem-statement] 900 Finn, N. and P. Thubert, "Deterministic Networking Problem 901 Statement", draft-ietf-detnet-problem-statement-02 (work 902 in progress), September 2017. 904 [I-D.ietf-detnet-security] 905 Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell, 906 J., Austad, H., Stanton, K., and N. Finn, "Deterministic 907 Networking (DetNet) Security Considerations", draft-ietf- 908 detnet-security-00 (work in progress), October 2017. 910 [I-D.ietf-detnet-use-cases] 911 Grossman, E., Gunther, C., Thubert, P., Wetterwald, P., 912 Raymond, J., Korhonen, J., Kaneko, Y., Das, S., Zha, Y., 913 Varga, B., Farkas, J., Goetz, F., Schmitt, J., Vilajosana, 914 X., Mahmoodi, T., Spirou, S., Vizarreta, P., Huang, D., 915 Geng, X., Dujovne, D., and M. Seewald, "Deterministic 916 Networking Use Cases", draft-ietf-detnet-use-cases-13 917 (work in progress), September 2017. 919 [I-D.varga-detnet-service-model] 920 Varga, B. and J. Farkas, "DetNet Service Model", draft- 921 varga-detnet-service-model-02 (work in progress), May 922 2017. 924 [IEC62439] 925 IEC, "Industrial communication networks - High 926 availability automation networks - Part 3: Parallel 927 Redundancy Protocol (PRP) and High-availability Seamless 928 Redundancy (HSR) - IEC62439-3", 2012, 929 . 931 [IEEE802.1TSNTG] 932 IEEE Standards Association, "IEEE 802.1 Time-Sensitive 933 Networks Task Group", 2013, 934 . 936 [ISA100.11a] 937 ISA/IEC, "ISA100.11a, Wireless Systems for Automation, 938 also IEC 62734", 2011, < http://www.isa100wci.org/en- 939 US/Documents/PDF/3405-ISA100-WirelessSystems-Future-broch- 940 WEB-ETSI.aspx>. 942 [ODVA] http://www.odva.org/, "The organization that supports 943 network technologies built on the Common Industrial 944 Protocol (CIP) including EtherNet/IP.". 946 [Profinet] 947 http://us.profinet.com/technology/profinet/, "PROFINET is 948 a standard for industrial networking in automation.", 949 . 951 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 952 Requirement Levels", BCP 14, RFC 2119, 953 DOI 10.17487/RFC2119, March 1997, 954 . 956 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 957 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 958 October 2014, . 960 [WirelessHART] 961 www.hartcomm.org, "Industrial Communication Networks - 962 Wireless Communication Network and Communication Profiles 963 - WirelessHART - IEC 62591", 2010. 965 Author's Address 967 Pascal Thubert (editor) 968 Cisco Systems, Inc 969 Building D (Regus) 45 Allee des Ormes 970 MOUGINS - Sophia Antipolis 971 FRANCE 973 Phone: +33 4 97 23 26 34 974 Email: pthubert@cisco.com