idnits 2.17.00 (12 Aug 2021) /tmp/idnits20172/draft-ietf-raw-architecture-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == Line 840 has weird spacing: '...-- Node z-- ...' == Line 845 has weird spacing: '... Node z-- ...' -- The document date (14 January 2022) is 127 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Missing Reference: 'PCE' is mentioned on line 747, but not defined == Outdated reference: A later version (-05) exists of draft-ietf-raw-technologies-04 == Outdated reference: A later version (-05) exists of draft-ietf-raw-use-cases-03 == Outdated reference: A later version (-05) exists of draft-irtf-panrg-path-properties-04 Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 RAW P. Thubert, Ed. 3 Internet-Draft Cisco Systems 4 Intended status: Informational G.Z. Papadopoulos 5 Expires: 18 July 2022 IMT Atlantique 6 14 January 2022 8 Reliable and Available Wireless Architecture 9 draft-ietf-raw-architecture-03 11 Abstract 13 Reliable and Available Wireless (RAW) provides for high reliability 14 and availability for IP connectivity over a wireless medium. The 15 wireless medium presents significant challenges to achieve 16 deterministic properties such as low packet error rate, bounded 17 consecutive losses, and bounded latency. This document defines the 18 RAW Architecture following an OODA loop that involves OAM, PCE, PSE 19 and PAREO functions. It builds on the DetNet Architecture and 20 discusses specific challenges and technology considerations needed to 21 deliver DetNet service utilizing scheduled wireless segments and 22 other media, e.g., frequency/time-sharing physical media resources 23 with stochastic traffic. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on 18 July 2022. 42 Copyright Notice 44 Copyright (c) 2022 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 49 license-info) in effect on the date of publication of this document. 50 Please review these documents carefully, as they describe your rights 51 and restrictions with respect to this document. Code Components 52 extracted from this document must include Revised BSD License text as 53 described in Section 4.e of the Trust Legal Provisions and are 54 provided without warranty as described in the Revised BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 2. The RAW problem . . . . . . . . . . . . . . . . . . . . . . . 5 60 2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 5 61 2.1.1. Acronyms . . . . . . . . . . . . . . . . . . . . . . 6 62 2.1.2. Link and Direction . . . . . . . . . . . . . . . . . 7 63 2.1.3. Path and Tracks . . . . . . . . . . . . . . . . . . . 7 64 2.1.4. Deterministic Networking . . . . . . . . . . . . . . 8 65 2.1.5. Reliability and Availability . . . . . . . . . . . . 9 66 2.1.6. OAM variations . . . . . . . . . . . . . . . . . . . 10 67 2.2. Reliability and Availability . . . . . . . . . . . . . . 11 68 2.2.1. High Availability Engineering Principles . . . . . . 12 69 2.2.2. Applying Reliability Concepts to Networking . . . . . 14 70 2.2.3. Reliability in the Context of RAW . . . . . . . . . . 15 71 2.3. Routing Time Scale vs. Forwarding Time Scale . . . . . . 16 72 3. The RAW Conceptual Model . . . . . . . . . . . . . . . . . . 18 73 4. The OODA Loop . . . . . . . . . . . . . . . . . . . . . . . . 20 74 4.1. Observe: The RAW OAM . . . . . . . . . . . . . . . . . . 21 75 4.2. Orient: The Path Computation Engine . . . . . . . . . . . 22 76 4.3. Decide: The Path Selection Engine . . . . . . . . . . . . 22 77 4.4. Act: The PAREO Functions . . . . . . . . . . . . . . . . 24 78 4.4.1. Packet Replication . . . . . . . . . . . . . . . . . 25 79 4.4.2. Packet Elimination . . . . . . . . . . . . . . . . . 26 80 4.4.3. Promiscuous Overhearing . . . . . . . . . . . . . . . 26 81 4.4.4. Constructive Interference . . . . . . . . . . . . . . 27 82 5. Security Considerations . . . . . . . . . . . . . . . . . . . 27 83 5.1. Forced Access . . . . . . . . . . . . . . . . . . . . . . 27 84 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27 85 7. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 28 86 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 28 87 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 28 88 9.1. Normative References . . . . . . . . . . . . . . . . . . 28 89 9.2. Informative References . . . . . . . . . . . . . . . . . 30 90 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 31 92 1. Introduction 94 Deterministic Networking is an attempt to emulate the properties of a 95 serial link over a switched fabric, by providing a bounded latency 96 and eliminating congestion loss, even when co-existing with best- 97 effort traffic. It is getting traction in various industries 98 including professional A/V, manufacturing, online gaming, and 99 smartgrid automation, enabling cost and performance optimizations 100 (e.g., vs. loads of P2P cables). 102 Bringing determinism in a packet network means eliminating the 103 statistical effects of multiplexing that result in probabilistic 104 jitter and loss. This can be approached with a tight control of the 105 physical resources to maintain the amount of traffic within a 106 budgeted volume of data per unit of time that fits the physical 107 capabilities of the underlying network, and the use of time-shared 108 resources (bandwidth and buffers) per circuit, and/or by shaping and/ 109 or scheduling the packets at every hop. 111 This innovation was initially introduced on wired networks, with IEEE 112 802.1 Time Sensitive networking (TSN) - for Ethernet LANs - and IETF 113 DetNet. But the wired and the wireless media are fundamentally 114 different at the physical level and in the possible abstractions that 115 can be built for IPv6 [IPoWIRELESS]. Nevertheless, deterministic 116 capabilities are required in a number of wireless use cases as well 117 [RAW-USE-CASES]. With new scheduled radios such as TSCH and OFDMA 118 [RAW-TECHNOS] being developped to provide determinism over wireless 119 links at the lower layers, providing DetNet capabilities is now 120 becoming possible. 122 Wireless networks operate on a shared medium where uncontrolled 123 interference, including the self-induced multipath fading cause 124 random transmission losses. Fixed and mobile obstacles and 125 reflectors may block or alter the signal, causing transient and 126 unpredictable variations of the throughput and packet delivery ratio 127 (PDR) of a wireless link. This adds new dimensions to the 128 statistical effects that affect the quality and reliability of the 129 link. Multiple links and transmissions must be used, and the 130 challenge is to provide enough diversity and redundancy to ensure the 131 timely packet delivery while preserving energy and optimizing the use 132 of the shared spectrum. 134 Reliable and Available Wireless (RAW) takes up the challenge of 135 providing highly available and reliable end-to-end performances in a 136 network with scheduled wireless segments. To defeat those additional 137 causes of transmission delay and loss, RAW leverages deterministic 138 layer-2 capabilities while controlling the use of diversity in the 139 spatial, time, code, radio technology, and frequency domains from 140 layer-3. 142 While the generic "Deterministic Networking Problem Statement" 143 [RFC8557] applies to both the wired and the wireless media, the 144 methods to achieve RAW must extend those used to support time- 145 sensitive networking over wires, as a RAW solution has to address 146 less consistent transmissions, energy conservation and shared 147 spectrum efficiency. 149 RAW provides DetNet elements that are specialized for IPv6 flows 150 [IPv6] over selected deterministic radios technologies [RAW-TECHNOS]. 151 Conceptually, RAW is agnostic to the radio layer underneath though 152 the capability to schedule transmissions is assumed. How the PHY is 153 programmed to do so, and whether the radio is single-hop or meshed, 154 are unknown at the IP layer and not part of the RAW abstraction. 155 Nevertheless, cross-layer optimizations may take place to ensure 156 proper link awareness (think, link quality) and packet handling 157 (think, scheduling). 159 The "Deterministic Networking Architecture" [RFC8655] is composed of 160 three planes: the Application (User) Plane, the Controller Plane, and 161 the Network Plane. The RAW Architecture extends the DetNet Network 162 Plane, to accommodate one or multiple hops of homogeneous or 163 heterogeneous wireless technologies, e.g. a Wi-Fi6 Mesh or parallel 164 CBRS access links federated by a 5G backhaul. 166 RAW and DetNet route application flows that require a special 167 treatment along the paths that will provide that treatment. This may 168 be seen as a form of Path Aware Networking and may be subject to 169 impediments documented in [RFC9049]. 171 The establishment of a path is not in-scope for RAW. It may be the 172 product of a centralized Controller Plane as described for DetNet. 173 As opposed to wired networks, the action of installing a path over a 174 set of wireless links may be very slow relative to the speed at which 175 the radio conditions vary, and it makes sense in the wireless case to 176 provide redundant forwarding solutions along a complex path and to 177 leave it to the Network Plane to select which of those forwarding 178 solutions are to be used for a given packet based on the current 179 conditions. 181 RAW distinguishes the longer time scale at which routes are computed 182 from the the shorter forwarding time scale where per-packet decisions 183 are made. RAW operates within the Network Plane at the forwarding 184 time scale on one DetNet flow over a complex path called a Track. 185 The Track is preestablished and installed by means outside of the 186 scope of RAW; it may be strict or loose depending on whether each or 187 just a subset of the hops are observed and controlled by RAW. 189 The RAW Architecture is based on an abstract OODA Loop (Observe, 190 Orient, Decide, Act). The generic concept involves: 192 1. Network Plane measurement protocols for Operations, 193 Administration and Maintenance (OAM) to Observe some or all hops 194 along a Track as well as the end-to-end packet delivery 196 2. Controller plane elements to reports the links statistics to a 197 Path computation Element (PCE) in a centralized controller that 198 computes and installs the Tracks and provides meta data to Orient 199 the routing decision 201 3. A Runtime distributed Path Selection Engine (PSE) that Decides 202 which subTrack to use for the next packet(s) that are routed 203 along the Track 205 4. Packet (hybrid) ARQ, Replication, Elimination and Ordering 206 Dataplane actions that operate at the DetNet Service Layer to 207 increase the reliability of the end-to-end transmissions. The 208 RAW architecture also covers in-situ signalling when the decision 209 is Acted by a node that down the Track from the PSE. 211 The overall OODA Loop optimizes the use of redundancy to achieve the 212 required reliability and availability Service Level Agreement (SLA) 213 while minimizing the use of constrained resources such as spectrum 214 and battery. 216 This document presents the RAW problem and associated terminology in 217 Section 2, and elaborates in Section 4 on the OODA loop based on the 218 RAW conceptual model presented in Section 3. 220 2. The RAW problem 222 2.1. Terminology 224 RAW reuses terminology defined for DetNet in the "Deterministic 225 Networking Architecture" [RFC8655], e.g., PREOF for Packet 226 Replication, Elimination and Ordering Functions. 228 RAW also reuses terminology defined for 6TiSCH in [6TiSCH-ARCHI] such 229 as the term Track. A Track associates a complex path with PAREO and 230 shaping operations. The concept is agnostic to the underlaying 231 technology and applies to any fully or partially wireless mesh, 232 including, e.g., a Wi-Fi mesh. RAW specifies strict and loose Tracks 233 depending on whether the path is fully controlled by RAW or traverses 234 an opaque network where RAW cannot observe and control the individual 235 hops. 237 RAW uses the following terminology and acronyms: 239 2.1.1. Acronyms 241 2.1.1.1. ARQ 243 Automatic Repeat Request, enabling an acknowledged transmission and 244 retries. ARQ is a typical model at Layer-2 on a wireless medium. 245 ARQ is typically implemented hop-by-hop and not end-to-end in 246 wireless networks. Else, it introduces excessive indetermination in 247 latency, but a limited number of retries within a bounded time may be 248 used within end-to-end constraints. 250 2.1.1.2. OAM 252 OAM stands for Operations, Administration, and Maintenance, and 253 covers the processes, activities, tools, and standards involved with 254 operating, administering, managing and maintaining any system. This 255 document uses the terms Operations, Administration, and Maintenance, 256 in conformance with the 'Guidelines for the Use of the "OAM" Acronym 257 in the IETF' [RFC6291] and the system observed by the RAW OAM is the 258 Track. 260 2.1.1.3. OODA 262 Observe, Orient, Decide, Act. The OODA Loop is a conceptual cyclic 263 model developed by USAF Colonel John Boyd, and that is applicable in 264 multiple domains where agility can provide benefits against brute 265 force. 267 2.1.1.4. PAREO 269 Packet (hybrid) ARQ, Replication, Elimination and Ordering. PAREO is 270 a superset Of DetNet's PREOF that includes radio-specific techniques 271 such as short range broadcast, MUMIMO, PHY rate and other Modulation 272 Coding Scheme (MCS) adaptation, constructive interference and 273 overhearing, which can be leveraged separately or combined to 274 increase the reliability. 276 2.1.2. Link and Direction 278 2.1.2.1. Flapping 280 In the context of RAW, a link flaps when the reliability of the 281 wireless connectivity drops abruptly for a short period of time, 282 typically of a subsecond to seconds duration. 284 2.1.2.2. Uplink 286 Connection from end-devices to a data communication equipment. In 287 the context of wireless, uplink refers to the connection between a 288 station (STA) and a controller (AP) or a User Equipment (UE) to a 289 Base Station (BS) such as a 3GPP 5G gNodeB (gNb). 291 2.1.2.3. Downlink 293 The reverse direction from uplink. 295 2.1.2.4. Downstream 297 Following the direction of the flow data path along a Track. 299 2.1.2.5. Upstream 301 Against the direction of the flow data path along a Track. 303 2.1.3. Path and Tracks 305 2.1.3.1. Path 307 Quoting section 1.1.3 of [INT-ARCHI]: 309 | At a given moment, all the IP datagrams from a particular source 310 | host to a particular destination host will typically traverse the 311 | same sequence of gateways. We use the term "path" for this 312 | sequence. Note that a path is uni-directional; it is not unusual 313 | to have different paths in the two directions between a given host 314 | pair. 316 Section 2 of [I-D.irtf-panrg-path-properties] points to a longer, 317 more modern definition of path, which begins as follows: 319 | A sequence of adjacent path elements over which a packet can f be 320 | transmitted, starting and ending with a node. A path is 321 | unidirectional. Paths are time-dependent, i.e., the sequence of 322 | path elements over which packets are sent from one node to another 323 | may change. A path is defined between two nodes. 325 It follows that the general acceptance of a path is a linear sequence 326 of nodes, as opposed to a multi-dimensional graph. In the context of 327 this document, a path is observed by following one copy of a packet 328 that is injected in a Track and possibly replicated within. 330 2.1.3.2. Track 332 A networking graph that can be followed to transport packets with 333 equivalent treatment; as opposed to the definition of a path above, a 334 Track is not necessarily a linear sequence. It may contain multiple 335 paths that may fork and rejoin, for instance to enable the RAW PAREO 336 operations. 338 In DetNet [RFC8655] terms, a Track has the following properties: 340 * A Track has one Ingress and one Egress nodes, which operate as 341 DetNet Edge nodes. 343 * A Track is reversible, meaning that packets can be routed against 344 the flow of data packets, e.g., to carry OAM measurements or 345 control messages back to the Ingress. 347 * The vertices of the Track are DetNet Relay nodes that operate at 348 the DetNet Service sublayer and provide the PAREO functions. 350 * The topological edges of the graph are serial sequences of DetNet 351 Transit nodes that operate at the DetNet Forwarding sublayer. 353 2.1.3.3. SubTrack 355 A Track within a Track. The RAW PSE selects a subTrack on a per- 356 packet or a per-collection of packets basis to provide the desired 357 reliability for the transported flows. 359 2.1.3.4. Segment 361 A serial path formed by a topological edge of a Track. East-West 362 Segments are oriented from Ingress (East) to Egress (West). North/ 363 South Segments can be bidirectional; to avoid loops, measures must be 364 taken to ensure that a given packet flows either Northwards or 365 Southwards along a bidirectional Segment, but never bounces back. 367 2.1.4. Deterministic Networking 369 This document reuses the terminology in section 2 of [RFC8557] and 370 section 4.1.2 of [RFC8655] for deterministic networking and 371 deterministic networks. 373 2.1.4.1. Flow 375 A collection of consecutive IP packets defined by the upper layers 376 and signaled by the same 5 or 6-tuple, see section 5.1 of [RFC8939]. 377 Packets of the same flow must be placed on the same Track to receive 378 an equivalent treatment from Ingress to Egress within the Track. 379 Multiple flows may be transported along the same Track. The subTrack 380 that is selected for the flow may change over time under the control 381 of the PSE. 383 2.1.4.2. Deterministic Flow Identifier (L2) 385 A tuple identified by a stream_handle, and provided by a bridge, in 386 accordance with IEEE 802.1CB. The tuple comprises at least src MAC, 387 dst MAC, VLAN ID, and L2 priority. Continuous streams are 388 characterized by bandwidth and max packet size; scheduled streams are 389 characterized by a repeating pattern of timed transmissions. 391 2.1.4.3. Deterministic Flow Identifier (L3) 393 See section 3.3 of [DetNet-DP]. The classical IP 5-tuple that 394 identifies a flow comprises the src IP, dst IP, src port, dest port, 395 and the upper layer protocol (ULP). DetNet uses a 6-tuple where the 396 extra field is the DSCP field in the packet. The IPv6 flow label is 397 not used for that purpose. 399 2.1.4.4. TSN 401 TSN stands for Time Sensitive Networking and denotes the efforts at 402 IEEE 802 for deterministic networking, originally for use on 403 Ethernet. Wireless TSN (WTSN) denotes extensions of the TSN work on 404 wireless media such as the selected RAW technologies [RAW-TECHNOS]. 406 2.1.5. Reliability and Availability 408 In the context of the RAW work, Reliability and Availability are 409 defined as follows: 411 2.1.5.1. Service Level Agreement 413 In the context of RAW, an SLA (service level agreement) is a contract 414 between a provider, the network, and a client, the application flow, 415 about measurable metrics such as latency boundaries, consecutive 416 losses, and packet delivery ratio (PDR). 418 2.1.5.2. Service Level Objective 420 A service level objective (SLO) is one term in the SLA, for which 421 specific network setting and operations are implemented. For 422 instance, a dynamic tuning of the packet redundancy will address an 423 SLO of consecutive losses in a row by augmenting the chances of 424 delivery of a packet that follows a loss.). 426 2.1.5.3. Service Level Indicator 428 A service level indicator (SLI) measures the compliance of an SLO to 429 the terms of the contrast. It can be for instance the statistics of 430 individual losses and losses in a row as time series.). 432 2.1.5.4. Reliability 434 Reliability is a measure of the probability that an item will perform 435 its intended function for a specified interval under stated 436 conditions (SLA). RAW expresses reliability in terms of Mean Time 437 Between Failure (MTBF) and Maximum Consecutive Failures (MCF). More 438 in [NASA].). 440 2.1.5.5. Available 442 That is exempt of unscheduled outage or derivation from the terms of 443 the SLA. A basic expectation for a RAW network is that the flow is 444 maintained in the face of any single breakage or flapping. 446 2.1.5.6. Availability 448 Availability is a measure of the relative amount of time where a RAW 449 Network operates in stated condition (SLA), expressed as 450 (uptime)/(uptime+downtime). Because a serial wireless path may not 451 be good enough to provide the required reliability, and even 2 452 parallel paths may not be over a longer period of time, the RAW 453 availability implies a journey that is a lot more complex than 454 following a serial path. 456 2.1.6. OAM variations 458 2.1.6.1. Active OAM 460 See [RFC7799]. In the context of RAW, Active OAM is used to observe 461 a particular Track, subTrack, or Segment of a Track regardless of 462 whether it is used for traffic at that time. 464 2.1.6.2. In-Band OAM 466 An active OAM packet is considered in-band for the monitored Track 467 when it traverses the same set of links and interfaces and if the OAM 468 packet receives the same QoS and PAREO treatment as the packets of 469 the data flows that are injected in the Track. 471 2.1.6.3. Out-of-Band OAM 473 Out-of-band OAM is an active OAM whose path is not topologically 474 congruent to the Track, or its test packets receive a QoS and/or 475 PAREO treatment that is different from that of the packets of the 476 data flows that are injected in the Track, or both. 478 2.1.6.4. Limited OAM 480 An active OAM packet is a Limited OAM packet when it observes the RAW 481 operation over a node, a segment, or a subTrack of the Track, though 482 not from Ingress to Egress. It is injected in the datapath and 483 extracted from the datapath around the particular function or 484 subnetwork (e.g., around a relay providing a service layer 485 replication point) that is being tested. 487 2.1.6.5. Upstream OAM 489 An upstream OAM packet is an Out-of-Band OAM packet that traverses 490 the Track from egress to ingress on the reverse direction, to capture 491 and report OAM measurements upstream. The collection may capture all 492 information along the whole Track, or it may only learn select data 493 across all, or only a particular subTrack, or Segment of a Track. 495 2.1.6.6. Residence Time 497 A residence time (RT) is defined as the time period between the 498 reception of a packet starts and the transmission of the packet 499 begins. In the context of RAW, RT is useful for a transit node, not 500 ingress or egress. 502 2.1.6.7. Additional References 504 [DetNet-OAM] provides additional terminology related to OAM in the 505 context of DetNet and by extension of RAW, whereas [RFC7799] defines 506 the Active, Passive, and Hybrid OAM methods. 508 2.2. Reliability and Availability 509 2.2.1. High Availability Engineering Principles 511 The reliability criteria of a critical system pervades through its 512 elements, and if the system comprises a data network then the data 513 network is also subject to the inherited reliability and availability 514 criteria. It is only natural to consider the art of high 515 availability engineering and apply it to wireless communications in 516 the context of RAW. 518 There are three principles [pillars] of high availability 519 engineering: 521 1. elimination of single points of failure 522 2. reliable crossover 523 3. prompt detection of failures as they occur. 525 These principles are common to all high availability systems, not 526 just ones with Internet technology at the center. Examples of both 527 non-Internet and Internet are included. 529 2.2.1.1. Elimination of Single Points of Failure 531 Physical and logical components in a system happen to fail, either as 532 the effect of wear and tear, when used beyond acceptable limits, or 533 due to a software bug. It is necessary to decouple component failure 534 from system failure to avoid the latter. This allows failed 535 components to be restored while the rest of the system continues to 536 function. 538 IP Routers leverage routing protocols to compute alternate routes in 539 case of a failure. There is a rather open-ended issue over alternate 540 routes -- for example, when links are cabled through the same 541 conduit, they form a shared risk link group (SRLG), and will share 542 the same fate if the bundle is cut. The same effect can happen with 543 virtual links that end up in a same physical transport through the 544 games of encapsulation. In a same fashion, an interferer or an 545 obstacle may affect multiple wireless transmissions at the same time, 546 even between different sets of peers. 548 Intermediate network Nodes such as routers, switches and APs, wire 549 bundles and the air medium itself can become single points of 550 failure. For High Availability, it is thus required to use 551 physically link- and Node-disjoint paths; in the wireless space, it 552 is also required to use the highest possible degree of diversity 553 (time, space, code, frequency, channel width) in the transmissions 554 over the air to combat the additional causes of transmission loss. 556 From an economics standpoint, executing this principle properly 557 generally increases capitalization expense because of the redundant 558 equipment. In a constrained network where the waste of energy and 559 bandwidth should be minimized, an excessive use of redundant links 560 must be avoided; for RAW this means that the extra bandwidth must be 561 used wisely and with parcimony. 563 2.2.1.2. Reliable Crossover 565 Having a backup equipment has a limited value unless it can be 566 reliably switched into use within the down-time parameters. IP 567 Routers execute reliable crossover continuously because the routers 568 will use any alternate routes that are available [RFC0791]. This is 569 due to the stateless nature of IP datagrams and the dissociation of 570 the datagrams from the forwarding routes they take. The "IP Fast 571 Reroute Framework" [FRR] analyzes mechanisms for fast failure 572 detection and path repair for IP Fast-Reroute, and discusses the case 573 of multiple failures and SRLG. Examples of FRR techniques include 574 Remote Loop-Free Alternate [RLFA-FRR] and backup label-switched path 575 (LSP) tunnels for the local repair of LSP tunnels using RSVP-TE 576 [RFC4090]. 578 Deterministic flows, on the contrary, are attached to specific paths 579 where dedicated resources are reserved for each flow. This is why 580 each DetNet path must inherently provide sufficient redundancy to 581 provide the guaranteed SLA at all times. The DetNet PREOF typically 582 leverages 1+1 redundancy whereby a packet is sent twice, over non- 583 congruent paths. This avoids the gap during the fast reroute 584 operation, but doubles the traffic in the network. 586 In the case of RAW, the expectation is that multiple transient faults 587 may happen in overlapping time windows, in which case the 1+1 588 redundancy with delayed reestablishment of the second path will not 589 provide the required guarantees. The Data Plane must be configured 590 with a sufficient degree of redundancy to select an alternate 591 redundant path immediately upon a fault, without the need for a slow 592 intervention from the controller plane. 594 2.2.1.3. Prompt Notification of Failures 596 The execution of the two above principles is likely to render a 597 system where the user will rarely see a failure. But someone needs 598 to in order to direct maintenance. 600 There are many reasons for system monitoring (FCAPS for fault, 601 configuration, accounting, performance, security is a handy mental 602 checklist) but fault monitoring is sufficient reason. 604 "An Architecture for Describing Simple Network Management Protocol 605 (SNMP) Management Frameworks" [STD 62] describes how to use SNMP to 606 observe and correct long-term faults. 608 "Overview and Principles of Internet Traffic Engineering" [TE] 609 discusses the importance of measurement for network protection, and 610 provides abstract an method for network survivability with the 611 analysis of a traffic matrix as observed by SNMP, probing techniques, 612 FTP, IGP link state advertisements, and more. 614 Those measurements are needed in the context of RAW to inform the 615 controller and make the long term reactive decision to rebuild a 616 complex path. But RAW itself operates in the Network Plane at a 617 faster time scale. To act on the Data Plane, RAW needs live 618 information from the Operational Plane , e.g., using Dynamic Link 619 Exchange Protocol (DLEP) [DLEP] to obtain timely and accurate 620 knowledge of the characteristics of the link (speed, state, etc.). 622 2.2.2. Applying Reliability Concepts to Networking 624 The terms Reliability and Availability are defined for use in RAW in 625 Section 2.1 and the reader is invited to read [NASA] for more details 626 on the general definition of Reliability. Practically speaking a 627 number of nines is often used to indicate the reliability of a data 628 link, e.g., 5 nines indicate a Packet Delivery Ratio (PDR) of 629 99.999%. 631 This number is typical in a wired environment where the loss is due 632 to a random event such as a solar particle that affects the 633 transmission of a particular frame, but does not affect the previous 634 or next frame, nor frames transmitted on other links. Note that the 635 QoS requirements in RAW may include a bounded latency, and a packet 636 that arrives too late is a fault and not considered as delivered. 638 For a periodic networking pattern such as an automation control loop, 639 this number is proportional to the Mean Time Between Failures (MTBF). 640 When a single fault can have dramatic consequences, the MTBF 641 expresses the chances that the unwanted fault event occurs. In data 642 networks, this is rarely the case. Packet loss cannot never be fully 643 avoided and the systems are built to resist to one loss, e.g., using 644 redundancy with Retries (HARQ) or Packet Replication and Elimination 645 (PRE), or, in a typical control loop, by linear interpolation from 646 the previous measurements. 648 But the linear interpolation method cannot resist multiple 649 consecutive losses, and a high MTBF is desired as a guarantee that 650 this will not happen, IOW that the number of losses-in-a-row can be 651 bounded. In that case, what is really desired is a Maximum 652 Consecutive Failures (MCF). If the number of losses in a row passes 653 the MCF, the control loop has to abort and the system, e.g., the 654 production line, may need to enter an emergency stop condition. 656 Engineers that build automated processes may use the network 657 reliability expressed in nines or as an MTBF as a proxy to indicate 658 an MCF, e.g., as described in section 7.4 of the "Deterministic 659 Networking Use Cases" [RFC8578]. 661 2.2.3. Reliability in the Context of RAW 663 In contrast with wired networks, errors in transmission are the 664 predominant source of packet loss in wireless networks. 666 The root cause for the loss may be of multiple origins, calling for 667 the use of different forms of diversity: 669 Multipath Fading A destructive interference by a reflection of the 670 original signal. 672 A radio signal may be received directly (line-of-sight) and/or as 673 a reflection on a physical structure (echo). The reflections take 674 a longer path and are delayed by the extra distance divided by the 675 speed of light in the medium. Depending on the frequency, the 676 echo lands with a different phase which may add up to 677 (constructive interference) or cancel the direct signal 678 (destructive interference). 680 The affected frequencies depend on the relative position of the 681 sender, the receiver, and all the reflecting objects in the 682 environment. A given hop will suffer from multipath fading for 683 multiple packets in a row till a phyqical movement changes the 684 reflection patterns. 686 Co-channel Interference Energy in the spectrum used for the 687 transmission confuses the receiver. 689 The wireless medium itself is a Shared Risk Link Group (SRLG) for 690 nearby users of the same spectrum, as an interference may affect 691 multiple co-channel transmissions between different peers within 692 the interference domain of the interferer, possibly even when they 693 use different technologies. 695 Obstacle in Fresnel Zone The optimal transmission happens when the 696 Fresnel Zone between the sender and the receiver is free of 697 obstacles. 699 As long as a physical object (e.g., a metallic trolley between 700 peers) that affects the transmission is not removed, the quality 701 of the link is affected. 703 In an environment that is rich of metallic structures and mobile 704 objects, a single radio link will provide a fuzzy service, meaning 705 that it cannot be trusted to transport the traffic reliably over a 706 long period of time. 708 Transmission losses are typically not independent, and their nature 709 and duration are unpredictable; as long as a physical object (e.g., a 710 metallic trolley between peers) that affects the transmission is not 711 removed, or as long as the interferer (e.g., a radar) keeps 712 transmitting, a continuous stream of packets will be affected. 714 The key technique to combat those unpredictable losses is diversity. 715 Different forms of diversity are necessary to combat different causes 716 of loss and the use of diversity must be maximized to optimize the 717 PDR. 719 A single packet may be sent at different times (time diversity) over 720 diverse paths (spatial diversity) that rely on diverse radio channels 721 (frequency diversity) and diverse PHY technologies, e.g., narrowband 722 vs. spread spectrum, or diverse codes. Using time diversity will 723 defeat short-term interferences; spatial diversity combats very local 724 causes such as multipath fading; narrowband and spread spectrum are 725 relatively innocuous to one another and can be used for diversity in 726 the presence of the other. 728 2.3. Routing Time Scale vs. Forwarding Time Scale 730 With DetNet, the Controller Plane Function that handles the routing 731 computation and maintenance (the PCE) can be centralized and can 732 reside outside the network. In a wireless mesh, the path to the PCE 733 can be expensive and slow, possibly going across the whole mesh and 734 back. Reaching to the PCE can also be slow in regards to the speed 735 of events that affect the forwarding operation at the radio layer. 737 Due to that cost and latency, the Controller Plane is not expected to 738 be sensitive/reactive to transient changes. The abstraction of a 739 link at the routing level is expected to use statistical metrics that 740 aggregate the behavior of a link over long periods of time, and 741 represent its properties as shades of gray as opposed to numerical 742 values such as a link quality indicator, or a boolean value for 743 either up or down. 745 +----------------+ 746 | Controller | 747 | [PCE] | 748 +----------------+ 749 ^ 750 | 751 Slow 752 | 753 _-._-._-._-._-._-. | ._-._-._-._-._-._-._-._-._-._-._-._- 754 _-._-._-._-._-._-._-. | _-._-._-._-._-._-._-._-._-._-._-._- 755 | 756 Expensive 757 | 758 .... | ....... 759 .... . | . ....... 760 .... v ... 761 .. A-------B-------C---D .. 762 ... / \ / / \ .. 763 . I ----M-------N--***-- E .. 764 .. \ \ / / ... 765 .. P--***--Q----------R .... 766 .. .... 767 . <----- Fast -------> .... 768 ....... .... 769 ................. 771 *** = flapping at this time 773 Figure 1: Time Scales 775 In the case of wireless, the changes that affect the forwarding 776 decision can happen frequently and often for short durations, e.g., a 777 mobile object moves between a transmitter and a receiver, and will 778 cancel the line of sight transmission for a few seconds, or a radar 779 measures the depth of a pool and interferes on a particular channel 780 for a split second. 782 There is thus a desire to separate the long term computation of the 783 route and the short term forwarding decision. In that model, the 784 routing operation computes a complex Track that enables multiple Non- 785 Equal Cost Multi-Path (N-ECMP) forwarding solutions, and leaves it to 786 the Data Plane to make the per-packet decision of which of these 787 possibilities should be used. 789 In the wired world, and more specifically in the context of Traffic 790 Engineering (TE), an alternate path can be used upon the detection of 791 a failure in the main path, e.g., using OAM in MPLS-TP or BFD over a 792 collection of SD-WAN tunnels. RAW formalizes a forwarding time scale 793 that is an order(s) of magnitude shorter than the controller plane 794 routing time scale, and separates the protocols and metrics that are 795 used at both scales. Routing can operate on long term statistics 796 such as delivery ratio over minutes to hours, but as a first 797 approximation can ignore flapping. On the other hand, the RAW 798 forwarding decision is made at the scale of the packet rate, and uses 799 information that must be pertinent at the present time for the 800 current transmission(s). 802 3. The RAW Conceptual Model 804 RAW inherits the conceptual model described in section 4 of the 805 DetNet Architecture [RFC8655]. RAW extends the DetNet service layer 806 to provide additional agility against transmission loss. 808 A RAW Network Plane may be strict or loose, depending on whether RAW 809 observes and takes actions on all hops or not. For instance, the 810 packets between two wireless entities may be relayed over a wired 811 infrastructure such as a Wi-Fi extended service set (ESS) or a 5G 812 Core; in that case, RAW observes and controls the transmission over 813 the wireless first and last hops, as well as end-to-end metrics such 814 as latency, jitter, and delivery ratio. This operation is loose 815 since the structure and properties of the wired infrastructure are 816 ignored, and may be either controlled by other means such as DetNet/ 817 TSN, or neglected in the face of the wireless hops. 819 A Controller Plane Function (CPF) called the Path Computation Element 820 (PCE) [RFC4655] interacts with RAW Nodes over a Southbound API. The 821 RAW Nodes are DetNet relays that are capable of additional diversity 822 mechanisms and measurement functions related to the radio interface, 823 in particular the PAREO diversity mechanisms. 825 The PCE defines a complex Track between an Ingress End System and an 826 Egress End System, and indicates to the RAW Nodes where the PAREO 827 operations may be actionned in the Network Plane. The Track may be 828 expressed loosely to enable traversing a non-RAW subnetwork. In that 829 case, the expectation is that the non-RAW subnetwork can be neglected 830 in the RAW computation, that is, considered infinitely fast, reliable 831 and/or available in comparison with the links between RAW nodes. 833 CPF CPF CPF CPF 835 Southbound API 836 _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._- 837 _-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._-._- 839 RAW --z RAW --z RAW --z RAW 840 z-- Node z-- Node z-- Node z-- Node --z 841 Ingress --z / / z-- Egress 842 End \ \ .. . End 843 Node ---z / / .. .. . z-- Node 844 z-- RAW --z RAW ( non-RAW ) -- RAW --z 845 Node z-- Node --- ( Nodes ) Node 846 ... . 847 --z wireless wired 848 z-- link --- link 850 Figure 2: RAW Nodes 852 The Link-Layer metrics are reported to the PCE in a time-aggregated, 853 e.g., statistical fashion. Example Link-Layer metrics include 854 typical Link bandwidth (the medium speed depends dynamically on the 855 PHY mode), number of flows (bandwidth that can be reserved for a 856 flomw depends on the number and size of flows sharing the spectrum) 857 and average and mean squared deviation of availability and 858 reliability figures such as Packet Delivery Ratio (PDR) over long 859 periods of time. 861 Based on those metrics, the PCE installs the Track with enough 862 redundant forwarding solutions to ensure that the Network Plane can 863 reliably deliver the packets within a System Level Agreement (SLA) 864 associated to the flows that it transports. The SLA defines end-to- 865 end reliability and availability requirements, where reliability may 866 be expressed as a successful delivery in order and within a bounded 867 delay of at least one copy of a packet. 869 Depending on the use case and the SLA, the Track may comprise non-RAW 870 segments, either interleaved inside the Track, or all the way to the 871 Egress End Node (e.g., a server in the Internet). RAW observes the 872 Lower-Layer Links between RAW nodes (typically, radio links) and the 873 end-to-end Network Layer operation to decide at all times which of 874 the PAREO diversity schemes is actioned by which RAW Nodes. 876 Once a Track is established, per-segment and end-to-end reliability 877 and availability statistics are periodically reported to the PCE to 878 assure that the SLA can be met or have it recompute the Track if not. 880 4. The OODA Loop 882 The RAW Architecture is structured as an OODA Loop (Observe, Orient, 883 Decide, Act). It involves: 885 1. Network Plane measurement protocols for Operations, 886 Administration and Maintenance (OAM) to Observe some or all hops 887 along a Track as well as the end-to-end packet delivery, more in 888 Section 4.1; 890 2. Controller plane elements to reports the links statistics to a 891 Path computation Element (PCE) in a centralized controller that 892 computes and installs the Tracks and provides meta data to Orient 893 the routing decision, more in Section 4.2; 895 3. A Runtime distributed Path Selection Engine (PSE) thar Decides 896 which subTrack to use for the next packet(s) that are routed 897 along the Track, more in Section 4.3; 899 4. Packet (hybrid) ARQ, Replication, Elimination and Ordering 900 Dataplane actions that operate at the DetNet Service Layer to 901 increase the reliability o fthe end-to-end transmission. The RAW 902 architecture also covers in-situ signalling when the decision is 903 Acted by a node that down the Track from the PSE, more in 904 Section 4.4. 906 +-------> Orient (PCE) --------+ 907 | link stats, | 908 | pre-trained model | 909 | ... | 910 | | 911 | v 912 Observe (OAM) Decide (PSE) 913 ^ | 914 | | 915 | | 916 +-------- Act (PAREO) <--------+ 917 At DetNet 918 Service sublayer 920 Figure 3: The RAW OODA Loop 922 The overall OODA Loop optimizes the use of redundancy to achieve the 923 required reliability and availability Service Level Agreement (SLA) 924 while minimizing the use of constrained resources such as spectrum 925 and battery. 927 4.1. Observe: The RAW OAM 929 RAW In-situ OAM operation in the Network Plane may observe either a 930 full Track or subTracks that are being used at this time. As packets 931 may be load balanced, replicated, eliminated, and / or fragmented for 932 Network Coding (NC) forward error correction (FEC), the RAW In-situ 933 operation needs to be able to signal which operation occured to an 934 individual packet. 936 Active RAW OAM may be needed to observe the unused segments and 937 evaluate the desirability of a rerouting decision. 939 Finally, the RAW Service Layer Assurance may observe the individual 940 PAREO operation of a relay node to ensure that it is conforming; this 941 might require injecting an OAM packet at an upstream point inside the 942 Track and extracting that packet at another point downstream before 943 it reaches the egress. 945 This observation feeds the RAW PSE that makes the decision on which 946 PAREO function is actioned at which RAW Node, for one a small 947 continuous series of packets. 949 ... .. 950 RAN 1 ----- ... .. ... 951 / . .. .... 952 +-------+ / . .. .... +------+ 953 |Ingress|- . ..... |Egress| 954 | End |------ RAN 2 -- . Internet ....---| End | 955 |System |- .. ..... |System| 956 +-------+ \ . ...... +------+ 957 \ ... ... ..... 958 RAN n -------- ... ..... 960 <------------------> <--------------------> 961 Observed by OAM Opaque to OAM 963 Figure 4: Observed Links in Radio Access Protection 965 In the case of a End-to-End Protection in a Wireless Mesh, the Track 966 is strict and congruent with the path so all links are observed. 967 Conversely, in the case of Radio Access Protection, the Track is 968 Loose and in that case only the first hop is observed; the rest of 969 the path is abstracted and considered infinitely reliable. 971 In the case of the Radio Access Protection, only the first hop is 972 protected; the loss of a packet that was sent over one of the 973 possible first hops is attributed to that first hop, even if a 974 particular loss effectively happens farther down the path. 976 The Links that are not observed by OAM are opaque to it, meaning that 977 the OAM information is carried across and possibly echoed as data, 978 but there is no information capture in intermediate nodes. In the 979 example above, the Internet is opaque and not controlled by RAW; 980 still the RAW OAM measures the end-to-end latency and delivery ratio 981 for packets sent via each if RAN 1, RAN 2 and RAN 3, and determines 982 whether a packet should be sent over either or a collection of those 983 access links. 985 4.2. Orient: The Path Computation Engine 987 RAW separates the path computation time scale at which a complex path 988 is recomputed from the path selection time scale at which the 989 forwarding decision is taken for one or a few packets (see in 990 Section 2.3). 992 The path computation is out of scope, but RAW expects that the 993 Controller plane protocol that installs the Track also provides 994 related knowledge in the form of meta data about the links, segments 995 and possible subTracks. That meta data can be a pre-digested 996 statistical model, and may include prediction of future flaps and 997 packet loss, as well as recommended actions when that happens. 999 The meta data may include: 1001 * Pre-Determined subTracks to match predictable error profiles 1003 * Pre-Trained models 1005 * Link Quality Statistics and their projected evolution 1007 The Track is installed with measurable objectives that are computed 1008 by the PCE to achieve the RAW SLA. The objectives can be expressed 1009 as any of maximum number of packet lost in a row, bounded latency, 1010 maximal jitter, maximum nmuber of interleaved out of order packets, 1011 average number of copies received at the elimination point, and 1012 maximal delay between the first and the last received copy of the 1013 same packet. 1015 4.3. Decide: The Path Selection Engine 1017 The RAW OODA Loop operates at the path selection time scale to 1018 provide agility vs. the brute force approach of flooding the whole 1019 Track. The OODA Loop controls, within the redundant solutions that 1020 are proposed by the PCE, which will be used for each packet to 1021 provide a Reliable and Available service while minimizing the waste 1022 of constrained resources. 1024 To that effect, RAW defines the Path Selection Engine (PSE) that is 1025 the counterpart of the PCE to perform rapid local adjustments of the 1026 forwarding tables within the diversity that the PCE has selected for 1027 the Track. The PSE enables to exploit the richer forwarding 1028 capabilities with PAREO and scheduled transmissions at a faster time 1029 scale over the smaller domain that is the Track, in either a loose or 1030 a strict fashion. 1032 Compared to the PCE, the PSE operates on metrics that evolve faster, 1033 but that need to be advertised at a fast rate but only locally, 1034 within the Track. The forwarding decision may also change rapidly, 1035 but with a scope that is also contained within the Track, with no 1036 visibility to the other Tracks and flows in the network. This is as 1037 opposed to the PCE that must observe the whole network and optimize 1038 all the Tracks globally, which can only be done at a slow pace and 1039 using long-term statistical metrics, as presented in Table 1. 1041 +===============+========================+===================+ 1042 | | PCE (Not in Scope) | PSE (In Scope) | 1043 +===============+========================+===================+ 1044 | Operation | Centralized | Source-Routed or | 1045 | | | Distributed | 1046 +---------------+------------------------+-------------------+ 1047 | Communication | Slow, expensive | Fast, local | 1048 +---------------+------------------------+-------------------+ 1049 | Time Scale | hours and above | seconds and below | 1050 +---------------+------------------------+-------------------+ 1051 | Network Size | Large, many Tracks to | Small, within one | 1052 | | optimize globally | Track | 1053 +---------------+------------------------+-------------------+ 1054 | Considered | Averaged, Statistical, | Instant values / | 1055 | Metrics | Shade of grey | boolean condition | 1056 +---------------+------------------------+-------------------+ 1058 Table 1: PCE vs. PSE 1060 The PSE sits in the DetNet Service sub-Layer of Edge and Relay Nodes. 1061 On the one hand, it operates on the packet flow, learning the Track 1062 and path selection information from the packet, possibly making local 1063 decision and retagging the packet to indicate so. On the other hand, 1064 the PSE interacts with the lower layers and with its peers to obtain 1065 up-to-date information about its radio links and the quality of the 1066 overall Track, respectively, as illustrated in Figure 5. 1068 | 1069 packet | going 1070 down the | stack 1071 +==========v==========+=====================+=====================+ 1072 | (iOAM + iCTRL) | (L2 Triggers, DLEP) | (oOAM) | 1073 +==========v==========+=====================+=====================+ 1074 | Learn from Learn from | 1075 | packet tagging Maintain end-to-end | 1076 +----------v----------+ Forwarding OAM packets | 1077 | Forwarding decision < State +---------^-----------| 1078 +----------v----------+ | Enrich or | 1079 + Retag Packet | Learn abstracted > Regenerate | 1080 | and Forward | metrics about Links | OAM packets | 1081 +..........v..........+..........^..........+.........^.v.........+ 1082 | Lower layers | 1083 +..........v.....................^....................^.v.........+ 1084 frame | sent Frame | L2 Ack oOAM | | packet 1085 over | wireless In | In | | and out 1086 v | | v 1088 Figure 5: PSE 1090 4.4. Act: The PAREO Functions 1092 RAW may control whether and how to use packet replication and 1093 elimination (PRE), Automatic Repeat reQuest (ARQ), Hybrid ARQ (HARQ) 1094 that includes Forward Error Correction (FEC) and coding, and other 1095 wireless-specific techniques such as overhearing and constructive 1096 interferences, in order to increase the reliabiility and availability 1097 of the end-to-end transmission. 1099 Collectively, those function are called PAREO for Packet (hybrid) 1100 ARQ, Replication, Elimination and Ordering. By tuning dynamically 1101 the use of PAREO functions, RAW avoids the waste of critical 1102 resources such as spectrum and energy while providing that the 1103 guaranteed SLA, e.g., by adding redundancy only when a spike of loss 1104 is observed. 1106 In a nutshell, PAREO establishes several paths in a network to 1107 provide redundancy and parallel transmissions to bound the end-to-end 1108 delay to traverse the network. Optionally, promiscuous listening 1109 between paths is possible, such that the Nodes on one path may 1110 overhear transmissions along the other path. Considering the 1111 scenario shown in Figure 6, many different paths are possible to 1112 traverse the network from ingress to egress. A simple way to benefit 1113 from this topology could be to use the two independent paths via 1114 Nodes A, C, E and via B, D, F. But more complex paths are possible 1115 by interleaving transmissions from the lower level of the path to the 1116 upper level. 1118 (A) -- (C) -- (E) 1119 / \ 1120 Ingress = | | | = Egress 1121 \ / 1122 (B) -- (D) -- (F) 1124 Figure 6: A Ladder Shape with Two Parallel Paths 1126 PAREO may also take advantage of the shared properties of the 1127 wireless medium to compensate for the potential loss that is incurred 1128 with radio transmissions. 1130 For instance, when the source sends to Node A, Node B may listen 1131 promiscuously and get a second chance to receive the frame without an 1132 additional transmission. Note that B would not have to listen if it 1133 already received that particular frame at an earlier timeslot in a 1134 dedicated transmission towards B. 1136 The PAREO model can be implemented in both centralized and 1137 distributed scheduling approaches. In the centralized approach, a 1138 Path Computation Element (PCE) scheduler calculates a Track and 1139 schedules the communication. In the distributed approach, the Track 1140 is computed within the network, and signaled in the packets, e.g., 1141 using BIER-TE, Segment Routing, or a Source Routing Header. 1143 4.4.1. Packet Replication 1145 By employing a Packet Replication procedure, a Node forwards a copy 1146 of each data packet to more than one successor. To do so, each Node 1147 (i.e., Ingress and intermediate Node) sends the data packet multiple 1148 times as separate unicast transmissions. For instance, in Figure 7, 1149 the Ingress Node is transmitting the packet to both successors, nodes 1150 A and B, at two different times. 1152 ===> (A) => (C) => (E) === 1153 // \\// \\// \\ 1154 Ingress //\\ //\\ Egress 1155 \\ // \\ // \\ // 1156 ===> (B) => (D) => (F) === 1158 Figure 7: Packet Replication 1160 An example schedule is shown in Table 2. This way, the transmission 1161 leverages with the time and spatial forms of diversity. 1163 +=========+======+======+======+======+======+======+======+ 1164 | Channel | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 1165 +=========+======+======+======+======+======+======+======+ 1166 | 0 | S->A | S->B | B->C | B->D | C->F | E->R | F->R | 1167 +---------+------+------+------+------+------+------+------+ 1168 | 1 | | A->C | A->D | C->E | D->E | D->F | | 1169 +---------+------+------+------+------+------+------+------+ 1171 Table 2: Packet Replication: Sample schedule 1173 4.4.2. Packet Elimination 1175 The replication operation increases the traffic load in the network, 1176 due to packet duplications. This may occur at several stages inside 1177 the Track, and to avoid an explosion of the number of copies, a 1178 Packet Elimination procedure must be applied as well. To this aim, 1179 once a Node receives the first copy of a data packet, it discards the 1180 subsequent copies. 1182 The logical functions of Replication and Elimination may be 1183 collocated in an intermediate Node, the Node first eliminating the 1184 redundant copies and then sending the packet exactly once to each of 1185 the selected successors. 1187 4.4.3. Promiscuous Overhearing 1189 Considering that the wireless medium is broadcast by nature, any 1190 neighbor of a transmitter may overhear a transmission. By employing 1191 the Promiscuous Overhearing operation, the next hops have additional 1192 opportunities to capture the data packets. In Figure 8, when Node A 1193 is transmitting to its DP (Node C), the AP (Node D) and its sibling 1194 (Node B) may decode this data packet as well. As a result, by 1195 employing corellated paths, a Node may have multiple opportunities to 1196 receive a given data packet. 1198 ===> (A) ====> (C) ====> (E) ==== 1199 // ^ | \\ \\ 1200 Ingress | | \\ Egress 1201 \\ | v \\ // 1202 ===> (B) ====> (D) ====> (F) ==== 1204 Figure 8: Unicast with Overhearing 1206 Variations on the same idea such as link-layer anycast and multicast 1207 may also be used to reach more than one next-hop with a single frame. 1209 4.4.4. Constructive Interference 1211 Constructive Interference can be seen as the reverse of Promiscuous 1212 Overhearing, and refers to the case where two senders transmit the 1213 exact same signal in a fashion that the emitted symbols add up at the 1214 receiver and permit a reception that would not be possible with a 1215 single sender at the same PHY mode and the same power level. 1217 Constructive Interference was proposed on 5G, Wi-Fi7 and even tested 1218 on IEEE Std 802.14.5. The hard piece is to synchronize the senders 1219 to the point that the signals are emitted at slightly different time 1220 to offset the difference of propagation delay that corresponds to the 1221 difference of distance of the transmitters to the receiver at the 1222 speed of light to the point that the symbols are superposed long 1223 enough to be recognizable. 1225 5. Security Considerations 1227 RAW uses all forms of diversity including radio technology and 1228 physical path to increase the reliability and availability in the 1229 face of unpredictable conditions. While this is not done 1230 specifically to defeat an attacker, the amount of diversity used in 1231 RAW makes an attack harder to achieve. 1233 5.1. Forced Access 1235 RAW will typically select the cheapest collection of links that 1236 matches the requested SLA, for instance, leverage free WI-Fi vs. paid 1237 3GPP access. By defeating the cheap connectivity (e.g., PHY-layer 1238 interference) the attacker can force an End System to use the paid 1239 access and increase the cost of the transmission for the user. 1241 6. IANA Considerations 1243 This document has no IANA actions. 1245 7. Contributors 1247 The editor wishes to thank: 1249 Xavi Vilajosana: Wireless Networks Research Lab, Universitat Oberta 1250 de Catalunya 1252 Remous-Aris Koutsiamanis: IMT Atlantique 1254 Nicolas Montavont: IMT Atlantique 1256 Rex Buddenberg: Individual contributor 1258 Greg Mirsky: ZTE 1260 for their contributions to the text and ideas exposed in this 1261 document. 1263 8. Acknowledgments 1265 TBD 1267 9. References 1269 9.1. Normative References 1271 [6TiSCH-ARCHI] 1272 Thubert, P., Ed., "An Architecture for IPv6 over the Time- 1273 Slotted Channel Hopping Mode of IEEE 802.15.4 (6TiSCH)", 1274 RFC 9030, DOI 10.17487/RFC9030, May 2021, 1275 . 1277 [INT-ARCHI] 1278 Braden, R., Ed., "Requirements for Internet Hosts - 1279 Communication Layers", STD 3, RFC 1122, 1280 DOI 10.17487/RFC1122, October 1989, 1281 . 1283 [RAW-TECHNOS] 1284 Thubert, P., Cavalcanti, D., Vilajosana, X., Schmitt, C., 1285 and J. Farkas, "Reliable and Available Wireless 1286 Technologies", Work in Progress, Internet-Draft, draft- 1287 ietf-raw-technologies-04, 3 August 2021, 1288 . 1291 [RAW-USE-CASES] 1292 Papadopoulos, G. Z., Thubert, P., Theoleyre, F., and C. J. 1293 Bernardos, "RAW use cases", Work in Progress, Internet- 1294 Draft, draft-ietf-raw-use-cases-03, 20 October 2021, 1295 . 1298 [RFC4655] Farrel, A., Vasseur, J.-P., and J. Ash, "A Path 1299 Computation Element (PCE)-Based Architecture", RFC 4655, 1300 DOI 10.17487/RFC4655, August 2006, 1301 . 1303 [RFC6291] Andersson, L., van Helvoort, H., Bonica, R., Romascanu, 1304 D., and S. Mansfield, "Guidelines for the Use of the "OAM" 1305 Acronym in the IETF", BCP 161, RFC 6291, 1306 DOI 10.17487/RFC6291, June 2011, 1307 . 1309 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1310 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1311 May 2016, . 1313 [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", 1314 RFC 8578, DOI 10.17487/RFC8578, May 2019, 1315 . 1317 [IPv6] Deering, S. and R. Hinden, "Internet Protocol, Version 6 1318 (IPv6) Specification", STD 86, RFC 8200, 1319 DOI 10.17487/RFC8200, July 2017, 1320 . 1322 [RFC8557] Finn, N. and P. Thubert, "Deterministic Networking Problem 1323 Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019, 1324 . 1326 [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, 1327 "Deterministic Networking Architecture", RFC 8655, 1328 DOI 10.17487/RFC8655, October 2019, 1329 . 1331 [RFC8939] Varga, B., Ed., Farkas, J., Berger, L., Fedyk, D., and S. 1332 Bryant, "Deterministic Networking (DetNet) Data Plane: 1333 IP", RFC 8939, DOI 10.17487/RFC8939, November 2020, 1334 . 1336 [RFC9049] Dawkins, S., Ed., "Path Aware Networking: Obstacles to 1337 Deployment (A Bestiary of Roads Not Taken)", RFC 9049, 1338 DOI 10.17487/RFC9049, June 2021, 1339 . 1341 9.2. Informative References 1343 [RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, 1344 DOI 10.17487/RFC0791, September 1981, 1345 . 1347 [TE] Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X. 1348 Xiao, "Overview and Principles of Internet Traffic 1349 Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002, 1350 . 1352 [STD 62] Harrington, D., Presuhn, R., and B. Wijnen, "An 1353 Architecture for Describing Simple Network Management 1354 Protocol (SNMP) Management Frameworks", STD 62, RFC 3411, 1355 DOI 10.17487/RFC3411, December 2002, 1356 . 1358 [RFC4090] Pan, P., Ed., Swallow, G., Ed., and A. Atlas, Ed., "Fast 1359 Reroute Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 1360 DOI 10.17487/RFC4090, May 2005, 1361 . 1363 [FRR] Shand, M. and S. Bryant, "IP Fast Reroute Framework", 1364 RFC 5714, DOI 10.17487/RFC5714, January 2010, 1365 . 1367 [RLFA-FRR] Bryant, S., Filsfils, C., Previdi, S., Shand, M., and N. 1368 So, "Remote Loop-Free Alternate (LFA) Fast Reroute (FRR)", 1369 RFC 7490, DOI 10.17487/RFC7490, April 2015, 1370 . 1372 [DetNet-DP] 1373 Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S. 1374 Bryant, "Deterministic Networking (DetNet) Data Plane 1375 Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020, 1376 . 1378 [DLEP] Ratliff, S., Jury, S., Satterwhite, D., Taylor, R., and B. 1379 Berry, "Dynamic Link Exchange Protocol (DLEP)", RFC 8175, 1380 DOI 10.17487/RFC8175, June 2017, 1381 . 1383 [I-D.irtf-panrg-path-properties] 1384 Enghardt, T. and C. Kraehenbuehl, "A Vocabulary of Path 1385 Properties", Work in Progress, Internet-Draft, draft-irtf- 1386 panrg-path-properties-04, 25 October 2021, 1387 . 1390 [IPoWIRELESS] 1391 Thubert, P., "IPv6 Neighbor Discovery on Wireless 1392 Networks", Work in Progress, Internet-Draft, draft- 1393 thubert-6man-ipv6-over-wireless-11, 15 December 2021, 1394 . 1397 [DetNet-OAM] 1398 Mirsky, G., Theoleyre, F., Papadopoulos, G. Z., Bernardos, 1399 C. J., Varga, B., and J. Farkas, "Framework of Operations, 1400 Administration and Maintenance (OAM) for Deterministic 1401 Networking (DetNet)", Work in Progress, Internet-Draft, 1402 draft-ietf-detnet-oam-framework-05, 14 October 2021, 1403 . 1406 [NASA] Adams, T., "RELIABILITY: Definition & Quantitative 1407 Illustration", . 1410 Authors' Addresses 1412 Pascal Thubert (editor) 1413 Cisco Systems, Inc 1414 Building D 1415 45 Allee des Ormes - BP1200 1416 06254 MOUGINS - Sophia Antipolis 1417 France 1419 Phone: +33 497 23 26 34 1420 Email: pthubert@cisco.com 1422 Georgios Z. Papadopoulos 1423 IMT Atlantique 1424 Office B00 - 114A 1425 2 Rue de la Chataigneraie 1426 35510 Cesson-Sevigne - Rennes 1427 France 1429 Phone: +33 299 12 70 04 1430 Email: georgios.papadopoulos@imt-atlantique.fr