idnits 2.17.00 (12 Aug 2021) /tmp/idnits43455/draft-ietf-ippm-ioam-data-10.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (July 13, 2020) is 676 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 601 -- Looks like a reference, but probably isn't: '1' on line 605 -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2' -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX' == Outdated reference: A later version (-03) exists of draft-brockners-opsawg-ioam-deployment-01 == Outdated reference: draft-ietf-ntp-packet-timestamps has been published as RFC 8877 == Outdated reference: draft-ietf-nvo3-geneve has been published as RFC 8926 == Outdated reference: A later version (-12) exists of draft-ietf-nvo3-vxlan-gpe-09 == Outdated reference: A later version (-08) exists of draft-ietf-sfc-proof-of-transit-06 == Outdated reference: A later version (-06) exists of draft-spiegel-ippm-ioam-rawexport-03 Summary: 0 errors (**), 0 flaws (~~), 7 warnings (==), 5 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm F. Brockners, Ed. 3 Internet-Draft S. Bhandari, Ed. 4 Intended status: Standards Track Cisco 5 Expires: January 14, 2021 T. Mizrahi, Ed. 6 Huawei 7 July 13, 2020 9 Data Fields for In-situ OAM 10 draft-ietf-ippm-ioam-data-10 12 Abstract 14 In-situ Operations, Administration, and Maintenance (IOAM) records 15 operational and telemetry information in the packet while the packet 16 traverses a path between two points in the network. This document 17 discusses the data fields and associated data types for in-situ OAM. 18 In-situ OAM data fields can be encapsulated into a variety of 19 protocols such as NSH, Segment Routing, Geneve, IPv6 (via extension 20 header), or IPv4. In-situ OAM can be used to complement OAM 21 mechanisms based on e.g. ICMP or other types of probe packets. 23 Status of This Memo 25 This Internet-Draft is submitted in full conformance with the 26 provisions of BCP 78 and BCP 79. 28 Internet-Drafts are working documents of the Internet Engineering 29 Task Force (IETF). Note that other groups may also distribute 30 working documents as Internet-Drafts. The list of current Internet- 31 Drafts is at https://datatracker.ietf.org/drafts/current/. 33 Internet-Drafts are draft documents valid for a maximum of six months 34 and may be updated, replaced, or obsoleted by other documents at any 35 time. It is inappropriate to use Internet-Drafts as reference 36 material or to cite them other than as "work in progress." 38 This Internet-Draft will expire on January 14, 2021. 40 Copyright Notice 42 Copyright (c) 2020 IETF Trust and the persons identified as the 43 document authors. All rights reserved. 45 This document is subject to BCP 78 and the IETF Trust's Legal 46 Provisions Relating to IETF Documents 47 (https://trustee.ietf.org/license-info) in effect on the date of 48 publication of this document. Please review these documents 49 carefully, as they describe your rights and restrictions with respect 50 to this document. Code Components extracted from this document must 51 include Simplified BSD License text as described in Section 4.e of 52 the Trust Legal Provisions and are provided without warranty as 53 described in the Simplified BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 58 2. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 3 59 3. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 4 60 4. Scope, Applicability, and Assumptions . . . . . . . . . . . . 5 61 5. IOAM Data-Fields, Types, Nodes . . . . . . . . . . . . . . . 6 62 5.1. IOAM Data-Fields and Option-Types . . . . . . . . . . . . 6 63 5.2. IOAM-Domains and types of IOAM Nodes . . . . . . . . . . 7 64 5.3. IOAM-Namespaces . . . . . . . . . . . . . . . . . . . . . 8 65 5.4. IOAM Trace Option-Types . . . . . . . . . . . . . . . . . 10 66 5.4.1. Pre-allocated and Incremental Trace Option-Types . . 13 67 5.4.2. IOAM node data fields and associated formats . . . . 17 68 5.4.2.1. Hop_Lim and node_id short format . . . . . . . . 17 69 5.4.2.2. ingress_if_id and egress_if_id . . . . . . . . . 18 70 5.4.2.3. timestamp seconds . . . . . . . . . . . . . . . . 18 71 5.4.2.4. timestamp subseconds . . . . . . . . . . . . . . 18 72 5.4.2.5. transit delay . . . . . . . . . . . . . . . . . . 19 73 5.4.2.6. namespace specific data . . . . . . . . . . . . . 19 74 5.4.2.7. queue depth . . . . . . . . . . . . . . . . . . . 19 75 5.4.2.8. Checksum Complement . . . . . . . . . . . . . . . 20 76 5.4.2.9. Hop_Lim and node_id wide . . . . . . . . . . . . 20 77 5.4.2.10. ingress_if_id and egress_if_id wide . . . . . . . 21 78 5.4.2.11. namespace specific data wide . . . . . . . . . . 21 79 5.4.2.12. buffer occupancy . . . . . . . . . . . . . . . . 22 80 5.4.2.13. Opaque State Snapshot . . . . . . . . . . . . . . 22 81 5.4.3. Examples of IOAM node data . . . . . . . . . . . . . 23 82 5.5. IOAM Proof of Transit Option-Type . . . . . . . . . . . . 25 83 5.5.1. IOAM Proof of Transit Type 0 . . . . . . . . . . . . 27 84 5.6. IOAM Edge-to-Edge Option-Type . . . . . . . . . . . . . . 28 85 6. Timestamp Formats . . . . . . . . . . . . . . . . . . . . . . 30 86 6.1. PTP Truncated Timestamp Format . . . . . . . . . . . . . 30 87 6.2. NTP 64-bit Timestamp Format . . . . . . . . . . . . . . . 31 88 6.3. POSIX-based Timestamp Format . . . . . . . . . . . . . . 33 89 7. IOAM Data Export . . . . . . . . . . . . . . . . . . . . . . 34 90 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 34 91 8.1. IOAM Option-Type Registry . . . . . . . . . . . . . . . . 35 92 8.2. IOAM Trace-Type Registry . . . . . . . . . . . . . . . . 35 93 8.3. IOAM Trace-Flags Registry . . . . . . . . . . . . . . . . 36 94 8.4. IOAM POT-Type Registry . . . . . . . . . . . . . . . . . 36 95 8.5. IOAM POT-Flags Registry . . . . . . . . . . . . . . . . . 36 96 8.6. IOAM E2E-Type Registry . . . . . . . . . . . . . . . . . 37 97 8.7. IOAM Namespace-ID Registry . . . . . . . . . . . . . . . 37 98 9. Management and Deployment Considerations . . . . . . . . . . 38 99 10. Security Considerations . . . . . . . . . . . . . . . . . . . 38 100 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 39 101 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 40 102 12.1. Normative References . . . . . . . . . . . . . . . . . . 40 103 12.2. Informative References . . . . . . . . . . . . . . . . . 40 104 Contributors' Addresses . . . . . . . . . . . . . . . . . . . . . 42 105 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 44 107 1. Introduction 109 This document defines data fields for "in-situ" Operations, 110 Administration, and Maintenance (IOAM). In-situ OAM records OAM 111 information within the packet while the packet traverses a particular 112 network domain. The term "in-situ" refers to the fact that the OAM 113 data is added to the data packets rather than being sent within 114 packets specifically dedicated to OAM. IOAM is to complement 115 mechanisms such as Ping or Traceroute. In terms of "active" or 116 "passive" OAM, "in-situ" OAM can be considered a hybrid OAM type. 117 "In-situ" mechanisms do not require extra packets to be sent. IOAM 118 adds information to the already available data packets and therefore 119 cannot be considered passive. In terms of the classification given 120 in [RFC7799] IOAM could be portrayed as Hybrid Type 1. IOAM 121 mechanisms can be leveraged where mechanisms using e.g. ICMP do not 122 apply or do not offer the desired results, such as proving that a 123 certain traffic flow takes a pre-defined path, SLA verification for 124 the live data traffic, detailed statistics on traffic distribution 125 paths in networks that distribute traffic across multiple paths, or 126 scenarios in which probe traffic is potentially handled differently 127 from regular data traffic by the network devices. 129 IOAM use cases and mechanisms have expanded as this document matured, 130 resulting in additional flags and options that could trigger creation 131 of additional packets dedicated to OAM. The term IOAM continues to 132 be used for such mechanisms, in addition to the "in-situ" mechanisms 133 that motivated this terminology. 135 2. Contributors 137 This document was the collective effort of several authors. The text 138 and content were contributed by the editors and the co-authors listed 139 below. The contact information of the co-authors appears at the end 140 of this document. 142 o Carlos Pignataro 144 o Mickey Spiegel 145 o Barak Gafni 147 o Jennifer Lemon 149 o Hannes Gredler 151 o John Leddy 153 o Stephen Youell 155 o David Mozes 157 o Petr Lapukhov 159 o Remy Chang 161 o Daniel Bernier 163 3. Conventions 165 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 166 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 167 document are to be interpreted as described in [RFC2119]. 169 Abbreviations used in this document: 171 E2E Edge to Edge 173 Geneve: Generic Network Virtualization Encapsulation 174 [I-D.ietf-nvo3-geneve] 176 IOAM: In-situ Operations, Administration, and Maintenance 178 MTU: Maximum Transmit Unit 180 NSH: Network Service Header [RFC8300] 182 OAM: Operations, Administration, and Maintenance 184 PMTU Path MTU 186 POT: Proof of Transit 188 SFC: Service Function Chain 190 SID: Segment Identifier 192 SR: Segment Routing 193 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 194 Extension [I-D.ietf-nvo3-vxlan-gpe] 196 4. Scope, Applicability, and Assumptions 198 IOAM deployment assumes a set of constraints, requirements, and 199 guiding principles which are described in this section. 201 Scope: This document defines the data fields and associated data 202 types for in-situ OAM. The in-situ OAM data field can be 203 encapsulated in a variety of protocols, including NSH, Segment 204 Routing, Geneve, IPv6, or IPv4. Specification details for these 205 different protocols are outside the scope of this document. 207 Deployment domain (or scope) of in-situ OAM deployment: IOAM is a 208 network domain focused feature, with "network domain" being a set of 209 network devices or entities within a single administration. For 210 example, a network domain can include an enterprise campus using 211 physical connections between devices or an overlay network using 212 virtual connections / tunnels for connectivity between said devices. 213 A network domain is defined by its perimeter or edge. Designers of 214 protocol encapsulations for IOAM specify mechanisms to ensure that 215 IOAM data stays within an IOAM domain. In addition, the operator of 216 such a domain is expected to put provisions in place to ensure that 217 IOAM data does not leak beyond the edge of an IOAM domain using,for 218 example, packet filtering methods. The operator has to consider the 219 potential operational impact of IOAM to mechanisms such as ECMP 220 processing (e.g. load-balancing schemes based on packet length could 221 be impacted by the increased packet size due to IOAM), path MTU (i.e. 222 ensure that the MTU of all links within a domain is sufficiently 223 large to support the increased packet size due to IOAM) and ICMP 224 message handling (i.e. in case of IPv6, IOAM support for ICMPv6 Echo 225 Request/Reply is desired which would translate into ICMPv6 extensions 226 to enable IOAM-Data-Fields to be copied from an Echo Request message 227 to an Echo Reply message). 229 IOAM control points: IOAM-Data-Fields are added to or removed from 230 the live user traffic by the devices which form the edge of a domain. 231 Devices which form an IOAM-Domain can add, update or remove IOAM- 232 Data-Fields. Edge devices of an IOAM-Domain can be hosts or network 233 devices. 235 Traffic-sets that IOAM is applied to: IOAM can be deployed on all or 236 only on subsets of the live user traffic. Using IOAM on a selected 237 set of traffic (e.g., per interface, based on an access control list 238 or flow specification defining a specific set of traffic, etc.) could 239 be useful in deployments where the cost of processing IOAM-Data- 240 Fields by encapsulating, transit, or decapsulating node(s) might be a 241 concern from a performance or operational perspective. Thus limiting 242 the amount of traffic IOAM is applied to could be beneficial in some 243 deployments. 245 Encapsulation independence: The definition of IOAM-Data-Fields is 246 independent from the protocols the IOAM-Data-Fields are encapsulated 247 into. IOAM-Data-Fields can be encapsulated into several 248 encapsulating protocols. The specification of how IOAM-Data-Fields 249 are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is 250 outside the scope of this document. 252 Layering: If several encapsulation protocols (e.g., in case of 253 tunneling) are stacked on top of each other, IOAM-Data-Fields could 254 be present at multiple layers. The behavior follows the ships-in- 255 the-night model, i.e. IOAM-Data-Fields in one layer are independent 256 from IOAM-Data-Fields in another layer. Layering allows operators to 257 instrument the protocol layer they want to measure. The different 258 layers could, but do not have to, share the same IOAM encapsulation 259 mechanisms. 261 IOAM implementation: The definition of the IOAM-Data-Fields take the 262 specifics of devices with hardware data planes and software data 263 planes into account. 265 5. IOAM Data-Fields, Types, Nodes 267 This section details IOAM-related nomenclature and describes data 268 types such as IOAM-Data-Fields, IOAM-Types, IOAM-Namespaces as well 269 as the different types of IOAM nodes. 271 5.1. IOAM Data-Fields and Option-Types 273 An IOAM-Data-Field is a set of bits with a defined format and 274 meaning, which can be stored at a certain place in a packet for the 275 purpose of IOAM. 277 To accommodate the different uses of IOAM, IOAM-Data-Fields fall into 278 different categories. In IOAM these categories are referred to as 279 IOAM-Option-Types. A common registry is maintained for IOAM-Option- 280 Types, see Section 8.1 for details. Corresponding to these IOAM- 281 Option-Types, different IOAM-Data-Fields are defined. IOAM-Data- 282 Fields can be encapsulated into a variety of protocols, such as NSH, 283 Geneve, IPv6, etc. The definition of how IOAM-Data-Fields are 284 encapsulated into other protocols is outside the scope of this 285 document. 287 This document defines four IOAM-Option-Types: 289 o Pre-allocated Trace Option-Type 291 o Incremental Trace Option-Type 293 o Proof of Transit (POT) Option-Type 295 o Edge-to-Edge (E2E) Option-Type 297 5.2. IOAM-Domains and types of IOAM Nodes 299 IOAM is expected to be deployed in a specific domain. The part of 300 the network which employs IOAM is referred to as the "IOAM-Domain". 301 One or more IOAM-Option-Types are added to a packet upon entering the 302 IOAM-Domain and are removed from the packet when exiting the domain. 303 Within the IOAM-Domain, the IOAM-Data-Fields MAY be updated by 304 network nodes that the packet traverses. An IOAM-Domain consists of 305 "IOAM encapsulating nodes", "IOAM decapsulating nodes" and "IOAM 306 transit nodes". The role of a node (i.e. encapsulating, transit, 307 decapsulating) is defined within an IOAM-Namespace (see below). A 308 node can have different roles in different IOAM-Namespaces. 310 A device which adds at least one IOAM-Option-Type to the packet is 311 called the "IOAM encapsulating node", whereas a device which removes 312 an IOAM-Option-Type is referred to as the "IOAM decapsulating node". 313 Nodes within the domain which are aware of IOAM data and read and/or 314 write or process the IOAM data are called "IOAM transit nodes". IOAM 315 nodes which add or remove the IOAM-Data-Fields can also update the 316 IOAM-Data-Fields at the same time. Or in other words, IOAM 317 encapsulating or decapsulating nodes can also serve as IOAM transit 318 nodes at the same time. Note that not every node in an IOAM domain 319 needs to be an IOAM transit node. For example, a deployment might 320 require that packets traverse a set of firewalls which support IOAM. 321 In that case, only the set of firewall nodes would be IOAM transit 322 nodes rather than all nodes. 324 An "IOAM encapsulating node" incorporates one or more IOAM-Option- 325 Types (from the list of IOAM-Types, see Section 8.1) into packets 326 that IOAM is enabled for. If IOAM is enabled for a selected subset 327 of the traffic, the IOAM encapsulating node is responsible for 328 applying the IOAM functionality to the selected subset. 330 An "IOAM transit node" updates one or more of the IOAM-Data-Fields. 331 If both the Pre-allocated and the Incremental Trace Option-Types are 332 present in the packet, each IOAM transit node based on configuration 333 and available implementation of IOAM populates IOAM trace data in 334 either Pre-allocated or Incremental Trace Option-Type but not both. 335 A transit node MUST ignore IOAM-Option-Types that it does not 336 understand. A transit node MUST NOT add new IOAM-Option-Types to a 337 packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT 338 change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type. 340 An "IOAM decapsulating node" removes IOAM-Option-Type(s) from 341 packets. 343 The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating 344 node is always performed within a specific IOAM-Namespace. This 345 means that an IOAM node which is e.g. an IOAM-decapsulating node for 346 IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove 347 the IOAM-Option-Types for IOAM-Namespace "A" from the packet. Note 348 that this applies even for IOAM-Option-Types that the node does not 349 understand, for example an IOAM-Option-Type other than the four 350 described above, that is added in a future revision. An IOAM 351 decapsulating node situated at the edge of an IOAM domain MUST remove 352 all IOAM-Option-Types and associated encapsulation headers for all 353 IOAM-Namespaces from the packet. 355 IOAM-Namespaces allow for a namespace-specific definition and 356 interpretation of IOAM-Data-Fields. An interface-id could for 357 example point to a physical interface (e.g., to understand which 358 physical interface of an aggregated link is used when receiving or 359 transmitting a packet) whereas in another case it could refer to a 360 logical interface (e.g., in case of tunnels). Please refer to 361 Section 5.3 for details on IOAM-Namespaces. 363 5.3. IOAM-Namespaces 365 A subset or all of the IOAM-Option-Types and their corresponding 366 IOAM-Data-Fields can be associated to an IOAM-Namespace. IOAM- 367 Namespaces add further context to IOAM-Option-Types and associated 368 IOAM-Data-Fields. Any IOAM-Namespace MUST interpret the IOAM-Option- 369 Types and associated IOAM-Data-Fields per the definition in this 370 document. IOAM-Namespaces group nodes to support different 371 deployment approaches of IOAM (see a few example use-cases below) as 372 well as resolve issues which can occur due to IOAM-Data-Fields not 373 being globally unique (e.g. IOAM node identifiers do not have to be 374 globally unique). IOAM-Data-Fields significance is always within a 375 particular IOAM-Namespace. 377 An IOAM-Namespace is identified by a 16-bit namespace identifier 378 (Namespace-ID). IOAM-Namespace identifiers MUST be present and 379 populated in all IOAM-Option-Types. The Namespace-ID value is 380 divided into two sub-ranges: 382 o An operator-assigned range from 0x0001 to 0x7FFF 384 o An IANA-assigned range from 0x8000 to 0xFFFF 385 The IANA-assigned range is intended to allow future extensions to 386 have new and interoperable IOAM functionality, while the operator- 387 assigned range is intended to be domain specific, and managed by the 388 network operator. The Namespace-ID value of 0x0000 is default and 389 known to all the nodes implementing IOAM. 391 Namespace identifiers allow devices which are IOAM capable to 392 determine: 394 o whether IOAM-Option-Type(s) need to be processed by a device: If 395 the Namespace-ID contained in a packet does not match any 396 Namespace-ID the node is configured to operate on, then the node 397 MUST NOT change the contents of the IOAM-Data-Fields. 399 o which IOAM-Option-Type needs to be processed/updated in case there 400 are multiple IOAM-Option-Types present in the packet. Multiple 401 IOAM-Option-Types can be present in a packet in case of 402 overlapping IOAM-Domains or in case of a layered IOAM deployment. 404 o whether IOAM-Option-Type(s) has to be removed from the packet, 405 e.g. at a domain edge or domain boundary. 407 IOAM-Namespaces support several different uses: 409 o IOAM-Namespaces can be used by an operator to distinguish 410 different operational domains. Devices at domain edges can filter 411 on Namespace-IDs to provide for proper IOAM-Domain isolation. 413 o IOAM-Namespaces provide additional context for IOAM-Data-Fields 414 and thus ensure that IOAM-Data-Fields are unique and can be 415 interpreted properly by management stations or network 416 controllers. While, for example, the node identifier field 417 (node_id, see below) does not need to be unique in a deployment 418 (e.g. if an operator wishes to use different node identifiers for 419 different IOAM layers, even within the same device; or node 420 identifiers might not be unique for other organizational reasons, 421 such as after a merger of two formerly separated organizations), 422 the combination of node_id and Namespace-ID will always be unique. 423 Similarly, IOAM-Namespaces can be used to define how certain IOAM- 424 Data-Fields are interpreted: IOAM offers three different timestamp 425 format options. The Namespace-ID can be used to determine the 426 timestamp format. IOAM-Data-Fields (e.g. buffer occupancy) which 427 do not have a unit associated are to be interpreted within the 428 context of a IOAM-Namespace. 430 o IOAM-Namespaces can be used to identify different sets of devices 431 (e.g., different types of devices) in a deployment: If an operator 432 desires to insert different IOAM-Data-Fields based on the device, 433 the devices could be grouped into multiple IOAM-Namespaces. This 434 could be due to the fact that the IOAM feature set differs between 435 different sets of devices, or it could be for reasons of optimized 436 space usage in the packet header. It could also stem from 437 hardware or operational limitations on the size of the trace data 438 that can be added and processed, preventing collection of a full 439 trace for a flow. 441 * Assigning different IOAM Namespace-IDs to different sets of 442 nodes or network partitions and using the Namespace-ID as a 443 selector at the IOAM encapsulating node, a full trace for a 444 flow could be collected and constructed via partial traces in 445 different packets of the same flow. Example: An operator could 446 choose to group the devices of a domain into two IOAM- 447 Namespaces, in a way that on average, only every second hop 448 would be recorded by any device. To retrieve a full view of 449 the deployment, the captured IOAM-Data-Fields of the two IOAM- 450 Namespaces need to be correlated. 452 * Assigning different IOAM Namespace-IDs to different sets of 453 nodes or network partitions and using a separate instance of an 454 IOAM-Option-Type for each Namespace-ID, a full trace for a flow 455 could be collected and constructed via partial traces from each 456 IOAM-Option-Type in each of the packets in the flow. Example: 457 An operator could choose to group the devices of a domain into 458 two IOAM-Namespaces, in a way that each IOAM-Namespace is 459 represented by one of two IOAM-Option-Types in the packet. 460 Each node would record data only for the IOAM-Namespace that it 461 belongs to, ignoring the other IOAM-Option-Type with a IOAM- 462 Namespace to which it doesn't belong. To retrieve a full view 463 of the deployment, the captured IOAM-Data-Fields of the two 464 IOAM-Namespaces need to be correlated. 466 5.4. IOAM Trace Option-Types 468 "IOAM tracing data" is expected to be either collected at every IOAM 469 transit node that a packet traverses to ensure visibility into the 470 entire path a packet takes within an IOAM-Domain. I.e., in a typical 471 deployment all nodes in an IOAM-Domain would participate in IOAM and 472 thus be IOAM transit nodes, IOAM encapsulating or IOAM decapsulating 473 nodes. If not all nodes within a domain support IOAM functionality 474 as defined in this document, IOAM tracing information (i.e., node 475 data, see below) will only be collected on those nodes which support 476 IOAM functionality as defined in this document. Nodes which do not 477 support IOAM functionality as defined in this document will forward 478 the packet without any changes to the IOAM-Data-Fields. The maximum 479 number of hops and the minimum path MTU of the IOAM domain is assumed 480 to be known. An overflow indicator (O-bit) is defined as one of the 481 ways to deal with situations where the PMTU was underestimated, i.e. 482 where the number of hops which are IOAM capable exceeds the available 483 space in the packet. 485 To optimize hardware and software implementations, IOAM tracing is 486 defined as two separate options. Any deployment MAY choose to 487 configure and support one or both of the following options. 489 Pre-allocated Trace-Option: This trace option is defined as a 490 container of node data fields (see below) with pre-allocated space 491 for each node to populate its information. This option is useful 492 for implementations where it is efficient to allocate the space 493 once and index into the array to populate the data during transit 494 (e.g., software forwarders often fall into this class). The IOAM 495 encapsulating node allocates space for Pre-allocated Trace Option- 496 Type in the packet and sets corresponding fields in this IOAM- 497 Option-Type. The IOAM encapsulating node allocates an array which 498 is used to store operational data retrieved from every node while 499 the packet traverses the domain. IOAM transit nodes update the 500 content of the array, and possibly update the checksums of outer 501 headers. A pointer which is part of the IOAM trace data, points 502 to the next empty slot in the array. An IOAM transit node that 503 updates the content of the pre-allocated option also updates the 504 value of the pointer, which specifies where the next IOAM transit 505 node fills in its data. The "node data list" array (see below) in 506 the packet is populated iteratively as the packet traverses the 507 network, starting with the last entry of the array, i.e., "node 508 data list [n]" is the first entry to be populated, "node data list 509 [n-1]" is the second one, etc. 511 Incremental Trace-Option: This trace option is defined as a 512 container of node data fields where each node allocates and pushes 513 its node data immediately following the option header. This type 514 of trace recording is useful for some of the hardware 515 implementations as it eliminates the need for the transit network 516 elements to read the full array in the option and allows for 517 arbitrarily long packets as the MTU allows. The IOAM 518 encapsulating node allocates space for the Incremental Trace 519 Option-Type. Based on operational state and configuration, the 520 IOAM encapsulating node sets the fields in the Option-Type that 521 control what IOAM-Data-Fields have to be collected and how large 522 the node data list can grow. IOAM transit nodes push their node 523 data to the node data list, decrease the remaining length 524 available to subsequent nodes and adjust the lengths and possibly 525 checksums in outer headers. 527 A particular implementation of IOAM MAY choose to support only one of 528 the two trace option types. In the event that both options are 529 utilized at the same time, the Incremental Trace-Option MUST be 530 placed before the Pre-allocated Trace-Option. Deployments which mix 531 devices with either the Incremental Trace-Option or the Pre-allocated 532 Trace-Option could result in both Option-Types being present in a 533 packet. Given that the operator knows which equipment is deployed in 534 a particular IOAM, the operator will decide by means of configuration 535 which type(s) of trace options will be used for a particular domain. 537 Every node data entry holds information for a particular IOAM transit 538 node that is traversed by a packet. The IOAM decapsulating node 539 removes the IOAM-Option-Type(s) and processes and/or exports the 540 associated data. Like all IOAM-Data-Fields, the IOAM-Data-Fields of 541 the IOAM-Trace-Option-Types are defined in the context of an IOAM- 542 Namespace. 544 IOAM tracing can collect the following types of information: 546 o Identification of the IOAM node. An IOAM node identifier can 547 match to a device identifier or a particular control point or 548 subsystem within a device. 550 o Identification of the interface that a packet was received on, 551 i.e. ingress interface. 553 o Identification of the interface that a packet was sent out on, 554 i.e. egress interface. 556 o Time of day when the packet was processed by the node as well as 557 the transit delay. Different definitions of processing time are 558 feasible and expected, though it is important that all devices of 559 an in-situ OAM domain follow the same definition. 561 o Generic data: Format-free information where syntax and semantic of 562 the information is defined by the operator in a specific 563 deployment. For a specific IOAM-Namespace, all IOAM nodes have to 564 interpret the generic data the same way. Examples for generic 565 IOAM data include geo-location information (location of the node 566 at the time the packet was processed), buffer queue fill level or 567 cache fill level at the time the packet was processed, or even a 568 battery charge level. 570 o Information to detect whether IOAM trace data was added at every 571 hop or whether certain hops in the domain weren't IOAM transit 572 nodes. 574 5.4.1. Pre-allocated and Incremental Trace Option-Types 576 The IOAM Pre-allocated Trace-Option and the IOAM Incremental Trace- 577 Option have similar formats. Except where noted below, the internal 578 formats and fields of the two trace options are identical. Both 579 Trace-Options consist of a fixed size "trace option header" and a 580 variable data space to store gathered data, the "node data list". An 581 IOAM transit node (that is not an IOAM encapsulating node or IOAM 582 decapsulating node) MUST NOT modify any of the fields in the fixed 583 size "trace option header", other than "flags" and "RemainingLen", 584 i.e. an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen, 585 IOAM-Trace-Type, or Reserved fields. 587 Pre-allocated and incremental trace option headers: 589 0 1 2 3 590 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 591 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 592 | Namespace-ID |NodeLen | Flags | RemainingLen| 593 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 594 | IOAM-Trace-Type | Reserved | 595 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 597 The trace option data MUST be 4-octet aligned: 599 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 600 | | | 601 | node data list [0] | | 602 | | | 603 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 604 | | a 605 | node data list [1] | t 606 | | a 607 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 608 ~ ... ~ S 609 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 610 | | a 611 | node data list [n-1] | c 612 | | e 613 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 614 | | | 615 | node data list [n] | | 616 | | | 617 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 618 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 619 Namespace-ID value of 0x0000 is defined as the default value and 620 MUST be known to all the nodes implementing IOAM. For any other 621 Namespace-ID value that does not match any Namespace-ID the node 622 is configured to operate on, the node MUST NOT change the contents 623 of the IOAM-Data-Fields. 625 NodeLen: 5-bit unsigned integer. This field specifies the length of 626 data added by each node in multiples of 4-octets, excluding the 627 length of the "Opaque State Snapshot" field. 629 If IOAM-Trace-Type bit 22 is not set, then NodeLen specifies the 630 actual length added by each node. If IOAM-Trace-Type bit 22 is 631 set, then the actual length added by a node would be (NodeLen + 632 length of the "Opaque State Snapshot" field) in 4 octet units. 634 For example, if 3 IOAM-Trace-Type bits are set and none of them 635 are wide, then NodeLen would be 3. If 3 IOAM-Trace-Type bits are 636 set and 2 of them are wide, then NodeLen would be 5. 638 An IOAM encapsulating node MUST set NodeLen. 640 A node receiving an IOAM Pre-allocated or Incremental Trace-Option 641 relies on the NodeLen value, or it can ignore the NodeLen value 642 and calculate the node length from the IOAM-Trace-Type bits (see 643 below). 645 Flags 4-bit field. Flags are allocated by IANA, as specified in 646 Section 8.3. This document allocates a single flag as follows: 648 Bit 0 "Overflow" (O-bit) (most significant bit). If there are 649 not enough octets left to record node data, the network element 650 MUST NOT add any fields and MUST set the overflow "O-bit" to 651 "1" in the IOAM-Trace-Option header. This is useful for 652 transit nodes to ignore further processing of the option. 654 RemainingLen: 7-bit unsigned integer. This field specifies the data 655 space in multiples of 4-octets remaining for recording the node 656 data, before the node data list is considered to have overflowed. 657 Given that the sender knows the path MTU (PMTU), the sender MAY 658 set the initial value of RemainingLen according to the number of 659 node data bytes allowed before exceeding the MTU. Subsequent 660 nodes can carry out a simple comparison between RemainingLen and 661 NodeLen, along with the length of the "Opaque State Snapshot" if 662 applicable, to determine whether or not data can be added by this 663 node. When node data is added, the node MUST decrease 664 RemainingLen by the amount of data added. In the pre-allocated 665 trace option, RemainingLen is used to derive the offset in data 666 space to record the node data element. Specifically, the 667 recording of the node data element would start from RemainingLen - 668 NodeLen - sizeof(opaque snapshot) in 4 octet units. If 669 RemainingLen in a pre-allocated trace option exceeds the length of 670 the option, as specified in the preceding header, then the node 671 MUST NOT add any fields. 673 IOAM-Trace-Type: A 24-bit identifier which specifies which data 674 types are used in this node data list. 676 The IOAM-Trace-Type value is a bit field. The following bits are 677 defined in this document, with details on each bit described in 678 the Section 5.4.2. The order of packing the data fields in each 679 node data element follows the bit order of the IOAM-Trace-Type 680 field, as follows: 682 Bit 0 (Most significant bit) When set, indicates presence of 683 Hop_Lim and node_id (short format) in the node data. 685 Bit 1 When set, indicates presence of ingress_if_id and 686 egress_if_id (short format) in the node data. 688 Bit 2 When set, indicates presence of timestamp seconds in the 689 node data. 691 Bit 3 When set, indicates presence of timestamp subseconds in 692 the node data. 694 Bit 4 When set, indicates presence of transit delay in the node 695 data. 697 Bit 5 When set, indicates presence of IOAM-Namespace specific 698 data (short format) in the node data. 700 Bit 6 When set, indicates presence of queue depth in the node 701 data. 703 Bit 7 When set, indicates presence of the Checksum Complement 704 node data. 706 Bit 8 When set, indicates presence of Hop_Lim and node_id in 707 wide format in the node data. 709 Bit 9 When set, indicates presence of ingress_if_id and 710 egress_if_id in wide format in the node data. 712 Bit 10 When set, indicates presence of IOAM-Namespace specific 713 data in wide format in the node data. 715 Bit 11 When set, indicates presence of buffer occupancy in the 716 node data. 718 Bit 12-21 Undefined. An IOAM encapsulating node MUST set the 719 value of each of these bits to 0. If an IOAM transit 720 node receives a packet with one or more of these bits set 721 to 1, it MUST either: 723 1. Add corresponding node data filled with the reserved 724 value 0xFFFFFFFF, after the node data fields for the 725 IOAM-Trace-Type bits defined above, such that the 726 total node data added by this node in units of 727 4-octets is equal to NodeLen, or 729 2. Not add any node data fields to the packet, even for 730 the IOAM-Trace-Type bits defined above. 732 Bit 22 When set, indicates presence of variable length Opaque 733 State Snapshot field. 735 Bit 23 Reserved: MUST be set to zero upon transmission and 736 ignored upon receipt. 738 Section 5.4.2 describes the IOAM-Data-Types and their formats. 739 Within an IOAM-Domain possible combinations of these bits making 740 the IOAM-Trace-Type can be restricted by configuration knobs. 742 Reserved: 8-bits. An IOAM encapsulating node MUST set the value to 743 zero upon transmission. IOAM transit nodes MUST ignore the 744 received value. 746 Node data List [n]: Variable-length field. This is a list of node 747 data elements where the content of each node data element is 748 determined by the IOAM-Trace-Type. The order of packing the data 749 fields in each node data element follows the bit order of the 750 IOAM-Trace-Type field. Each node MUST prepend its node data 751 element in front of the node data elements that it received, such 752 that the transmitted node data list begins with this node's data 753 element as the first populated element in the list. The last node 754 data element in this list is the node data of the first IOAM 755 capable node in the path. Populating the node data list in this 756 way ensures that the order of node data list is the same for 757 incremental and pre-allocated trace options. In the pre-allocated 758 trace option, the index contained in RemainingLen identifies the 759 offset for current active node data to be populated. 761 5.4.2. IOAM node data fields and associated formats 763 All the IOAM-Data-Fields MUST be 4-octet aligned. If a node which is 764 supposed to update an IOAM-Data-Field is not capable of populating 765 the value of a field set in the IOAM-Trace-Type, the field value MUST 766 be set to 0xFFFFFFFF for 4-octet fields or 0xFFFFFFFFFFFFFFFF for 767 8-octet fields, indicating that the value is not populated, except 768 when explicitly specified in the field description below. 770 Some IOAM-Data-Fields defined below, such as interface identifiers or 771 IOAM-Namespace specific data, are defined in both "short format" as 772 well as "wide format". Their use is not exclusive. A deployment 773 could choose to leverage both. For example, ingress_if_id_(short 774 format) could be an identifier for the physical interface, whereas 775 ingress_if_id_(wide format) could be an identifier for a logical sub- 776 interface of that physical interface. 778 Data fields and associated data types for each of the IOAM-Data- 779 Fields are specified in the following sections. 781 5.4.2.1. Hop_Lim and node_id short format 783 The "Hop_Lim and node_id short format" field is a 4-octet field that 784 is defined as follows: 786 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 787 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 788 | Hop_Lim | node_id | 789 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 791 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 792 in the packet at the node that records this data. Hop Limit 793 information is used to identify the location of the node in the 794 communication path. This is copied from the lower layer, e.g., 795 TTL value in IPv4 header or hop limit field from IPv6 header of 796 the packet when the packet is ready for transmission. The 797 semantics of the Hop_Lim field depend on the lower layer protocol 798 that IOAM is encapsulated into, and therefore its specific 799 semantics are outside the scope of this memo. The value of this 800 field MUST be set to 0xff when the lower level does not have a 801 TTL/Hop limit equivalent field. 803 node_id: 3-octet unsigned integer. Node identifier field to 804 uniquely identify a node within the IOAM-Namespace and associated 805 IOAM-Domain. The procedure to allocate, manage and map the 806 node_ids is beyond the scope of this document. 808 5.4.2.2. ingress_if_id and egress_if_id 810 The "ingress_if_id and egress_if_id" field is a 4-octet field that is 811 defined as follows: 813 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 814 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 815 | ingress_if_id | egress_if_id | 816 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 818 ingress_if_id: 2-octet unsigned integer. Interface identifier to 819 record the ingress interface the packet was received on. 821 egress_if_id: 2-octet unsigned integer. Interface identifier to 822 record the egress interface the packet is forwarded out of. 824 Note that due to the fact that IOAM uses its own IOAM-Namespaces for 825 IOAM-Data-Fields, data fields like interface identifiers can be used 826 in a flexible way to represent system resources that are associated 827 with ingressing or egressing packets, i.e. ingress_if_id could 828 represent a physical interface, a virtual or logical interface, or 829 even a queue. 831 5.4.2.3. timestamp seconds 833 The "timestamp seconds" field is a 4-octet unsigned integer field. 834 Absolute timestamp in seconds that specifies the time at which the 835 packet was received by the node. This field has three possible 836 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX 837 [POSIX]. The three timestamp formats are specified in Section 6. In 838 all three cases, the Timestamp Seconds field contains the 32 most 839 significant bits of the timestamp format that is specified in 840 Section 6. If a node is not capable of populating this field, it 841 assigns the value 0xFFFFFFFF. Note that this is a legitimate value 842 that is valid for 1 second in approximately 136 years; the analyzer 843 has to correlate several packets or compare the timestamp value to 844 its own time-of-day in order to detect the error indication. 846 5.4.2.4. timestamp subseconds 848 The "timestamp subseconds" field is a 4-octet unsigned integer field. 849 Absolute timestamp in subseconds that specifies the time at which the 850 packet was received by the node. This field has three possible 851 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX 852 [POSIX]. The three timestamp formats are specified in Section 6. In 853 all three cases, the Timestamp Subseconds field contains the 32 least 854 significant bits of the timestamp format that is specified in 855 Section 6. If a node is not capable of populating this field, it 856 assigns the value 0xFFFFFFFF. Note that this is a legitimate value 857 in the NTP format, valid for approximately 233 picoseconds in every 858 second. If the NTP format is used the analyzer has to correlate 859 several packets in order to detect the error indication. 861 5.4.2.5. transit delay 863 The "transit delay" field is a 4-octet unsigned integer in the range 864 0 to 2^31-1. It is the time in nanoseconds the packet spent in the 865 transit node. This can serve as an indication of the queuing delay 866 at the node. If the transit delay exceeds 2^31-1 nanoseconds then 867 the top bit 'O' is set to indicate overflow and value set to 868 0x80000000. When this field is part of the data field but a node 869 populating the field is not able to fill it, the field position in 870 the field MUST be filled with value 0xFFFFFFFF to mean not populated. 872 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 873 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 874 |O| transit delay | 875 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 877 5.4.2.6. namespace specific data 879 The "namespace specific data" field is a 4-octet field which can be 880 used by the node to add IOAM-Namespace specific data. This 881 represents a "free-format" 4-octet bit field with its semantics 882 defined in the context of a specific IOAM-Namespace. 884 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 885 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 886 | namespace specific data | 887 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 889 5.4.2.7. queue depth 891 The "queue depth" field is a 4-octet unsigned integer field. This 892 field indicates the current length of the egress interface queue of 893 the interface from where the packet is forwarded out. The queue 894 depth is expressed as the current amount of memory buffers used by 895 the queue (a packet could consume one or more memory buffers, 896 depending on its size). 898 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 899 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 900 | queue depth | 901 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 903 5.4.2.8. Checksum Complement 905 The "Checksum Complement" field is a 4-octet node data which contains 906 a 4-octet Checksum Complement field. The Checksum Complement is 907 useful when IOAM is transported over encapsulations that make use of 908 a UDP transport, such as VXLAN-GPE or Geneve. Without the Checksum 909 Complement, nodes adding IOAM node data update the UDP Checksum field 910 following the recommendation of the encapsulation protocols. When 911 the Checksum Complement is present, an IOAM encapsulating node or 912 IOAM transit node adding node data MUST carry out one of the 913 following two alternatives in order to maintain the correctness of 914 the UDP Checksum value: 916 1. Recompute the UDP Checksum field. 918 2. Use the Checksum Complement to make a checksum-neutral update in 919 the UDP payload; the Checksum Complement is assigned a value that 920 complements the rest of the node data fields that were added by 921 the current node, causing the existing UDP Checksum field to 922 remain correct. 924 IOAM decapsulating nodes MUST recompute the UDP Checksum field, since 925 they do not know whether previous hops modified the UDP Checksum 926 field or the Checksum Complement field. 928 Checksum Complement fields are used in a similar manner in [RFC7820] 929 and [RFC7821]. 931 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 932 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 933 | Checksum Complement | 934 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 936 5.4.2.9. Hop_Lim and node_id wide 938 The "Hop_Lim and node_id wide" field is an 8-octet field defined as 939 follows: 941 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 942 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 943 | Hop_Lim | node_id ~ 944 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 945 ~ node_id (contd) | 946 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 948 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit value 949 in the packet at the node that records this data. Hop Limit 950 information is used to identify the location of the node in the 951 communication path. This is copied from the lower layer for e.g. 952 TTL value in IPv4 header or hop limit field from IPv6 header of 953 the packet. The semantics of the Hop_Lim field depend on the 954 lower layer protocol that IOAM is encapsulated into, and therefore 955 its specific semantics are outside the scope of this memo. The 956 value of this field MUST be set to 0xff when the lower level does 957 not have a TTL/Hop limit equivalent field. 959 node_id: 7-octet unsigned integer. Node identifier field to 960 uniquely identify a node within the IOAM-Namespace and associated 961 IOAM-Domain. The procedure to allocate, manage and map the 962 node_ids is beyond the scope of this document. 964 5.4.2.10. ingress_if_id and egress_if_id wide 966 The "ingress_if_id and egress_if_id wide" field is an 8-octet field 967 which is defined as follows: 969 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 970 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 971 | ingress_if_id | 972 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 973 | egress_if_id | 974 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 976 ingress_if_id: 4-octet unsigned integer. Interface identifier to 977 record the ingress interface the packet was received on. 979 egress_if_id: 4-octet unsigned integer. Interface identifier to 980 record the egress interface the packet is forwarded out of. 982 5.4.2.11. namespace specific data wide 984 The "namespace specific data wide" field is an 8-octet field which 985 can be used by the node to add IOAM-Namespace specific data. This 986 represents a "free-format" 8-octet bit field with its semantics 987 defined in the context of a specific IOAM-Namespace. 989 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 990 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 991 | namespace specific data ~ 992 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 993 ~ namespace specific data (contd) | 994 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 996 5.4.2.12. buffer occupancy 998 The "buffer occupancy" field is a 4-octet unsigned integer field. 999 This field indicates the current status of the occupancy of the 1000 common buffer pool used by a set of queues. The units of this field 1001 are implementation specific. Hence, the units are interpreted within 1002 the context of an IOAM-Namespace and/or node-id if used. The authors 1003 acknowledge that in some operational cases there is a need for the 1004 units to be consistent across a packet path through the network, 1005 hence RECOMMEND the implementations to use standard units such as 1006 Bytes. 1008 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1009 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1010 | buffer occupancy | 1011 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1013 5.4.2.13. Opaque State Snapshot 1015 The "Opaque State Snapshot" is a variable length field and follows 1016 the fixed length IOAM-Data-Fields defined above. It allows the 1017 network element to store an arbitrary state in the node data field, 1018 without a pre-defined schema. The schema is to be defined within the 1019 context of an IOAM-Namespace. The schema needs to be made known to 1020 the analyzer by some out-of-band mechanism. The specification of 1021 this mechanism is beyond the scope of this document. A 24-bit 1022 "Schema Id" field, interpreted within the context of an IOAM- 1023 Namespace, indicates which particular schema is used, and has to be 1024 configured on the network element by the operator. 1026 0 1 2 3 1027 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1028 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1029 | Length | Schema ID | 1030 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1031 | | 1032 | | 1033 | Opaque data | 1034 ~ ~ 1035 . . 1036 . . 1037 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1039 Length: 1-octet unsigned integer. It is the length in multiples of 1040 4-octets of the Opaque data field that follows Schema Id. 1042 Schema ID: 3-octet unsigned integer identifying the schema of Opaque 1043 data. 1045 Opaque data: Variable length field. This field is interpreted as 1046 specified by the schema identified by the Schema ID. 1048 When this field is part of the data field but a node populating the 1049 field has no opaque state data to report, the Length MUST be set to 0 1050 and the Schema ID MUST be set to 0xFFFFFF to mean no schema. 1052 5.4.3. Examples of IOAM node data 1054 An entry in the "node data list" array can have different formats, 1055 following the needs of the deployment. Some deployments might only 1056 be interested in recording the node identifiers, whereas others might 1057 be interested in recording node identifier and timestamp. The 1058 section provides example entries of the "node data list". 1060 0xD40000: IOAM-Trace-Type is 0xD40000 (0b110101000000000000000000) 1061 then the format of node data is: 1063 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1064 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1065 | Hop_Lim | node_id | 1066 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1067 | ingress_if_id | egress_if_id | 1068 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1069 | timestamp subseconds | 1070 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1071 | namespace specific data | 1072 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1074 0xC00000: IOAM-Trace-Type is 0xC00000 (0b110000000000000000000000) 1075 then the format is: 1077 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1078 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1079 | Hop_Lim | node_id | 1080 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1081 | ingress_if_id | egress_if_id | 1082 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1084 0x900000: IOAM-Trace-Type is 0x900000 (0b100100000000000000000000) 1085 then the format is: 1087 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1088 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1089 | Hop_Lim | node_id | 1090 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1091 | timestamp subseconds | 1092 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1094 0x840000: IOAM-Trace-Type is 0x840000 (0b100001000000000000000000) 1095 then the format is: 1097 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1098 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1099 | Hop_Lim | node_id | 1100 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1101 | namespace specific data | 1102 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1104 0x940000: IOAM-Trace-Type is 0x940000 (0b100101000000000000000000) 1105 then the format is: 1107 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1108 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1109 | Hop_Lim | node_id | 1110 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1111 | timestamp subseconds | 1112 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1113 | namespace specific data | 1114 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1116 0x308002: IOAM-Trace-Type is 0x308002 (0b001100001000000000000010) 1117 then the format is: 1119 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1120 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1121 | timestamp seconds | 1122 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1123 | timestamp subseconds | 1124 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1125 | Hop_Lim | node_id | 1126 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1127 | node_id(contd) | 1128 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1129 | Length | Schema Id | 1130 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1131 | | 1132 | | 1133 | Opaque data | 1134 ~ ~ 1135 . . 1136 . . 1137 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1139 5.5. IOAM Proof of Transit Option-Type 1141 IOAM Proof of Transit Option-Type is to support path or service 1142 function chain [RFC7665] verification use cases. Proof-of-transit 1143 leverages mechanisms like Shamir's Secret Sharing Schema (SSSS) 1144 [SSS]. For further information on Proof-of-transit, please refer to 1145 [I-D.ietf-sfc-proof-of-transit]. While details on how the IOAM data 1146 for the Proof-of-transit option is processed at IOAM encapsulating, 1147 decapsulating and transit nodes are outside the scope of the 1148 document, all of these approaches share the need to uniquely identify 1149 a packet as well as iteratively operate on a set of information that 1150 is handed from node to node. Correspondingly, two pieces of 1151 information are added as IOAM-Data-Fields to the packet: 1153 o Random: Unique identifier for the packet (e.g., 64-bits allow for 1154 the unique identification of 2^64 packets). 1156 o Cumulative: Information which is handed from node to node and 1157 updated by every node according to a verification algorithm. 1159 The IOAM Proof-of-Transit Option-Type consist of a fixed size "IOAM 1160 proof of transit option header" and "IOAM proof of transit option 1161 data fields": 1163 IOAM proof of transit option header: 1165 0 1 2 3 1166 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1167 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1168 | Namespace-ID |IOAM POT Type | IOAM POT flags| 1169 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1171 IOAM proof of transit Option-Type IOAM-Data-Fields MUST be 1172 4-octet aligned: 1174 0 1 2 3 1175 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1176 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1177 | POT Option data field determined by IOAM-POT-Type | 1178 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1180 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1181 Namespace-ID value of 0x0000 is defined as the default value and 1182 MUST be known to all the nodes implementing IOAM. For any other 1183 Namespace-ID value that does not match any Namespace-ID the node 1184 is configured to operate on, the node MUST NOT change the contents 1185 of the IOAM-Data-Fields. 1187 IOAM POT Type: 8-bit identifier of a particular POT variant that 1188 specifies the POT data that is included. This document defines 1189 POT Type 0: 1191 0: POT data is a 16 Octet field as described below. 1193 If a node receives an IOAM POT Type value that it does not 1194 understand, the node MUST NOT change the contents of the IOAM- 1195 Data-Fields. 1197 IOAM POT flags: 8-bit. Following flags are defined: 1199 Bit 0 "Profile-to-use" (P-bit) (most significant bit). For IOAM 1200 POT types that use a maximum of two profiles to drive 1201 computation, indicates which POT-profile is used. The two 1202 profiles are numbered 0, 1. 1204 Bit 1-7 Reserved: MUST be set to zero upon transmission and 1205 ignored upon receipt. 1207 POT Option data: Variable-length field. The type of which is 1208 determined by the IOAM-POT-Type. 1210 5.5.1. IOAM Proof of Transit Type 0 1212 IOAM proof of transit option of IOAM POT Type 0: 1214 0 1 2 3 1215 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1216 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1217 | Namespace-ID |IOAM POT Type=0|P|R R R R R R R| 1218 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1219 | Random | | 1220 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 1221 | Random(contd) | O 1222 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 1223 | Cumulative | | 1224 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1225 | Cumulative (contd) | | 1226 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 1228 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1229 Namespace-ID value of 0x0000 is defined as the default value and 1230 MUST be known to all the nodes implementing IOAM. For any other 1231 Namespace-ID value that does not match any Namespace-ID the node 1232 is configured to operate on, the node MUST NOT change the contents 1233 of the IOAM-Data-Fields. 1235 IOAM POT Type: 8-bit identifier of a particular POT variant that 1236 specifies the POT data that is included. This section defines the 1237 POT data when the IOAM POT Type is set to the value 0. 1239 P bit: 1-bit. "Profile-to-use" (P-bit) (most significant bit). 1240 Indicates which POT-profile is used to generate the Cumulative. 1241 Any node participating in POT will have a maximum of 2 profiles 1242 configured that drive the computation of cumulative. The two 1243 profiles are numbered 0, 1. This bit conveys whether profile 0 or 1244 profile 1 is used to compute the Cumulative. 1246 R (7 bits): 7-bit IOAM POT flags for future use. MUST be set to 1247 zero upon transmission and ignored upon receipt. 1249 Random: 64-bit Per packet Random number. 1251 Cumulative: 64-bit Cumulative that is updated at specific nodes by 1252 processing per packet Random number field and configured 1253 parameters. 1255 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 1256 feasible and could be required for certain deployments (e.g. in case 1257 of space constraints in the encapsulation protocols used). Future 1258 documents could introduce different sizes of data for "proof of 1259 transit". 1261 5.6. IOAM Edge-to-Edge Option-Type 1263 The IOAM Edge-to-Edge Option-Type is to carry data that is added by 1264 the IOAM encapsulating node and interpreted by IOAM decapsulating 1265 node. The IOAM transit nodes MAY process the data but MUST NOT 1266 modify it. 1268 The IOAM Edge-to-Edge Option-Type consist of a fixed size "IOAM Edge- 1269 to-Edge Option-Type header" and "IOAM Edge-to-Edge Option-Type data 1270 fields": 1272 IOAM Edge-to-Edge Option-Type header: 1274 0 1 2 3 1275 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1277 | Namespace-ID | IOAM-E2E-Type | 1278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1280 IOAM Edge-to-Edge Option-Type IOAM-Data-Fields MUST 1281 be 4-octet aligned: 1283 0 1 2 3 1284 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1285 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1286 | E2E Option data field determined by IOAM-E2E-Type | 1287 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1289 Namespace-ID: 16-bit identifier of an IOAM-Namespace. The 1290 Namespace-ID value of 0x0000 is defined as the default value and 1291 MUST be known to all the nodes implementing IOAM. For any other 1292 Namespace-ID value that does not match any Namespace-ID the node 1293 is configured to operate on, then the node MUST NOT change the 1294 contents of the IOAM-Data-Fields. 1296 IOAM-E2E-Type: A 16-bit identifier which specifies which data types 1297 are used in the E2E option data. The IOAM-E2E-Type value is a bit 1298 field. The order of packing the E2E option data field elements 1299 follows the bit order of the IOAM-E2E-Type field, as follows: 1301 Bit 0 (Most significant bit) When set indicates presence of a 1302 64-bit sequence number added to a specific "packet group" 1303 which is used to detect packet loss, packet reordering, 1304 or packet duplication within the group. The "packet 1305 group" is deployment dependent and defined at the IOAM 1306 encapsulating node e.g. by n-tuple based classification 1307 of packets. 1309 Bit 1 When set indicates presence of a 32-bit sequence number 1310 added to a specific "packet group" which is used to 1311 detect packet loss, packet reordering, or packet 1312 duplication within that group. The "packet group" is 1313 deployment dependent and defined at the IOAM 1314 encapsulating node e.g. by n-tuple based classification 1315 of packets. 1317 Bit 2 When set indicates presence of timestamp seconds, 1318 representing the time at which the packet entered the 1319 IOAM domain. Within the IOAM encapsulating node, the 1320 time that the timestamp is retrieved can depend on the 1321 implementation. Some possibilities are: 1) the time at 1322 which the packet was received by the node, 2) the time at 1323 which the packet was transmitted by the node, 3) when a 1324 tunnel encapsulation is used, the point at which the 1325 packet is encapsulated into the tunnel. Each 1326 implementation has to document when the E2E timestamp 1327 that is going to be put in the packet is retrieved. This 1328 4-octet field has three possible formats; based on either 1329 PTP [IEEE1588v2], NTP [RFC5905], or POSIX [POSIX]. The 1330 three timestamp formats are specified in Section 6. In 1331 all three cases, the Timestamp Seconds field contains the 1332 32 most significant bits of the timestamp format that is 1333 specified in Section 6. If a node is not capable of 1334 populating this field, it assigns the value 0xFFFFFFFF. 1335 Note that this is a legitimate value that is valid for 1 1336 second in approximately 136 years; the analyzer has to 1337 correlate several packets or compare the timestamp value 1338 to its own time-of-day in order to detect the error 1339 indication. 1341 Bit 3 When set indicates presence of timestamp subseconds, 1342 representing the time at which the packet entered the 1343 IOAM domain. This 4-octet field has three possible 1344 formats; based on either PTP [IEEE1588v2], NTP [RFC5905], 1345 or POSIX [POSIX]. The three timestamp formats are 1346 specified in Section 6. In all three cases, the 1347 Timestamp Subseconds field contains the 32 least 1348 significant bits of the timestamp format that is 1349 specified in Section 6. If a node is not capable of 1350 populating this field, it assigns the value 0xFFFFFFFF. 1351 Note that this is a legitimate value in the NTP format, 1352 valid for approximately 233 picoseconds in every second. 1353 If the NTP format is used the analyzer has to correlate 1354 several packets in order to detect the error indication. 1356 Bit 4-15 Undefined. An IOAM encapsulating node MUST set the value 1357 of these bits to zero upon transmission and ignore upon 1358 receipt. 1360 E2E Option data: Variable-length field. The type of which is 1361 determined by the IOAM-E2E-Type. 1363 6. Timestamp Formats 1365 The IOAM-Data-Fields include a timestamp field which is represented 1366 in one of three possible timestamp formats. It is assumed that the 1367 management plane is responsible for determining which timestamp 1368 format is used. 1370 6.1. PTP Truncated Timestamp Format 1372 The Precision Time Protocol (PTP) [IEEE1588v2] uses an 80-bit 1373 timestamp format. The truncated timestamp format is a 64-bit field, 1374 which is the 64 least significant bits of the 80-bit PTP timestamp. 1375 The PTP truncated format is specified in Section 4.3 of 1376 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1377 for the sake of completeness. 1379 0 1 2 3 1380 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1381 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1382 | Seconds | 1383 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1384 | Nanoseconds | 1385 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1387 Figure 1: PTP [IEEE1588v2] Truncated Timestamp Format 1389 Timestamp field format: 1391 Seconds: specifies the integer portion of the number of seconds 1392 since the epoch. 1394 + Size: 32 bits. 1396 + Units: seconds. 1398 Nanoseconds: specifies the fractional portion of the number of 1399 seconds since the epoch. 1401 + Size: 32 bits. 1403 + Units: nanoseconds. The value of this field is in the range 0 1404 to (10^9)-1. 1406 Epoch: 1408 The PTP [IEEE1588v2] epoch is 1 January 1970 00:00:00 TAI, which 1409 is 31 December 1969 23:59:51.999918 UTC. 1411 Resolution: 1413 The resolution is 1 nanosecond. 1415 Wraparound: 1417 This time format wraps around every 2^32 seconds, which is roughly 1418 136 years. The next wraparound will occur in the year 2106. 1420 Synchronization Aspects: 1422 It is assumed that nodes that run this protocol are synchronized 1423 among themselves. Nodes MAY be synchronized to a global reference 1424 time. Note that if PTP [IEEE1588v2] is used for synchronization, 1425 the timestamp MAY be derived from the PTP-synchronized clock, 1426 allowing the timestamp to be measured with respect to the clock of 1427 an PTP Grandmaster clock. 1429 The PTP truncated timestamp format is not affected by leap 1430 seconds. 1432 6.2. NTP 64-bit Timestamp Format 1434 The Network Time Protocol (NTP) [RFC5905] timestamp format is 64 bits 1435 long. This format is specified in Section 4.2.1 of 1436 [I-D.ietf-ntp-packet-timestamps], and the details are presented below 1437 for the sake of completeness. 1439 0 1 2 3 1440 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1441 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1442 | Seconds | 1443 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1444 | Fraction | 1445 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1447 Figure 2: NTP [RFC5905] 64-bit Timestamp Format 1449 Timestamp field format: 1451 Seconds: specifies the integer portion of the number of seconds 1452 since the epoch. 1454 + Size: 32 bits. 1456 + Units: seconds. 1458 Fraction: specifies the fractional portion of the number of 1459 seconds since the epoch. 1461 + Size: 32 bits. 1463 + Units: the unit is 2^(-32) seconds, which is roughly equal to 1464 233 picoseconds. 1466 Epoch: 1468 The epoch is 1 January 1900 at 00:00 UTC. 1470 Resolution: 1472 The resolution is 2^(-32) seconds. 1474 Wraparound: 1476 This time format wraps around every 2^32 seconds, which is roughly 1477 136 years. The next wraparound will occur in the year 2036. 1479 Synchronization Aspects: 1481 Nodes that use this timestamp format will typically be 1482 synchronized to UTC using NTP [RFC5905]. Thus, the timestamp MAY 1483 be derived from the NTP-synchronized clock, allowing the timestamp 1484 to be measured with respect to the clock of an NTP server. 1486 The NTP timestamp format is affected by leap seconds; it 1487 represents the number of seconds since the epoch minus the number 1488 of leap seconds that have occurred since the epoch. The value of 1489 a timestamp during or slightly after a leap second could be 1490 temporarily inaccurate. 1492 6.3. POSIX-based Timestamp Format 1494 This timestamp format is based on the POSIX time format [POSIX]. The 1495 detailed specification of the timestamp format used in this document 1496 is presented below. 1498 0 1 2 3 1499 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 1500 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1501 | Seconds | 1502 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1503 | Microseconds | 1504 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 1506 Figure 3: POSIX-based Timestamp Format 1508 Timestamp field format: 1510 Seconds: specifies the integer portion of the number of seconds 1511 since the epoch. 1513 + Size: 32 bits. 1515 + Units: seconds. 1517 Microseconds: specifies the fractional portion of the number of 1518 seconds since the epoch. 1520 + Size: 32 bits. 1522 + Units: the unit is microseconds. The value of this field is in 1523 the range 0 to (10^6)-1. 1525 Epoch: 1527 The epoch is 1 January 1970 00:00:00 TAI, which is 31 December 1528 1969 23:59:51.999918 UTC. 1530 Resolution: 1532 The resolution is 1 microsecond. 1534 Wraparound: 1536 This time format wraps around every 2^32 seconds, which is roughly 1537 136 years. The next wraparound will occur in the year 2106. 1539 Synchronization Aspects: 1541 It is assumed that nodes that use this timestamp format run the 1542 Linux operating system, and hence use the POSIX time. In some 1543 cases nodes MAY be synchronized to UTC using a synchronization 1544 mechanism that is outside the scope of this document, such as NTP 1545 [RFC5905]. Thus, the timestamp MAY be derived from the NTP- 1546 synchronized clock, allowing the timestamp to be measured with 1547 respect to the clock of an NTP server. 1549 The POSIX-based timestamp format is affected by leap seconds; it 1550 represents the number of seconds since the epoch minus the number 1551 of leap seconds that have occurred since the epoch. The value of 1552 a timestamp during or slightly after a leap second could be 1553 temporarily inaccurate. 1555 7. IOAM Data Export 1557 IOAM nodes collect information for packets traversing a domain that 1558 supports IOAM. IOAM decapsulating nodes as well as IOAM transit 1559 nodes can choose to retrieve IOAM information from the packet, 1560 process the information further and export the information using 1561 e.g., IPFIX. The mechanisms and associated data formats for 1562 exporting IOAM data is outside the scope of this document. 1564 Raw data export of IOAM data using IPFIX is discussed in 1565 [I-D.spiegel-ippm-ioam-rawexport]. 1567 8. IANA Considerations 1569 This document requests the following IANA Actions. 1571 IANA is requested to define a registry group named "In-Situ OAM 1572 (IOAM) Protocol Parameters". 1574 This group will include the following registries: 1576 IOAM Option-Type 1578 IOAM Trace-Type 1580 IOAM Trace-Flags 1581 IOAM POT-Type 1583 IOAM POT-Flags 1585 IOAM E2E-Type 1587 IOAM Namespace-ID 1589 New registries in this group can be created via RFC Required process 1590 as per [RFC8126]. 1592 The subsequent sub-sections detail the registries herein contained. 1594 8.1. IOAM Option-Type Registry 1596 This registry defines 128 code points for the IOAM Option-Type field 1597 for identifying IOAM Option-Types as explained in Section 5. The 1598 following code points are defined in this draft: 1600 0 IOAM Pre-allocated Trace Option-Type 1602 1 IOAM Incremental Trace Option-Type 1604 2 IOAM POT Option-Type 1606 3 IOAM E2E Option-Type 1608 4 - 127 are available for assignment via RFC Required process as per 1609 [RFC8126]. 1611 8.2. IOAM Trace-Type Registry 1613 This registry defines code point for each bit in the 24-bit IOAM- 1614 Trace-Type field for Pre-allocated trace option and Incremental trace 1615 option defined in Section 5.4. The meaning of Bits 0 - 11 for trace 1616 type are defined in this document in Paragraph 5 of Section 5.4.1: 1618 Bit 0 hop_Lim and node_id in short format 1620 Bit 1 ingress_if_id and egress_if_id in short format 1622 Bit 2 timestamp seconds 1624 Bit 3 timestamp subseconds 1626 Bit 4 transit delay 1628 Bit 5 namespace specific data in short format 1629 Bit 6 queue depth 1631 Bit 7 checksum complement 1633 Bit 8 hop_Lim and node_id in wide format 1635 Bit 9 ingress_if_id and egress_if_id in wide format 1637 Bit 10 namespace specific data in wide format 1639 Bit 11 buffer occupancy 1641 Bit 22 variable length Opaque State Snapshot 1643 Bit 23 reserved 1645 The meaning for Bits 12 - 21 are available for assignment via RFC 1646 Required process as per [RFC8126]. 1648 8.3. IOAM Trace-Flags Registry 1650 This registry defines code points for each bit in the 4 bit flags for 1651 the Pre-allocated trace option and for the Incremental trace option 1652 defined in Section 5.4. The meaning of Bit 0 (the most significant 1653 bit) for trace flags is defined in this document in Paragraph 3 of 1654 Section 5.4.1: 1656 Bit 0 "Overflow" (O-bit) 1658 Bit 1 - 3 are available for assignment via RFC Required process as 1659 per [RFC8126]. 1661 8.4. IOAM POT-Type Registry 1663 This registry defines 256 code points to define IOAM POT Type for 1664 IOAM proof of transit option Section 5.5. The code point value 0 is 1665 defined in this document: 1667 0: 16 Octet POT data 1669 1 - 255 are available for assignment via RFC Required process as per 1670 [RFC8126]. 1672 8.5. IOAM POT-Flags Registry 1674 This registry defines code points for each bit in the 8 bit flags for 1675 IOAM POT option defined in Section 5.5. The meaning of Bit 0 for 1676 IOAM POT flags is defined in this document in Section 5.5: 1678 Bit 0 "Profile-to-use" (P-bit) 1680 The meaning for Bits 1 - 7 are available for assignment via RFC 1681 Required process as per [RFC8126]. 1683 8.6. IOAM E2E-Type Registry 1685 This registry defines code points for each bit in the 16 bit IOAM- 1686 E2E-Type field for IOAM E2E option Section 5.6. The meaning of Bit 0 1687 - 3 are defined in this document: 1689 Bit 0 64-bit sequence number 1691 Bit 1 32-bit sequence number 1693 Bit 2 timestamp seconds 1695 Bit 3 timestamp subseconds 1697 The meaning of Bits 4 - 15 are available for assignment via RFC 1698 Required process as per [RFC8126]. 1700 8.7. IOAM Namespace-ID Registry 1702 IANA is requested to set up an "IOAM Namespace-ID Registry", 1703 containing 16-bit values. The meaning of Bit 0 is defined in this 1704 document. IANA is requested to reserve the values 0x0001 to 0x7FFF 1705 for private use (managed by operators), as specified in Section 5.3 1706 of the current document. Registry entries for the values 0x8000 to 1707 0xFFFF are to be assigned via the "Expert Review" policy defined in 1708 [RFC8126]. Upon a new allocation request, the responsible AD will 1709 appoint a designated expert, who will review the allocation request. 1710 The expert will post the request on the IPPM mailing list, and 1711 possibly on other relevant mailing lists, to allow for community 1712 feedback. Based on the review, the expert will either approve or 1713 deny the request. The intention is that any allocation will be 1714 accompanied by a published RFC. But in order to allow for the 1715 allocation of values prior to the RFC being approved for publication, 1716 the designated expert can approve allocations once it seems clear 1717 that an RFC will be published. 1719 0: default namespace (known to all IOAM nodes) 1721 0x0001 - 0x7FFF: reserved for private use 1723 0x8000 - 0xFFFF: unassigned 1725 9. Management and Deployment Considerations 1727 This document defines the structure and use of IOAM data fields. 1728 This document does not define the encapsulation of IOAM data fields 1729 into different protocols. Management and deployment aspects for IOAM 1730 have to be considered within the context of the protocol IOAM data 1731 fields are encapsulated into and as such, are out of scope for this 1732 document. For a discussion of IOAM deployment, please also refer to 1733 [I-D.brockners-opsawg-ioam-deployment], which outlines a framework 1734 for IOAM deployment and provides best current practices. 1736 10. Security Considerations 1738 As discussed in [RFC7276], a successful attack on an OAM protocol in 1739 general, and specifically on IOAM, can prevent the detection of 1740 failures or anomalies, or create a false illusion of nonexistent 1741 ones. In particular, these threats are applicable by compromising 1742 the integrity of IOAM data, either by maliciously modifying IOAM 1743 options in transit, or by injecting packets with maliciously 1744 generated IOAM options 1746 The Proof of Transit Option-Type (Section Section 5.5) is used for 1747 verifying the path of data packets. The security considerations of 1748 POT are further discussed in [I-D.ietf-sfc-proof-of-transit]. 1750 From a confidentiality perspective, although IOAM options do not 1751 contain user data, they can be used for network reconnaissance, 1752 allowing attackers to collect information about network paths, 1753 performance, queue states, buffer occupancy and other information. 1754 Moreover, if IOAM data leaks from the IOAM domain it could enable 1755 reconnaissance beyond the scope of the IOAM domain. Note that in 1756 case IOAM is used in "Direct Exporting" mode 1757 [I-D.ioamteam-ippm-ioam-direct-export], the IOAM related trace 1758 information would not be available in the customer data packets, but 1759 would trigger export of packet related IOAM information at every 1760 node, thus restricting the potential threat to the management plane 1761 and mitigating the leakage threat. IOAM data exporting and the way 1762 it is secured is outside the scope of this document. 1764 IOAM can be used as a means for implementing Denial of Service (DoS) 1765 attacks, or for amplifying them. For example, a malicious attacker 1766 can add an IOAM header to packets in order to consume the resources 1767 of network devices that take part in IOAM or entities that receive, 1768 collect or analyze the IOAM data. Another example is a packet length 1769 attack, in which an attacker pushes headers associated with IOAM 1770 Option-Types into data packets, causing these packets to be increased 1771 beyond the MTU size, resulting in fragmentation or in packet drops. 1773 Since IOAM options can include timestamps, if network devices use 1774 synchronization protocols then any attack on the time protocol 1775 [RFC7384] can compromise the integrity of the timestamp-related data 1776 fields. 1778 At the management plane, attacks can be set up by misconfiguring or 1779 by maliciously configuring IOAM-enabled nodes in a way that enables 1780 other attacks. Thus, IOAM configuration has to be secured in a way 1781 that authenticates authorized users and verifies the integrity of 1782 configuration procedures. 1784 The current document does not define a specific IOAM encapsulation. 1785 It has to be noted that some IOAM encapsulation types can introduce 1786 specific security considerations. A specification that defines an 1787 IOAM encapsulation is expected to address the respective 1788 encapsulation-specific security considerations. 1790 Notably, in most cases IOAM is expected to be deployed in specific 1791 network domains, thus confining the potential attack vectors to 1792 within the network domain. A limited administrative domain provides 1793 the operator with the means to select, monitor, and control the 1794 access of all the network devices, making these devices trusted by 1795 the operator. Indeed, in order to limit the scope of threats 1796 mentioned above to within the current network domain the network 1797 operator is expected to enforce policies that prevent IOAM traffic 1798 from leaking outside of the IOAM domain, and prevent IOAM data from 1799 outside the domain to be processed and used within the domain. 1801 The security considerations of a system that deploys IOAM, much like 1802 any system, has to be reviewed on a per-deployment-scenario basis, 1803 based on a systems-specific threat analysis, which can lead to 1804 specific security solutions that are beyond the scope of the current 1805 document. For example, in an IOAM deployment that is not confined to 1806 a single LAN, but spans multiple inter-connected sites, the inter- 1807 site links can be secured (e.g., by IPsec) in order to avoid external 1808 threats. 1810 11. Acknowledgements 1812 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 1813 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 1814 Nadahalli, LJ Wobker, Erik Nordmark, Vengada Prasad Govindan, Andrew 1815 Yourtchenko, Aviv Kfir, Tianran Zhou and Zhenbin (Robin) for the 1816 comments and advice. 1818 This document leverages and builds on top of several concepts 1819 described in [I-D.kitamura-ipv6-record-route]. The authors would 1820 like to acknowledge the work done by the author Hiroshi Kitamura and 1821 people involved in writing it. 1823 The authors would like to gracefully acknowledge useful review and 1824 insightful comments received from Joe Clarke, Al Morton, Tom Herbert, 1825 Haoyu Song, Mickey Spiegel and Barak Gafni. 1827 12. References 1829 12.1. Normative References 1831 [IEEE1588v2] 1832 Institute of Electrical and Electronics Engineers, "IEEE 1833 Std 1588-2008 - IEEE Standard for a Precision Clock 1834 Synchronization Protocol for Networked Measurement and 1835 Control Systems", IEEE Std 1588-2008, 2008, 1836 . 1839 [POSIX] Institute of Electrical and Electronics Engineers, "IEEE 1840 Std 1003.1-2008 (Revision of IEEE Std 1003.1-2004) - IEEE 1841 Standard for Information Technology - Portable Operating 1842 System Interface (POSIX(R))", IEEE Std 1003.1-2008, 2008, 1843 . 1846 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1847 Requirement Levels", BCP 14, RFC 2119, 1848 DOI 10.17487/RFC2119, March 1997, 1849 . 1851 [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch, 1852 "Network Time Protocol Version 4: Protocol and Algorithms 1853 Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010, 1854 . 1856 [RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for 1857 Writing an IANA Considerations Section in RFCs", BCP 26, 1858 RFC 8126, DOI 10.17487/RFC8126, June 2017, 1859 . 1861 12.2. Informative References 1863 [I-D.brockners-opsawg-ioam-deployment] 1864 Brockners, F., Bhandari, S., and d. 1865 daniel.bernier@bell.ca, "In-situ OAM Deployment", draft- 1866 brockners-opsawg-ioam-deployment-01 (work in progress), 1867 March 2020. 1869 [I-D.ietf-ntp-packet-timestamps] 1870 Mizrahi, T., Fabini, J., and A. Morton, "Guidelines for 1871 Defining Packet Timestamps", draft-ietf-ntp-packet- 1872 timestamps-09 (work in progress), March 2020. 1874 [I-D.ietf-nvo3-geneve] 1875 Gross, J., Ganga, I., and T. Sridhar, "Geneve: Generic 1876 Network Virtualization Encapsulation", draft-ietf- 1877 nvo3-geneve-16 (work in progress), March 2020. 1879 [I-D.ietf-nvo3-vxlan-gpe] 1880 Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol 1881 Extension for VXLAN", draft-ietf-nvo3-vxlan-gpe-09 (work 1882 in progress), December 2019. 1884 [I-D.ietf-sfc-proof-of-transit] 1885 Brockners, F., Bhandari, S., Mizrahi, T., Dara, S., and S. 1886 Youell, "Proof of Transit", draft-ietf-sfc-proof-of- 1887 transit-06 (work in progress), June 2020. 1889 [I-D.ioamteam-ippm-ioam-direct-export] 1890 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 1891 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 1892 OAM Direct Exporting", draft-ioamteam-ippm-ioam-direct- 1893 export-00 (work in progress), October 2019. 1895 [I-D.kitamura-ipv6-record-route] 1896 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 1897 Option Extension", draft-kitamura-ipv6-record-route-00 1898 (work in progress), November 2000. 1900 [I-D.spiegel-ippm-ioam-rawexport] 1901 Spiegel, M., Brockners, F., Bhandari, S., and R. 1902 Sivakolundu, "In-situ OAM raw data export with IPFIX", 1903 draft-spiegel-ippm-ioam-rawexport-03 (work in progress), 1904 March 2020. 1906 [RFC7276] Mizrahi, T., Sprecher, N., Bellagamba, E., and Y. 1907 Weingarten, "An Overview of Operations, Administration, 1908 and Maintenance (OAM) Tools", RFC 7276, 1909 DOI 10.17487/RFC7276, June 2014, 1910 . 1912 [RFC7384] Mizrahi, T., "Security Requirements of Time Protocols in 1913 Packet Switched Networks", RFC 7384, DOI 10.17487/RFC7384, 1914 October 2014, . 1916 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 1917 Chaining (SFC) Architecture", RFC 7665, 1918 DOI 10.17487/RFC7665, October 2015, 1919 . 1921 [RFC7799] Morton, A., "Active and Passive Metrics and Methods (with 1922 Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799, 1923 May 2016, . 1925 [RFC7820] Mizrahi, T., "UDP Checksum Complement in the One-Way 1926 Active Measurement Protocol (OWAMP) and Two-Way Active 1927 Measurement Protocol (TWAMP)", RFC 7820, 1928 DOI 10.17487/RFC7820, March 2016, 1929 . 1931 [RFC7821] Mizrahi, T., "UDP Checksum Complement in the Network Time 1932 Protocol (NTP)", RFC 7821, DOI 10.17487/RFC7821, March 1933 2016, . 1935 [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., 1936 "Network Service Header (NSH)", RFC 8300, 1937 DOI 10.17487/RFC8300, January 2018, 1938 . 1940 [SSS] Wikipedia, "Shamir's Secret Sharing", 1941 . 1943 Contributors' Addresses 1945 Carlos Pignataro 1946 Cisco Systems, Inc. 1947 7200-11 Kit Creek Road 1948 Research Triangle Park, NC 27709 1949 United States 1951 Email: cpignata@cisco.com 1953 Mickey Spiegel 1954 Barefoot Networks, an Intel company 1955 4750 Patrick Henry Drive 1956 Santa Clara, CA 95054 1957 US 1959 Email: mickey.spiegel@intel.com 1961 Barak Gafni 1962 Mellanox Technologies, Inc. 1963 350 Oakmead Parkway, Suite 100 1964 Sunnyvale, CA 94085 1965 U.S.A. 1967 Email: gbarak@mellanox.com 1969 Jennifer Lemon 1970 Broadcom 1971 270 Innovation Drive 1972 San Jose, CA 95134 1973 US 1975 Email: jennifer.lemon@broadcom.com 1977 Hannes Gredler 1978 RtBrick Inc. 1980 Email: hannes@rtbrick.com 1982 John Leddy 1983 United States 1985 Email: john@leddy.net 1987 Stephen Youell 1988 JP Morgan Chase 1989 25 Bank Street 1990 London E14 5JP 1991 United Kingdom 1993 Email: stephen.youell@jpmorgan.com 1995 David Mozes 1997 Email: mosesster@gmail.com 1999 Petr Lapukhov 2000 Facebook 2001 1 Hacker Way 2002 Menlo Park, CA 94025 2003 US 2004 Email: petr@fb.com 2006 Remy Chang 2007 Barefoot Networks 2008 4750 Patrick Henry Drive 2009 Santa Clara, CA 95054 2010 US 2012 Email: remy@barefootnetworks.com 2014 Daniel Bernier 2015 Bell Canada 2016 Canada 2018 Email: daniel.bernier@bell.ca 2020 Authors' Addresses 2022 Frank Brockners (editor) 2023 Cisco Systems, Inc. 2024 Hansaallee 249, 3rd Floor 2025 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 2026 Germany 2028 Email: fbrockne@cisco.com 2030 Shwetha Bhandari (editor) 2031 Cisco Systems, Inc. 2032 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 2033 Bangalore, KARNATAKA 560 087 2034 India 2036 Email: shwethab@cisco.com 2038 Tal Mizrahi (editor) 2039 Huawei 2040 8-2 Matam 2041 Haifa 3190501 2042 Israel 2044 Email: tal.mizrahi.phd@gmail.com