idnits 2.17.00 (12 Aug 2021) /tmp/idnits16796/draft-brockners-inband-oam-data-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (October 30, 2016) is 2029 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- -- Looks like a reference, but probably isn't: '0' on line 465 -- Looks like a reference, but probably isn't: '1' on line 379 == Outdated reference: A later version (-03) exists of draft-brockners-inband-oam-requirements-01 == Outdated reference: A later version (-05) exists of draft-brockners-inband-oam-transport-01 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group F. Brockners 3 Internet-Draft S. Bhandari 4 Intended status: Experimental C. Pignataro 5 Expires: May 3, 2017 Cisco 6 H. Gredler 7 RtBrick Inc. 8 J. Leddy 9 Comcast 10 S. Youell 11 JMPC 12 T. Mizrahi 13 Marvell 14 D. Mozes 15 Mellanox Technologies Ltd. 16 P. Lapukhov 17 Facebook 18 R. Chang 19 Barefoot Networks 20 October 30, 2016 22 Data Formats for In-situ OAM 23 draft-brockners-inband-oam-data-02 25 Abstract 27 In-situ Operations, Administration, and Maintenance (OAM) records 28 operational and telemetry information in the packet while the packet 29 traverses a path between two points in the network. This document 30 discusses the data types and data formats for in-situ OAM data 31 records. In-situ OAM data records can be embedded into a variety of 32 transports such as NSH, Segment Routing, VXLAN-GPE, native IPv6 (via 33 extension header), or IPv4. In-situ OAM is to complement current 34 out-of-band OAM mechanisms based on ICMP or other types of probe 35 packets. 37 Status of This Memo 39 This Internet-Draft is submitted in full conformance with the 40 provisions of BCP 78 and BCP 79. 42 Internet-Drafts are working documents of the Internet Engineering 43 Task Force (IETF). Note that other groups may also distribute 44 working documents as Internet-Drafts. The list of current Internet- 45 Drafts is at http://datatracker.ietf.org/drafts/current/. 47 Internet-Drafts are draft documents valid for a maximum of six months 48 and may be updated, replaced, or obsoleted by other documents at any 49 time. It is inappropriate to use Internet-Drafts as reference 50 material or to cite them other than as "work in progress." 52 This Internet-Draft will expire on May 3, 2017. 54 Copyright Notice 56 Copyright (c) 2016 IETF Trust and the persons identified as the 57 document authors. All rights reserved. 59 This document is subject to BCP 78 and the IETF Trust's Legal 60 Provisions Relating to IETF Documents 61 (http://trustee.ietf.org/license-info) in effect on the date of 62 publication of this document. Please review these documents 63 carefully, as they describe your rights and restrictions with respect 64 to this document. Code Components extracted from this document must 65 include Simplified BSD License text as described in Section 4.e of 66 the Trust Legal Provisions and are provided without warranty as 67 described in the Simplified BSD License. 69 Table of Contents 71 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 72 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 3 73 3. In-situ OAM Data Types and Data Format . . . . . . . . . . . 4 74 3.1. In-situ OAM Tracing Options . . . . . . . . . . . . . . . 4 75 3.1.1. Pre-allocated Trace Option . . . . . . . . . . . . . 6 76 3.1.2. Incremental Trace Option . . . . . . . . . . . . . . 9 77 3.1.3. In-situ OAM node data element format . . . . . . . . 11 78 3.1.4. Examples of In-situ OAM node data . . . . . . . . . . 14 79 3.2. In-situ OAM Proof of Transit Option . . . . . . . . . . . 16 80 3.3. In-situ OAM Edge-to-Edge Option . . . . . . . . . . . . . 18 81 4. In-situ OAM Data Export . . . . . . . . . . . . . . . . . . . 18 82 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19 83 6. Manageability Considerations . . . . . . . . . . . . . . . . 19 84 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 85 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 86 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 87 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 88 9.2. Informative References . . . . . . . . . . . . . . . . . 20 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 20 91 1. Introduction 93 This document defines data record types for "in-situ" Operations, 94 Administration, and Maintenance (OAM). In-situ OAM records OAM 95 information within the packet while the packet traverses a particular 96 network domain. The term "in-situ" refers to the fact that the OAM 97 data is added to the data packets rather than is being sent within 98 packets specifically dedicated to OAM. A discussion of the 99 motivation and requirements for in-situ OAM can be found in 100 [I-D.brockners-inband-oam-requirements]. In-situ OAM is to 101 complement "out-of-band" or "active" mechanisms such as ping or 102 traceroute, or more recent active probing mechanisms as described in 103 [I-D.lapukhov-dataplane-probe]. In-situ OAM mechanisms can be 104 leveraged where current out-of-band mechanisms do not apply or do not 105 offer the desired results, such as proving that a certain set of 106 traffic takes a pre-defined path, SLA verification for the live data 107 traffic, detailed statistics on traffic distribution paths in 108 networks that distribute traffic across multiple paths, or scenarios 109 where probe traffic is potentially handled differently from regular 110 data traffic by the network devices. 112 This document defines the data types and data formats for in-situ OAM 113 data records. The in-situ OAM data records can be transported by a 114 variety of transport protocols, including NSH, Segment Routing, 115 VXLAN-GPE, IPv6, IPv4. Encapsulation details for these different 116 transport protocols are outside the scope of this document. 118 2. Conventions 120 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 121 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 122 document are to be interpreted as described in [RFC2119]. 124 Abbreviations used in this document: 126 MTU: Maximum Transmit Unit 128 NSH: Network Service Header 130 OAM: Operations, Administration, and Maintenance 132 SFC: Service Function Chain 134 SID: Segment Identifier 136 SR: Segment Routing 138 TLV: Type-Length-Value 140 VXLAN-GPE: Virtual eXtensible Local Area Network, Generic Protocol 141 Extension 143 3. In-situ OAM Data Types and Data Format 145 This section defines in-situ OAM data types and data formats of the 146 data records required for in-situ OAM. The different uses of in-situ 147 OAM require the definition of different types of data. The in-situ 148 OAM data format for the data being carried corresponds to the three 149 main categories of in-situ OAM data defined in 150 [I-D.brockners-inband-oam-requirements], which are : edge-to-edge, 151 per node, and for selected nodes only. 153 Transport options for in-situ OAM data are found in 154 [I-D.brockners-inband-oam-transport]. In-situ OAM data is defined as 155 options in Type-Length-Value (TLV) format. The TLV format for each 156 of the three different types of in-situ OAM data is defined in this 157 document. 159 In-situ OAM is expected to be deployed in a specific domain rather 160 than on the overall Internet. The part of the network which employs 161 in situ OAM is referred to as the "in-situ OAM-domain". In-situ OAM 162 data is added to a packet upon entering the in-situ OAM-domain and is 163 removed from the packet when exiting the domain. Within the in-situ 164 OAM-domain, the in-situ OAM data may be updated by network nodes that 165 the packet traverses. The device which adds in-situ OAM data 166 container to the packet to capture in-situ OAM data is called the 167 "in-situ OAM encapsulating node", whereas the device which removes 168 the in-situ OAM data container is referred to as the "in-situ OAM 169 decapsulating node". Nodes within the domain which are aware of in- 170 situ OAM data and read and/or write or process the in-situ OAM data 171 are called "in-situ OAM transit nodes". Note that not every node in 172 an in-situ OAM domain needs to be an in-situ OAM transit node. For 173 example, a Segment Routing deployment might require the segment 174 routing path to be verified. In that case, only the SR nodes would 175 also be in-situ OAM transit nodes rather than all nodes. 177 3.1. In-situ OAM Tracing Options 179 "In-situ OAM tracing data" is expected to be collected at every node 180 that a packet traverses, i.e., in a typical deployment all nodes in 181 an in-situ OAM-domain would participate in in-situ OAM and thus be 182 in-situ OAM transit nodes, in-situ OAM encapsulating or in-situ OAM 183 decapsulating nodes. The maximum network diameter of the in-situ OAM 184 domain is assumed to be known. 186 To optimize hardware and software implementations tracing is defined 187 as two separate options. Any deployment MAY choose to configure and 188 support one or both of the following options. An implementation of 189 the transport protocol that carries these in-situ OAM data MAY choose 190 to support only one of the options. In the event that both options 191 are utilized at the same time, the Incremental Trace Option MUST be 192 placed before the Pre-allocated Trace Option. 194 Pre-allocated Trace Option: This trace option is defined as a 195 container of node-data elements with pre-allocated space for each 196 node to populate its information. This option is useful for 197 software implementations where it is efficient to allocate the 198 space once and index into the array to populate the data during 199 transit. The in-situ OAM encapsulating node allocates the option 200 header and sets the fields in the option header. The in situ OAM 201 encapsulating node allocates an array which is to store 202 operational data retrieved from every node while the packet 203 traverses the domain. In-situ OAM transit nodes update the 204 content of the array. A pointer which is part of the in-situ OAM 205 trace data points to the next empty slot in the array, which is 206 where the next in-situ OAM transit node fills in its data. 208 Incremental Trace Option: This trace options is defined as a 209 container of node-data elements where each node allocates and 210 pushes its node data immediately following the option header. The 211 number of node-data recorded and maximum number of node data that 212 can be recorded are written into the option header. This format 213 of trace recording is useful for some of the hardware 214 implementations as this eliminates the need for the transit 215 network elements to read the full array in the option and allows 216 for arbitrarily long packets as the MTU allows. The in-situ OAM 217 encapsulating node allocates the option header. The in-situ OAM 218 encapsulating node based on operational state and configuration 219 sets the fields in the header to control how large the node data 220 list can grow. In-situ OAM transit nodes pushes its node data to 221 the node data list and increments the number of node data records 222 in the header. 224 Every node data entry is to hold information for a particular in situ 225 OAM transit node that is traversed by a packet. The in-situ OAM 226 decapsulating node removes the in-situ OAM data and process and/or 227 export the metadata. In-situ OAM data uses its own name-space for 228 information such as node identifier or interface identifier. This 229 allows for a domain-specific definition and interpretation. For 230 example: In one case an interface-id could point to a physical 231 interface (e.g., to understand which physical interface of an 232 aggregated link is used when receiving or transmitting a packet) 233 whereas in another case it could refer to a logical interface (e.g., 234 in case of tunnels). 236 The following in-situ OAM data is defined for in-situ OAM tracing: 238 o Identification of the in-situ OAM node. An in-situ OAM node 239 identifier can match to a device identifier or a particular 240 control point or subsystem within a device. 242 o Identification of the interface that a packet was received on. 244 o Identification of the interface that a packet was sent out on. 246 o Time of day when the packet was processed by the node. Different 247 definitions of processing time are feasible and expected, though 248 it is important that all devices of an in-situ OAM domain follow 249 the same definition. 251 o Generic data: Format-free information where syntax and semantic of 252 the information is defined by the operator in a specific 253 deployment. For a specific deployment, all in-situ OAM nodes 254 should interpret the generic data the same way. Examples for 255 generic in-situ OAM data include geo-location information 256 (location of the node at the time the packet was processed), 257 buffer queue fill level or cache fill level at the time the packet 258 was processed, or even a battery charge level. 260 o A mechanism to detect whether in-situ OAM trace data was added at 261 every hop or whether certain hops in the domain weren't in-situ 262 OAM transit nodes. 264 The "Node data List" array in the packet is populated iteratively as 265 the packet traverses the network, starting with the last entry of the 266 array, i.e., "Node data List [n]" is the first entry to be populated, 267 "Node data List [n-1]" is the second one, etc. 269 3.1.1. Pre-allocated Trace Option 270 In-situ OAM Pre-allocated Trace Option: 272 0 1 2 3 273 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 274 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 275 | Option Type | Opt Data Len | Elements-left | Reserved1 | 276 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 277 | OAM-Trace-Type | Reserved2 | 278 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 279 | | | 280 | Node data List [0] | | 281 | | | 282 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ D 283 | | a 284 | Node data List [1] | t 285 | | a 286 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 287 ~ ... ~ S 288 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ p 289 | | a 290 | Node data List [n-1] | c 291 | | e 292 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 293 | | | 294 | Node data List [n] | | 295 | | | 296 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 298 Option Type: 8-bit identifier of the type of option. Option number 299 is defined based on the encapsulation protocol. 301 Opt Data Len: 8-bit unsigned integer. Length of the Option Data 302 field of this option, in octets. 304 Elements-left: 8-bit unsigned integer. A pointer that indicates the 305 next data recording point in the data space of the packet in 306 octets. It is the index into the "Node data List" array shown 307 above. 309 Reserved1: 8-bit unused field in this document and MUST be set to 310 zero. 312 OAM-trace-type: 16-bit identifier of a particular trace element 313 variant. 315 The trace type value is a bit field. The following bit fields are 316 defined in this document, with details on each field described in 317 the Section 3.1.3. The order of packing the trace data in each 318 Node-data element follows the bit order for setting each trace 319 data element. 321 Bit 0 When set indicates presence of Hop_Lim and node_id in the 322 Node data. 324 Bit 1 When set indicates presence of ingress_if_id and 325 egress_if_id in the Node data. 327 Bit 2 When set indicates presence of timestamp seconds in the 328 Node data 330 Bit 3 When set indicates presence of timestamp nanoseconds in 331 the Node data. 333 Bit 4 When set indicates presence of transit delay in the Node 334 data. 336 Bit 5 When set indicates presence of app_data in the Node data. 338 Bit 6 When set indicates presence of queue depth in the Node 339 data. 341 Bit 7 - 14 Undefined in this document. 343 Bit 15 When set indicates wide data format for all the node data 344 elements that are present. When unset indicates short 345 data format for all the node data elements that are 346 present. 348 Section 3.1.4 describes the format of a number of trace types. 350 Reserved2: 16-bit unused field in this document and MUST be set to 351 zero. 353 Node data List [n]: Variable-length field. The format of which is 354 determined by the OAM Type representing the n-th Node data in the 355 Node data List. The Node data List is encoded starting from the 356 last Node data of the path. The first element of the node data 357 list (Node data List [0]) contains the last node of the path while 358 the last node data of the Node data List (Node data List[n]) 359 contains the first Node data of the path traced. The index 360 contained in "Elements-left" identifies the current active Node 361 data to be populated. 363 3.1.2. Incremental Trace Option 365 In-situ OAM Incremental Trace Option: 367 0 1 2 3 368 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 369 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 370 | Option Type | Opt Data Len | Maximum Length| Flags | 371 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 372 | OAM Trace Type | Reserved | 373 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 374 | | 375 | Node data List [0] | 376 | | 377 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 378 | | 379 | Node data List [1] | 380 | | 381 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 382 ~ ... ~ 383 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 384 | | 385 | Node data List [n-1] | 386 | | 387 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 388 | | 389 | Node data List [n] | 390 | | 391 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 393 Option Type: 8-bit identifier of the type of option. Option number 394 is defined based on the encapsulation protocol. 396 Opt Data Len: 8-bit unsigned integer. Length of the Option Data 397 field of this option, in octets. 399 Maximum Length: 8-bit unsigned integer. This field specifies the 400 maximum length of the node data list in octets. Given that the 401 sender knows the minimum path MTU, the sender can set the maximum 402 of node data bytes allowed before exceeding the MTU. Thus, a 403 simple comparison between "Opt data Len" and "Max Length" allows 404 to decide whether or not data could be added. 406 Flags 8-bit field. Following flags are defined: 408 1 "Overflow" (O-bit) (least significant bit). This bit is set by 409 the network element if the number of records on the packet is 410 at the maximum limit as specified by the packet: i.e., the 411 packet is already "full" of telemetry information. This is 412 useful for transit nodes to ignore further processing of the 413 option. If inserting a new node data record would cause "Opt 414 Data Len" to exceed "Max Length", no record is added and the 415 overflow "O-bit" must be set to "1" in the header. 417 OAM-trace-type: 16-bit identifier of a particular trace element 418 variant. 420 The trace type value is a bit field. The following bit fields are 421 defined in this document, with details on each field described in 422 the Section 3.1.3. The order of packing the trace data in each 423 Node-data element follows the bit order for setting each trace 424 data element. 426 Bit 0 When set indicates presence of Hop_Lim and node_id in the 427 Node data. 429 Bit 1 When set indicates presence of ingress_if_id and 430 egress_if_id in the Node data. 432 Bit 2 When set indicates presence of timestamp seconds in the 433 Node data 435 Bit 3 When set indicates presence of timestamp nanoseconds in 436 the Node data. 438 Bit 4 When set indicates presence of transit delay in the Node 439 data. 441 Bit 5 When set indicates presence of app_data in the Node data. 443 Bit 6 When set indicates presence of queue depth in the Node 444 data. 446 Bit 7 When set indicates presence of variable length Opaque 447 State Snapshot field. 449 Bit 8-14 Undefined in this draft. 451 Bit 15 When set indicates wide data format for all the node data 452 elements that are present. When unset indicates short 453 data format for all the node data elements that are 454 present. 456 Section 3.1.4 describes the format of a number of trace types. 458 Reserved: 2 bytes unused field in this document and MUST be set to 459 zero. 461 Node data List [n]: Variable-length field. The format of which is 462 determined by the OAM Type representing the n-th Node data in the 463 Node data List. The Node data List is encoded starting from the 464 last Node data of the path. The first element of the node data 465 list (Node data List [0]) contains the last node of the path while 466 the last node data of the Node data List (Node data List[n]) 467 contains the first Node data of the path traced. 469 3.1.3. In-situ OAM node data element format 471 The in-situ OAM node data elements are defined in 2 formats - short 472 and wide that is selected by bit 15 in the OAM-trace-type field. All 473 the data records MUST be 4-byte aligned in both the formats. 475 Data type and format for each of the data records in short format is 476 shown below: 478 Hop_Lim and node_id: 4-octet field defined as follows: 480 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 481 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 482 | Hop_Lim | node_id | 483 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 485 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 486 value in the packet at the node that records this data. Hop 487 Limit information is used to identify the location of the node 488 in the communication path. This is copied from the lower layer 489 for e.g. TTL value in IPv4 header or hop limit field from IPv6 490 header of the packet. 492 node_id: 3-octet unsigned integer. Node identifier field to 493 uniquely identify a node within in-situ OAM domain. The 494 procedure to allocate, manage and map the node_ids is beyond 495 the scope of this document. 497 ingress_if_id and egress_if_id: 4-octet field defined as follows: 499 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 500 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 501 | ingress_if_id | egress_if_id | 502 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 503 ingress_if_id: 2-octet unsigned integer. Interface identifier to 504 record the ingress interface the packet was received on. 506 egress_if_id: 2-octet unsigned integer. Interface identifier to 507 record the egress interface the packet is forwarded out of. 509 timestamp seconds: 4-octet unsigned integer. Absolute timestamp in 510 seconds that specifies the time at which the packet was received 511 by the node. The format of this field is identical to the most 512 significant 32 bits of 64 least significant bits of the 513 [IEEE1588v2]. This truncated format consists of a 32-bit seconds 514 field. As defined in [IEEE1588v2], the timestamp specifies the 515 number of seconds elapsed since 1 January 1970 00:00:00 according 516 to the International Atomic Time (TAI). 518 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 519 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 520 | timestamp seconds | 521 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 523 timestamp nanoseconds: 4-octet unsigned integer in the range 0 to 524 10^9-1. This timestamp specifies the fractional part of the wall 525 clock time at which the packet was received by the node in units 526 of nanoseconds. It is nanoseconds that are recorded in 32 least 527 significant bits of absolute time as per [IEEE1588v2]. This 528 fields allows for delay computation between any two nodes in the 529 network when the nodes are time synchronized. 531 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 532 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 533 | timestamp nanoseconds | 534 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 536 transit delay: 4-octet unsigned integer in the range 0 to 2^30-1. 537 It is the time in nanoseconds packet spent in transiting a node. 538 This can serve to give an indication of queuing delay at the node. 539 If the transit delay exceeds 2^30-1 nanoseconds then the top bit 540 'O' is set to indicate overflow. 542 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 543 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 544 |O| transit delay | 545 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 547 app_data: 4-octet placeholder which can be used by the node to add 548 application specific data 550 queue depth: 4-octet unsigned integer field. This field indicates 551 the length of the egress interface queue of the interface where 552 the packet is forwarded out of. 554 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 555 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 556 | queue depth | 557 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 559 Data type and format for each of the elements in wide format follows 560 when Most Significant Bit (MSB) i.e., bit 15 of OAM-Trace-Type is 561 set: 563 Hop_Lim and node_id: 8-octet field defined as follows: 565 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 566 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 567 | Hop_Lim | node_id ~ 568 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 569 ~ node_id (contd) | 570 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 572 Hop_Lim: 1-octet unsigned integer. It is set to the Hop Limit 573 value in the packet at the node that records this data. Hop 574 Limit information is used to identify the location of the node 575 in the communication path. This is copied from the lower layer 576 for e.g. TTL value in IPv4 header or hop limit field from IPv6 577 header of the packet. 579 node_id: 7-octet unsigned integer. Node identifier field to 580 uniquely identify a node within in-situ OAM domain. The 581 procedure to allocate, manage and map the node_ids is beyond 582 the scope of this document. 584 ingress_if_id and egress_if_id: 8-octet field defined as follows: 586 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 587 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 588 | ingress_if_id | 589 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 590 | egress_if_id | 591 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 593 ingress_if_id: 4-octet unsigned integer. Interface identifier to 594 record the ingress interface the packet was received on. 596 egress_if_id: 4-octet unsigned integer. Interface identifier to 597 record the egress interface the packet is forwarded out of. 599 app_data: 8-octet placeholder which can be used by the node to add 600 application specific data. 602 Opaque State Snapshot: Variable length field. It allows the network 603 element to store arbitrary state in the node data record, without 604 a pre-defined schema. The schema needs to made known to the 605 analyzer by some out-of-band means. The 24-bit "Schema Id" field 606 in the record is supposed to let the analyzer know which 607 particular schema to use, and it is expected to be configured on 608 the network element by the operator. This ID is expected to be 609 configured on the device by the network operator. 611 0 1 2 3 612 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 613 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 614 | Length | Schema ID | 615 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 616 | | 617 | | 618 | Opaque data | 619 ~ ~ 620 . . 621 . . 622 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 624 Length: 1-octet unsigned integer. It is the length of the Opaque 625 data field that follows Schema Id. It MUST always be a 626 multiple of 4. 628 Schema ID: 3-octet unsigned integer identifying the schema of 629 Opaque data. 631 Opaque data: Variable length field. This field is interpreted as 632 specified by the schema identified by the Schema ID. 634 The fields - timestamp seconds, timestamp nanoseconds and transit 635 delay have the same format as defined in short format. 637 3.1.4. Examples of In-situ OAM node data 639 An entry in the "Node data List" array can have different formats, 640 following the needs of the deployment. Some deployments might only 641 be interested in recording the node identifiers, whereas others might 642 be interested in recording node identifier and timestamp. The 643 section defines different formats that an entry in "Node data List" 644 can take. 646 0x002B: In-situ OAM-trace-type is 0x2B then the format of node data 647 is: 649 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 650 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 651 | Hop_Lim | node_id | 652 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 653 | ingress_if_id | egress_if_id | 654 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 655 | timestamp nanoseconds | 656 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 657 | app_data | 658 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 660 0x0003: In-situ OAM-trace-type is 0x0003 then the format is: 662 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 663 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 664 | Hop_Lim | node_id | 665 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 666 | ingress_if_id | egress_if_id | 667 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 669 0x0009: In-situ OAM-trace-type is 0x0009 then the format is: 671 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 672 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 673 | Hop_Lim | node_id | 674 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 675 | timestamp nanoseconds | 676 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 678 0x0021: In-situ OAM-trace-type is 0x0021 then the format is: 680 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 681 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 682 | Hop_Lim | node_id | 683 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 684 | app_data | 685 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 687 0x0029: In-situ OAM-trace-type is 0x0029 then the format is: 689 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 690 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 691 | Hop_Lim | node_id | 692 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 693 | timestamp nanoseconds | 694 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 695 | app_data | 696 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 698 0x104D: In-situ OAM-trace-type is 0x104D then the format is: 700 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 701 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 702 | Hop_Lim | node_id | 703 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 704 | node_id(contd) | 705 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 706 | timestamp seconds | 707 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 708 | timestamp nanoseconds | 709 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 710 | Length | Schema Id | 711 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 712 | | 713 | | 714 | Opaque data | 715 ~ ~ 716 . . 717 . . 718 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 720 3.2. In-situ OAM Proof of Transit Option 722 In-situ OAM Proof of Transit data is to support the path or service 723 function chain [RFC7665] verification use cases. Proof-of-transit 724 uses methods like nested hashing or nested encryption of the in-situ 725 OAM data or mechanisms such as Shamir's Secret Sharing Schema (SSSS). 726 While details on how the in-situ OAM data for the proof of transit 727 option is processed at in-situ OAM encapsulating, decapsulating and 728 transit nodes are outside the scope of the document, all of these 729 approaches share the need to uniquely identify a packet as well as 730 iteratively operate on a set of information that is handed from node 731 to node. Correspondingly, two pieces of information are added as in- 732 situ OAM data to the packet: 734 o Random: Unique identifier for the packet (e.g., 64-bits allow for 735 the unique identification of 2^64 packets). 737 o Cumulative: Information which is handed from node to node and 738 updated by every node according to a verification algorithm. 740 In-situ OAM Proof of Transit option: 742 0 1 2 3 743 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 744 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 745 | Option Type | Opt Data Len | POT type = 0 |P| reserved | 746 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 747 | Random | | 748 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P 749 | Random(contd) | O 750 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ T 751 | Cumulative | | 752 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 753 | Cumulative (contd) | | 754 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+ 756 Option Type: 8-bit identifier of the type of option. 758 Opt Data Len: 8-bit unsigned integer. Length of the Option Data 759 field of this option, in octets. 761 POT Type: 8-bit identifier of a particular POT variant that dictates 762 the POT data that is included. This document defines POT Type 0: 764 0: POT data is a 16 Octet field as described below. 766 Profile to use (P): 1-bit. Indicates which POT-profile is used to 767 generate the Cumulative. Any node participating in POT will have 768 a maximum of 2 profiles configured that drive the computation of 769 cumulative. The two profiles are numbered 0, 1. This bit conveys 770 whether profile 0 or profile 1 is used to compute the Cumulative. 772 Reserved: 7-bit. Reserved for future use. 774 Random: 64-bit Per packet Random number. 776 Cumulative: 64-bit Cumulative that is updated at specific nodes by 777 processing per packet Random number field and configured 778 parameters. 780 Note: Larger or smaller sizes of "Random" and "Cumulative" data are 781 feasible and could be required for certain deployments (e.g. in case 782 of space constraints in the transport protocol used). Future 783 versions of this document will address different sizes of data for 784 "proof of transit". 786 3.3. In-situ OAM Edge-to-Edge Option 788 The in-situ OAM Edge-to-Edge Option is to carry data which is to be 789 interpreted only by the in-situ OAM encapsulating and in-situ OAM 790 decapsulating node, but not by in-situ OAM transit nodes. 792 Currently only sequence numbers use the in-situ OAM Edge-to-Edge 793 option. In order to detect packet loss, packet reordering, or packet 794 duplication in an in-situ OAM-domain, sequence numbers can be added 795 to packets of a particular tube (see 796 [I-D.hildebrand-spud-prototype]). Each tube leverages a dedicated 797 namespace for its sequence numbers. 799 0 1 2 3 800 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 801 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 802 | Option Type | Opt Data Len | OAM-E2E-Type | reserved | 803 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 804 | E2E Option data format determined by iOAM-E2E-Type ~ 805 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 807 Option Type: 8-bit identifier of the type of option. 809 Opt Data Len: 8-bit unsigned integer. Length of the Option Data 810 field of this option, in octets. 812 OAM-E2E-Type: 8-bit identifier of a particular in situ OAM E2E 813 variant. 815 0: E2E option data is a 64-bit sequence number added to a 816 specific tube which is used to identify packet loss and 817 reordering for that tube. 819 Reserved: 8-bit. (Reserved Octet) Reserved octet for future use. 821 4. In-situ OAM Data Export 823 In-situ OAM nodes collect information for packets traversing a domain 824 that supports in-situ OAM. The device at the domain edge (which 825 could also be an end-host) which receives a packet with in-situ OAM 826 information chooses how to process the in-situ OAM data collected 827 within the packet. This decapsulating node can simply discard the 828 information collected, can process the information further, or export 829 the information using e.g., IPFIX. 831 The discussion of in-situ OAM data processing and export is left for 832 a future version of this document. 834 5. IANA Considerations 836 IANA considerations will be added in a future version of this 837 document. 839 6. Manageability Considerations 841 Manageability considerations will be addressed in a later version of 842 this document.. 844 7. Security Considerations 846 Security considerations will be addressed in a later version of this 847 document. For a discussion of security requirements of in-situ OAM, 848 please refer to [I-D.brockners-inband-oam-requirements]. 850 8. Acknowledgements 852 The authors would like to thank Eric Vyncke, Nalini Elkins, Srihari 853 Raghavan, Ranganathan T S, Karthik Babu Harichandra Babu, Akshaya 854 Nadahalli, LJ Wobker, Erik Nordmark, and Andrew Yourtchenko for the 855 comments and advice. This document leverages and builds on top of 856 several concepts described in [I-D.kitamura-ipv6-record-route]. The 857 authors would like to acknowledge the work done by the author Hiroshi 858 Kitamura and people involved in writing it. 860 9. References 862 9.1. Normative References 864 [I-D.brockners-inband-oam-requirements] 865 Brockners, F., Bhandari, S., Dara, S., Pignataro, C., 866 Gredler, H., Leddy, J., and S. Youell, "Requirements for 867 In-band OAM", draft-brockners-inband-oam-requirements-01 868 (work in progress), July 2016. 870 [IEEE1588v2] 871 Institute of Electrical and Electronics Engineers, 872 "1588-2008 - IEEE Standard for a Precision Clock 873 Synchronization Protocol for Networked Measurement and 874 Control Systems", IEEE Std 1588-2008, 2008, 875 . 878 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 879 Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ 880 RFC2119, March 1997, 881 . 883 9.2. Informative References 885 [I-D.brockners-inband-oam-transport] 886 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 887 Leddy, J., and S. Youell, "Encapsulations for In-band OAM 888 Data", draft-brockners-inband-oam-transport-01 (work in 889 progress), July 2016. 891 [I-D.hildebrand-spud-prototype] 892 Hildebrand, J. and B. Trammell, "Substrate Protocol for 893 User Datagrams (SPUD) Prototype", draft-hildebrand-spud- 894 prototype-03 (work in progress), March 2015. 896 [I-D.kitamura-ipv6-record-route] 897 Kitamura, H., "Record Route for IPv6 (PR6) Hop-by-Hop 898 Option Extension", draft-kitamura-ipv6-record-route-00 899 (work in progress), November 2000. 901 [I-D.lapukhov-dataplane-probe] 902 Lapukhov, P. and r. remy@barefootnetworks.com, "Data-plane 903 probe for in-band telemetry collection", draft-lapukhov- 904 dataplane-probe-01 (work in progress), June 2016. 906 [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function 907 Chaining (SFC) Architecture", RFC 7665, DOI 10.17487/ 908 RFC7665, October 2015, 909 . 911 Authors' Addresses 912 Frank Brockners 913 Cisco Systems, Inc. 914 Hansaallee 249, 3rd Floor 915 DUESSELDORF, NORDRHEIN-WESTFALEN 40549 916 Germany 918 Email: fbrockne@cisco.com 920 Shwetha Bhandari 921 Cisco Systems, Inc. 922 Cessna Business Park, Sarjapura Marathalli Outer Ring Road 923 Bangalore, KARNATAKA 560 087 924 India 926 Email: shwethab@cisco.com 928 Carlos Pignataro 929 Cisco Systems, Inc. 930 7200-11 Kit Creek Road 931 Research Triangle Park, NC 27709 932 United States 934 Email: cpignata@cisco.com 936 Hannes Gredler 937 RtBrick Inc. 939 Email: hannes@rtbrick.com 941 John Leddy 942 Comcast 944 Email: John_Leddy@cable.comcast.com 946 Stephen Youell 947 JP Morgan Chase 948 25 Bank Street 949 London E14 5JP 950 United Kingdom 952 Email: stephen.youell@jpmorgan.com 953 Tal Mizrahi 954 Marvell 955 6 Hamada St. 956 Yokneam 20692 957 Israel 959 Email: talmi@marvell.com 961 David Mozes 962 Mellanox Technologies Ltd. 964 Email: davidm@mellanox.com 966 Petr Lapukhov 967 Facebook 968 1 Hacker Way 969 Menlo Park, CA 94025 970 US 972 Email: petr@fb.com 974 Remy Chang 975 Barefoot Networks 976 2185 Park Boulevard 977 Palo Alto, CA 94306 978 US