idnits 2.17.00 (12 Aug 2021) /tmp/idnits54145/draft-spiegel-ippm-ioam-rawexport-05.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (12 July 2021) is 313 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-17) exists of draft-ietf-ippm-ioam-data-14 == Outdated reference: A later version (-07) exists of draft-ietf-ippm-ioam-direct-export-05 == Outdated reference: A later version (-07) exists of draft-ietf-ippm-ioam-ipv6-options-05 == Outdated reference: A later version (-10) exists of draft-ietf-sfc-ioam-nsh-05 == Outdated reference: A later version (-05) exists of draft-weis-ippm-ioam-eth-04 -- Obsolete informational reference (is this intentional?): RFC 5226 (Obsoleted by RFC 8126) Summary: 0 errors (**), 0 flaws (~~), 6 warnings (==), 2 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm M. Spiegel 3 Internet-Draft Barefoot Networks, an Intel company 4 Intended status: Informational F. Brockners 5 Expires: 13 January 2022 Cisco 6 S. Bhandari 7 Thoughtspot 8 R. Sivakolundu 9 Cisco 10 12 July 2021 12 In-situ OAM raw data export with IPFIX 13 draft-spiegel-ippm-ioam-rawexport-05 15 Abstract 17 In-situ Operations, Administration, and Maintenance (IOAM) records 18 operational and telemetry information in the packet while the packet 19 traverses a path between two points in the network. This document 20 discusses how In-situ Operations, Administration, and Maintenance 21 (IOAM) information can be exported in raw, i.e. uninterpreted, format 22 from network devices to systems, such as monitoring or analytics 23 systems using IPFIX. 25 Status of This Memo 27 This Internet-Draft is submitted in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF). Note that other groups may also distribute 32 working documents as Internet-Drafts. The list of current Internet- 33 Drafts is at https://datatracker.ietf.org/drafts/current/. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 This Internet-Draft will expire on 13 January 2022. 42 Copyright Notice 44 Copyright (c) 2021 IETF Trust and the persons identified as the 45 document authors. All rights reserved. 47 This document is subject to BCP 78 and the IETF Trust's Legal 48 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 49 license-info) in effect on the date of publication of this document. 50 Please review these documents carefully, as they describe your rights 51 and restrictions with respect to this document. Code Components 52 extracted from this document must include Simplified BSD License text 53 as described in Section 4.e of the Trust Legal Provisions and are 54 provided without warranty as described in the Simplified BSD License. 56 Table of Contents 58 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 59 1.1. Requirements . . . . . . . . . . . . . . . . . . . . . . 4 60 1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5 61 2. Conventions . . . . . . . . . . . . . . . . . . . . . . . . . 6 62 3. IPFIX for IOAM raw data export . . . . . . . . . . . . . . . 6 63 3.1. Key IPFIX information elements leveraged for IOAM raw data 64 export . . . . . . . . . . . . . . . . . . . . . . . . . 6 65 3.2. New IPFIX information elements leveraged for IOAM raw data 66 export . . . . . . . . . . . . . . . . . . . . . . . . . 7 67 3.2.1. ioamReportFlags . . . . . . . . . . . . . . . . . . . 7 68 3.2.2. ioamEncapsulationType . . . . . . . . . . . . . . . . 8 69 3.2.3. ioamPreallocatedTraceData . . . . . . . . . . . . . . 8 70 3.2.4. ioamIncrementalTraceData . . . . . . . . . . . . . . 9 71 3.2.5. ioamE2EData . . . . . . . . . . . . . . . . . . . . . 9 72 3.2.6. ioamPOTData . . . . . . . . . . . . . . . . . . . . . 10 73 3.2.7. ioamDirectExportData . . . . . . . . . . . . . . . . 10 74 3.2.8. ipHeaderPacketSectionWithPadding . . . . . . . . . . 11 75 3.2.9. ethernetFrameSection . . . . . . . . . . . . . . . . 12 76 4. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 13 77 4.1. Fixed Length IP Packet . . . . . . . . . . . . . . . . . 13 78 4.2. Variable Length IP Packet (length < 255) . . . . . . . . 13 79 4.3. Variable Length IP Packet (length > 255) . . . . . . . . 14 80 4.4. Variable Length ETHERNET Packet (length < 255) . . . . . 15 81 4.5. Variable Length IP Packet with Fixed Length IOAM 82 Incremental Trace Data . . . . . . . . . . . . . . . . . 16 83 4.6. Variable Length IP Packet with Variable Length IOAM 84 Incremental Trace Data . . . . . . . . . . . . . . . . . 17 85 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 86 6. Manageability Considerations . . . . . . . . . . . . . . . . 19 87 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19 88 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 89 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19 90 9.1. Normative References . . . . . . . . . . . . . . . . . . 19 91 9.2. Informative References . . . . . . . . . . . . . . . . . 20 92 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21 94 1. Introduction 96 In-situ Operations, Administration, and Maintenance (IOAM) records 97 operational and telemetry information in the packet while the packet 98 traverses a path between two points in the network. IOAM data fields 99 are defined in [I-D.ietf-ippm-ioam-data]. This document discusses 100 how In-situ Operations, Administration, and Maintenance (IOAM) 101 information can be exported in raw format, i.e. uninterpreted format, 102 from network devices to systems, such as monitoring or analytics 103 systems using IPFIX [RFC7011]. 105 "Raw export of IOAM data" refers to a mode of operation where a node 106 exports the IOAM data as it is received in the packet. The exporting 107 node neither interprets, aggregates nor reformats the IOAM data 108 before it is exported. Raw export of IOAM data is to support an 109 operational model where the processing and interpretation of IOAM 110 data is decoupled from the operation of encapsulating/updating/ 111 decapsulating IOAM data, which is also referred to as IOAM data-plane 112 operation. The figure below shows the separation of concerns for 113 IOAM export: Exporting IOAM data is performed by the "IOAM node" 114 which performs IOAM data-plane operation, whereas the interpretation 115 of IOAM data is performed by the IOAM data processing system. The 116 separation of concerns is to off-load interpretation, aggregation and 117 formatting of IOAM data from the node which performs data-plane 118 operations. In other words, a node which is focused on data-plane 119 operations, i.e. forwarding of packets and handling IOAM data will 120 not be tasked to also interpret the IOAM data, but can leave this 121 task to another system. Note that for scalability reasons, a single 122 IOAM node could choose to export IOAM data to several IOAM data 123 processing systems. 125 +-------------+ 126 | Monitoring/ | 127 | Analytics | 128 | system | 129 +-------------+ 130 ^ 131 | Processed/interpreted/aggregated 132 | IOAM data 133 | 134 +-------------+ 135 | IOAM data | 136 | processing | 137 | system | 138 +-------------+ 139 ^ 140 | Raw export of 141 | IOAM data 142 | 143 +------------+ 144 | | 145 ..--| IOAM node |--.. 146 | | 147 +------------+ 149 IOAM node: IOAM encapsulating, IOAM decapsulating or 150 IOAM transit node. 152 IOAM data processing system: System that receives raw IOAM data 153 and provides for formatting, aggregation and 154 interpretation of the IOAM data. 156 Monitoring/Analytics system: System that receives telemetry and 157 other operational information from a variety of 158 sources and provides for correlation and 159 interpretation of the data received. 161 Raw export of IOAM data is typically generated by network devices at 162 the edges of the network. Deployment and use-case dependent, such as 163 in case of direct export [I-D.ietf-ippm-ioam-direct-export] or in 164 cases where the operator is interested in dropped packets, raw export 165 of IOAM data may be generated by IOAM transit nodes. 167 1.1. Requirements 169 Requirements for raw export of IOAM data: 171 * Export all IOAM information contained in a packet. 173 * Export a specific IOAM data type - Incremental Trace type, 174 Preallocated Trace type, Proof of Transit type, Edge to Edge type, 175 Direct Export type. 177 * Export IOAM trace data associated with a packet, even if that data 178 was never included in a transmitted or received packet in the 179 network, for example in case of direct export. 181 * Support coalescing of the IOAM data from multiple packets into a 182 single raw export packet. 184 * Support export of additional parts of the packet, other than the 185 IOAM data as part of the raw export. This could be parts of the 186 packet header and/or parts of the packet payload. This additional 187 information provides context to the IOAM data (e.g. to be used for 188 flow identification) and is to enable the IOAM data processing 189 system to perform further analysis on the received data. 191 * Report the reason why IOAM data was exported. The "reason for 192 export" is to complement the IOAM data retrieved from the packet. 193 For example, if a packet was dropped by a node due to congestion, 194 it could be helpful to export the IOAM data of this dropped packet 195 along with an indication that the packet that the IOAM data 196 belongs to was dropped due to congestion. 198 1.2. Scope 200 This document discusses raw export of IOAM data using IPFIX. 202 The following is considered out of scope for this document: 204 * Protocols other than IPFIX for raw export of IOAM data. 206 * Interpretation or aggregation of IOAM data prior to exporting. 208 * Configuration of network devices so that they can determine when 209 to generate IOAM reports, and what information to include in those 210 reports. 212 * Events that trigger generation of IOAM reports. 214 * Selection of particular destinations within distributed telemetry 215 monitoring systems, to which IOAM reports will be sent. 217 * Export format for flow statistics or processed/interpreted/ 218 aggregated IOAM data. 220 2. Conventions 222 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 223 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 224 document are to be interpreted as described in [RFC2119]. 226 Abbreviations used in this document: 228 E2E: Edge to Edge 230 IOAM: In-situ Operations, Administration, and Maintenance 232 MTU: Maximum Transmit Unit 234 OAM: Operations, Administration, and Maintenance 236 POT: Proof of Transit 238 3. IPFIX for IOAM raw data export 240 IPFIX, being a generic export protocol, can export any Information 241 Elements as long as they are described in the information model. The 242 IPFIX protocol is well suited for and is defined as the protocol for 243 exporting packet samples in [RFC5476]. 245 IPFIX/PSAMP [RFC7011], [RFC5476] already define many of the 246 information elements needed for exporting sections of packets needed 247 for deriving context and raw IOAM data export. This document 248 specifies extensions of the IPFIX information model for meeting the 249 requirements in Section 1.1. 251 3.1. Key IPFIX information elements leveraged for IOAM raw data export 253 The existing IPFIX Information Elements that are required for IOAM 254 raw data export are listed here. Their details are available in 255 IANA's IPFIX registry [IANA-IPFIX]. 257 The existing IPFIX Information Elements used to carry the sections of 258 the packets including IOAM data within it are as follows: 260 313 - ipHeaderPacketSection 262 315 - dataLinkFrameSection 264 The following Information Elements will be used to provide context to 265 the ipHeaderPacketSection and dataLinkFrameSection as described in 266 [IANA-IPFIX]: 268 408 - dataLinkFrameType 270 409 - sectionOffset 272 410 - sectionExportedOctets 274 The following Information Element will be used to provide forwarding 275 status of the flow and any attached reasons. 277 89 - forwardingStatus 279 3.2. New IPFIX information elements leveraged for IOAM raw data export 281 IOAM data raw export using IPFIX requires a set of new information 282 elements which are described in this section. 284 3.2.1. ioamReportFlags 286 Description: 288 This Information Element describes properties associated with an IOAM 289 report. 291 The ioamReportFlags data type is an 8-bit field. The following bits 292 are defined here: 294 Bit 0 Dropped Association - Dropped packet of interest. 296 Bit 1 Congested Queue Association - Indicates the presence of 297 congestion on a monitored queue. 299 Bit 2 Tracked Flow Association - Matched a flow of interest. 301 Bit 3-7 Reserved 303 IANA is requested to create a new subregistry for IOAM Report Flags 304 and fill it with the initial list from the description. New 305 assignments for IOAM Encapsulation Types are administered by IANA 306 through Expert Review [RFC5226] i.e., review by one of a group of 307 experts designated by an IETF Area Director. 309 Abstract Data Type: unsigned8 311 Data Type Semantics: flags 313 ElementId: TBD1 315 Status: current 317 3.2.2. ioamEncapsulationType 319 Description: 321 This Information Element specifies the type of encapsulation to 322 interpret ioamPreallocatedTraceData, ioamIncrementalTraceData, 323 ioamE2EData, ioamPOTData, ioamDirectExportData. 325 The following ioamEncapsulationType values are defined here: 327 0 None : IOAM data follows format defined in 328 [I-D.ietf-ippm-ioam-data] 330 1 GRE : IOAM data follows format defined in [I-D.weis-ippm-ioam-eth] 332 2 IPv6 : IOAM data follows format defined in 333 [I-D.ietf-ippm-ioam-ipv6-options] 335 3 VXLAN-GPE : IOAM data follows format defined in 336 [I-D.brockners-ippm-ioam-vxlan-gpe] 338 4 GENEVE Option: IOAM data follows format defined in 339 [I-D.brockners-ippm-ioam-geneve] 341 5 GENEVE Next Protocol: IOAM data follows format defined in 342 [I-D.weis-ippm-ioam-eth] 344 6 NSH : IOAM data follows format defined in [I-D.ietf-sfc-ioam-nsh] 346 IANA is requested to create a new subregistry for IOAM Encapsulation 347 Types and fill it with the initial list from the description. New 348 assignments for IOAM Encapsulation Types are administered by IANA 349 through Expert Review [RFC5226] i.e., review by one of a group of 350 experts designated by an IETF Area Director. 352 Abstract Data Type: unsigned8 354 Data Type Semantics: identifier 356 ElementId: TBD2 358 Status: current 360 3.2.3. ioamPreallocatedTraceData 362 Description: 364 This Information Element carries n octets of IOAM Preallocated Trace 365 data defined in [I-D.ietf-ippm-ioam-data]. 367 The format of the data is determined by the ioamEncapsulationType 368 information element, if present. When the ioamEncapsulationType 369 information element is present and has a value other than "None", and 370 with sufficient length, this element may also report octets from 371 subsequent headers and payload. If no ioamEncapsulationType 372 information element is present, then the encapsulation type shall be 373 assumed to be "None" and this information element only contains 374 octets from the IOAM Preallocated Trace Option. 376 Abstract Data Type: octetArray 378 ElementId: TBD3 380 Status: current 382 3.2.4. ioamIncrementalTraceData 384 Description: 386 This Information Element carries n octets of IOAM Incremental Trace 387 data defined in [I-D.ietf-ippm-ioam-data]. 389 The format of the data is determined by the ioamEncapsulationType 390 information element, if present. When the ioamEncapsulationType 391 information element is present and has a value other than "None", and 392 with sufficient length, this element may also report octets from 393 subsequent headers and payload. If no ioamEncapsulationType 394 information element is present, then the encapsulation type shall be 395 assumed to be "None" and this information element only contains 396 octets from the IOAM Incremental Trace Option. 398 Abstract Data Type: octetArray 400 ElementId: TBD4 402 Status: current 404 3.2.5. ioamE2EData 406 Description: 408 This Information Element carries n octets of IOAM E2E data defined in 409 [I-D.ietf-ippm-ioam-data]. 411 The format of the data is determined by the ioamEncapsulationType 412 information element, if present. When the ioamEncapsulationType 413 information element is present and has a value other than "None", and 414 with sufficient length, this element may also report octets from 415 subsequent headers and payload. If no ioamEncapsulationType 416 information element is present, then the encapsulation type shall be 417 assumed to be "None" and this information element only contains 418 octets from the IOAM Edge-to-Edge Option. 420 Abstract Data Type: octetArray 422 ElementId: TBD5 424 Status: current 426 3.2.6. ioamPOTData 428 Description: 430 This Information Element carries n octets of IOAM POT data defined in 431 [I-D.ietf-ippm-ioam-data]. 433 The format of the data is determined by the ioamEncapsulationType 434 information element, if present. When the ioamEncapsulationType 435 information element is present and has a value other than "None", and 436 with sufficient length, this element may also report octets from 437 subsequent headers and payload. If no ioamEncapsulationType 438 information element is present, then the encapsulation type shall be 439 assumed to be "None" and this information element only contains 440 octets from the IOAM Proof of Transit Option. 442 Abstract Data Type: octetArray 444 ElementId: TBD6 446 Status: current 448 3.2.7. ioamDirectExportData 450 Description: 452 This Information Element carries n octets of IOAM Direct Export data 453 defined in [I-D.ietf-ippm-ioam-direct-export]. 455 In addition to the fields from the IOAM Direct Export Option header 456 in the packet, this information element includes all of the trace 457 data from the exporting node, based on the IOAM-Trace-Type value. 458 This data is appended inside ioamDirectExportData following the bit 459 order of the IOAM-Trace-Type field, similar to the way that IOAM 460 encapsulating nodes append trace data in Incremental Trace Option 461 headers. 463 The format of the data is determined by the ioamEncapsulationType 464 information element, if present. When the ioamEncapsulationType 465 information element is present and has a value other than "None", and 466 with sufficient length, this element may also report octets from 467 subsequent headers and payload. If no ioamEncapsulationType 468 information element is present, then the encapsulation type shall be 469 assumed to be "None" and this information element only contains 470 octets from the IOAM Direct Export Option plus the corresponding 471 trace data. 473 Abstract Data Type: octetArray 475 ElementId: TBD7 477 Status: current 479 3.2.8. ipHeaderPacketSectionWithPadding 481 Description: 483 This Information Element carries a series of n octets from the IP 484 header of a sampled packet, starting sectionOffset octets into the IP 485 header. 487 However, if no sectionOffset field corresponding to this Information 488 Element is present, then a sectionOffset of zero applies, and the 489 octets MUST be from the start of the IP header. 491 With sufficient length, this element also reports octets from the IP 492 payload. However, full packet capture of arbitrary packet streams is 493 explicitly out of scope per the Security Considerations sections of 494 [RFC5477] and [RFC2804]. 496 When this Information Element has a fixed length, this MAY include 497 padding octets that are used to fill out that fixed length. 499 When this information element has a variable length, the variable 500 length MAY include up to 3 octets of padding, used to preserve 501 4-octet alignment of subequent Information Elements or subsequent 502 records within the same set. 504 In either case of fixed or variable length, the amount of populated 505 octets MAY be specified in the sectionExportedOctets field 506 corresponding to this Information Element, in which case the 507 remainder (if any) MUST be padding. If there is no 508 sectionExportedOctets field corresponding to this Information 509 Element, then all octets MUST be populated unless the total length of 510 the IP packet is less than the fixed length of this Information 511 Element, in which case the remainder MUST be padding. 513 Abstract Data Type: octetArray 515 ElementId: TBD8 517 Status: current 519 3.2.9. ethernetFrameSection 521 Description: 523 This Information Element carries a series of n octets from the IEEE 524 802.3 Ethernet frame of a sampled packet, starting after the preamble 525 and start frame delimiter (SFD), plus sectionOffset octets into the 526 frame if there is a sectionOffset field corresponding to this 527 Information Element. 529 With sufficient length, this element also reports octets from the 530 Ethernet payload. However, full packet capture of arbitrary packet 531 streams is explicitly out of scope per the Security Considerations 532 sections of [RFC5477] and [RFC2804]. 534 When this Information Element has a fixed length, this MAY include 535 padding octets that are used to fill out that fixed length. 537 When this information element has a variable length, the variable 538 length MAY include up to 3 octets of padding, used to preserve 539 4-octet alignment of subequent Information Elements or subsequent 540 records within the same set. 542 In either case of fixed or variable length, the amount of populated 543 octets MAY be specified in the sectionExportedOctets field 544 corresponding to this Information Element, in which case the 545 remainder (if any) MUST be padding. If there is no 546 sectionExportedOctets field corresponding to this Information 547 Element, then all octets MUST be populated unless the total length of 548 the Ethernet frame is less than the fixed length of this Information 549 Element, in which case the remainder MUST be padding. 551 Abstract Data Type: octetArray 553 ElementId: TBD9 554 Status: current 556 4. Examples 558 This section shows a set of examples of how IOAM information along 559 with other parts of the packet can be carried using IPFIX. 561 4.1. Fixed Length IP Packet 563 This example shows a fixed length IP packet. IOAM data is part of 564 the ipHeaderPacketSection. 566 0 1 2 3 567 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 568 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+--- 569 | Version Number | Length | | 570 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 571 | Export Time (seconds) |IPFIX 572 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 573 | Sequence Number |Header 574 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 575 | Observation Domain ID | | 576 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 577 | Set ID ( = Template ID) | Length |SetHdr 578 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 579 |ioamReportFlags| fwdingStatus | sectionExportedOctets | | 580 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 581 | ipHeaderPacketSection (start) | | 582 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 583 | ... | | 584 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 585 | ipHeaderPacketSection (end) | | 586 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 587 |ioamReportFlags| fwdingStatus | sectionExportedOctets | | 588 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 589 | ipHeaderPacketSection (start) | | 590 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2 591 | ... | | 592 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 593 | ipHeaderPacketSection (end) | | 594 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 596 4.2. Variable Length IP Packet (length < 255) 598 This examples shows a variable length IP packet, with length < 255 599 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding. 601 0 1 2 3 602 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 603 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 604 | Version Number | Length | | 605 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 606 | Export Time (seconds) |IPFIX 607 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 608 | Sequence Number |Header 609 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 610 | Observation Domain ID | | 611 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 612 | Set ID ( = Template ID) | Length |SetHdr 613 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 614 |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | 615 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 616 | ipHeaderPacketSectionWithPadding (start) | | 617 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 618 | ... | | 619 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 620 | ipHeaderPacketSectionWithPadding (end) | | 621 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 622 |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | 623 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 624 | ipHeaderPacketSectionWithPadding (start) | | 625 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record2 626 | ... | | 627 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 628 | ipHeaderPacketSectionWithPadding (end) | | 629 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 631 4.3. Variable Length IP Packet (length > 255) 633 This examples shows a variable length IP packet, with length > 255 634 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding. 636 0 1 2 3 637 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 638 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 639 | Version Number | Length | | 640 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 641 | Export Time (seconds) |IPFIX 642 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 643 | Sequence Number |Header 644 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 645 | Observation Domain ID | | 646 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 647 | Set ID ( = Template ID) | Length |SetHdr 648 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 649 |ioamReportFlags| fwdingStatus | paddingOctets | | 650 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 651 | paddingOctets | 255 | Length (0 to 65535) | | 652 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 653 | ipHeaderPacketSectionWithPadding (start) |Record1 654 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 655 | ... | | 656 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 657 | ipHeaderPacketSectionWithPadding (end) | | 658 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 660 4.4. Variable Length ETHERNET Packet (length < 255) 662 This examples shows a variable length Ethernet packet, with length < 663 255 bytes. IOAM data is part of the ethernetFrameSection. 665 0 1 2 3 666 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 667 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 668 | Version Number | Length | | 669 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 670 | Export Time (seconds) |IPFIX 671 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 672 | Sequence Number |Header 673 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 674 | Observation Domain ID | | 675 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 676 | Set ID ( = Template ID) | Length |SetHdr 677 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 678 |ioamReportFlags| fwdingStatus | paddingOctets | Length (< 255)| | 679 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 680 | ethernetFrameSection (start) | | 681 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 682 | ... | | 683 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 684 | ethernetFrameSection (end) | | 685 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+-- 687 4.5. Variable Length IP Packet with Fixed Length IOAM Incremental Trace 688 Data 690 This examples shows a variable length IP packet with length < 255 691 bytes and fixed length ioamIncrementalTraceData carried separately. 693 0 1 2 3 694 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 695 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 696 | Version Number | Length | | 697 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 698 | Export Time (seconds) |IPFIX 699 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 700 | Sequence Number |Header 701 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 702 | Observation Domain ID | | 703 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 704 | Set ID ( = Template ID) | Length |SetHdr 705 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 706 | ioamIncrementalTraceData (start) | | 707 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 708 | ... | | 709 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 710 | ioamIncrementalTraceData (end) | | 711 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 712 |ioamReportFlags| fwdingStatus |ioamEncapType | Length (< 255)| | 713 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 714 | ipHeaderPacketSectionWithPadding (start) | | 715 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 716 | ... | | 717 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 718 | ipHeaderPacketSectionWithPadding (end) | | 719 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 721 4.6. Variable Length IP Packet with Variable Length IOAM Incremental 722 Trace Data 724 This examples shows a variable length IP packet with length < 255 725 bytes and variable length ioamIncrementalTraceData with length < 255 726 bytes carried separately. 728 0 1 2 3 729 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 730 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 731 | Version Number | Length | | 732 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 733 | Export Time (seconds) |IPFIX 734 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Message 735 | Sequence Number |Header 736 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 737 | Observation Domain ID | | 738 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 739 | Set ID ( = Template ID) | Length |SetHdr 740 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 741 |ioamReportFlags| fwdingStatus |ioamEncapType | Length (< 255)| | 742 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 743 | ioamIncrementalTraceData (start) | | 744 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 745 | ... | | 746 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 747 | ioamIncrementalTraceData (end) | | 748 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Record1 749 | paddingOctets | Length (< 255)| | 750 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 751 | ipHeaderPacketSectionWithPadding (start) | | 752 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 753 | ... | | 754 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 755 | ipHeaderPacketSectionWithPadding (end) | | 756 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+--+--- 758 5. IANA Considerations 760 IANA is requested to allocate code points for the following 761 Information Elements in [IANA-IPFIX]: 763 TBD1 ioamReportFlags 765 TBD2 ioamEncapsulationType 767 TBD3 ioamPreallocatedTraceData 769 TBD4 ioamIncrementalTraceData 771 TBD5 ioamE2EData 773 TBD6 ioamPOTData 775 TBD7 ioamDirectExportData 776 TBD8 ipHeaderPacketSectionWithPadding 778 TBD9 ethernetFrameSection 780 See Section 3.2 for further details. 782 IANA is requested to create subregistries for ioamReportFlags defined 783 in Section 3.2.1 and ioamEncapsulationType defined in Section 3.2.2. 785 6. Manageability Considerations 787 Manageability considerations will be addressed in a later version of 788 this document. 790 7. Security Considerations 792 Security considerations will be addressed in a later version of this 793 document. 795 8. Acknowledgements 797 The authors would like to thank Barak Gafni, Tal Mizrahi, John Lemon, 798 and Aviv Kfir for their thoughts and comments on raw IOAM data 799 export. 801 9. References 803 9.1. Normative References 805 [I-D.ietf-ippm-ioam-data] 806 Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields 807 for In-situ OAM", Work in Progress, Internet-Draft, draft- 808 ietf-ippm-ioam-data-14, 24 June 2021, 809 . 812 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 813 Requirement Levels", BCP 14, RFC 2119, 814 DOI 10.17487/RFC2119, March 1997, 815 . 817 [RFC5476] Claise, B., Ed., Johnson, A., and J. Quittek, "Packet 818 Sampling (PSAMP) Protocol Specifications", RFC 5476, 819 DOI 10.17487/RFC5476, March 2009, 820 . 822 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 823 "Specification of the IP Flow Information Export (IPFIX) 824 Protocol for the Exchange of Flow Information", STD 77, 825 RFC 7011, DOI 10.17487/RFC7011, September 2013, 826 . 828 9.2. Informative References 830 [I-D.brockners-ippm-ioam-geneve] 831 Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., 832 Nainar, N., Gredler, H., Leddy, J., Youell, S., Mizrahi, 833 T., Lapukhov, P., Gafni, B., Kfir, A., and M. Spiegel, 834 "Geneve encapsulation for In-situ OAM Data", Work in 835 Progress, Internet-Draft, draft-brockners-ippm-ioam- 836 geneve-05, 19 November 2020, . 839 [I-D.brockners-ippm-ioam-vxlan-gpe] 840 Brockners, F., Bhandari, S., Govindan, V., Pignataro, C., 841 Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Kfir, A., 842 Gafni, B., Lapukhov, P., and M. Spiegel, "VXLAN-GPE 843 Encapsulation for In-situ OAM Data", Work in Progress, 844 Internet-Draft, draft-brockners-ippm-ioam-vxlan-gpe-03, 4 845 November 2019, . 848 [I-D.ietf-ippm-ioam-direct-export] 849 Song, H., Gafni, B., Zhou, T., Li, Z., Brockners, F., 850 Bhandari, S., Sivakolundu, R., and T. Mizrahi, "In-situ 851 OAM Direct Exporting", Work in Progress, Internet-Draft, 852 draft-ietf-ippm-ioam-direct-export-05, 12 July 2021, 853 . 856 [I-D.ietf-ippm-ioam-ipv6-options] 857 Bhandari, S., Brockners, F., Pignataro, C., Gredler, H., 858 Leddy, J., Youell, S., Mizrahi, T., Kfir, A., Gafni, B., 859 Lapukhov, P., Spiegel, M., Krishnan, S., Asati, R., and M. 860 Smith, "In-situ OAM IPv6 Options", Work in Progress, 861 Internet-Draft, draft-ietf-ippm-ioam-ipv6-options-05, 21 862 February 2021, . 865 [I-D.ietf-sfc-ioam-nsh] 866 Brockners, F. and S. Bhandari, "Network Service Header 867 (NSH) Encapsulation for In-situ OAM (IOAM) Data", Work in 868 Progress, Internet-Draft, draft-ietf-sfc-ioam-nsh-05, 12 869 December 2020, . 872 [I-D.weis-ippm-ioam-eth] 873 Weis, B., Brockners, F., Hill, C., Bhandari, S., Govindan, 874 V., Pignataro, C., Gredler, H., Leddy, J., Youell, S., 875 Mizrahi, T., Kfir, A., Gafni, B., Lapukhov, P., and M. 876 Spiegel, "EtherType Protocol Identification of In-situ OAM 877 Data", Work in Progress, Internet-Draft, draft-weis-ippm- 878 ioam-eth-04, 13 May 2020, . 881 [IANA-IPFIX] 882 "IP Flow Information Export (IPFIX) Entities", 883 . 885 [RFC2804] IAB and IESG, "IETF Policy on Wiretapping", RFC 2804, 886 DOI 10.17487/RFC2804, May 2000, 887 . 889 [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an 890 IANA Considerations Section in RFCs", RFC 5226, 891 DOI 10.17487/RFC5226, May 2008, 892 . 894 [RFC5477] Dietz, T., Claise, B., Aitken, P., Dressler, F., and G. 895 Carle, "Information Model for Packet Sampling Exports", 896 RFC 5477, DOI 10.17487/RFC5477, March 2009, 897 . 899 Authors' Addresses 901 Mickey Spiegel 902 Barefoot Networks, an Intel company 903 4750 Patrick Henry Drive 904 Santa Clara, CA, 95054 905 United States of America 907 Email: mickey.spiegel@intel.com 909 Frank Brockners 910 Cisco Systems, Inc. 911 Hansaallee 249, 3rd Floor 912 40549 DUESSELDORF, NORDRHEIN-WESTFALEN 913 Germany 915 Email: fbrockne@cisco.com 917 Shwetha Bhandari (editor) 918 Thoughtspot 919 3rd Floor, Indiqube Orion, 24th Main Rd, Garden Layout, HSR Layout 920 Bangalore, KARNATAKA 560 102 921 India 923 Email: shwetha.bhandari@thoughtspot.com 925 Ramesh Sivakolundu 926 Cisco Systems, Inc. 927 170 West Tasman Dr. 928 SAN JOSE, CA 95134, 929 United States of America 931 Email: sramesh@cisco.com