idnits 2.17.00 (12 Aug 2021) /tmp/idnits16639/draft-song-ippm-ioam-scalability-00.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (May 18, 2017) is 1829 days in the past. Is this intentional? Checking references for intended status: Experimental ---------------------------------------------------------------------------- == Outdated reference: A later version (-07) exists of draft-brockners-inband-oam-data-02 == Outdated reference: A later version (-03) exists of draft-brockners-inband-oam-requirements-02 Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 ippm H. Song, Ed. 3 Internet-Draft T. Zhou 4 Intended status: Experimental Huawei 5 Expires: November 19, 2017 May 18, 2017 7 On Scalability of In-situ OAM 8 draft-song-ippm-ioam-scalability-00 10 Abstract 12 This document describes several scalability issues in current in-situ 13 OAM documents and proposes corresponding solutions. Specifically, we 14 extend in-situ OAM to support more standard tracing data than is 15 currently defined and add new features to avoid limitations on MTU, 16 bandwidth, forwarding path length, and node processing capability. 18 Status of This Memo 20 This Internet-Draft is submitted in full conformance with the 21 provisions of BCP 78 and BCP 79. 23 Internet-Drafts are working documents of the Internet Engineering 24 Task Force (IETF). Note that other groups may also distribute 25 working documents as Internet-Drafts. The list of current Internet- 26 Drafts is at http://datatracker.ietf.org/drafts/current/. 28 Internet-Drafts are draft documents valid for a maximum of six months 29 and may be updated, replaced, or obsoleted by other documents at any 30 time. It is inappropriate to use Internet-Drafts as reference 31 material or to cite them other than as "work in progress." 33 This Internet-Draft will expire on November 19, 2017. 35 Copyright Notice 37 Copyright (c) 2017 IETF Trust and the persons identified as the 38 document authors. All rights reserved. 40 This document is subject to BCP 78 and the IETF Trust's Legal 41 Provisions Relating to IETF Documents 42 (http://trustee.ietf.org/license-info) in effect on the date of 43 publication of this document. Please review these documents 44 carefully, as they describe your rights and restrictions with respect 45 to this document. Code Components extracted from this document must 46 include Simplified BSD License text as described in Section 4.e of 47 the Trust Legal Provisions and are provided without warranty as 48 described in the Simplified BSD License. 50 Table of Contents 52 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 53 2. Current iOAM Limitations . . . . . . . . . . . . . . . . . . 2 54 2.1. Data Type Limitation . . . . . . . . . . . . . . . . . . 2 55 2.2. Node Data Size Limitation . . . . . . . . . . . . . . . . 3 56 2.3. Node Processing Limitation . . . . . . . . . . . . . . . 3 57 3. Scalable Data Type Extension . . . . . . . . . . . . . . . . 4 58 3.1. Data Type Bitmap . . . . . . . . . . . . . . . . . . . . 4 59 3.2. Scalable Data Type Extension Use Cases . . . . . . . . . 5 60 3.3. Consideration for Data Packing . . . . . . . . . . . . . 5 61 4. Segment In-situ OAM . . . . . . . . . . . . . . . . . . . . . 6 62 4.1. Segment and Hops . . . . . . . . . . . . . . . . . . . . 6 63 4.2. Considerations for Data Handling . . . . . . . . . . . . 7 64 4.3. Segment iOAM Use Cases . . . . . . . . . . . . . . . . . 7 65 5. In-situ OAM Sampling and Data Validation . . . . . . . . . . 8 66 5.1. Valid Node Bitmap and Valid Data Bitmap . . . . . . . . . 8 67 5.2. iOAM Sampling and Data Validation Use Cases . . . . . . . 9 68 6. Security Considerations . . . . . . . . . . . . . . . . . . . 9 69 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 70 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 10 71 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 10 72 10. Informative References . . . . . . . . . . . . . . . . . . . 10 73 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 75 1. Introduction 77 In-situ OAM (iOAM) [I-D.brockners-inband-oam-requirements] records 78 OAM information within user packets while the packets traverse a 79 network. The data types and data formats for in-situ OAM data 80 records have been defined in [I-D.brockners-inband-oam-data]. We 81 identify several scalability issues of the current iOAM specification 82 and propose solutions in this draft. 84 2. Current iOAM Limitations 86 2.1. Data Type Limitation 88 Currently 11 data types and associated formats (including wide format 89 and short format of the same data) are defined in 90 [I-D.brockners-inband-oam-data] . The presence of data is indicated 91 by a 16-bit bitmap in the "OAM-Trace-Type" field. 93 In the current specification only five bits are left to identify new 94 data types. Moreover, some data is forced to be bundled together as 95 a single unit to save bitmap space and pack data to the ideal size 96 (e.g., the hop limit and the node id are bundled, and the ingress 97 interface id and the egress interface id are bundled), regardless of 98 the fact that an application may only ask for a part of the data. 99 Last but not the least, each data is forced to be 4-byte aligned for 100 easier access, resulting in waste of header space in many cases. 102 Since the data plane bandwidth, the data plane packet processing, and 103 the management plane data handling are all precious yet scarce 104 resource, the scheme should strive to be simple and precise. The 105 application should be able to control the exact type and format of 106 data it needs to collect and analyze. It is conceivable that more 107 types of data may be introduced in the future. However, the current 108 scheme cannot support it after all the bits in the bitmap are used 109 up. 111 Currently, bit 7 is used to indicate the presence of variable length 112 opaque state snapshot data. While this data field can be used to 113 store arbitrary data, the data is difficult to be standardized and 114 another schema is needed to decode the data, which may lead to low 115 data plane performance. 117 2.2. Node Data Size Limitation 119 The total size of data is limited by the MTU. When the number of 120 required data types is large and the forwarding path length is long, 121 it is possible that there is not enough space in the iOAM header to 122 save the data. The current proposal is to label the overflow status 123 and stop adding new node data to the packet, leading to loss of 124 information. 126 Even if the header has enough space to hold the iOAM data, the 127 overhead may be too large and consume too much bandwidth. For 128 example, if we assume moderate 16 bytes of data per node, a path with 129 length of 10 will need 160 bytes to hold the data. This will inflate 130 small 64-byte packets by more than three times. Therefore, we need 131 to limit the iOAM data overhead without sacrificing the data 132 collection capability. 134 2.3. Node Processing Limitation 136 iOAM can designate the flow to add the iOAM header and collect data 137 on the flow forwarding path. The flow can have arbitrary 138 granularity. However, processing the data can be a heavy burden for 139 the network nodes, especially when some data needs to be calculated 140 by the node (e.g., the transit delay). If the flow traffic is heavy, 141 the node may not be able to handle the iOAM processing so many 142 performance issues may occur, such as long latency and packet drop. 144 Although it is good for the OAM applications to gain the detailed 145 information on every packet at every node, in many cases, such 146 information is often repetitive and redundant. The large quantity of 147 data would also burden the management plane which needs to collect 148 and stream the data for analytics. It is also possible that some 149 nodes cannot provide the requested data at all. So a trade-off is 150 needed to balance the performance impact and the data availability 151 and completeness. 153 3. Scalable Data Type Extension 155 Based on the observation in Section 2.1, we propose a method for data 156 type encoding which can solve the current limitation and address 157 future data requirements. 159 3.1. Data Type Bitmap 161 Bitmap is simple and efficient data structure for high performance 162 data plane implementation. The base bitmap size is kept to be 16 163 bits. We use one bit to indicate a single type of data in a single 164 format. The last bit in the bitmap (i.e., bit 15), if set, is used 165 to indicate the presence of the next data type bitmap, which is 32 166 bits long. In the second bitmap, bit 31 is again reserved to 167 indicate a third bitmap, and so on. With each extra bitmap, 31 more 168 data types can be defined. 170 Figure 1 shows an example of the in-situ OAM header format with two 171 extended OAM trace type fields. Except the OAM Trace Type fields, 172 all other fields remain the same as defined in 173 [I-D.brockners-inband-oam-data]. 175 0 1 2 3 176 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 177 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 178 | Base OAM Trace Type |1| Length Field | Flags | 179 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 180 | Extended OAM Trace Type 1 |1| 181 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 182 | Extended OAM Trace Type 2 |0| 183 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 184 | | 185 | Node Data List [] | 186 | | 187 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 189 Figure 1: Extended OAM Trace Type Header Format 191 The specification of the Base OAM Trace Type is the same as the OAM 192 Trace Type in [I-D.brockners-inband-oam-data] except the last bit, 193 which is defined as follows: 195 o Bit 15: When set indicates presence of next bit map. 197 The OAM trace type fields are labeled as Base OAM Trace Type, 198 Extended OAM Trace Type 1, Extended OAM Trace Type 2, and so on. The 199 Base OAM Trace Type is always present. If no data type is asked by 200 the application in Extended OAM Trace Type n and beyond, then the 201 last bit in the previous bitmap is set to 1 and these extended fields 202 are not included in the header. On the other hand, to eliminate 203 ambiguity, if any data is asked for by the application in Extended 204 OAM Trace Type n, then Extended OAM Trace Type 1 to (n-1) must be 205 included in the header, even though no data type in these bitmaps are 206 needed (i.e., all zero bitmap except the last bit). 208 The actual data in a node is packed together in the same order as 209 listed in the OAM Trace Type bitmap. Each node is padded to be the 210 multiple of 4 bytes. 212 3.2. Scalable Data Type Extension Use Cases 214 New types of data can be potentially added and standardized, which 215 demand new bits allocated in the OAM Trace Type bitmaps. Some 216 examples are listed here. 218 o Metered flow bandwidth. 220 o Time gap between two consecutive flow packets. 222 o Remaining time budget to the packet delivery deadline. 224 o Buffer occupancy on the Node. 226 o Queue depth on each level of hierarchical QoS queues. 228 o Packet jitter at the Node. 230 o Other node statistics. 232 3.3. Consideration for Data Packing 234 The length of each data must be the multiple of 2 bytes. However, 235 allowing different data type to have different length, while 236 efficient in storage, makes data alignment and packing difficult. 238 If we can define the maximum number of data types that can be carried 239 per packet, the offset of each data in the node can be pre-calculated 240 and carried in the iOAM header. The overhead can be justified by the 241 overall space saving of the node data list. Otherwise, each data's 242 offset in the node must be calculated in each device, with the help 243 of a table which stores the size of each data type. We can also 244 arrange the bitmap to reflect the data availability order in the 245 system (e.g., the bit for egress_if_id must be after the bit for 246 ingress_if_id), so in a pipeline-based system, the required data can 247 be packed one after one. 249 4. Segment In-situ OAM 251 Based on the observation in Section 2.2, we propose a method to limit 252 the size of the node data list. 254 4.1. Segment and Hops 256 A hop is a node on a flow's forwarding path which is capable of 257 processing iOAM data. A segment is a fixed number hops on a flow's 258 forwarding path. While working in the "per hop" mode, the segment 259 size (SSize) and the remaining hops (RHop), is added to the iOAM 260 header at the edge. Initially, RHop is equal to SSize. At each hop, 261 if RH is not zero, the node data is added to the node data list at 262 the corresponding location and then RH is decremented by 1. If RH is 263 equal to 0 when receiving the packet, the node needs to remove (in 264 incremental trace option) or clear (in pre-allocated trace option) 265 the iOAM node data list and reset RHop to SSize. Then the node will 266 add its data to the node data list as if it is the edge node. 268 Figure 2 shows the proposed in-situ OAM header format. The last bit 269 (bit 31) in the Flags field is used to indicate the current header is 270 a segment iOAM header. In this context, the third byte of the first 271 word is partitioned into two 4-bit piece. The first piece is used to 272 save the segment size and the second piece is used to save the 273 remaining hops. This limits the maximum segment size to 15. 275 0 1 2 3 276 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 277 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 278 | Base OAM Trace Type |0| SSize | RHop | Flags |1| 279 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 280 | | 281 | Node Data List [] | 282 | | 283 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 285 Figure 2: Segment iOAM Header Format 287 4.2. Considerations for Data Handling 289 At any hop when RHop is equal to 0, the node data list is copied from 290 the iOAM header. The data can be encapsulated and reported to the 291 controller or the edge node as configured. The encapsulation and 292 report method is beyond the scope of this draft but should be comply 293 with the method used by the iOAM edge node. 295 The actual size of the last segment may not be equal to SSize but 296 this is not a problem. 298 4.3. Segment iOAM Use Cases 300 Segment iOAM is necessary in the following example scenarios: 302 o Segment iOAM can be used to detect at which segment the flow 303 packet is dropped. If the SSize is set to 1, then the exact drop 304 node can be identified. The iOAM data before the dropping point 305 is also retained. 307 o The path MTU allows to add at most k node data in the list to 308 avoid fragmentation. Therefore SSize is set to k and at each hop 309 where RHop is 0, the node data list is retrieved and sent in a 310 standalone packet. 312 o A flow contains mainly short packets and travels a long path. It 313 would be inefficient to keep a large node data list in the packet 314 so the network bandwidth utilization rate is low. In this case, 315 segment iOAM can be used to limit the ratio of the iOAM data to 316 the flow packet payload. 318 o The network allows at most n bytes budget for the iOAM data. 319 There is a tradeoff between the number of data types that can be 320 collected and the number of hops for data collecting. The segment 321 size is therefore necessary to meet the application's data 322 requirement (i.e., SSize * Node Data Size < n). 324 5. In-situ OAM Sampling and Data Validation 326 Based on the observation in Section 1.3, the source edge node should 327 be able to define either the period or the probability to add the 328 iOAM header to the selected flow packet. In this way, only a subset 329 of the flow/sec packets would carry the OAM data, which not only 330 reduces the overall iOAM data quantity but also reduces the 331 processing work load of the network nodes. 333 5.1. Valid Node Bitmap and Valid Data Bitmap 335 It is possible that even an iOAM capable node will not add data to 336 the node data list as requested. In some cases, a node can be too 337 busy to handle the data request or some types of the requested data 338 is not available. Therefore, we propose to add two bitmaps, a valid 339 node bitmap and a valid data bit, to the iOAM specification. 341 The Node Valid Bitmap is inserted before the Node Data List as shown 342 in Figure 3. Each bit in the bitmap corresponds to a hop on the 343 packet's forwarding path. The bits are listed in the same order as 344 the hop on the packet's forwarding path. The bitmap is cleared to 345 all zero at first. If a hop can add data to the Node Data List, the 346 corresponding bit in Node Valid Bitmap is set to 1. The bit location 347 for a hop can be calculated from the length field (e.g, the bit index 348 is equal to SSize-RHop).The valid node data items in the node data 349 list is equal to the number of 1's in the Node Valid Bitmap. 351 0 1 2 3 352 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 353 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 354 | Base OAM Trace Type |0| Length Field | Flags | 355 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 356 | Valid Node Bitmap | 357 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 358 | | 359 | Node Data List [] | 360 | | 361 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 363 Figure 3: Segment iOAM Header Format 365 For each node data in the node data list, a Valid Data Bitmap is 366 added before the node data. The number of bits in the Valid Data 367 Bitmap is equal to the number of 1's in the OAM Trace Type bitmaps 368 (excluding the next trace type bitmap indicator bits). When the bit 369 is set, the corresponding data is valid in the node; otherwise, the 370 corresponding data is invalid so the management plane should ignore 371 it after the data is collected. 373 The size of the bitmap can be padded to two or four bytes, which 374 allow up to 16 or 32 types of data to be included in a node. 376 5.2. iOAM Sampling and Data Validation Use Cases 378 We give some examples to show the usefulness of in-situ OAM sampling 379 and data validation features. 381 o An application needs to track a flow's forwarding path and knows 382 the path will not change frequently, so it sets a low sampling 383 rate to periodically insert the iOAM header to request the node 384 ID. 386 o In a heterogeneous data plane, some nodes support to provide data 387 x but the other nodes do not support it. However, an application 388 is still interested in collecting data x if available. In this 389 case, iOAM header can still be configured to ask for data x but 390 the nodes that cannot provide the data simply invalidates it by 391 resetting the corresponding bit in the valid data bitmap. 393 o Multiple sampling rate and multiple data request schema can be 394 defined for a flow based on applications requirements and the data 395 property, so for a flow packet, there can be no iOAM header or 396 different iOAM headers. The node does not need to process all 397 data all the time. 399 o For security reason, a node decides to not participate in the iOAM 400 data collection. While it processes the other iOAM header fields 401 as usual, it does not set the node valid bit in the Node Valid 402 Bitmap and add node data to the Node Data List. 404 6. Security Considerations 406 There is no extra security considerations beyond those have been 407 identified by in-situ OAM protocol. 409 7. IANA Considerations 411 This memo includes no request to IANA. 413 8. Acknowledgments 415 9. Contributors 417 The document is inspired by numerous discussions with James N. 418 Guichard. He also provided significant comments and suggestions to 419 help improve this document. 421 10. Informative References 423 [I-D.brockners-inband-oam-data] 424 Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., 425 Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, 426 P., and R. <>, "Data Formats for In-situ OAM", draft- 427 brockners-inband-oam-data-02 (work in progress), October 428 2016. 430 [I-D.brockners-inband-oam-requirements] 431 Brockners, F., Bhandari, S., Dara, S., Pignataro, C., 432 Gredler, H., Leddy, J., Youell, S., Mozes, D., Mizrahi, 433 T., <>, P., and r. remy@barefootnetworks.com, 434 "Requirements for In-situ OAM", draft-brockners-inband- 435 oam-requirements-02 (work in progress), October 2016. 437 Authors' Addresses 439 Haoyu Song (editor) 440 Huawei 441 2330 Central Expressway 442 Santa Clara, 95050 443 USA 445 Email: haoyu.song@huawei.com 447 Tianran Zhou 448 Huawei 449 156 Beiqing Road 450 Beijing, 100095 451 P.R. China 453 Email: zhoutianran@huawei.com