idnits 2.17.00 (12 Aug 2021) /tmp/idnits45115/draft-fioccola-rfc8889bis-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- ** The abstract seems to contain references ([RFC8889]), which it shouldn't. Please replace those with straight textual mentions of the documents in question. Miscellaneous warnings: ---------------------------------------------------------------------------- -- The document date (February 17, 2022) is 93 days in the past. Is this intentional? Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) == Outdated reference: A later version (-04) exists of draft-fioccola-rfc8321bis-02 ** Downref: Normative reference to an Informational RFC: RFC 5474 == Outdated reference: A later version (-17) exists of draft-song-opsawg-ifit-framework-16 == Outdated reference: A later version (-09) exists of draft-zhou-ippm-enhanced-alternate-marking-08 Summary: 2 errors (**), 0 flaws (~~), 3 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group G. Fioccola, Ed. 3 Internet-Draft Huawei Technologies 4 Obsoletes: 8889 (if approved) M. Cociglio 5 Intended status: Standards Track Telecom Italia 6 Expires: August 21, 2022 A. Sapio 7 Intel Corporation 8 R. Sisto 9 Politecnico di Torino 10 T. Zhou 11 Huawei Technologies 12 February 17, 2022 14 Multipoint Alternate-Marking Method 15 draft-fioccola-rfc8889bis-02 17 Abstract 19 This document generalizes and expands Alternate-Marking methodology 20 to measure any kind of unicast flow whose packets can follow several 21 different paths in the network -- in wider terms, a multipoint-to- 22 multipoint network. For this reason, the technique here described is 23 called "Multipoint Alternate Marking". This document obsoletes 24 [RFC8889]. 26 Status of This Memo 28 This Internet-Draft is submitted in full conformance with the 29 provisions of BCP 78 and BCP 79. 31 Internet-Drafts are working documents of the Internet Engineering 32 Task Force (IETF). Note that other groups may also distribute 33 working documents as Internet-Drafts. The list of current Internet- 34 Drafts is at https://datatracker.ietf.org/drafts/current/. 36 Internet-Drafts are draft documents valid for a maximum of six months 37 and may be updated, replaced, or obsoleted by other documents at any 38 time. It is inappropriate to use Internet-Drafts as reference 39 material or to cite them other than as "work in progress." 41 This Internet-Draft will expire on August 21, 2022. 43 Copyright Notice 45 Copyright (c) 2022 IETF Trust and the persons identified as the 46 document authors. All rights reserved. 48 This document is subject to BCP 78 and the IETF Trust's Legal 49 Provisions Relating to IETF Documents 50 (https://trustee.ietf.org/license-info) in effect on the date of 51 publication of this document. Please review these documents 52 carefully, as they describe your rights and restrictions with respect 53 to this document. Code Components extracted from this document must 54 include Simplified BSD License text as described in Section 4.e of 55 the Trust Legal Provisions and are provided without warranty as 56 described in the Simplified BSD License. 58 Table of Contents 60 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 61 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 62 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 63 2.1. Correlation with RFC 5644 . . . . . . . . . . . . . . . . 5 64 3. Flow Classification . . . . . . . . . . . . . . . . . . . . . 6 65 4. Multipoint Performance Measurement . . . . . . . . . . . . . 9 66 4.1. Monitoring Network . . . . . . . . . . . . . . . . . . . 9 67 5. Multipoint Packet Loss . . . . . . . . . . . . . . . . . . . 10 68 6. Network Clustering . . . . . . . . . . . . . . . . . . . . . 11 69 6.1. Algorithm for Clusters Partition . . . . . . . . . . . . 12 70 7. Timing Aspects . . . . . . . . . . . . . . . . . . . . . . . 16 71 8. Multipoint Delay and Delay Variation . . . . . . . . . . . . 17 72 8.1. Delay Measurements on a Multipoint-Paths Basis . . . . . 18 73 8.1.1. Single-Marking Measurement . . . . . . . . . . . . . 18 74 8.2. Delay Measurements on a Single-Packet Basis . . . . . . . 18 75 8.2.1. Single- and Double-Marking Measurement . . . . . . . 18 76 8.2.2. Hashing Selection Method . . . . . . . . . . . . . . 19 77 9. Results of the Multipoint Alternate Marking Experiment . . . 20 78 10. A Closed-Loop Performance-Management Approach . . . . . . . . 21 79 11. Security Considerations . . . . . . . . . . . . . . . . . . . 22 80 12. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23 81 13. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 23 82 14. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 23 83 15. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 84 15.1. Normative References . . . . . . . . . . . . . . . . . . 23 85 15.2. Informative References . . . . . . . . . . . . . . . . . 24 86 Appendix A. Changes Log . . . . . . . . . . . . . . . . . . . . 25 87 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 25 89 1. Introduction 91 The Alternate-Marking method, as described in 92 [I-D.fioccola-rfc8321bis], is applicable to a point-to-point path. 93 The extension proposed in this document applies to the most general 94 case of multipoint-to-multipoint path and enables flexible and 95 adaptive performance measurements in a managed network. 97 The Alternate-Marking methodology described in 98 [I-D.fioccola-rfc8321bis] allows the synchronization of the 99 measurements in different points by dividing the packet flow into 100 batches. So it is possible to get coherent counters and show what is 101 happening in every marking period for each monitored flow. The 102 monitoring parameters are the packet counter and timestamps of a flow 103 for each marking period. Note that additional details about the 104 applicability of the Alternate-Marking methodology are described in 105 [I-D.fioccola-rfc8321bis] while implementation details can be found 106 in the paper "AM-PM: Efficient Network Telemetry using Alternate 107 Marking" [IEEE-Network-PNPM]. 109 There are some applications of the Alternate-Marking method where 110 there are a lot of monitored flows and nodes. Multipoint Alternate 111 Marking aims to reduce these values and makes the performance 112 monitoring more flexible in case a detailed analysis is not needed. 113 For instance, by considering n measurement points and m monitored 114 flows, the order of magnitude of the packet counters for each time 115 interval is n*m*2 (1 per color). The number of measurement points 116 and monitored flows may vary and depends on the portion of the 117 network we are monitoring (core network, metro network, access 118 network) and the granularity (for each service, each customer). So 119 if both n and m are high values, the packet counters increase a lot, 120 and Multipoint Alternate Marking offers a tool to control these 121 parameters. 123 The approach presented in this document is applied only to unicast 124 flows and not to multicast. Broadcast, Unknown Unicast, and 125 Multicast (BUM) traffic is not considered here, because traffic 126 replication is not covered by the Multipoint Alternate-Marking 127 method. Furthermore, it can be applicable to anycast flows, and 128 Equal-Cost Multipath (ECMP) paths can also be easily monitored with 129 this technique. 131 [I-D.fioccola-rfc8321bis] applies to point-to-point unicast flows and 132 BUM traffic. For BUM traffic, the basic method of 133 [I-D.fioccola-rfc8321bis] can easily be applied link by link and 134 therefore split the multicast flow tree distribution into separate 135 unicast point-to-point links. While this document and its Clustered 136 Alternate-Marking method is valid for multipoint-to-multipoint 137 unicast flows, anycast, and ECMP flows. 139 Therefore, the Alternate-Marking method can be extended to any kind 140 of multipoint-to-multipoint paths, and the network-clustering 141 approach presented in this document is the formalization of how to 142 implement this property and allow a flexible and optimized 143 performance measurement support for network management in every 144 situation. 146 Without network clustering, it is possible to apply Alternate Marking 147 only for all the network or per single flow. Instead, with network 148 clustering, it is possible to use the partition of the network into 149 clusters at different levels in order to perform the needed degree of 150 detail. In some circumstances, it is possible to monitor a 151 multipoint network by analyzing the network clustering, without 152 examining in depth. In case of problems (packet loss is measured or 153 the delay is too high), the filtering criteria could be specified 154 more in order to perform a detailed analysis by using a different 155 combination of clusters up to a per-flow measurement as described in 156 [I-D.fioccola-rfc8321bis]. 158 This approach fits very well with the Closed-Loop Network and 159 Software-Defined Network (SDN) paradigm, where the SDN orchestrator 160 and the SDN controllers are the brains of the network and can manage 161 flow control to the switches and routers and, in the same way, can 162 calibrate the performance measurements depending on the desired 163 accuracy. An SDN controller application can orchestrate how 164 accurately the network performance monitoring is set up by applying 165 the Multipoint Alternate Marking as described in this document. 167 It is important to underline that, as an extension of 168 [I-D.fioccola-rfc8321bis], this is a methodology document, so the 169 mechanism that can be used to transmit the counters and the 170 timestamps is out of scope here, and the implementation is open. 171 Several options are possible -- e.g., see "Enhanced Alternate Marking 172 Method" [I-D.zhou-ippm-enhanced-alternate-marking]. 174 This document assumes that the blocks are created according to a 175 fixed timer as per [I-D.fioccola-rfc8321bis]. The switching after a 176 fixed number of packets is an additional possibility but it is out of 177 scope here. 179 Note that the fragmented packets case can be managed with the 180 Alternate-Marking methodology. The same considerations of 181 [I-D.fioccola-rfc8321bis] apply also in the case of Multipoint 182 Alternate Marking. As defined in [I-D.fioccola-rfc8321bis] the 183 marking node MUST mark all the fragments except in the case of 184 fragmentation within the network domain, in that event it is 185 suggested to mark only the first fragment. 187 1.1. Requirements Language 189 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 190 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 191 "OPTIONAL" in this document are to be interpreted as described in BCP 192 14 [RFC2119] [RFC8174] when, and only when, they appear in all 193 capitals, as shown here. 195 2. Terminology 197 The definitions of the basic terms are identical to those found in 198 Alternate Marking [I-D.fioccola-rfc8321bis]. It is to be remembered 199 that [I-D.fioccola-rfc8321bis] is valid for point-to-point unicast 200 flows and BUM traffic. 202 The important new terms that need to be explained are listed below: 204 Multipoint Alternate Marking: Extension to 205 [I-D.fioccola-rfc8321bis], valid for multipoint-to-multipoint 206 unicast flows, anycast, and ECMP flows. It can also be referred 207 to as Clustered Alternate Marking. 209 Flow definition: The concept of flow is generalized in this 210 document. The identification fields are selected without any 211 constraints and, in general, the flow can be a multipoint-to- 212 multipoint flow, as a result of aggregate point-to-point flows. 214 Monitoring Network: Identified with the nodes of the network that 215 are the measurement points (MPs) and the links that are the 216 connections between MPs. The monitoring network graph depends on 217 the flow definition, so it can represent a specific flow or the 218 entire network topology as aggregate of all the flows. 220 Cluster: Smallest identifiable subnetwork of the entire monitoring 221 network graph that still satisfies the condition that the number 222 of packets that go in is the same as the number that go out. 224 Multipoint metrics: Packet loss, delay and delay variation are 225 extended to the case of multipoint flows. It is possible to 226 compute these metrics on the basis of multipoint paths in order to 227 associate the measurements to a cluster, a combination of 228 clusters, or the entire monitored network. For delay and delay 229 variation, it is also possible to define the metrics on a single- 230 packet basis, and it means that the multipoint path is used to 231 easily couple packets between input and output nodes of a 232 multipoint path. 234 The next section highlights the correlation with the terms used in 235 RFC 5644 [RFC5644]. 237 2.1. Correlation with RFC 5644 239 RFC 5644 [RFC5644] is limited to active measurements using a single 240 source packet or stream. Its scope is also limited to observations 241 of corresponding packets along the path (spatial metric) and at one 242 or more destinations (one-to-group) along the path. 244 Instead, the scope of this memo is to define multiparty metrics for 245 passive and hybrid measurements in a group-to-group topology with 246 multiple sources and destinations. 248 RFC 5644 [RFC5644] introduces metric names that can be reused here 249 but have to be extended and rephrased to be applied to the Alternate- 250 Marking schema: 252 a. the multiparty metrics are not only one-to-group metrics but can 253 be also group-to-group metrics; 255 b. the spatial metrics, used for measuring the performance of 256 segments of a source to destination path, are applied here to 257 group-to-group segments (called clusters). 259 3. Flow Classification 261 A unicast flow is identified by all the packets having a set of 262 common characteristics. This definition is inspired by RFC 7011 263 [RFC7011]. 265 As an example, by considering a flow as all the packets sharing the 266 same source IP address or the same destination IP address, it is easy 267 to understand that the resulting pattern will not be a point-to-point 268 connection, but a point-to-multipoint or multipoint-to-point 269 connection. 271 In general, a flow can be defined by a set of selection rules used to 272 match a subset of the packets processed by the network device. These 273 rules specify a set of Layer 3 and Layer 4 header fields 274 (identification fields) and the relative values that must be found in 275 matching packets. 277 The choice of the identification fields directly affects the type of 278 paths that the flow would follow in the network. In fact, it is 279 possible to relate a set of identification fields with the pattern of 280 the resulting graphs, as listed in Figure 1. 282 A TCP 5-tuple usually identifies flows following either a single path 283 or a point-to-point multipath (in the case of load balancing). On 284 the contrary, a single source address selects aggregate flows 285 following a point-to-multipoint, while a multipoint-to-point can be 286 the result of a matching on a single destination address. In the 287 case where a selection rule and its reverse are used for 288 bidirectional measurements, they can correspond to a point-to- 289 multipoint in one direction and a multipoint-to-point in the opposite 290 direction. 292 So the flows to be monitored are selected into the monitoring points 293 using packet selection rules, which can also change the pattern of 294 the monitored network. 296 Note that, more generally, the flow can be defined at different 297 levels based on the potential encapsulation, and additional 298 conditions that are not in the packet header can also be included as 299 part of matching criteria. 301 The Alternate-Marking method is applicable only to a single path (and 302 partially to a one-to-one multipath), so the extension proposed in 303 this document is suitable also for the most general case of 304 multipoint-to-multipoint, which embraces all the other patterns of 305 Figure 1. 307 point-to-point single path 308 +------+ +------+ +------+ 309 ---<> R1 <>----<> R2 <>----<> R3 <>--- 310 +------+ +------+ +------+ 312 point-to-point multipath 313 +------+ 314 <> R2 <> 315 / +------+ \ 316 / \ 317 +------+ / \ +------+ 318 ---<> R1 <> <> R4 <>--- 319 +------+ \ / +------+ 320 \ / 321 \ +------+ / 322 <> R3 <> 323 +------+ 325 point-to-multipoint 326 +------+ 327 <> R4 <>--- 328 / +------+ 329 +------+ / 330 <> R2 <> 331 / +------+ \ 332 +------+ / \ +------+ 333 ---<> R1 <> <> R5 <>--- 334 +------+ \ +------+ 335 \ +------+ 336 <> R3 <> 337 +------+ \ 338 \ +------+ 339 <> R6 <>--- 340 +------+ 342 multipoint-to-point 343 +------+ 344 ---<> R1 <> 345 +------+ \ 346 \ +------+ 347 <> R4 <> 348 / +------+ \ 349 +------+ / \ +------+ 350 ---<> R2 <> <> R6 <>--- 351 +------+ / +------+ 352 +------+ / 353 <> R5 <> 354 / +------+ 355 +------+ / 356 ---<> R3 <> 357 +------+ 359 multipoint-to-multipoint 360 +------+ +------+ 361 ---<> R1 <> <> R6 <>--- 362 +------+ \ / +------+ 363 \ +------+ / 364 <> R4 <> 365 +------+ \ 366 +------+ \ +------+ 367 ---<> R2 <> <> R7 <>--- 368 +------+ \ / +------+ 369 \ +------+ / 370 <> R5 <> 371 / +------+ \ 372 +------+ / \ +------+ 373 ---<> R3 <> <> R8 <>--- 374 +------+ +------+ 376 Figure 1: Flow Classification 378 The case of unicast flow is considered in Figure 1. The anycast flow 379 is also in scope, because there is no replication and only a single 380 node from the anycast group receives the traffic, so it can be viewed 381 as a special case of unicast flow. Furthermore, an ECMP flow is in 382 scope by definition, since it is a point-to-multipoint unicast flow. 384 4. Multipoint Performance Measurement 386 By using the Alternate-Marking method, only point-to-point paths can 387 be monitored. To have an IP (TCP/UDP) flow that follows a point-to- 388 point path, we have to define, with a specific value, 5 389 identification fields (IP Source, IP Destination, Transport Protocol, 390 Source Port, Destination Port). 392 Multipoint Alternate Marking enables the performance measurement for 393 multipoint flows selected by identification fields without any 394 constraints (even the entire network production traffic). It is also 395 possible to use multiple marking points for the same monitored flow. 397 4.1. Monitoring Network 399 The monitoring network is deduced from the production network by 400 identifying the nodes of the graph that are the measurement points, 401 and the links that are the connections between measurement points. 403 There are some techniques that can help with the building of the 404 monitoring network (as an example, see [I-D.ietf-ippm-route]). In 405 general, there are different options: the monitoring network can be 406 obtained by considering all the possible paths for the traffic or 407 periodically checking the traffic (e.g. daily, weekly, monthly) and 408 updating the graph as appropriate, but this is up to the Network 409 Management System (NMS) configuration. 411 So a graph model of the monitoring network can be built according to 412 the Alternate-Marking method: the monitored interfaces and links are 413 identified. Only the measurement points and links where the traffic 414 has flowed have to be represented in the graph. 416 Figure 2 shows a simple example of a monitoring network graph: 418 +------+ 419 <> R6 <>--- 420 / +------+ 421 +------+ +------+ / 422 <> R2 <>---<> R4 <> 423 / +------+ \ +------+ \ 424 / \ \ +------+ 425 +------+ / +------+ \ +------+ <> R7 <>--- 426 ---<> R1 <>---<> R3 <>---<> R5 <> +------+ 427 +------+ \ +------+ \ +------+ \ 428 \ \ \ +------+ 429 \ \ <> R8 <>--- 430 \ \ +------+ 431 \ \ 432 \ \ +------+ 433 \ <> R9 <>--- 434 \ +------+ 435 \ 436 \ +------+ 437 <> R10 <>--- 438 +------+ 440 Figure 2: Monitoring Network Graph 442 Each monitoring point is characterized by the packet counter that 443 refers only to a marking period of the monitored flow. Also, it is 444 assumed that there be a monitoring point at all possible egress 445 points of the multipoint monitored network. 447 The same is also applicable for the delay, but it will be described 448 in the following sections. 450 The rest of the document assumes that the traffic is going from left 451 to right in order to simplify the explanation. But the analysis done 452 for one direction applies equally to all directions. 454 5. Multipoint Packet Loss 456 Since all the packets of the considered flow leaving the network have 457 previously entered the network, the number of packets counted by all 458 the input nodes is always greater than, or equal to, the number of 459 packets counted by all the output nodes. Noninitial fragments are 460 not considered here. 462 The assumption is the use of the Alternate-Marking method. In the 463 case of no packet loss occurring in the marking period, if all the 464 input and output points of the network domain to be monitored are 465 measurement points, the sum of the number of packets on all the 466 ingress interfaces equals the number on egress interfaces for the 467 monitored flow. In this circumstance, if no packet loss occurs, the 468 intermediate measurement points only have the task of splitting the 469 measurement. 471 It is possible to define the Network Packet Loss of one monitored 472 flow for a single period. In a packet network, the number of lost 473 packets is the number of packets counted by the input nodes minus the 474 number of packets counted by the output nodes. This is true for 475 every packet flow in each marking period. 477 The monitored network packet loss with n input nodes and m output 478 nodes is given by: 480 PL = (PI1 + PI2 +...+ PIn) - (PO1 + PO2 +...+ POm) 482 where: 484 PL is the network packet loss (number of lost packets) 486 PIi is the number of packets flowed through the i-th input node in 487 this period 489 POj is the number of packets flowed through the j-th output node in 490 this period 492 The equation is applied on a per-time-interval basis and a per-flow 493 basis: 495 The reference interval is the Alternate-Marking period, as defined 496 in [I-D.fioccola-rfc8321bis]. 498 The flow definition is generalized here. Indeed, as described 499 before, a multipoint packet flow is considered, and the 500 identification fields can be selected without any constraints. 502 6. Network Clustering 504 The previous equation can determine the number of packets lost 505 globally in the monitored network, exploiting only the data provided 506 by the counters in the input and output nodes. 508 In addition, it is also possible to leverage the data provided by the 509 other counters in the network to converge on the smallest 510 identifiable subnetworks where the losses occur. These subnetworks 511 are named "clusters". 513 A cluster graph is a subnetwork of the entire monitoring network 514 graph that still satisfies the packet loss equation (introduced in 515 the previous section), where PL in this case is the number of packets 516 lost in the cluster. As for the entire monitoring network graph, the 517 cluster is defined on a per-flow basis. 519 For this reason, a cluster should contain all the arcs emanating from 520 its input nodes and all the arcs terminating at its output nodes. 521 This ensures that we can count all the packets (and only those) 522 exiting an input node again at the output node, whatever path they 523 follow. 525 In a completely monitored unidirectional network (a network where 526 every network interface is monitored), each network device 527 corresponds to a cluster, and each physical link corresponds to two 528 clusters (one for each device). 530 Clusters can have different sizes depending on the flow-filtering 531 criteria adopted. 533 Moreover, sometimes clusters can be optionally simplified. For 534 example, when two monitored interfaces are divided by a single router 535 (one is the input interface, the other is the output interface, and 536 the router has only these two interfaces), instead of counting 537 exactly twice, upon entering and leaving, it is possible to consider 538 a single measurement point. In this case, we do not care about the 539 internal packet loss of the router. 541 It is worth highlighting that it might also be convenient to define 542 clusters based on the topological information so that they are 543 applicable to all the possible flows in the monitored network. 545 6.1. Algorithm for Clusters Partition 547 A simple algorithm can be applied in order to split our monitoring 548 network into clusters. This can be done for each direction 549 separately. The clusters partition is based on the monitoring 550 network graph, which can be valid for a specific flow or can also be 551 general and valid for the entire network topology. 553 It is a two-step algorithm: 555 o Group the links where there is the same starting node; 557 o Join the grouped links with at least one ending node in common. 559 Considering that the links are unidirectional, the first step implies 560 listing all the links as connections between two nodes and grouping 561 the different links if they have the same starting node. Note that 562 it is possible to start from any link, and the procedure will work. 563 Following this classification, the second step implies eventually 564 joining the groups classified in the first step by looking at the 565 ending nodes. If different groups have at least one common ending 566 node, they are put together and belong to the same set. After the 567 application of the two steps of the algorithm, each one of the 568 composed sets of links, together with the endpoint nodes, constitutes 569 a cluster. 571 In our monitoring network graph example, it is possible to identify 572 the clusters partition by applying this two-step algorithm. 574 The first step identifies the following groups: 576 1. Group 1: (R1-R2), (R1-R3), (R1-R10) 578 2. Group 2: (R2-R4), (R2-R5) 580 3. Group 3: (R3-R5), (R3-R9) 582 4. Group 4: (R4-R6), (R4-R7) 584 5. Group 5: (R5-R8) 586 And then, the second step builds the clusters partition (in 587 particular, we can underline that Groups 2 and 3 connect together, 588 since R5 is in common): 590 1. Cluster 1: (R1-R2), (R1-R3), (R1-R10) 592 2. Cluster 2: (R2-R4), (R2-R5), (R3-R5), (R3-R9) 594 3. Cluster 3: (R4-R6), (R4-R7) 596 4. Cluster 4: (R5-R8) 598 The flow direction here considered is from left to right. For the 599 opposite direction, the same reasoning can be applied, and in this 600 example, you get the same clusters partition. 602 In the end, the following 4 clusters are obtained: 604 Cluster 1 605 +------+ 606 <> R2 <>--- 607 / +------+ 609 / 610 +------+ / +------+ 611 ---<> R1 <>---<> R3 <>--- 612 +------+ \ +------+ 613 \ 614 \ 615 \ 616 \ 617 \ 618 \ 619 \ 620 \ 621 \ +------+ 622 <> R10 <>--- 623 +------+ 625 Cluster 2 626 +------+ +------+ 627 ---<> R2 <>---<> R4 <>--- 628 +------+ \ +------+ 629 \ 630 +------+ \ +------+ 631 ---<> R3 <>---<> R5 <>--- 632 +------+ \ +------+ 633 \ 634 \ 635 \ 636 \ 637 \ +------+ 638 <> R9 <>--- 639 +------+ 641 Cluster 3 642 +------+ 643 <> R6 <>--- 644 / +------+ 645 +------+ / 646 ---<> R4 <> 647 +------+ \ 648 \ +------+ 649 <> R7 <>--- 650 +------+ 652 Cluster 4 653 +------+ 655 ---<> R5 <> 656 +------+ \ 657 \ +------+ 658 <> R8 <>--- 659 +------+ 661 Figure 3: Clusters Example 663 There are clusters with more than two nodes as well as two-node 664 clusters. In the two-node clusters, the loss is on the link (Cluster 665 4). In more-than-two-node clusters, the loss is on the cluster, but 666 we cannot know in which link (Cluster 1, 2, or 3). 668 In this way, the calculation of packet loss can be made on a cluster 669 basis. Note that the packet counters for each marking period permit 670 calculating the packet rate on a cluster basis, so Committed 671 Information Rate (CIR) and Excess Information Rate (EIR) could also 672 be deduced on a cluster basis. 674 Obviously, by combining some clusters in a new connected subnetwork 675 (called a "super cluster"), the packet-loss rule is still true. 677 In this way, in a very large network, there is no need to configure 678 detailed filter criteria to inspect the traffic. You can check a 679 multipoint network and, in case of problems, go deep with a step-by- 680 step cluster analysis, but only for the cluster or combination of 681 clusters where the problem happens. 683 In summary, once a flow is defined, the algorithm to build the 684 clusters partition is based on topological information; therefore, it 685 considers all the possible links and nodes crossed by the given flow, 686 even if there is no traffic. So, if the flow does not enter or 687 traverse all the nodes, the counters have a nonzero value for the 688 involved nodes and a zero value for the other nodes without traffic; 689 but in the end, all the formulas are still valid. 691 The algorithm described above is an iterative clustering algorithm, 692 but it is also possible to apply a recursive clustering algorithm by 693 using the node-node adjacency matrix representation 694 [IEEE-ACM-ToN-MPNPM]. 696 The complete and mathematical analysis of the possible algorithms for 697 clusters partition, including the considerations in terms of 698 efficiency and a comparison between the different methods, is in the 699 paper [IEEE-ACM-ToN-MPNPM]. 701 7. Timing Aspects 703 It is important to consider the timing aspects, since out-of-order 704 packets happen and have to be handled as well, as described in 705 [I-D.fioccola-rfc8321bis]. 707 However, in a multisource situation, an additional issue has to be 708 considered. With multipoint path, the egress nodes will receive 709 alternate marked packets in random order from different ingress 710 nodes, and this must not affect the measurement. 712 So, if we analyze a multipoint-to-multipoint path with more than one 713 marking node, it is important to recognize the reference measurement 714 interval. In general, the measurement interval for describing the 715 results is the interval of the marking node that is more aligned with 716 the start of the measurement, as reported in Figure 4. 718 Note that the mark switching approach based on a fixed timer is 719 considered in this document. 721 time -> start stop 722 T(R1) |-------------| 723 T(R2) |-------------| 724 T(R3) |------------| 726 Figure 4: Measurement Interval 728 In Figure 4, it is assumed that the node with the earliest clock (R1) 729 identifies the right starting and ending times of the measurement, 730 but it is just an assumption, and other possibilities could occur. 731 So, in this case, T(R1) is the measurement interval, and its 732 recognition is essential in order to make comparisons with other 733 active/passive/hybrid Packet Loss metrics. 735 Regarding the timing constraints of the methodology, 736 [I-D.fioccola-rfc8321bis] already describes two contributions that 737 are taken into account: the clock error between network devices and 738 the network delay between the measurement points. 740 When we expand to a multipoint environment, we have to consider that 741 there are more marking nodes that mark the traffic based on 742 synchronized clock time. But, due to different synchronization 743 issues that may happen, the marking batches can be of different 744 lengths and with different offsets when they get mixed in a 745 multipoint flow. The additional gap that results between the sources 746 can be incorporated into A, which is the maximum clock skew between 747 the network devices, as already defined in [I-D.fioccola-rfc8321bis]. 749 ...BBBBBBBBB | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA | BBBBBBBBB... 750 |<======================================>| 751 | L | 752 ...=========>|<==================><==================>|<==========... 753 | L/2 L/2 | 754 |<====>| |<====>| 755 d | | d 756 |<========================>| 757 available counting interval 759 Figure 5: Timing Aspects 761 Moreover, it is assumed that each path of the multipoint flow can 762 still be represented with a distinct normal distribution. So, for 763 the aggregate multipoint path, the combination of normal 764 distributions result in a new normal distribution. Under this 765 assumption, the definition of the guard band d is still applicable as 766 defined in [I-D.fioccola-rfc8321bis] and is given by: 768 d = A + D_avg + 3*D_stddev, 770 where A is the clock accuracy, D_avg is the average value of the 771 network delay, and D_stddev is the standard deviation of the delay. 773 As shown in Figure 5 and according to [I-D.fioccola-rfc8321bis], the 774 condition that must be satisfied to enable the method to function 775 properly is that the available counting interval must be > 0, and 776 that means: 778 L - 2d > 0. 780 This formula needs to be verified for each measurement point on the 781 multipoint path. 783 Note that the timing considerations are valid for both packet loss 784 and delay measurements. 786 8. Multipoint Delay and Delay Variation 788 The same line of reasoning can be applied to delay and delay 789 variation. Similarly to the delay measurements defined in 790 [I-D.fioccola-rfc8321bis], the marking batches anchor the samples to 791 a particular period, and this is the time reference that can be used. 793 It is important to highlight that both delay and delay-variation 794 measurements make sense in a multipoint path. The delay variation is 795 calculated by considering the same packets selected for measuring the 796 delay. 798 In general, it is possible to perform delay and delay-variation 799 measurements on the basis of multipoint paths or single packets: 801 o Delay measurements on the basis of multipoint paths mean that the 802 delay value is representative of an entire multipoint path (e.g., 803 the whole multipoint network, a cluster, or a combination of 804 clusters). 806 o Delay measurements on a single-packet basis mean that you can use 807 a multipoint path just to easily couple packets between input and 808 output nodes of a multipoint path, as described in the following 809 sections. 811 8.1. Delay Measurements on a Multipoint-Paths Basis 813 8.1.1. Single-Marking Measurement 815 Mean delay and mean delay-variation measurements can also be 816 generalized to the case of multipoint flows. It is possible to 817 compute the average one-way delay of packets in one block, a cluster, 818 or the entire monitored network. 820 The average latency can be measured as the difference between the 821 weighted averages of the mean timestamps of the sets of output and 822 input nodes. This means that, in the calculation, it is possible to 823 weigh the timestamps by considering the number of packets for each 824 endpoints. 826 Note that, since the one-way delay value is representative of a 827 multipoint path, it is possible to calculate the two-way delay of a 828 multipoint path by summing the one-way delays of the two directions, 829 similarly to [I-D.fioccola-rfc8321bis]. 831 8.2. Delay Measurements on a Single-Packet Basis 833 8.2.1. Single- and Double-Marking Measurement 835 Delay and delay-variation measurements relative to only one picked 836 packet per period (both single and double marked) can be performed in 837 the multipoint scenario, with some limitations: 839 Single marking based on the first/last packet of the interval 840 would not work, because it would not be possible to agree on the 841 first packet of the interval. 843 Double marking or multiplexed marking would work, but each 844 measurement would only give information about the delay of a 845 single path. However, by repeating the measurement multiple 846 times, it is possible to get information about all the paths in 847 the multipoint flow. This can be done in the case of a point-to- 848 multipoint path, but it is more difficult to achieve in the case 849 of a multipoint-to-multipoint path because of the multiple source 850 routers. 852 If we would perform a delay measurement for more than one picked 853 packet in the same marking period, and especially if we want to get 854 delay measurements on a multipoint-to-multipoint basis, neither the 855 single- nor the double-marking method is useful in the multipoint 856 scenario, since they would not be representative of the entire flow. 857 The packets can follow different paths with various delays, and in 858 general it can be very difficult to recognize marked packets in a 859 multipoint-to-multipoint path, especially in the case when there is 860 more than one per period. 862 A desirable option is to monitor simultaneously all the paths of a 863 multipoint path in the same marking period; for this purpose, hashing 864 can be used, as reported in the next section. 866 Note that, since the one-way delay measurement is done on a single- 867 packet basis, it is always possible to calculate the two-way delay 868 but it is not immediate since it is necessary to couple the 869 measurement on each single path with the opposite direction. In this 870 case the NMS can do the calculation. 872 8.2.2. Hashing Selection Method 874 RFCs 5474 [RFC5474] and 5475 [RFC5475] introduce sampling and 875 filtering techniques for IP packet selection. 877 The hash-based selection methodologies for delay measurement can work 878 in a multipoint-to-multipoint path and MAY be used either coupled to 879 mean delay or stand-alone. 881 [I-D.mizrahi-ippm-marking] introduces how to use the hash method (RFC 882 5474 [RFC5474] and RFC 5475 [RFC5475]) combined with the Alternate- 883 Marking method for point-to-point flows. It is also called Mixed 884 Hashed Marking: the coupling of a marking method and hashing 885 technique is very useful, because the marking batches anchor the 886 samples selected with hashing, and this simplifies the correlation of 887 the hashing packets along the path. 889 It is possible to use a basic-hash or a dynamic-hash method. One of 890 the challenges of the basic approach is that the frequency of the 891 sampled packets may vary considerably. For this reason, the dynamic 892 approach has been introduced for point-to-point flows in order to 893 have the desired and almost fixed number of samples for each 894 measurement period. Using the hash-based sampling, the number of 895 samples may vary a lot because it depends on the packet rate that is 896 variable. The dynamic approach helps to have an almost fixed number 897 of samples for each marking period, and this is a better option for 898 making regular measurements over time. In the hash-based sampling, 899 Alternate Marking is used to create periods, so that hash-based 900 samples are divided into batches, which allows anchoring the selected 901 samples to their period. Moreover, in the dynamic hash-based 902 sampling, by dynamically adapting the length of the hash value, the 903 number of samples is bounded in each marking period. 905 In a multipoint environment, the hashing selection MAY be the 906 solution for performing delay measurements on specific packets and 907 overcoming the single- and double-marking limitations. 909 9. Results of the Multipoint Alternate Marking Experiment 911 The methodology described in the previous sections can be applied to 912 various performance measurement problems, as also explained in 913 [I-D.fioccola-rfc8321bis]. 915 Either one or two flag bits might be available for marking in 916 different deployments: 918 One flag: packet loss measurement SHOULD be done as described in 919 Section 5 by applying the network clustering partition described 920 in Section 6. While delay measurement MAY be done according to 921 the Mean delay calculation representative of the multipoint path, 922 as described in Section 8.1.1. Single-marking method based on the 923 first/last packet of the interval cannot be applied, as mentioned 924 in Section 8.2.1. 926 Two flags: packet loss measurement SHOULD be done as described in 927 Section 5 by applying the network clustering partition described 928 in Section 6. While delay measurement SHOULD be done on a single 929 packet basis according to double-marking method Section 8.2.1. In 930 this case the Mean delay calculation (Section 8.1.1) MAY also be 931 used as a representative value of a multipoint path. 933 One flag and hash-based selection: packet loss measurement SHOULD 934 be done as described in Section 5 by applying the network 935 clustering partition described in Section 6. Hash-based selection 936 methodologies, introduced in Section 8.2.2, MAY be used for delay 937 measurement. 939 The experiment with Multipoint Alternate Marking methodologies 940 confirmed the benefits of the Alternate Marking methodology described 941 in [I-D.fioccola-rfc8321bis], as its extension to the general case of 942 multipoint-to-multipoint scenarios. 944 The Multipoint Alternate Marking Method is RECOMMENDED only for 945 controlled domains, as per [I-D.fioccola-rfc8321bis]. 947 10. A Closed-Loop Performance-Management Approach 949 The Multipoint Alternate-Marking framework that is introduced in this 950 document adds flexibility to Performance Management (PM), because it 951 can reduce the order of magnitude of the packet counters. This 952 allows an SDN orchestrator to supervise, control, and manage PM in 953 large networks. 955 The monitoring network can be considered as a whole or split into 956 clusters that are the smallest subnetworks (group-to-group segments), 957 maintaining the packet-loss property for each subnetwork. The 958 clusters can also be combined in new, connected subnetworks at 959 different levels, depending on the detail we want to achieve. 961 An SDN controller or a Network Management System (NMS) can calibrate 962 performance measurements, since they are aware of the network 963 topology. They can start without examining in depth. In case of 964 necessity (packet loss is measured or the delay is too high), the 965 filtering criteria could be immediately reconfigured in order to 966 perform a partition of the network by using clusters and/or different 967 combinations of clusters. In this way, the problem can be localized 968 in a specific cluster or a single combination of clusters, and a more 969 detailed analysis can be performed step by step by successive 970 approximation up to a point-to-point flow detailed analysis. This is 971 the so-called "closed loop". 973 This approach can be called "network zooming" and can be performed in 974 two different ways: 976 1) change the traffic filter and select more detailed flows; 978 2) activate new measurement points by defining more specified 979 clusters. 981 The network-zooming approach implies that some filters or rules are 982 changed and that therefore there is a transient time to wait once the 983 new network configuration takes effect. This time can be determined 984 by the Network Orchestrator/Controller, based on the network 985 conditions. 987 For example, if the network zooming identifies the performance 988 problem for the traffic coming from a specific source, we need to 989 recognize the marked signal from this specific source node and its 990 relative path. For this purpose, we can activate all the available 991 measurement points and better specify the flow filter criteria (i.e., 992 5-tuple). As an alternative, it can be enough to select packets from 993 the specific source for delay measurements; in this case, it is 994 possible to apply the hashing technique, as mentioned in the previous 995 sections. 997 [I-D.song-opsawg-ifit-framework] defines an architecture where the 998 centralized Data Collector and Network Management can apply the 999 intelligent and flexible Alternate-Marking algorithm as previously 1000 described. 1002 As for [I-D.fioccola-rfc8321bis], it is possible to classify the 1003 traffic and mark a portion of the total traffic. For each period, 1004 the packet rate and bandwidth are calculated from the number of 1005 packets. In this way, the network orchestrator becomes aware if the 1006 traffic rate surpasses limits. In addition, more precision can be 1007 obtained by reducing the marking period; indeed, some implementations 1008 use a marking period of 1 sec or less. 1010 In addition, an SDN controller could also collect the measurement 1011 history. 1013 It is important to mention that the Multipoint Alternate Marking 1014 framework also helps Traffic Visualization. Indeed, this methodology 1015 is very useful for identifying which path or cluster is crossed by 1016 the flow. 1018 11. Security Considerations 1020 This document specifies a method of performing measurements that does 1021 not directly affect Internet security or applications that run on the 1022 Internet. However, implementation of this method must be mindful of 1023 security and privacy concerns, as explained in 1024 [I-D.fioccola-rfc8321bis]. 1026 12. IANA Considerations 1028 This document has no IANA actions. 1030 13. Contributors 1032 Greg Mirsky 1033 Ericsson 1034 Email: gregimirsky@gmail.com 1036 Tal Mizrahi 1037 Huawei Technologies 1038 Email: tal.mizrahi.phd@gmail.com 1040 Xiao Min 1041 ZTE Corp. 1042 Email: xiao.min2@zte.com.cn 1044 14. Acknowledgements 1046 The authors would like to thank Martin Duke and Tommy Pauly for their 1047 assistance and their detailed and precious reviews. 1049 15. References 1051 15.1. Normative References 1053 [I-D.fioccola-rfc8321bis] 1054 Fioccola, G., Cociglio, M., Mirsky, G., Mizrahi, T., Zhou, 1055 T., and X. Min, "Alternate-Marking Method", draft- 1056 fioccola-rfc8321bis-02 (work in progress), February 2022. 1058 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1059 Requirement Levels", BCP 14, RFC 2119, 1060 DOI 10.17487/RFC2119, March 1997, 1061 . 1063 [RFC5474] Duffield, N., Ed., Chiou, D., Claise, B., Greenberg, A., 1064 Grossglauser, M., and J. Rexford, "A Framework for Packet 1065 Selection and Reporting", RFC 5474, DOI 10.17487/RFC5474, 1066 March 2009, . 1068 [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. 1069 Raspall, "Sampling and Filtering Techniques for IP Packet 1070 Selection", RFC 5475, DOI 10.17487/RFC5475, March 2009, 1071 . 1073 [RFC5644] Stephan, E., Liang, L., and A. Morton, "IP Performance 1074 Metrics (IPPM): Spatial and Multicast", RFC 5644, 1075 DOI 10.17487/RFC5644, October 2009, 1076 . 1078 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 1079 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 1080 May 2017, . 1082 15.2. Informative References 1084 [I-D.ietf-ippm-route] 1085 Alvarez-Hamelin, J. I., Morton, A., Fabini, J., Pignataro, 1086 C., and R. Geib, "Advanced Unidirectional Route Assessment 1087 (AURA)", draft-ietf-ippm-route-10 (work in progress), 1088 August 2020. 1090 [I-D.mizrahi-ippm-marking] 1091 Mizrahi, T., Fioccola, G., Cociglio, M., Chen, M., and G. 1092 Mirsky, "Marking Methods for Performance Measurement", 1093 draft-mizrahi-ippm-marking-00 (work in progress), October 1094 2021. 1096 [I-D.song-opsawg-ifit-framework] 1097 Song, H., Qin, F., Chen, H., Jin, J., and J. Shin, "In- 1098 situ Flow Information Telemetry", draft-song-opsawg-ifit- 1099 framework-16 (work in progress), October 2021. 1101 [I-D.zhou-ippm-enhanced-alternate-marking] 1102 Zhou, T., Fioccola, G., Liu, Y., Lee, S., Cociglio, M., 1103 and W. Li, "Enhanced Alternate Marking Method", draft- 1104 zhou-ippm-enhanced-alternate-marking-08 (work in 1105 progress), January 2022. 1107 [IEEE-ACM-ToN-MPNPM] 1108 IEEE/ACM TRANSACTION ON NETWORKING, "Multipoint Passive 1109 Monitoring in Packet Networks", 1110 DOI 10.1109/TNET.2019.2950157, 2019. 1112 [IEEE-Network-PNPM] 1113 IEEE Network, "AM-PM: Efficient Network Telemetry using 1114 Alternate Marking", DOI 10.1109/MNET.2019.1800152, 2019. 1116 [RFC7011] Claise, B., Ed., Trammell, B., Ed., and P. Aitken, 1117 "Specification of the IP Flow Information Export (IPFIX) 1118 Protocol for the Exchange of Flow Information", STD 77, 1119 RFC 7011, DOI 10.17487/RFC7011, September 2013, 1120 . 1122 [RFC8889] Fioccola, G., Ed., Cociglio, M., Sapio, A., and R. Sisto, 1123 "Multipoint Alternate-Marking Method for Passive and 1124 Hybrid Performance Monitoring", RFC 8889, 1125 DOI 10.17487/RFC8889, August 2020, 1126 . 1128 Appendix A. Changes Log 1130 Changes from RFC 8889 include: 1132 o Minor editorial changes 1134 o Removed section on "Examples of application" 1136 Changes in v-(01) include: 1138 o Considerations on BUM traffic 1140 o Reference to RFC8321bis for the fragmentation part 1142 o Revised section on "Delay Measurements on a Single-Packet Basis" 1144 o Revised section on "Timing Aspects" 1146 Changes in v-(02) include: 1148 o Clarified the formula in the section on "Timing Aspects" to be 1149 aligned with RFC 8321 1151 o Considerations on two-way delay measurements in both sections 8.1 1152 and 8.2 on delay measurements 1154 o Clarified in section 4.1 on "Monitoring Network" that the 1155 description is done for one direction but it can easily be 1156 extended to all direction 1158 o New section on "Results of the Multipoint Alternate Marking 1159 Experiment" 1161 Authors' Addresses 1163 Giuseppe Fioccola (editor) 1164 Huawei Technologies 1165 Riesstrasse, 25 1166 Munich 80992 1167 Germany 1169 Email: giuseppe.fioccola@huawei.com 1170 Mauro Cociglio 1171 Telecom Italia 1172 Via Reiss Romoli, 274 1173 Torino 10148 1174 Italy 1176 Email: mauro.cociglio@telecomitalia.it 1178 Amedeo Sapio 1179 Intel Corporation 1180 4750 Patrick Henry Dr. 1181 Santa Clara, CA 95054 1182 USA 1184 Email: amedeo.sapio@intel.com 1186 Riccardo Sisto 1187 Politecnico di Torino 1188 Corso Duca degli Abruzzi, 24 1189 Torino 10129 1190 Italy 1192 Email: riccardo.sisto@polito.it 1194 Tianran Zhou 1195 Huawei Technologies 1196 156 Beiqing Rd. 1197 Beijing 100095 1198 China 1200 Email: zhoutianran@huawei.com