idnits 2.17.00 (12 Aug 2021) /tmp/idnits38320/draft-geib-ippm-metrictest-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** You're using the IETF Trust Provisions' Section 6.b License Notice from 12 Sep 2009 rather than the newer Notice from 28 Dec 2009. (See https://trustee.ietf.org/license-info/) Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- == It seems as if not all pages are separated by form feeds - found 0 form feeds but 25 pages Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (February 11, 2010) is 4475 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Unused Reference: 'RFC2680' is defined on line 848, but no explicit reference was found in the text == Unused Reference: 'RFC2681' is defined on line 851, but no explicit reference was found in the text ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) Summary: 3 errors (**), 0 flaws (~~), 4 warnings (==), 1 comment (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 1 Internet Engineering Task Force R. Geib, Ed. 2 Internet-Draft Deutsche Telekom 3 Intended status: Informational A. Morton 4 Expires: August 15, 2010 AT&T Labs 5 R. Fardid 6 Covad Communications 7 February 11, 2010 9 IPPM standard advancement testing 10 draft-geib-ippm-metrictest-02 12 Abstract 14 This document specifies tests to determine if multiple independent 15 instantiations of a performance metric RFC have implemented the 16 specifications in the same way. This is the performance metric 17 equivalent of interoperability, required to advance RFCs along the 18 standards track. Results from different implementations of metric 19 RFCs will be collected under the same underlying network conditions 20 and compared using state of the art statistical methods. The goal is 21 an evaluation of the metric RFC itself, whether its definitions are 22 clear and unambiguous to implementors and therefore a candidate for 23 advancement on the IETF standards track. 25 Status of this Memo 27 This Internet-Draft is submitted to IETF in full conformance with the 28 provisions of BCP 78 and BCP 79. 30 Internet-Drafts are working documents of the Internet Engineering 31 Task Force (IETF), its areas, and its working groups. Note that 32 other groups may also distribute working documents as Internet- 33 Drafts. 35 Internet-Drafts are draft documents valid for a maximum of six months 36 and may be updated, replaced, or obsoleted by other documents at any 37 time. It is inappropriate to use Internet-Drafts as reference 38 material or to cite them other than as "work in progress." 40 The list of current Internet-Drafts can be accessed at 41 http://www.ietf.org/ietf/1id-abstracts.txt. 43 The list of Internet-Draft Shadow Directories can be accessed at 44 http://www.ietf.org/shadow.html. 46 This Internet-Draft will expire on August 15, 2010. 48 Copyright Notice 50 Copyright (c) 2010 IETF Trust and the persons identified as the 51 document authors. All rights reserved. 53 This document is subject to BCP 78 and the IETF Trust's Legal 54 Provisions Relating to IETF Documents 55 (http://trustee.ietf.org/license-info) in effect on the date of 56 publication of this document. Please review these documents 57 carefully, as they describe your rights and restrictions with respect 58 to this document. Code Components extracted from this document must 59 include Simplified BSD License text as described in Section 4.e of 60 the Trust Legal Provisions and are provided without warranty as 61 described in the BSD License. 63 Table of Contents 65 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 66 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 5 67 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 5 68 3. Verification of conformance to a metric specification . . . . 7 69 3.1. Tests of an individual implementation against a metric 70 specification . . . . . . . . . . . . . . . . . . . . . . 7 71 3.2. Test setup resulting in identical live network testing 72 conditions . . . . . . . . . . . . . . . . . . . . . . . . 9 73 3.3. Tests of two or more different implementations against 74 a metric specification . . . . . . . . . . . . . . . . . . 12 75 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 13 76 3.5. Recommended Metric Verification Measurement Process . . . 14 77 3.6. Miscellaneous . . . . . . . . . . . . . . . . . . . . . . 17 78 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18 79 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 18 80 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18 81 7. Security Considerations . . . . . . . . . . . . . . . . . . . 18 82 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18 83 8.1. Normative References . . . . . . . . . . . . . . . . . . . 18 84 8.2. Informative References . . . . . . . . . . . . . . . . . . 19 85 Appendix A. An example on a One-way Delay metric validation . . . 20 86 A.1. Compliance to Metric specification requirements . . . . . 20 87 A.2. Examples related to statistical tests for One-way Delay . 22 88 Appendix B. Glossary . . . . . . . . . . . . . . . . . . . . . . 24 89 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24 91 1. Introduction 93 The Internet Standards Process RFC2026 [RFC2026] requires that for a 94 IETF specification to advance beyond the Proposed Standard level, at 95 least two genetically unrelated implementations must be shown to 96 interoperate correctly with all features and options. There are two 97 distinct reasons for this requirement. 99 In the case of a protocol specification, the notion of 100 "interoperability" is reasonably intuitive - the implementations must 101 successfully "talk to each other", while exercising all features and 102 options. To achieve interoperability, two implementors need to 103 interpret the protocol specifications in equivalent ways. In the 104 case of IP Performance Metrics (IPPM), this definition of 105 interoperability is only useful for test and control protocols like 106 OWAMP [RFC4656] and TWAMP [RFC5357]. 108 A metric specification RFC describes one or more metric definitions, 109 methods of measurement and a way to report the results of 110 measurement. One example would be a way to test and report the One- 111 way Delay that data packets incur while being sent from one network 112 location to another, One-way Delay Metric. 114 In the case of metric specifications, exactly what constitutes 115 "interoperation" is less obvious. The IETF has not yet agreed on how 116 to judge metric specification "interoperability" in the context of 117 the IETF Standards Process. This draft provides methods which should 118 be suitable to evaluate metric specifications for standards track 119 advancement. The methods proposed here MAY be generally applicable 120 to metric specification RFCs beyond those developed under the IPPM 121 Framework [RFC2330]. 123 Since many implementations of IP metrics are embedded in measurement 124 systems that do not interact with one another (they were built before 125 OWAMP and TWAMP), the interoperability evaluation called for in the 126 IETF standards process cannot be determined by observing that 127 independent implementations interact properly for various protocol 128 exchanges. Instead, verifying that different implementations give 129 statistically equivalent results under controlled measurement 130 conditions takes the place of interoperability observations. Even 131 when evaluating OWAMP and TWAMP RFCs for standards track advancement, 132 the methods described here are useful to evaluate the measurement 133 results because they have little value otherwise. 135 The standards advancement process aims at producing confidence that 136 the metric definitions and supporting material are clearly worded and 137 unambiguous, or reveals ways in which the metric definitions can be 138 revised to achieve clarity. The process also permits identification 139 of options that were not implemented, so that they can be removed 140 from the advancing specification. Thus, the product of this process 141 is information about the metric specification RFC itself: 142 determination of the specifications or definitions that are clear and 143 unambiguous and those that are not (as opposed to an evaluation of 144 the implementations which assist in the process). 146 This document defines a process to verify that implementations (or 147 practically, measurement systems) have interpreted the metric 148 specifications in equivalent ways, and produce equivalent results. 150 Testing for statistical equivalence requires ensuring identical test 151 setups (or awareness of differences) to the best possible extent. 152 Thus, producing identical test conditions is a core requirement. 153 Another important aspect of this process is to test individual 154 implementations against specific requirements in the metric 155 specifications using customized tests for each requirement. These 156 tests can distinguish equivalent interpretations of each specific 157 requirement. 159 Conclusions on equivalence are reached by two measures. 161 First, implementations are compared against individual metric 162 specifications to make sure that differences in implementation are 163 minimised or at least known. 165 Second, a test setup is proposed ensuring identical networking 166 conditions so that unknowns are minimized and comparisons are 167 simplified. The resulting separate data sets may be seen as samples 168 taken from the same underlying distribution. Using state of the art 169 statistical methods, the equivalence of the results is verified. To 170 illustrate application of the process and methods defined here, 171 evaluation of the One-way Delay Metric [RFC2679] is provided in an 172 Appendix. While test setups will vary with the metrics to be 173 validated, the general methodology of determining equivalent results 174 will not. Documents defining test setups to evaluate other metrics 175 should be developed once the process proposed here has been agreed 176 and approved. 178 Changes from -01 to -02 version 180 o Major editorial review, rewording and clarifications on all 181 contents. 183 o Additional text on parrallel testing using VLANs and GRE or 184 Pseudowire tunnels. 186 o Additional examples and a glossary. 188 Changes from -00 to -01 version 190 o Addition of a comparison of individual metric implementations 191 against the metric specification (trying to pick up problems and 192 solutions for metric advancement [morton-advance-metrics]). 194 o More emphasis on the requirement to carefully design and document 195 the measurement setup of the metric comparison. 197 o Proposal of testing conditions under identical WAN network 198 conditions using IP in IP tunneling or Pseudo Wires and parallel 199 measurement streams. 201 o Proposing the requirement to document the smallest resolution at 202 which an ADK test was passed by 95%. As no minimum resolution is 203 specified, IPPM metric compliance is not linked to a particular 204 performance of an implementation. 206 o Reference to RFC 2330 and RFC 2679 for the 95% confidence interval 207 as preferred criterion to decide on statistical equivalence 209 o Reducing the proposed statistical test to ADK with 95% confidence. 211 1.1. Requirements Language 213 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 214 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 215 document are to be interpreted as described in RFC 2119 [RFC2119]. 217 2. Basic idea 219 The implementation of a standard compliant metric is expected to meet 220 the requirements of the related metric specification. So before 221 comparing two metric implementations, each metric implementation is 222 individually compared against the metric specification. 224 Most metric specifications leave freedom to implementors on non- 225 fundamental aspects of an individual metric (or options). Comparing 226 different measurement results using a statistical test with the 227 assumption of identical test path and testing conditions requires 228 knowledge of all differences in the overall test setup. Metric 229 specification options chosen by implementors have to be documented. 230 It is REQUIRED to use identical implementation options wherever 231 possible for any test proposed here. Calibrations proposed by metric 232 standards should be performed to further identify (and possibly 233 reduce) potential sources of errors in the test setup. 235 The Framework for IP Performance Metrics [RFC2330] expects that a 236 "methodology for a metric should have the property that it is 237 repeatable: if the methodology is used multiple times under identical 238 conditions, it should result in consistent measurements." This means 239 an implementation is expected to repeatedly measure a metric with 240 consistent results (repeatability with the same result). Small 241 deviations in the test setup are expected to lead to small deviations 242 in results only. To characterise statistical equivalence in the case 243 of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% 244 confidence interval. Quoting RFC 2679, "95 percent was chosen 245 because ... a particular confidence level should be specified so that 246 the results of independent implementations can be compared." 248 Two different implementations are expected to produce statistically 249 equivalent results if they both measure a metric under the same 250 networking conditions. Formulating in statistical terms: separate 251 metric implementations collect separate samples from the same 252 underlying statistical process (the same network conditions). The 253 statistical hypothesis to be tested is the expectation that both 254 samples do not expose statistically different properties. This 255 requires careful test design: 257 o The measurement test setup must be self-consistent to the largest 258 possible extent. To minimize the influence of the test and 259 measurement setup on the result, network conditions and paths MUST 260 be identical for the compared implementations to the largest 261 possible degree. This includes both the stability and non- 262 ambiguity of routes taken by the measurement packets. See RFC 263 2330 for a discussion on self-consistency. 265 o The error induced by the sample size must be small enough to 266 minimize its influence on the test result. This may have to be 267 respected, especially if two implementations measure with 268 different average probing rates. 270 o Every comparison must be repeated several times based on different 271 measurement data to avoid random indications of compatibility (or 272 the lack of it). 274 o To minimize the influence of implementation options on the result, 275 metric implementations SHOULD use identical options and parameters 276 for the metric under evaluation. 278 o The implementation with the lowest probing frequency determines 279 the smallest temporal interval for which samples can be compared. 281 The metric specifications themselves are the primary focus of 282 evaluation, rather than the implementations of metrics. The 283 documentation produced by the advancement process should identify 284 which metric definitions and supporting material were found to be 285 clearly worded and unambiguous, OR, it should identify ways in which 286 the metric specification text should be revised to achieve clarity 287 and unified interpretation. 289 The process should also permit identification of options that were 290 not implemented, so that they can be removed from the advancing 291 specification (this is an aspect more typical of protocol advancement 292 along the standards track). 294 Note that this document does not propose to base interoperability 295 indications of performance metric implementations on comparisons of 296 individual singletons. Individual singletons may be impacted by many 297 statistical effects while they are measured. Comparing two 298 singletons of different implementations may result in failures with 299 higher probability than comparing samples. 301 3. Verification of conformance to a metric specification 303 This section specifies how to verify compliance of two or more IPPM 304 implementations against a metric specification. This document only 305 proposes a general methodology. Compliance criteria to a specific 306 metric implementation need to be defined for each individual metric 307 specification. The only exception is the statistical test comparing 308 two metric implementations which are simultaneously tested. This 309 test is applicable without metric specific decision criteria. 311 3.1. Tests of an individual implementation against a metric 312 specification 314 A metric implementation MUST support the requirements classified as 315 "MUST" and "REQUIRED" of the related metric specification to be 316 compliant to the latter. 318 Further, supported options of a metric implementation SHOULD be 319 documented in sufficient detail. The documentation of chosen options 320 is RECOMMENDED to minimise (and recognise) differences in the test 321 setup if two metric implementations are compared. Further, this 322 documentation is used to validate and improve the underlying metric 323 specification option, to remove options which saw no implementation 324 or which are badly specified from the metric specification to be 325 promoted to a standard. This documentation SHOULD be made for all 326 implementation relevant specifications of a metric picked for a 327 comparison, which aren't explicitly marked as "MUST" or "REQUIRED" in 328 the metric specification. This applies for the following sections of 329 all metric specifications: 331 o Singleton Definition of the Metric. 333 o Sample Definition of the Metric. 335 o Statistics Definition of the Metric. As statistics are compared 336 by the test specified here, this documentation is required even in 337 the case, that the metric specification does not contain a 338 Statistics Definition. 340 o Timing and Synchronisation related specification (if relevant for 341 the Metric). 343 o Any other technical part present or missing in the metric 344 specification, which is relevant for the implementation of the 345 Metric. 347 RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric 348 implementations. A single IPPM conformant implementation MUST under 349 otherwise identical network conditions produce precise results for 350 repeated measurements of the same metric. 352 RFC 2330 prefers the "empirical distribution function" EDF to 353 describe collections of measurements. RFC 2330 determines, that 354 "unless otherwise stated, IPPM goodness-of-fit tests are done using 355 5% significance." The goodness of fit test determines by which 356 precision two or more samples of a metric implementation belong to 357 the same underlying distribution (of measured network performance 358 events). The goodness of fit test to be applied is the Anderson- 359 Darling K sample test (ADK sample test, K stands for the number of 360 samples to be compared) [ADK]. Please note that RFC 2330 and RFC 361 2679 apply an Anderson Darling goodness of fit test too. 363 The results of a repeated test with a single implementation MUST pass 364 an ADK sample test with confidence level of 95%. The resolution for 365 which the ADK test has been passed with the specified confidence 366 level MUST be documented. To formulate this differently: The 367 requirement is to document the smallest resolution, at which the 368 results of the tested metric implementation pass an ADK test with a 369 confidence level of 95%. The minimum resolution available in the 370 reported results from each implementation MUST be taken into account 371 in the ADK test. 373 3.2. Test setup resulting in identical live network testing conditions 375 Two major issues complicate tests for metric compliance across live 376 networks under identical testing conditions. One is the general 377 point that metric definition implementations cannot be conveniently 378 examined in field measurement scenarios. The other one is more 379 broadly described as "parallelism in devices and networks", including 380 mechanisms like those that achieve load balancing (see [RFC4818]). 382 This section proposes two measures to deal with both issues. 383 Tunneling mechanisms can be used to avoid parallel processing of 384 different flows in the network. Measuring by separate parallel probe 385 flows results in repeated collection of data. If both measures are 386 combined, WAN network conditions are identical for a number of 387 independent measurement flows, no matter what the network conditions 388 are in detail. 390 Any measurement setup MUST be made to avoid the probing traffic 391 itself to impede the metric measurement. The created measurement 392 load MUST NOT result in congestion at the access link connecting the 393 measurement implementation to the WAN. The created measurement load 394 MUST NOT overload the measurement implementation itself, eg. by 395 causing a high CPU load or by creating imprecisions due to internal 396 transmit (receive respectively) probe packet collisions. 398 IP in IP tunnels can be used to avoid Equal-Cost Multi-Path (ECMP) 399 routing of different measurement streams if they carry inner IP 400 packets from different senders in a single tunnel with the same outer 401 origin and destination address as well as the same port numbers. 403 >>>> Comment: The author is not an expert on tunneling and 404 appreciates guidance on the applicability of one or more of the 405 following protocols: IP in IP [RFC2003] or L2TP [RFC2661] or 406 [RFC3931].RFC 4928 [RFC4928] proposes measures how to avoid ECMP 407 treatment in MPLS networks. 409 By applying 802.1Q VLANs combined with an Ethernet port based tunnel 410 mechanism likeGeneric Routing Encapsulation (GRE) [RFC2784] or 411 Ethernet Pseudo Wires [RFC4448] the desired test environment can be 412 set up (see Figures 1 and 2). By this solution, Ethernet frames are 413 transmitted, containing the measurement packets. The IP packet size 414 of the metric implementation SHOULD be chosen small enough to avoid 415 fragmentation due to tghe added Ethernet and tunnel headers. 417 If tunneling is applied, two tunnels MUST carry all test traffic in 418 between the test site and the remote site. For example, if 802.1Q 419 Ethernet Virtual LANs (VLAN) are applied and the measurement streams 420 are carried in different VLANs, the GRE tunnel or Pseudo Wires 421 respectively MUST be set up in physical port mode to avoid set up of 422 Pseudo Wires per VLAN (which may see different paths due to ECMP 423 routing), see RFC 4448. The remote router and the Ethernet switch 424 shown in figure 2 must support 802.1Q in this set up. 426 The tunneled packets carry an overhead. To avoid fragmentation in 427 the Internet, it is suggested to limit the size of the test packets. 428 The following headers are added if VLANs and GRE tunnels are applied: 430 o Ethernet 802.1Q: 22 Byte. 432 o GRE Header: 8 Byte. 434 o IPv4 Header (outer IP header): 20 Byte. 436 o MPLS Labels may be added by a carrier. Each MPLS Label has a 437 length of 4 Bytes. By the time of writing, between 1 and 4 Labels 438 seems to be a fair guess of what's expectable. 440 Each test is repeated several times. WAN conditions may change over 441 time. Sequential testing is desirable, but may not be a useful 442 metric test option. However tests can be carried out by establishing 443 n different parallel measurement flows. Two or three linecards per 444 implementation serving to send or receive measurement flows should be 445 sufficient to create 5 or more parallel measurement flows. If three 446 linecards are used, each card sends and receives 2 flows. Other 447 options are to separate flows by DiffServ marks (without deploying 448 any QoS in the inner or outer tunnel) or using a single CBR flow and 449 evaluating every n-th singleton to belong to a specific measurement 450 flow. Tunneling setups like the one proposed by GRE encapsulated 451 multicast probing [GU+Duffield] should be applied (note that one or 452 more remote tunnel end points and the same number of additional 453 routers are required). 455 An illustration of a test setup with two tunnels and two flows 456 between two linecards of one implementation is given in Figure 1. 458 Implementation ,---. +--------+ 459 +~~~~~~~~~~~/ \~~~~~~| Remote | 460 +------->-----F2->-| / \ |->---+ | 461 | +---------+ | Tunnel 1( ) | | | 462 | | transmit|-F1->-| ( ) |->+ | | 463 | | LC1 | +~~~~~~~~~| |~~~~| | | | 464 | | receive |-<--+ ( ) | F1 F2 | 465 | +---------+ | |Internet | | | | | 466 *-------<-----+ F2 | | | | | | 467 +---------+ | | +~~~~~~~~~| |~~~~| | | | 468 | transmit|-* *-| | | |--+<-* | 469 | LC2 | | Tunnel 2( ) | | | 470 | receive |-<-F1-| \ / |<-* | 471 +---------+ +~~~~~~~~~~~\ /~~~~~~| Router | 472 `-+-' +--------+ 474 Illustration of a test setup with two tunnels and two flows F between 475 two linecards LC of one implementation. 477 Figure 1 479 Figure 2 shows the network elements required to set up GRE tunnels or 480 as shown by figure 1. 482 Implementation 484 +-----+ ,---. 485 | LC1 | / \ 486 +-----+ / \ +------+ 487 | +-------+ ( ) +-------+ |Remote| 488 +--------+ | | | | | | | | 489 |Ethernet| | Tunnel| |Internet | | Tunnel| | | 490 |Switch |--| Head |--| |--| Head |--| | 491 +--------+ | Router| | | | Router| | | 492 | | | ( ) | | |Router| 493 +-----+ +-------+ \ / +-------+ +------+ 494 | LC2 | \ / 495 +-----+ `-+-' 496 Illustration of a hardware setup to realise the test setup 497 illustrated by figure 1 with GRE tunnels or Pseudowires. 499 Figure 2 501 Some additional rules to calculate and compare samples have to be 502 respected to perform a metric test: 504 o To compare different probes of a common underlying distribution in 505 terms of metrics characterising a communication network requires 506 to respect the temporal nature for which the assumption of common 507 underlying distribution may hold. Any singletons or samples to be 508 compared MUST be captured within the same time interval. 510 o Whenever statistical events like singletons or rates are used to 511 characterise measured metrics of a time-interval, at least 5 512 singletons of a relevant metric SHOULD be present to ensure a 513 minimum confidence into the reported value (see Wikipedia on 514 confidence [Rule of thumb]). Note that this criterion also is to 515 be respected e.g. when comparing packet loss metrics. Any packet 516 loss measurement interval to be compared with the results of 517 another implementation SHOULD contain at least five lost packets 518 to have a minimum confidence that the observed loss rate wasn't 519 caused by a small number of random packet drops. 521 o The minimum number of singletons or samples to be compared by an 522 Anderson-Darling test SHOULD be 100 per tested metric 523 implementation. Note that the Anderson-Darling test detects small 524 differences in distributions fairly well and will fail for high 525 number of compared results (RFC2330 mentions an example with 8192 526 measurements where an Anderson-Darling test always failed). 528 o Generally, the Anderson-Darling test is sensitive to differences 529 in the accuracy or bias associated with varying implementations or 530 test conditions. These dissimilarities may result in differing 531 averages of samples to be compared. An example may be different 532 packet sizes, resulting in a constant delay difference between 533 compared samples. Therefore samples to be compared by an Anderson 534 Darling test MAY be calibrated by the difference of the average 535 values of the samples. Any calibration of this kind MUST be 536 documented in the test result. 538 3.3. Tests of two or more different implementations against a metric 539 specification 541 RFC2330 expects "a methodology for a given metric [to] exhibit 542 continuity if, for small variations in conditions, it results in 543 small variations in the resulting measurements. Slightly more 544 precisely, for every positive epsilon, there exists a positive delta, 545 such that if two sets of conditions are within delta of each other, 546 then the resulting measurements will be within epsilon of each 547 other." A small variation in conditions in the context of the metric 548 test proposed here can be seen as different implementations measuring 549 the same metric along the same path. 551 IPPM metric specification however allow for implementor options to 552 the largest possible degree. It can't be expected that two 553 implementors pick identical options for the implementations. 554 Implementors SHOULD to the highest degree possible pick the same 555 configurations for their systems when comparing their implementations 556 by a metric test. 558 In some cases, a goodness of fit test may not be possible or show 559 disappointing results. To clarify the difficulties arising from 560 different implementation options, the individual options picked for 561 every compared implementation SHOULD be documented in sufficient 562 detail. Based on this documentation, the underlying metric 563 specification should be improved before it is promoted to a standard. 565 The same statistical test as applicable to quantify precision of a 566 single metric implementation MUST be passed to compare metric 567 conformance of different implementations. To document compatibility, 568 the smallest measurement resolution at which the compared 569 implementations passed the ADK sample test MUST be documented. 571 For different implementations of the same metric, "variations in 572 conditions" are reasonably expected. The ADK test comparing samples 573 of the different implementations may result in a lower precision than 574 the test for precision of each implementation individually. 576 3.4. Clock synchronisation 578 Clock synchronization effects require special attention. Accuracy of 579 one-way active delay measurements for any metrics implementation 580 depends on clock synchronization between the source and destination 581 of tests. Ideally, one-way active delay measurement (RFC 2679, 582 [RFC2679]) test endpoints either have direct access to independent 583 GPS or CDMA-based time sources or indirect access to nearby NTP 584 primary (stratum 1) time sources, equipped with GPS receivers. 585 Access to these time sources may not be available at all test 586 locations associated with different Internet paths, for a variety of 587 reasons out of scope of this document. 589 When secondary (stratum 2 and above) time sources are used with NTP 590 running across the same network, whose metrics are subject to 591 comparative implementation tests, network impairments can affect 592 clock synchronization, distort sample one-way values and their 593 interval statistics. It is RECOMMENDED to discard sample one-way 594 delay values for any implementation, when one of the following 595 reliability conditions is met: 597 o Delay is measured and is finite in one direction, but not the 598 other. 600 o Absolute value of the difference between the sum of one-way 601 measurements in both directions and round-trip measurement is 602 greater than X% of the latter value. 604 Examination of the second condition requires RTT measurement for 605 reference, e.g., based on TWAMP (RFC5357, RFC 5357 [RFC5357]), in 606 conjunction with one-way delay measurement. 608 Specification of X% to strike a balance between identification of 609 unreliable one-way delay samples and misidentification of reliable 610 samples under a wide range of Internet path RTTs probably requires 611 further study. 613 An IPPM compliant metric implementation whose measurement requires 614 synchronized clocks is however expected to provide precise 615 measurement results. Any IPPM metric implementation MUST be of a 616 precision of 1 ms (+/- 500 us) with a confidence of 95% if the metric 617 is captured along an Internet path which is stable and not congested 618 during a measurement duration of an hour or more. [Editor: this 619 latter definition may avoid NTP (stratum 2 or worse) synchronized 620 IPPM implementations from becoming IPPM compliant. However internal 621 PC clock synched implementations can't be rejected that way. 623 >>>> Comment: Ideas on criteria to deal with the latter are welcome. 624 May drift be one, as GPS synched implementations shouldn't have one 625 or the same on origin and destination, respectively.] 627 3.5. Recommended Metric Verification Measurement Process 629 In order to meet their obligations under the IETF Standards Process 630 the IESG must be convinced that each metric specification advanced to 631 Draft Standard or Internet Standard status is clearly written, that 632 there are the required multiple verifiably equivalent 633 implementations, and that all options have been implemented. 635 In the context of this document, metrics are designed to measure some 636 characteristic of a data network. An aim of any metric definition 637 should be that it should be specified in a way that can reliably 638 measure the specific characteristic in a repeatable way. 640 Each metric, statistic or option of those to be validated MUST be 641 compared against a reference measurement or another implementation by 642 at least 5 different basic data sets, each one with sufficient size 643 to reach the specified level of confidence, as specified by this 644 document. 646 Finally, the metric definitions, embodied in the text of the RFCs, 647 are the objects that require evaluation and possible revision in 648 order to advance to the next step on the standards track. 650 IF two (or more) implementations do not measure an equivalent metric 651 as specified by this document, 653 AND sources of measurement error do not adequately explain the lack 654 of agreement, 656 THEN the details of each implementation should be audited along with 657 the exact definition text, to determine if there is a lack of clarity 658 that has caused the implementations to vary in a way that affects the 659 correspondence of the results. 661 IF there was a lack of clarity or multiple legitimate interpretations 662 of the definition text, 664 THEN the text should be modified and the resulting memo proposed for 665 consensus and advancement along the standards track. 667 The complete process of advancing a metric specification to a 668 standard as defined by this document is illustrated in Figure 3. 670 ,---. 671 / \ 672 ( Start ) 673 \ / Implementations 674 `-+-' +-------+ 675 | /| 1 `. 676 +---+----+ / +-------+ `.-----------+ ,-------. 677 | RFC | / |Check for | ,' was RFC `. YES 678 | | / |Equivalence.... clause x -----+ 679 | |/ +-------+ |under | `. clear? ,' | 680 | Metric \.....| 2 ....relevant | `---+---' +----+---+ 681 | Metric |\ +-------+ |identical | No | |Advance | 682 | Metric | \ |network | +---+---.--+spec | 683 | ... | \ |conditions | |Modify | +----+---+ 684 | | \ +-------+ | | |Spec | | 685 +--------+ \| n |.'+-----------+ +-------+ +--+-+ 686 +-------+ |DONE| 687 +----+ 689 Illustration of the metric standardisation process 691 Figure 3 693 Any recommendation for the advancement of a metric specification MUST 694 be accompanied by an implementation report, as is the case with all 695 requests for the advancement of IETF specifications. The 696 implementation report needs to include the tests performed, the 697 applied test setup, the specific metrics in the RFC and reports of 698 the tests performed with two or more implementations. The test plan 699 needs to specify the precision reached for each measured metric and 700 thus define the meaning of "statistically equivalent" for the 701 specific metrics being tested. 703 Ideally, the test plan would co-evolve with the development of the 704 metric, since that's when people have the most context in their 705 thinking regarding the different subtleties that can arise. 707 In particular, the implementation report MUST as a minimum document: 709 o The metric compared and the RFC specifying it. This includes 710 statements as required by the section "Tests of an individual 711 implementation against a metric specification" of this document. 713 o The measurement configuration and setup. 715 o A complete specification of the measurement stream (mean rate, 716 statistical distribution of packets, packet size or mean packet 717 size and their distribution), DSCP and any other measurement 718 stream properties which could result in deviating results. 719 Deviations in results can be caused also if chosen IP addresses 720 and ports of different implementations can result in different 721 layer 2 or layer 3 paths due to operation of Equal Cost Multi-Path 722 routing in an operational network. 724 o The duration of each measurement to be used for a metric 725 validation, the number of measurement points collected for each 726 metric during each measurement interval (i.e. the probe size) and 727 the level of confidence derived from this probe size for each 728 measurement interval. 730 o The result of the statistical tests performed for each metric 731 validation as required by the section "Tests of two or more 732 different implementations against a metric specification" of this 733 document. 735 o A parameterization of laboratory conditions and applied traffic 736 and network conditions allowing reproduction of these laboratory 737 conditions for readers of the implementation report. 739 o The documentation helping to improve metric specifications defined 740 by this section. 742 All of the tests for each set SHOULD be run in a test setup as 743 specified in the section "Test setup resulting in identical live 744 network testing conditions." 746 If a different test set up is chosen, it is RECOMMENDED to avoid 747 effects falsifying results of validation measurements caused by real 748 data networks (like parallelism in devices and networks). Data 749 networks may forward packets differently in the case of: 751 o Different packet sizes chosen for different metric 752 implementations. A proposed countermeasure is selecting the same 753 packet size when validating results of two samples or a sample 754 against an original distribution. 756 o Selection of differing IP addresses and ports used by different 757 metric implementations during metric validation tests. If ECMP is 758 applied on IP or MPLS level, different paths can result (note that 759 it may be impossible to detect an MPLS ECMP path from an IP 760 endpoint). A proposed counter measure is to connect the 761 measurement equipment to be compared by a NAT device, or 762 establishing a single tunnel to transport all measurement traffic 763 The aim is to have the same IP addresses and port for all 764 measurement packets or to avoid ECMP based local routing diversion 765 by using a layer 2 tunnel. 767 o Different IP options. 769 o Different DSCP. 771 o If the five measurements are captured by repeated measurements 772 instead of simultaneous ones: Changing pathes and load conditions 773 time. 775 3.6. Miscellaneous 777 In the case that a metric validation requires capturing rare events, 778 an impairment generator may have to be added to the test set up. 779 Inclusion of an impairment generator and the parameterisation of the 780 impairments generated MUST be documented. Rare events could be 781 packet duplications, packet loss rates above one digit percentages, 782 loss patterns or packet re-ordering and so on. 784 As specified above, 5 singletons are the recommended basis to 785 minimise interference of random events with the statistical test 786 proposed by this document. In the case of ratio measurements (like 787 packet loss), the underlying sum of basic events, against the which 788 the metric's monitored singletons are "rated", determines the 789 resolution of the test. A packet loss statistic with a resolution of 790 1% requires one packet loss statistic-datapoint to consist of 500 791 delay singletons (of which at least 5 were lost). To compare EDFs on 792 packet loss requires one hundred such statistics per flow. That 793 means, all in all at least 50 000 delay singletons are required per 794 single measurement flow. Live network packet loss is assumed to be 795 present during main traffic hours only. Let this interval be 5 796 hours. The required minimum rate of a single measurement flow in 797 that case is 2.8 packets/sec (assuming a loss of 1% during 5 hours). 798 If this measurement is too demanding under live network conditions, 799 an impairment generator should be used. 801 4. Acknowledgements 803 Gerhard Hasslinger commented a first version of this document, 804 suggested statistical tests and the evaluation of time series 805 information. Henk Uijterwaal pushed this work and Mike Hamilton, 806 Scott Bradner and Emile Stephan commented on versions of this draft 807 before initial publication. Carol Davids reviewed the 01 version of 808 this draft. 810 5. Contributors 812 Scott Bradner, Vern Paxson and Allison Mankin drafted bradner- 813 metrictest [bradner-metrictest], and major parts of it are included 814 in this document. 816 6. IANA Considerations 818 This memo includes no request to IANA. 820 7. Security Considerations 822 This draft does not raise any specific security issues. 824 8. References 826 8.1. Normative References 828 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, 829 October 1996. 831 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 832 3", BCP 9, RFC 2026, October 1996. 834 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 835 Requirement Levels", BCP 14, RFC 2119, March 1997. 837 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 838 "Framework for IP Performance Metrics", RFC 2330, 839 May 1998. 841 [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, 842 G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", 843 RFC 2661, August 1999. 845 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 846 Delay Metric for IPPM", RFC 2679, September 1999. 848 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 849 Packet Loss Metric for IPPM", RFC 2680, September 1999. 851 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 852 Delay Metric for IPPM", RFC 2681, September 1999. 854 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 855 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 856 March 2000. 858 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 859 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 861 [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, 862 "Encapsulation Methods for Transport of Ethernet over MPLS 863 Networks", RFC 4448, April 2006. 865 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 866 Zekauskas, "A One-way Active Measurement Protocol 867 (OWAMP)", RFC 4656, September 2006. 869 [RFC4818] Salowey, J. and R. Droms, "RADIUS Delegated-IPv6-Prefix 870 Attribute", RFC 4818, April 2007. 872 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 873 Cost Multipath Treatment in MPLS Networks", BCP 128, 874 RFC 4928, June 2007. 876 8.2. Informative References 878 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 879 Tests of fit, for continuous and discrete cases", 880 University of Washington, Technical Report No. 81, 881 May 1986. 883 [GU+Duffield] 884 Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE 885 Encapsulated Multicast Probing: A Scalable Technique for 886 Measuring One-Way Loss", SIGMETRICS'07 San Diego, 887 California, USA, June 2007. 889 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 890 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 891 RFC 5357, October 2008. 893 [Rule of thumb] 894 N., N., "Confidence interval", October 2008. 896 [bradner-metrictest] 897 Bradner, S., Mankin, A., and V. Paxson, "Advancement of 898 metrics specifications on the IETF Standards Track", 899 draft -morton-ippm-advance-metrics-00, (work in progress), 900 July 2007. 902 [morton-advance-metrics] 903 Morton, A., "Problems and Possible Solutions for Advancing 904 Metrics on the Standards Track", draft -bradner- 905 metricstest-03, (work in progress), July 2009. 907 Appendix A. An example on a One-way Delay metric validation 909 The text of this appendix is not binding. It is an example how parts 910 of a One-way Delay metric test could look like. 911 http://xml.resource.org/public/rfc/bibxml/ 913 A.1. Compliance to Metric specification requirements 915 One-way Delay, Loss threshold, RFC 2679 917 This test determines if implementations use the same configured 918 maximum waiting time delay from one measurement to another under 919 different delay conditions, and correctly declare packets arriving in 920 excess of the waiting time threshold as lost. See Section 3.5 of 921 RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. 923 (1) Configure a path with 1 sec one-way constant delay. 925 (2) Measure one-way delay with 2 or more implementations, using 926 identical waiting time thresholds for loss set at 2 seconds. 928 (3) Configure the path with 3 sec one-way delay. 930 (4) Repeat measurements. 932 (5) Observe that the increase measured in step 4 caused all packets 933 to be declared lost, and that all packets that arrive 934 successfully in step 2 are assigned a valid one-way delay. 936 One-way Delay, First-bit to Last bit, RFC 2679 938 This test determines if implementations register the same relative 939 increase in delay from one measurement to another under different 940 delay conditions. This test tends to cancel the sources of error 941 which may be present in an implementation. See Section 3.7.2 of 942 RFC2679, and Section 10.2 of RFC2330. 944 (1) Configure a path with X ms one-way constant delay, and ideally 945 including a low-speed link. 947 (2) Measure one-way delay with 2 or more implementations, using 948 identical options and equal size small packets (e.g., 100 octet 949 IP payload). 951 (3) Maintain the same path with X ms one-way delay. 953 (4) Measure one-way delay with 2 or more implementations, using 954 identical options and equal size large packets (e.g., 1500 octet 955 IP payload). 957 (5) Observe that the increase measured in steps 2 and 4 is 958 equivalent to the increase in ms expected due to the larger 959 serialization time for each implementation. Most of the 960 measurement errors in each system should cancel, if they are 961 stationary. 963 One-way Delay, RFC 2679 965 This test determines if implementations register the same relative 966 increase in delay from one measurement to another under different 967 delay conditions. This test tends to cancel the sources of error 968 which may be present in an implementation. This test is intended to 969 evaluate measurments in sections 3 and 4 of RFC2679. 971 (1) Configure a path with X ms one-way constant delay. 973 (2) Measure one-way delay with 2 or more implementations, using 974 identical options. 976 (3) Configure the path with X+Y ms one-way delay. 978 (4) Repeat measurements. 980 (5) Observe that the increase measured in steps 2 and 4 is ~Y ms for 981 each implementation. Most of the measurement errors in each 982 system should cancel, if they are stationary. 984 Error Calibration, RFC 2679 986 This is a simple check to determine if an implementation reports the 987 error calibration as required in Section 4.8 of RFC2679. Note that 988 the context (Type-P) must also be reported. 990 A.2. Examples related to statistical tests for One-way Delay 992 A one way delay measurement may pass an ADK test with a timestamp 993 resultion of 1 ms. The same test may fail, if timestamps with a 994 resolution of 100 microseconds are eavluated. The implementation 995 then is then conforming to the metric specification up to a timestamp 996 resolution of 1 ms. 998 Let's assume another one way delay measurement comparison between 999 implementation 1, probing with a frequency of 2 probes per second and 1000 implementation 2 probing at a rate of 2 probes every 3 minutes. To 1001 ensure reasonable confidence in results, sample metrics are 1002 calculated from at least 5 singletons per compared time interval. 1003 This means, sample delay values are calculated for each system for 1004 identical 6 minute intervals for the whole test duration. Per 6 1005 minute interval, the sample metric is calculated from 720 singletons 1006 for implementation 1 and from 6 singletons for implementation 2. 1007 Note, that if outliers are not filtered, moving averages are an 1008 option for an evaluation too. The minimum move of an averaging 1009 interval is three minutes in this example. 1011 The data in table 1 may result from measuring One-Way Delay with 1012 implementation 1 (see column Implemnt_1) and implementation 2 (see 1013 column implemnt_2). Each data point in the table represents a 1014 (rounded) average of the sampled delay values per interval. The 1015 resolution of the clock is one micro-second. The difference in the 1016 delay values may result eg. from different probe packet sizes. 1018 +------------+------------+-----------------------------+ 1019 | Implemnt_1 | Implemnt_2 | Implemnt_2 - Delta_Averages | 1020 +------------+------------+-----------------------------+ 1021 | 5000 | 6549 | 4997 | 1022 | 5008 | 6555 | 5003 | 1023 | 5012 | 6564 | 5012 | 1024 | 5015 | 6565 | 5013 | 1025 | 5019 | 6568 | 5016 | 1026 | 5022 | 6570 | 5018 | 1027 | 5024 | 6573 | 5021 | 1028 | 5026 | 6575 | 5023 | 1029 | 5027 | 6577 | 5025 | 1030 | 5029 | 6580 | 5028 | 1031 | 5030 | 6585 | 5033 | 1032 | 5032 | 6586 | 5034 | 1033 | 5034 | 6587 | 5035 | 1034 | 5036 | 6588 | 5036 | 1035 | 5038 | 6589 | 5037 | 1036 | 5039 | 6591 | 5039 | 1037 | 5041 | 6592 | 5040 | 1038 | 5043 | 6599 | 5047 | 1039 | 5046 | 6606 | 5054 | 1040 | 5054 | 6612 | 5060 | 1041 +------------+------------+-----------------------------+ 1043 Table 1 1045 Average values of sample metrics captured during identical time 1046 intervals are compared. This excludes random differences caused by 1047 differing probing intervals or differing temporal distance of 1048 singletons resulting from their Poisson distributed sending times. 1050 In the example, 20 values have been picked (note that at least 100 1051 values are recommended for a single run of a real test). Data must 1052 be ordered by ascending rank. The data of Implemnt_1 and Implemnt_2 1053 as shown in the first two columns of table 1 clearly fails an ADK 1054 test with 95% confidence. 1056 The results of Implemnt_2 are now reduced by difference of the 1057 averages of column 2 (rounded to 6581 us) and column 1 (rounded to 1058 5029 us), which is 1552 us. The result may be found in column 3 of 1059 table 1. Comparing column 1 and column 3 of the table by an ADK test 1060 shows, that the data contained in these columns passes an ADK tests 1061 with 95% confidence. 1063 Appendix B. Glossary 1065 +-------------+-----------------------------------------------------+ 1066 | ADK | Anderson-Darling K-Sample test, a test used to | 1067 | | check whether two samples have the same statistical | 1068 | | distribution. | 1069 | ECMP | Equal Cost Multipath, a load balancing mechanism | 1070 | | evaluating MPLS labels stacks, IP addresses and | 1071 | | ports. | 1072 | EDF | The "Empirical Distribution Function" of a set of | 1073 | | scalar measurements is a function F(x) which for | 1074 | | any x gives the fractional proportion of the total | 1075 | | measurements that were smaller than or equal as x. | 1076 | Metric | A measured quantity related to the performance and | 1077 | | reliability of the Internet, expressed by a value. | 1078 | | This could be a singleton (single value), a sample | 1079 | | of single values or a statistic based on a sample | 1080 | | of singletons. | 1081 | OWAMP | One-way Active Measurement Protocol, a protocol for | 1082 | | communication between IPPM measurement systems | 1083 | | specified by IPPM. | 1084 | OWD | One-Way Delay, a performance metric specified by | 1085 | | IPPM. | 1086 | Sample | A sample metric is derived from a given singleton | 1087 | metric | metric by evaluating a number of distinct instances | 1088 | | together. | 1089 | Singleton | A singleton metric is, in a sense, one atomic | 1090 | metric | measurement of this metric. | 1091 | Statistical | A 'statistical' metric is derived from a given | 1092 | metric | sample metric by computing some statistic of the | 1093 | | values defined by the singleton metric on the | 1094 | | sample. | 1095 | TWAMP | Two-way Active Measurement Protocol, a protocol for | 1096 | | communication between IPPM measurement systems | 1097 | | specified by IPPM. | 1098 +-------------+-----------------------------------------------------+ 1100 Table 2 1102 Authors' Addresses 1104 Ruediger Geib (editor) 1105 Deutsche Telekom 1106 Heinrich Hertz Str. 3-7 1107 Darmstadt, 64295 1108 Germany 1110 Phone: +49 6151 628 2747 1111 Email: Ruediger.Geib@telekom.de 1113 Al Morton 1114 AT&T Labs 1115 200 Laurel Avenue South 1116 Middletown, NJ 07748 1117 USA 1119 Phone: +1 732 420 1571 1120 Fax: +1 732 368 1192 1121 Email: acmorton@att.com 1122 URI: http://home.comcast.net/~acmacm/ 1124 Reza Fardid 1125 Covad Communications 1126 2510 Zanker Road 1127 San Jose, CA 95131 1128 USA 1130 Phone: +1 408 434-2042 1131 Email: RFardid@covad.com