idnits 2.17.00 (12 Aug 2021) /tmp/idnits48436/draft-ietf-ippm-metrictest-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year -- The document date (June 29, 2011) is 3978 days in the past. Is this intentional? -- Found something which looks like a code comment -- if you have code sections in the document, please surround them with '' and '' lines. Checking references for intended status: Proposed Standard ---------------------------------------------------------------------------- (See RFCs 3967 and 4897 for information about using normative references to lower-maturity documents in RFCs) -- Looks like a reference, but probably isn't: '0' on line 1400 == Unused Reference: 'RFC2680' is defined on line 1062, but no explicit reference was found in the text == Unused Reference: 'RFC2681' is defined on line 1065, but no explicit reference was found in the text == Unused Reference: 'RFC3931' is defined on line 1072, but no explicit reference was found in the text == Unused Reference: 'RFC4719' is defined on line 1086, but no explicit reference was found in the text ** Downref: Normative reference to an Informational RFC: RFC 2330 ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) ** Downref: Normative reference to an Informational RFC: RFC 4459 Summary: 4 errors (**), 0 flaws (~~), 5 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Internet Engineering Task Force R. Geib, Ed. 3 Internet-Draft Deutsche Telekom 4 Intended status: Standards Track A. Morton 5 Expires: December 31, 2011 AT&T Labs 6 R. Fardid 7 Cariden Technologies 8 A. Steinmitz 9 Deutsche Telekom 10 June 29, 2011 12 IPPM standard advancement testing 13 draft-ietf-ippm-metrictest-03 15 Abstract 17 This document specifies tests to determine if multiple independent 18 instantiations of a performance metric RFC have implemented the 19 specifications in the same way. This is the performance metric 20 equivalent of interoperability, required to advance RFCs along the 21 standards track. Results from different implementations of metric 22 RFCs will be collected under the same underlying network conditions 23 and compared using state of the art statistical methods. The goal is 24 an evaluation of the metric RFC itself, whether its definitions are 25 clear and unambiguous to implementors and therefore a candidate for 26 advancement on the IETF standards track. 28 Status of this Memo 30 This Internet-Draft is submitted in full conformance with the 31 provisions of BCP 78 and BCP 79. 33 Internet-Drafts are working documents of the Internet Engineering 34 Task Force (IETF). Note that other groups may also distribute 35 working documents as Internet-Drafts. The list of current Internet- 36 Drafts is at http://datatracker.ietf.org/drafts/current/. 38 Internet-Drafts are draft documents valid for a maximum of six months 39 and may be updated, replaced, or obsoleted by other documents at any 40 time. It is inappropriate to use Internet-Drafts as reference 41 material or to cite them other than as "work in progress." 43 This Internet-Draft will expire on December 31, 2011. 45 Copyright Notice 47 Copyright (c) 2011 IETF Trust and the persons identified as the 48 document authors. All rights reserved. 50 This document is subject to BCP 78 and the IETF Trust's Legal 51 Provisions Relating to IETF Documents 52 (http://trustee.ietf.org/license-info) in effect on the date of 53 publication of this document. Please review these documents 54 carefully, as they describe your rights and restrictions with respect 55 to this document. Code Components extracted from this document must 56 include Simplified BSD License text as described in Section 4.e of 57 the Trust Legal Provisions and are provided without warranty as 58 described in the Simplified BSD License. 60 Table of Contents 62 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 63 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 7 64 2. Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . 7 65 3. Verification of conformance to a metric specification . . . . 8 66 3.1. Tests of an individual implementation against a metric 67 specification . . . . . . . . . . . . . . . . . . . . . . 9 68 3.2. Test setup resulting in identical live network testing 69 conditions . . . . . . . . . . . . . . . . . . . . . . . . 11 70 3.3. Tests of two or more different implementations against 71 a metric specification . . . . . . . . . . . . . . . . . . 16 72 3.4. Clock synchronisation . . . . . . . . . . . . . . . . . . 17 73 3.5. Recommended Metric Verification Measurement Process . . . 18 74 3.6. Proposal to determine an "equivalence" threshold for 75 each metric evaluated . . . . . . . . . . . . . . . . . . 21 76 4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22 77 5. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . 22 78 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 79 7. Security Considerations . . . . . . . . . . . . . . . . . . . 23 80 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 23 81 8.1. Normative References . . . . . . . . . . . . . . . . . . . 23 82 8.2. Informative References . . . . . . . . . . . . . . . . . . 24 83 Appendix A. An example on a One-way Delay metric validation . . . 25 84 A.1. Compliance to Metric specification requirements . . . . . 25 85 A.2. Examples related to statistical tests for One-way Delay . 27 86 Appendix B. Anderson-Darling 2 sample C++ code . . . . . . . . . 29 87 Appendix C. Glossary . . . . . . . . . . . . . . . . . . . . . . 37 88 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 90 1. Introduction 92 The Internet Standards Process RFC2026 [RFC2026] requires that for a 93 IETF specification to advance beyond the Proposed Standard level, at 94 least two genetically unrelated implementations must be shown to 95 interoperate correctly with all features and options. This 96 requirement can be met by supplying: 98 o evidence that (at least a sub-set of) the specification has been 99 implemented by multiple parties, thus indicating adoption by the 100 IETF community and the extent of feature coverage. 102 o evidence that each feature of the specification is sufficiently 103 well-described to support interoperability, as demonstrated 104 through testing and/or user experience with deployment. 106 In the case of a protocol specification, the notion of 107 "interoperability" is reasonably intuitive - the implementations must 108 successfully "talk to each other", while exercising all features and 109 options. To achieve interoperability, two implementors need to 110 interpret the protocol specifications in equivalent ways. In the 111 case of IP Performance Metrics (IPPM), this definition of 112 interoperability is only useful for test and control protocols like 113 the One-Way Active Measurement Protocol, OWAMP [RFC4656], and the 114 Two-Way Active Measurement Protocol, TWAMP [RFC5357]. 116 A metric specification RFC describes one or more metric definitions, 117 methods of measurement and a way to report the results of 118 measurement. One example would be a way to test and report the One- 119 way Delay that data packets incur while being sent from one network 120 location to another, One-way Delay Metric. 122 In the case of metric specifications, the conditions that satisfy the 123 "interoperability" requirement are less obvious, and there was a need 124 for IETF agreement on practices to judge metric specification 125 "interoperability" in the context of the IETF Standards Process. 126 This memo provides methods which should be suitable to evaluate 127 metric specifications for standards track advancement. The methods 128 proposed here MAY be generally applicable to metric specification 129 RFCs beyond those developed under the IPPM Framework [RFC2330]. 131 Since many implementations of IP metrics are embedded in measurement 132 systems that do not interact with one another (they were built before 133 OWAMP and TWAMP), the interoperability evaluation called for in the 134 IETF standards process cannot be determined by observing that 135 independent implementations interact properly for various protocol 136 exchanges. Instead, verifying that different implementations give 137 statistically equivalent results under controlled measurement 138 conditions takes the place of interoperability observations. Even 139 when evaluating OWAMP and TWAMP RFCs for standards track advancement, 140 the methods described here are useful to evaluate the measurement 141 results because their validity would not be ascertained in typical 142 interoperability testing. 144 The standards advancement process aims at producing confidence that 145 the metric definitions and supporting material are clearly worded and 146 unambiguous, or reveals ways in which the metric definitions can be 147 revised to achieve clarity. The process also permits identification 148 of options that were not implemented, so that they can be removed 149 from the advancing specification. Thus, the product of this process 150 is information about the metric specification RFC itself: 151 determination of the specifications or definitions that are clear and 152 unambiguous and those that are not (as opposed to an evaluation of 153 the implementations which assist in the process). 155 This document defines a process to verify that implementations (or 156 practically, measurement systems) have interpreted the metric 157 specifications in equivalent ways, and produce equivalent results. 159 Testing for statistical equivalence requires ensuring identical test 160 setups (or awareness of differences) to the best possible extent. 161 Thus, producing identical test conditions is a core goal of the memo. 162 Another important aspect of this process is to test individual 163 implementations against specific requirements in the metric 164 specifications using customized tests for each requirement. These 165 tests can distinguish equivalent interpretations of each specific 166 requirement. 168 Conclusions on equivalence are reached by two measures. 170 First, implementations are compared against individual metric 171 specifications to make sure that differences in implementation are 172 minimised or at least known. 174 Second, a test setup is proposed ensuring identical networking 175 conditions so that unknowns are minimized and comparisons are 176 simplified. The resulting separate data sets may be seen as samples 177 taken from the same underlying distribution. Using state of the art 178 statistical methods, the equivalence of the results is verified. To 179 illustrate application of the process and methods defined here, 180 evaluation of the One-way Delay Metric [RFC2679] is provided in an 181 Appendix. While test setups will vary with the metrics to be 182 validated, the general methodology of determining equivalent results 183 will not. Documents defining test setups to evaluate other metrics 184 should be developed once the process proposed here has been agreed 185 and approved. 187 The metric RFC advancement process begins with a request for protocol 188 action accompanied by a memo that documents the supporting tests and 189 results. The procedures of [RFC2026] are expanded in[RFC5657], 190 including sample implementation and interoperability reports. 191 Section 3 of [morton-advance-metrics-01] can serve as a template for 192 a metric RFC report which accompanies the protocol action request to 193 the Area Director, including description of the test set-up, 194 procedures, results for each implementation and conclusions. 196 Changes from WG-02 to WG-03: 198 o Changes stemming from experiments that implemented this plan, in 199 general. 201 o Adoption of the VLAN loopback figure in the main body of the memo 202 (section 3.2). 204 Changes from WG-01 to WG-02: 206 o Clarification of the number of test streams recommended in section 207 3.2. 209 o Clarifications on testing details in sections 3.3 and 3.4. 211 o Spelling corrections throughout. 213 Changes from WG -00 to WG -01 draft 215 o Discussion on merits and requirements of a distributed lab test 216 using only local load generators. 218 o Proposal of metrics suitable for tests using the proposed 219 measurement configuration. 221 o Hint on delay caused by software based L2TPv3 implementation. 223 o Added an appendix with a test configuration allowing remote tests 224 comparing different implementations across the network. 226 o Proposal for maximum error of "equivalence", based on performance 227 comparison of identical implementations. This may be useful for 228 both ADK and non-ADK comparisons. 230 Changes from prior ID -02 to WG -00 draft 232 o Incorporation of aspects of reporting to support the protocol 233 action request in the Introduction and section 3.5 235 o Overhaul of section 3.2 regarding tunneling: Added generic 236 tunneling requirements and L2TPv3 as an example tunneling 237 mechanism fulfilling the tunneling requirements. Removed and 238 adapted some of the prior references to other tunneling protocols 240 o Softened a requirement within section 3.4 (MUST to SHOULD on 241 precision) and removed some comments of the authors. 243 o Updated contact information of one author and added a new author. 245 o Added example C++ code of an Anderson-Darling two sample test 246 implementation. 248 Changes from ID -01 to ID -02 version 250 o Major editorial review, rewording and clarifications on all 251 contents. 253 o Additional text on parallel testing using VLANs and GRE or 254 Pseudowire tunnels. 256 o Additional examples and a glossary. 258 Changes from ID -00 to ID -01 version 260 o Addition of a comparison of individual metric implementations 261 against the metric specification (trying to pick up problems and 262 solutions for metric advancement [morton-advance-metrics]). 264 o More emphasis on the requirement to carefully design and document 265 the measurement setup of the metric comparison. 267 o Proposal of testing conditions under identical WAN network 268 conditions using IP in IP tunneling or Pseudo Wires and parallel 269 measurement streams. 271 o Proposing the requirement to document the smallest resolution at 272 which an ADK test was passed by 95%. As no minimum resolution is 273 specified, IPPM metric compliance is not linked to a particular 274 performance of an implementation. 276 o Reference to RFC 2330 and RFC 2679 for the 95% confidence interval 277 as preferred criterion to decide on statistical equivalence 279 o Reducing the proposed statistical test to ADK with 95% confidence. 281 1.1. Requirements Language 283 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 284 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 285 document are to be interpreted as described in RFC 2119 [RFC2119]. 287 2. Basic idea 289 The implementation of a standard compliant metric is expected to meet 290 the requirements of the related metric specification. So before 291 comparing two metric implementations, each metric implementation is 292 individually compared against the metric specification. 294 Most metric specifications leave freedom to implementors on non- 295 fundamental aspects of an individual metric (or options). Comparing 296 different measurement results using a statistical test with the 297 assumption of identical test path and testing conditions requires 298 knowledge of all differences in the overall test setup. Metric 299 specification options chosen by implementors have to be documented. 300 It is REQUIRED to use identical implementation options wherever 301 possible for any test proposed here. Calibrations proposed by metric 302 standards should be performed to further identify (and possibly 303 reduce) potential sources of errors in the test setup. 305 The Framework for IP Performance Metrics [RFC2330] expects that a 306 "methodology for a metric should have the property that it is 307 repeatable: if the methodology is used multiple times under identical 308 conditions, it should result in consistent measurements." This means 309 an implementation is expected to repeatedly measure a metric with 310 consistent results (repeatability with the same result). Small 311 deviations in the test setup are expected to lead to small deviations 312 in results only. To characterise statistical equivalence in the case 313 of small deviations, RFC 2330 and [RFC2679] suggest to apply a 95% 314 confidence interval. Quoting RFC 2679, "95 percent was chosen 315 because ... a particular confidence level should be specified so that 316 the results of independent implementations can be compared." 318 Two different implementations are expected to produce statistically 319 equivalent results if they both measure a metric under the same 320 networking conditions. Formulating in statistical terms: separate 321 metric implementations collect separate samples from the same 322 underlying statistical process (the same network conditions). The 323 statistical hypothesis to be tested is the expectation that both 324 samples do not expose statistically different properties. This 325 requires careful test design: 327 o The measurement test setup must be self-consistent to the largest 328 possible extent. To minimize the influence of the test and 329 measurement setup on the result, network conditions and paths MUST 330 be identical for the compared implementations to the largest 331 possible degree. This includes both the stability and non- 332 ambiguity of routes taken by the measurement packets. See RFC 333 2330 for a discussion on self-consistency. 335 o To minimize the influence of implementation options on the result, 336 metric implementations SHOULD use identical options and parameters 337 for the metric under evaluation. 339 o The error induced by the sample size must be small enough to 340 minimize its influence on the test result. This may have to be 341 respected, especially if two implementations measure with 342 different average probing rates. 344 o The implementation with the lowest probing frequency determines 345 the smallest temporal interval for which samples can be compared. 347 o Repeat comparisons with several independent metric samples to 348 avoid random indications of compatibility (or the lack of it). 350 The metric specifications themselves are the primary focus of 351 evaluation, rather than the implementations of metrics. The 352 documentation produced by the advancement process should identify 353 which metric definitions and supporting material were found to be 354 clearly worded and unambiguous, OR, it should identify ways in which 355 the metric specification text should be revised to achieve clarity 356 and unified interpretation. 358 The process should also permit identification of options that were 359 not implemented, so that they can be removed from the advancing 360 specification (this is an aspect more typical of protocol advancement 361 along the standards track). 363 Note that this document does not propose to base interoperability 364 indications of performance metric implementations on comparisons of 365 individual singletons. Individual singletons may be impacted by many 366 statistical effects while they are measured. Comparing two 367 singletons of different implementations may result in failures with 368 higher probability than comparing samples. 370 3. Verification of conformance to a metric specification 372 This section specifies how to verify compliance of two or more IPPM 373 implementations against a metric specification. This document only 374 proposes a general methodology. Compliance criteria to a specific 375 metric implementation need to be defined for each individual metric 376 specification. The only exception is the statistical test comparing 377 two metric implementations which are simultaneously tested. This 378 test is applicable without metric specific decision criteria. 380 Several testing options exist to compare two or more implementations: 382 o Use a single test lab to compare the implementations and emulate 383 the Internet with an impairment generator. 385 o Use a single test lab to compare the implementations and measure 386 across the Internet. 388 o Use remotely separated test labs to compare the implementations 389 and emulate the Internet with two "identically" configured 390 impairment generators. 392 o Use remotely separated test labs to compare the implementations 393 and measure across the Internet. 395 o Use remotely separated test labs to compare the implementations 396 and measure across the Internet and include a single impairment 397 generator to impact all measurement flows in non discriminatory 398 way. 400 The first two approaches work, but cause higher expenses than the 401 other ones (due to travel and/or shipping+installation). For the 402 third option, ensuring two identically configured impairment 403 generators requires well defined test cases and possibly identical 404 hard- and software. 406 As documented in a test report [morton-testplan-rfc2679], the last 407 option was required to prove compatibility of two delay metric 408 implementations. An impairment generator is probably required when 409 testing compatibility of most other metrics and it therefore 410 RECOMMENDED to include an impairment generator in metric test set 411 ups. 413 3.1. Tests of an individual implementation against a metric 414 specification 416 A metric implementation MUST support the requirements classified as 417 "MUST" and "REQUIRED" of the related metric specification to be 418 compliant to the latter. 420 Further, supported options of a metric implementation SHOULD be 421 documented in sufficient detail. The documentation of chosen options 422 is RECOMMENDED to minimise (and recognise) differences in the test 423 setup if two metric implementations are compared. Further, this 424 documentation is used to validate and improve the underlying metric 425 specification option, to remove options which saw no implementation 426 or which are badly specified from the metric specification to be 427 promoted to a standard. This documentation SHOULD be made for all 428 implementation-relevant specifications of a metric picked for a 429 comparison that are not explicitly marked as "MUST" or "REQUIRED" in 430 the RFC text. This applies for the following sections of all metric 431 specifications: 433 o Singleton Definition of the Metric. 435 o Sample Definition of the Metric. 437 o Statistics Definition of the Metric. As statistics are compared 438 by the test specified here, this documentation is required even in 439 the case, that the metric specification does not contain a 440 Statistics Definition. 442 o Timing and Synchronisation related specification (if relevant for 443 the Metric). 445 o Any other technical part present or missing in the metric 446 specification, which is relevant for the implementation of the 447 Metric. 449 RFC2330 and RFC2679 emphasise precision as an aim of IPPM metric 450 implementations. A single IPPM conformant implementation MUST under 451 otherwise identical network conditions produce precise results for 452 repeated measurements of the same metric. 454 RFC 2330 prefers the "empirical distribution function" EDF to 455 describe collections of measurements. RFC 2330 determines, that 456 "unless otherwise stated, IPPM goodness-of-fit tests are done using 457 5% significance." The goodness of fit test determines by which 458 precision two or more samples of a metric implementation belong to 459 the same underlying distribution (of measured network performance 460 events). The goodness of fit test suggested for the metric test is 461 the Anderson-Darling K sample test (ADK sample test, K stands for the 462 number of samples to be compared) [ADK]. Please note that RFC 2330 463 and RFC 2679 apply an Anderson Darling goodness of fit test too. 465 The results of a repeated test with a single implementation MUST pass 466 an ADK sample test with confidence level of 95%. The conditions for 467 which the ADK test has been passed with the specified confidence 468 level MUST be documented. To formulate this differently: The 469 requirement is to document the set of parameters with the smallest 470 deviation, at which the results of the tested metric implementation 471 pass an ADK test with a confidence level of 95%. The minimum 472 resolution available in the reported results from each implementation 473 MUST be taken into account in the ADK test. 475 The test conditions which MUST be documented for a passed metric test 476 include: 478 o The metric resolution at which a test was passed (e.g. the 479 resolution of timestamps) 481 o The parameters modified by an impairment generator. 483 o The impairment generator parameter settings. 485 3.2. Test setup resulting in identical live network testing conditions 487 Two major issues complicate tests for metric compliance across live 488 networks under identical testing conditions. One is the general 489 point that metric definition implementations cannot be conveniently 490 examined in field measurement scenarios. The other one is more 491 broadly described as "parallelism in devices and networks", including 492 mechanisms like those that achieve load balancing (see [RFC4928]). 494 This section proposes two measures to deal with both issues. 495 Tunneling mechanisms can be used to avoid parallel processing of 496 different flows in the network. Measuring by separate parallel probe 497 flows results in repeated collection of data. If both measures are 498 combined, WAN network conditions are identical for a number of 499 independent measurement flows, no matter what the network conditions 500 are in detail. 502 Any measurement setup MUST be made to avoid the probing traffic 503 itself to impede the metric measurement. The created measurement 504 load MUST NOT result in congestion at the access link connecting the 505 measurement implementation to the WAN. The created measurement load 506 MUST NOT overload the measurement implementation itself, e.g., by 507 causing a high CPU load or by creating imprecisions due to internal 508 transmit (receive respectively) probe packet collisions. 510 Tunneling multiple flows reaching a network element on a single 511 physical port may allow to transmit all packets of the tunnel via the 512 same path. Applying tunnels to avoid undesired influence of standard 513 routing for measurement purposes is a concept known from literature, 514 see e.g. GRE encapsulated multicast probing [GU+Duffield]. An 515 existing IP in IP tunnel protocol can be applied to avoid Equal-Cost 516 Multi-Path (ECMP) routing of different measurement streams if it 517 meets the following criteria: 519 o Inner IP packets from different measurement implementations are 520 mapped into a single tunnel with single outer IP origin and 521 destination address as well as origin and destination port numbers 522 which are identical for all packets. 524 o An easily accessible commodity tunneling protocol allows to carry 525 out a metric test from more test sites. 527 o A low operational overhead may enable a broader audience to set up 528 a metric test with the desired properties. 530 o The tunneling protocol should be reliable and stable in set up and 531 operation to avoid disturbances or influence on the test results. 533 o The tunneling protocol should not incur any extra cost for those 534 interested in setting up a metric test. 536 An illustration of a test setup with two layer 2 tunnels and two 537 flows between two linecards of one implementation is given in 538 Figure 1. 540 Implementation ,---. +--------+ 541 +~~~~~~~~~~~/ \~~~~~~| Remote | 542 +------->-----F2->-| / \ |->---+ | 543 | +---------+ | Tunnel 1( ) | | | 544 | | transmit|-F1->-| ( ) |->+ | | 545 | | LC1 | +~~~~~~~~~| |~~~~| | | | 546 | | receive |-<--+ ( ) | F1 F2 | 547 | +---------+ | |Internet | | | | | 548 *-------<-----+ F2 | | | | | | 549 +---------+ | | +~~~~~~~~~| |~~~~| | | | 550 | transmit|-* *-| | | |--+<-* | 551 | LC2 | | Tunnel 2( ) | | | 552 | receive |-<-F1-| \ / |<-* | 553 +---------+ +~~~~~~~~~~~\ /~~~~~~| Router | 554 `-+-' +--------+ 556 Illustration of a test setup with two layer 2 tunnels. For 557 simplicity, only two linecards of one implementation and two flows F 558 between them are shown. 560 Figure 1 562 Figure 2 shows the network elements required to set up layer 2 563 tunnels as shown by figure 1. 565 Implementation 567 +-----+ ,---. 568 | LC1 | / \ 569 +-----+ / \ +------+ 570 | +-------+ ( ) +-------+ |Remote| 571 +--------+ | | | | | | | | 572 |Ethernet| | Tunnel| |Internet | | Tunnel| | | 573 |Switch |--| Head |--| |--| Head |--| | 574 +--------+ | Router| | | | Router| | | 575 | | | ( ) | | |Router| 576 +-----+ +-------+ \ / +-------+ +------+ 577 | LC2 | \ / 578 +-----+ `-+-' 579 Illustration of a hardware setup to realise the test setup 580 illustrated by figure 1 with layer 2 tunnels or Pseudowires. 582 Figure 2 584 The test set up successfully used during a delay metric test 585 [morton-testplan-rfc2679] is given as an example in figure 3. Note 586 that the shown set up allows a metric test between two remote sites. 588 +----+ +----+ +----+ +----+ 589 |LC10| |LC11| ,---. |LC20| |LC21| 590 +----+ +----+ / \ +-------+ +----+ +----+ 591 | V10 | V11 / \ | Tunnel| | V20 | V21 592 | | ( ) | Head | | | 593 +--------+ +------+ | | | Router|__+----------+ 594 |Ethernet| |Tunnel| |Internet | +---B---+ |Ethernet | 595 |Switch |--|Head |-| | | |Switch | 596 +-+--+---+ |Router| | | +---+---+ +--+--+----+ 597 |__| +--A---+ ( )--|Option.| |__| 598 \ / |Impair.| 599 Bridge \ / |Gener. | Bridge 600 V20 to V21 `-+-? +-------+ V10 to V11 602 Figure 3 604 In figure 3, LC10 identify measurement clients /line cards. V10 and 605 the others denote VLANs. All VLANs are using the same tunnel from A 606 to B and in the reverse direction. The remote site VLANs are 607 U-bridged at the local site Ethernet switch. The measurement packets 608 of site 1 travel tunnel A->B first, are U-bridged at site 2 and 609 travel tunnel B->A second. Measurement packets of site 2 travel 610 tunnel B->A first, are U-bridged at site 1 and travel tunnel A->B 611 second. So all measurement packets pass the same tunnel segments, 612 but in different segment order. 614 If tunneling is applied, two tunnels MUST carry all test traffic in 615 between the test site and the remote site. For example, if 802.1Q 616 Ethernet Virtual LANs (VLAN) are applied and the measurement streams 617 are carried in different VLANs, the IP tunnel or Pseudo Wires 618 respectively MUST be set up in physical port mode to avoid set up of 619 Pseudo Wires per VLAN (which may see different paths due to ECMP 620 routing), see RFC 4448. The remote router and the Ethernet switch 621 shown in figure 3 has to support 802.1Q in this set up. 623 The IP packet size of the metric implementation SHOULD be chosen 624 small enough to avoid fragmentation due to the added Ethernet and 625 tunnel headers. Otherwise, the impact of tunnel overhead on 626 fragmentation and interface MTU size MUST be understood and taken 627 into account (see [RFC4459]). 629 An Ethernet port mode IP tunnel carrying several 802.1Q VLANs each 630 containing measurement traffic of a single measurement system was 631 successfully applied when testing compatibility of two metric 632 implementations [morton-testplan-rfc2679]. 634 The following headers may have to be accounted for when calculating 635 total packet length, if VLANs and Ethernet over L2TPv3 tunnels are 636 applied: 638 o Ethernet 802.1Q: 22 Byte. 640 o L2TPv3 Header: 4-16 Byte for L2TPv3 data messages over IP; 16-28 641 Byte for L2TPv3 data messages over UDP. 643 o IPv4 Header (outer IP header): 20 Byte. 645 o MPLS Labels may be added by a carrier. Each MPLS Label has a 646 length of 4 Bytes. By the time of writing, between 1 and 4 Labels 647 seems to be a fair guess of what's expectable. 649 The applicability of one or more of the following tunneling protocols 650 may be investigated by interested parties if Ethernet over L2TPv3 is 651 felt to be not suitable: IP in IP [RFC2003] or Generic Routing 652 Encapsulation (GRE) [RFC2784]. RFC 4928 [RFC4928] proposes measures 653 how to avoid ECMP treatment in MPLS networks. 655 L2TP is a commodity tunneling protocol [RFC2661]. By the time of 656 writing, L2TPv3 [RFC3931]is the latest version of L2TP. If L2TPv3 is 657 applied, software based implementations of this protocol are not 658 suitable for the test set up, as such implementations may cause 659 incalculable delay shifts. 661 Ethernet Pseudo Wires may also be set up on MPLS networks [RFC4448]. 662 While there's no technical issue with this solution, MPLS interfaces 663 are mostly found in the network provider domain. Hence not all of 664 the above criteria to select a tunneling protocol are met. 666 Note that setting up a metric test environment isn't a plug and play 667 issue. Skilled networking engineers should be consulted and 668 involved, if a set up between remote sites is preferred. 670 Passing or failing an ADK test with 2 samples could be a random 671 result (note that [RFC2330] defines a sample as a set of singleton 672 metric values produced by a measurement stream, and we continue to 673 use this terminology here). The error margin of a statistical test 674 is higher if the number of samples it is based on is low (the number 675 of samples taken influences the so called "degree of freedom" of a 676 statistical test and a higher degree of freedom produces more 677 reliable results). To pass ADK with higher probability, the number 678 of samples collected per implementation under identical networking 679 conditions SHOULD be greater than 2. Hardware and load constraints 680 may enforce an upper limit on the number of simultaneous measurement 681 streams. The ADK test allows one to combine different samples (see 682 section 9 [ADK]) and then to run a two sample test between combined 683 samples. At least 4 samples per implementation captured under 684 identical networking conditions is RECOMMENDED when comparing 685 different metric implementations by a statistical test. 687 It is RECOMMENDED that tests be carried out by establishing N 688 different parallel measurement flows. Two or three linecards per 689 implementation serving to send or receive measurement flows should be 690 sufficient to create 4 or more parallel measurement flows. Other 691 options are to separate flows by DiffServ marks (without deploying 692 any QoS in the inner or outer tunnel) or using a single CBR flow and 693 evaluating every n-th singleton to belong to a specific measurement 694 flow. Note that a practical test indeed showed that ADK was passed 695 with 4 samples even if a 2 sample test 696 failed[morton-testplan-rfc2679]. 698 Some additional guidelines to calculate and compare samples to 699 perform a metric test are: 701 o To compare different probes of a common underlying distribution in 702 terms of metrics characterising a communication network requires 703 to respect the temporal nature for which the assumption of common 704 underlying distribution may hold. Any singletons or samples to be 705 compared MUST be captured within the same time interval. 707 o If statistical events like rates are used to characterise measured 708 metrics of a time-interval, its RECOMMENDED to pick as a minimum 5 709 singletons of a relevant metric to ensure a minimum confidence 710 into the reported value. The error margin of the determined rate 711 depends on the number singletons (refer to statistical textbooks 712 on Student's t-test). As an example, any packet loss measurement 713 interval to be compared with the results of another implementation 714 contains at least five lost packets to have some confidence that 715 the observed loss rate wasn't caused by a small number of random 716 packet drops. 718 o The minimum number of singletons or samples to be compared by an 719 Anderson-Darling test SHOULD be 100 per tested metric 720 implementation. Note that the Anderson-Darling test detects small 721 differences in distributions fairly well and will fail for high 722 number of compared results (RFC2330 mentions an example with 8192 723 measurements where an Anderson-Darling test always failed). 725 o Generally, the Anderson-Darling test is sensitive to differences 726 in the accuracy or bias associated with varying implementations or 727 test conditions. These dissimilarities may result in differing 728 averages of samples to be compared. An example may be different 729 packet sizes, resulting in a constant delay difference between 730 compared samples. Therefore samples to be compared by an Anderson 731 Darling test MAY be calibrated by the difference of the average 732 values of the samples. Any calibration of this kind MUST be 733 documented in the test result. 735 3.3. Tests of two or more different implementations against a metric 736 specification 738 RFC2330 expects "a methodology for a given metric [to] exhibit 739 continuity if, for small variations in conditions, it results in 740 small variations in the resulting measurements. Slightly more 741 precisely, for every positive epsilon, there exists a positive delta, 742 such that if two sets of conditions are within delta of each other, 743 then the resulting measurements will be within epsilon of each 744 other." A small variation in conditions in the context of the metric 745 test proposed here can be seen as different implementations measuring 746 the same metric along the same path. 748 IPPM metric specifications however allow for implementor options to 749 the largest possible degree. It cannot be expected that two 750 implementors allow 100% identical options in their implementations. 751 Testers SHOULD to the highest degree possible pick the same 752 configurations for their systems when comparing their implementations 753 by a metric test. 755 In some cases, a goodness of fit test may not be possible or show 756 disappointing results. To clarify the difficulties arising from 757 different implementation options, the individual options picked for 758 every compared implementation SHOULD be documented in sufficient 759 detail. Based on this documentation, the underlying metric 760 specification should be improved before it is promoted to a standard. 762 The same statistical test as applicable to quantify precision of a 763 single metric implementation MUST be used to compare metric result 764 equivalence for different implementations. To document 765 compatibility, the smallest measurement resolution at which the 766 compared implementations passed the ADK sample test MUST be 767 documented. 769 For different implementations of the same metric, "variations in 770 conditions" are reasonably expected. The ADK test comparing samples 771 of the different implementations MAY result in a lower precision than 772 the test for precision in the same-implementation comparison. 774 3.4. Clock synchronisation 776 Clock synchronization effects require special attention. Accuracy of 777 one-way active delay measurements for any metrics implementation 778 depends on clock synchronization between the source and destination 779 of tests. Ideally, one-way active delay measurement (RFC 2679, 780 [RFC2679]) test endpoints either have direct access to independent 781 GPS or CDMA-based time sources or indirect access to nearby NTP 782 primary (stratum 1) time sources, equipped with GPS receivers. 783 Access to these time sources may not be available at all test 784 locations associated with different Internet paths, for a variety of 785 reasons out of scope of this document. 787 When secondary (stratum 2 and above) time sources are used with NTP 788 running across the same network, whose metrics are subject to 789 comparative implementation tests, network impairments can affect 790 clock synchronization, distort sample one-way values and their 791 interval statistics. It is RECOMMENDED to discard sample one-way 792 delay values for any implementation, when one of the following 793 reliability conditions is met: 795 o Delay is measured and is finite in one direction, but not the 796 other. 798 o Absolute value of the difference between the sum of one-way 799 measurements in both directions and round-trip measurement is 800 greater than X% of the latter value. 802 Examination of the second condition requires RTT measurement for 803 reference, e.g., based on TWAMP (RFC5357, RFC 5357 [RFC5357]), in 804 conjunction with one-way delay measurement. 806 Specification of X% to strike a balance between identification of 807 unreliable one-way delay samples and misidentification of reliable 808 samples under a wide range of Internet path RTTs probably requires 809 further study. 811 An IPPM compliant metric implementation of an RFC that requires 812 synchronized clocks is expected to provide precise measurement 813 results. 815 IF an implementation publishes a specification of its precision, such 816 as "a precision of 1 ms (+/- 500 us) with a confidence of 95%", then 817 the specification SHOULD be met over a useful measurement duration. 818 For example, if the metric is measured along an Internet path which 819 is stable and not congested, then the precision specification SHOULD 820 be met over durations of an hour or more. 822 3.5. Recommended Metric Verification Measurement Process 824 In order to meet their obligations under the IETF Standards Process 825 the IESG must be convinced that each metric specification advanced to 826 Draft Standard or Internet Standard status is clearly written, that 827 there are a sufficient number of verified equivalent implementations, 828 and that options that have been implemented are documented. 830 In the context of this document, metrics are designed to measure some 831 characteristic of a data network. An aim of any metric definition 832 should be that it should be specified in a way that can reliably 833 measure the specific characteristic in a repeatable way across 834 multiple independent implementations. 836 Each metric, statistic or option of those to be validated MUST be 837 compared against a reference measurement or another implementation by 838 as specified by this document. 840 Finally, the metric definitions, embodied in the text of the RFCs, 841 are the objects that require evaluation and possible revision in 842 order to advance to the next step on the standards track. 844 IF two (or more) implementations do not measure an equivalent metric 845 as specified by this document, 847 AND sources of measurement error do not adequately explain the lack 848 of agreement, 849 THEN the details of each implementation should be audited along with 850 the exact definition text, to determine if there is a lack of clarity 851 that has caused the implementations to vary in a way that affects the 852 correspondence of the results. 854 IF there was a lack of clarity or multiple legitimate interpretations 855 of the definition text, 857 THEN the text should be modified and the resulting memo proposed for 858 consensus and (possible) advancement along the standards track. 860 Finally, all the findings MUST be documented in a report that can 861 support advancement on the standards track, similar to those 862 described in [RFC5657]. The list of measurement devices used in 863 testing satisfies the implementation requirement, while the test 864 results provide information on the quality of each specification in 865 the metric RFC (the surrogate for feature interoperability). 867 The complete process of advancing a metric specification to a 868 standard as defined by this document is illustrated in Figure 4. 870 ,---. 871 / \ 872 ( Start ) 873 \ / Implementations 874 `-+-' +-------+ 875 | /| 1 `. 876 +---+----+ / +-------+ `.-----------+ ,-------. 877 | RFC | / |Check for | ,' was RFC `. YES 878 | | / |Equivalence.... clause x ------+ 879 | |/ +-------+ |under | `. clear? ,' | 880 | Metric \.....| 2 ....relevant | `---+---' +----+-----+ 881 | Metric |\ +-------+ |identical | No | |Report | 882 | Metric | \ |network | +--+----+ |results + | 883 | ... | \ |conditions | |Modify | |Advance | 884 | | \ +-------+ | | |Spec +--+RFC | 885 +--------+ \| n |.'+-----------+ +-------+ |request(?)| 886 +-------+ +----------+ 888 Illustration of the metric standardisation process 890 Figure 4 892 Any recommendation for the advancement of a metric specification MUST 893 be accompanied by an implementation report, as is the case with all 894 requests for the advancement of IETF specifications. The 895 implementation report needs to include the tests performed, the 896 applied test setup, the specific metrics in the RFC and reports of 897 the tests performed with two or more implementations. The test plan 898 needs to specify the precision reached for each measured metric and 899 thus define the meaning of "statistically equivalent" for the 900 specific metrics being tested. 902 Ideally, the test plan would co-evolve with the development of the 903 metric, since that's when people have the most context in their 904 thinking regarding the different subtleties that can arise. 906 In particular, the implementation report MUST as a minimum document: 908 o The metric compared and the RFC specifying it. This includes 909 statements as required by the section "Tests of an individual 910 implementation against a metric specification" of this document. 912 o The measurement configuration and setup. 914 o A complete specification of the measurement stream (mean rate, 915 statistical distribution of packets, packet size or mean packet 916 size and their distribution), DSCP and any other measurement 917 stream properties which could result in deviating results. 918 Deviations in results can be caused also if chosen IP addresses 919 and ports of different implementations can result in different 920 layer 2 or layer 3 paths due to operation of Equal Cost Multi-Path 921 routing in an operational network. 923 o The duration of each measurement to be used for a metric 924 validation, the number of measurement points collected for each 925 metric during each measurement interval (i.e. the probe size) and 926 the level of confidence derived from this probe size for each 927 measurement interval. 929 o The result of the statistical tests performed for each metric 930 validation as required by the section "Tests of two or more 931 different implementations against a metric specification" of this 932 document. 934 o A parameterization of laboratory conditions and applied traffic 935 and network conditions allowing reproduction of these laboratory 936 conditions for readers of the implementation report. 938 o The documentation helping to improve metric specifications defined 939 by this section. 941 All of the tests for each set SHOULD be run in a test setup as 942 specified in the section "Test setup resulting in identical live 943 network testing conditions." 945 If a different test set up is chosen, it is RECOMMENDED to avoid 946 effects falsifying results of validation measurements caused by real 947 data networks (like parallelism in devices and networks). Data 948 networks may forward packets differently in the case of: 950 o Different packet sizes chosen for different metric 951 implementations. A proposed countermeasure is selecting the same 952 packet size when validating results of two samples or a sample 953 against an original distribution. 955 o Selection of differing IP addresses and ports used by different 956 metric implementations during metric validation tests. If ECMP is 957 applied on IP or MPLS level, different paths can result (note that 958 it may be impossible to detect an MPLS ECMP path from an IP 959 endpoint). A proposed counter measure is to connect the 960 measurement equipment to be compared by a NAT device, or 961 establishing a single tunnel to transport all measurement traffic 962 The aim is to have the same IP addresses and port for all 963 measurement packets or to avoid ECMP based local routing diversion 964 by using a layer 2 tunnel. 966 o Different IP options. 968 o Different DSCP. 970 o If the N measurements are captured using sequential measurements 971 instead of simultaneous ones, then the following factors come into 972 play: Time varying paths and load conditions. 974 3.6. Proposal to determine an "equivalence" threshold for each metric 975 evaluated 977 This section describes a proposal for maximum error of "equivalence", 978 based on performance comparison of identical implementations. This 979 comparison may be useful for both ADK and non-ADK comparisons. 981 Each metric tested by two or more implementations (cross- 982 implementation testing). 984 Each metric is also tested twice simultaneously by the *same* 985 implementation, using different Src/Dst Address pairs and other 986 differences such that the connectivity differences of the cross- 987 implementation tests are also experienced and measured by the same 988 implementation. 990 Comparative results for the same implementation represent a bound on 991 cross-implementation equivalence. This should be particularly useful 992 when the metric does *not* produces a continuous distribution of 993 singleton values, such as with a loss metric, or a duplication 994 metric. Appendix A indicates how the ADK will work for 0ne-way 995 delay, and should be likewise applicable to distributions of delay 996 variation. 998 Proposal: the implementation with the largest difference in 999 homogeneous comparison results is the lower bound on the equivalence 1000 threshold, noting that there may be other systematic errors to 1001 account for when comparing between implementations. 1003 Thus, when evaluating equivalence in cross-implementation results: 1005 Maximum_Error = Same_Implementation_Error + Systematic_Error 1007 and only the systematic error need be decided beforehand. 1009 In the case of ADK comparison, the largest same-implementation 1010 resolution of distribution equivalence can be used as a limit on 1011 cross-implementation resolutions (at the same confidence level). 1013 4. Acknowledgements 1015 Gerhard Hasslinger commented a first version of this document, 1016 suggested statistical tests and the evaluation of time series 1017 information. Matthias Wieser's thesis on a metric test resulted in 1018 new input for this draft. Henk Uijterwaal and Lars Eggert have 1019 encouraged and helped to orgainize this work. Mike Hamilton, Scott 1020 Bradner, David Mcdysan and Emile Stephan commented on this draft. 1021 Carol Davids reviewed the 01 version of the ID before it was promoted 1022 to WG draft. 1024 5. Contributors 1026 Scott Bradner, Vern Paxson and Allison Mankin drafted bradner- 1027 metrictest [bradner-metrictest], and major parts of it are included 1028 in this document. 1030 6. IANA Considerations 1032 This memo includes no request to IANA. 1034 7. Security Considerations 1036 This draft does not raise any specific security issues. 1038 8. References 1040 8.1. Normative References 1042 [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, 1043 October 1996. 1045 [RFC2026] Bradner, S., "The Internet Standards Process -- Revision 1046 3", BCP 9, RFC 2026, October 1996. 1048 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1049 Requirement Levels", BCP 14, RFC 2119, March 1997. 1051 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1052 "Framework for IP Performance Metrics", RFC 2330, 1053 May 1998. 1055 [RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, 1056 G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", 1057 RFC 2661, August 1999. 1059 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1060 Delay Metric for IPPM", RFC 2679, September 1999. 1062 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1063 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1065 [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip 1066 Delay Metric for IPPM", RFC 2681, September 1999. 1068 [RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. 1069 Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, 1070 March 2000. 1072 [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling 1073 Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. 1075 [RFC4448] Martini, L., Rosen, E., El-Aawar, N., and G. Heron, 1076 "Encapsulation Methods for Transport of Ethernet over MPLS 1077 Networks", RFC 4448, April 2006. 1079 [RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the- 1080 Network Tunneling", RFC 4459, April 2006. 1082 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1083 Zekauskas, "A One-way Active Measurement Protocol 1084 (OWAMP)", RFC 4656, September 2006. 1086 [RFC4719] Aggarwal, R., Townsley, M., and M. Dos Santos, "Transport 1087 of Ethernet Frames over Layer 2 Tunneling Protocol Version 1088 3 (L2TPv3)", RFC 4719, November 2006. 1090 [RFC4928] Swallow, G., Bryant, S., and L. Andersson, "Avoiding Equal 1091 Cost Multipath Treatment in MPLS Networks", BCP 128, 1092 RFC 4928, June 2007. 1094 [RFC5657] Dusseault, L. and R. Sparks, "Guidance on Interoperation 1095 and Implementation Reports for Advancement to Draft 1096 Standard", BCP 9, RFC 5657, September 2009. 1098 8.2. Informative References 1100 [ADK] Scholz, F. and M. Stephens, "K-sample Anderson-Darling 1101 Tests of fit, for continuous and discrete cases", 1102 University of Washington, Technical Report No. 81, 1103 May 1986. 1105 [GU+Duffield] 1106 Gu, Y., Duffield, N., Breslau, L., and S. Sen, "GRE 1107 Encapsulated Multicast Probing: A Scalable Technique for 1108 Measuring One-Way Loss", SIGMETRICS'07 San Diego, 1109 California, USA, June 2007. 1111 [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. 1112 Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)", 1113 RFC 5357, October 2008. 1115 [Rule of thumb] 1116 Hardy, M., "Confidence interval", March 2010. 1118 [bradner-metrictest] 1119 Bradner, S., Mankin, A., and V. Paxson, "Advancement of 1120 metrics specifications on the IETF Standards Track", 1121 draft -bradner-metricstest-03, (work in progress), 1122 July 2007. 1124 [morton-advance-metrics] 1125 Morton, A., "Problems and Possible Solutions for Advancing 1126 Metrics on the Standards Track", draft -morton-ippm- 1127 advance-metrics-00, (work in progress), July 2009. 1129 [morton-advance-metrics-01] 1130 Morton, A., "Lab Test Results for Advancing Metrics on the 1131 Standards Track", draft -morton-ippm-advance-metrics-01, 1132 (work in progress), June 2010. 1134 [morton-testplan-rfc2679] 1135 Ciavattone, L., Geib, R., Morton, A., and M. Wieser, "Test 1136 Plan and Results for Advancing RFC 2679 on the Standards 1137 Track", draft -morton-ippm-testplan-rfc2679-01, (work in 1138 progress), June 2011. 1140 Appendix A. An example on a One-way Delay metric validation 1142 The text of this appendix is not binding. It is an example how parts 1143 of a One-way Delay metric test could look like. 1144 http://xml.resource.org/public/rfc/bibxml/ 1146 A.1. Compliance to Metric specification requirements 1148 One-way Delay, Loss threshold, RFC 2679 1150 This test determines if implementations use the same configured 1151 maximum waiting time delay from one measurement to another under 1152 different delay conditions, and correctly declare packets arriving in 1153 excess of the waiting time threshold as lost. See Section 3.5 of 1154 RFC2679, 3rd bullet point and also Section 3.8.2 of RFC2679. 1156 (1) Configure a path with 1 sec one-way constant delay. 1158 (2) Measure one-way delay with 2 or more implementations, using 1159 identical waiting time thresholds for loss set at 2 seconds. 1161 (3) Configure the path with 3 sec one-way delay. 1163 (4) Repeat measurements. 1165 (5) Observe that the increase measured in step 4 caused all packets 1166 to be declared lost, and that all packets that arrive 1167 successfully in step 2 are assigned a valid one-way delay. 1169 One-way Delay, First-bit to Last bit, RFC 2679 1171 This test determines if implementations register the same relative 1172 increase in delay from one measurement to another under different 1173 delay conditions. This test tends to cancel the sources of error 1174 which may be present in an implementation. See Section 3.7.2 of 1175 RFC2679, and Section 10.2 of RFC2330. 1177 (1) Configure a path with X ms one-way constant delay, and ideally 1178 including a low-speed link. 1180 (2) Measure one-way delay with 2 or more implementations, using 1181 identical options and equal size small packets (e.g., 100 octet 1182 IP payload). 1184 (3) Maintain the same path with X ms one-way delay. 1186 (4) Measure one-way delay with 2 or more implementations, using 1187 identical options and equal size large packets (e.g., 1500 octet 1188 IP payload). 1190 (5) Observe that the increase measured in steps 2 and 4 is 1191 equivalent to the increase in ms expected due to the larger 1192 serialization time for each implementation. Most of the 1193 measurement errors in each system should cancel, if they are 1194 stationary. 1196 One-way Delay, RFC 2679 1198 This test determines if implementations register the same relative 1199 increase in delay from one measurement to another under different 1200 delay conditions. This test tends to cancel the sources of error 1201 which may be present in an implementation. This test is intended to 1202 evaluate measurments in sections 3 and 4 of RFC2679. 1204 (1) Configure a path with X ms one-way constant delay. 1206 (2) Measure one-way delay with 2 or more implementations, using 1207 identical options. 1209 (3) Configure the path with X+Y ms one-way delay. 1211 (4) Repeat measurements. 1213 (5) Observe that the increase measured in steps 2 and 4 is ~Y ms for 1214 each implementation. Most of the measurement errors in each 1215 system should cancel, if they are stationary. 1217 Error Calibration, RFC 2679 1219 This is a simple check to determine if an implementation reports the 1220 error calibration as required in Section 4.8 of RFC2679. Note that 1221 the context (Type-P) must also be reported. 1223 A.2. Examples related to statistical tests for One-way Delay 1225 A one way delay measurement may pass an ADK test with a timestamp 1226 resultion of 1 ms. The same test may fail, if timestamps with a 1227 resolution of 100 microseconds are eavluated. The implementation 1228 then is then conforming to the metric specification up to a timestamp 1229 resolution of 1 ms. 1231 Let's assume another one way delay measurement comparison between 1232 implementation 1, probing with a frequency of 2 probes per second and 1233 implementation 2 probing at a rate of 2 probes every 3 minutes. To 1234 ensure reasonable confidence in results, sample metrics are 1235 calculated from at least 5 singletons per compared time interval. 1236 This means, sample delay values are calculated for each system for 1237 identical 6 minute intervals for the whole test duration. Per 6 1238 minute interval, the sample metric is calculated from 720 singletons 1239 for implementation 1 and from 6 singletons for implementation 2. 1240 Note, that if outliers are not filtered, moving averages are an 1241 option for an evaluation too. The minimum move of an averaging 1242 interval is three minutes in this example. 1244 The data in table 1 may result from measuring One-Way Delay with 1245 implementation 1 (see column Implemnt_1) and implementation 2 (see 1246 column implemnt_2). Each data point in the table represents a 1247 (rounded) average of the sampled delay values per interval. The 1248 resolution of the clock is one micro-second. The difference in the 1249 delay values may result eg. from different probe packet sizes. 1251 +------------+------------+-----------------------------+ 1252 | Implemnt_1 | Implemnt_2 | Implemnt_2 - Delta_Averages | 1253 +------------+------------+-----------------------------+ 1254 | 5000 | 6549 | 4997 | 1255 | 5008 | 6555 | 5003 | 1256 | 5012 | 6564 | 5012 | 1257 | 5015 | 6565 | 5013 | 1258 | 5019 | 6568 | 5016 | 1259 | 5022 | 6570 | 5018 | 1260 | 5024 | 6573 | 5021 | 1261 | 5026 | 6575 | 5023 | 1262 | 5027 | 6577 | 5025 | 1263 | 5029 | 6580 | 5028 | 1264 | 5030 | 6585 | 5033 | 1265 | 5032 | 6586 | 5034 | 1266 | 5034 | 6587 | 5035 | 1267 | 5036 | 6588 | 5036 | 1268 | 5038 | 6589 | 5037 | 1269 | 5039 | 6591 | 5039 | 1270 | 5041 | 6592 | 5040 | 1271 | 5043 | 6599 | 5047 | 1272 | 5046 | 6606 | 5054 | 1273 | 5054 | 6612 | 5060 | 1274 +------------+------------+-----------------------------+ 1276 Table 1 1278 Average values of sample metrics captured during identical time 1279 intervals are compared. This excludes random differences caused by 1280 differing probing intervals or differing temporal distance of 1281 singletons resulting from their Poisson distributed sending times. 1283 In the example, 20 values have been picked (note that at least 100 1284 values are recommended for a single run of a real test). Data must 1285 be ordered by ascending rank. The data of Implemnt_1 and Implemnt_2 1286 as shown in the first two columns of table 1 clearly fails an ADK 1287 test with 95% confidence. 1289 The results of Implemnt_2 are now reduced by difference of the 1290 averages of column 2 (rounded to 6581 us) and column 1 (rounded to 1291 5029 us), which is 1552 us. The result may be found in column 3 of 1292 table 1. Comparing column 1 and column 3 of the table by an ADK test 1293 shows, that the data contained in these columns passes an ADK tests 1294 with 95% confidence. 1296 >>> Comment: Extensive averaging was used in this example, because of 1297 the vastly different sampling frequencies. As a result, the 1298 distributions compared do not exactly align with a metric in 1300 [RFC2679], but illustrate the ADK process adequately. 1302 Appendix B. Anderson-Darling 2 sample C++ code 1304 /* Routines for computing the Anderson-Darling 2 sample 1305 * test statistic. 1306 * 1307 * Implemented based on the description in 1308 * "Anderson-Darling K Sample Test" Heckert, Alan and 1309 * Filliben, James, editors, Dataplot Reference Manual, 1310 * Chapter 15 Auxiliary, NIST, 2004. 1311 * Official Reference by 2010 1312 * Heckert, N. A. (2001). Dataplot website at the 1313 * National Institute of Standards and Technology: 1314 * http://www.itl.nist.gov/div898/software/dataplot.html/ 1315 * June 2001. 1316 */ 1318 #include 1319 #include 1320 #include 1321 #include 1323 using namespace std; 1325 vector vec1, vec2; 1326 double adk_result; 1327 double adk_criterium = 1.993; 1329 /* vec1 and vec2 to be initialised with sample 1 and 1330 * sample 2 values in ascending order. 1331 */ 1333 /* example for iterating the vectors 1334 * for(vector::iterator it = vec1->begin(); 1335 * it != vec1->end(); it++ 1336 * { 1337 * cout << *it << endl; 1338 * } 1339 */ 1341 static int k, val_st_z_samp1, val_st_z_samp2, 1342 val_eq_z_samp1, val_eq_z_samp2, 1343 j, n_total, n_sample1, n_sample2, L, 1344 max_number_samples, line, maxnumber_z; 1345 static int column_1, column_2; 1346 static double adk, n_value, z, sum_adk_samp1, 1347 sum_adk_samp2, z_aux; 1348 static double H_j, F1j, hj, F2j, denom_1_aux, denom_2_aux; 1349 static bool next_z_sample2, equal_z_both_samples; 1350 static int stop_loop1, stop_loop2, stop_loop3,old_eq_line2, 1351 old_eq_line1; 1353 static double adk_criterium = 1.993; 1355 k = 2; 1356 n_sample1 = vec1->size() - 1; 1357 n_sample2 = vec2->size() - 1; 1359 // -1 because vec[0] is a dummy value 1361 n_total = n_sample1 + n_sample2; 1363 /* value equal to the line with a value = zj in sample 1. 1364 * Here j=1, so the line is 1. 1365 */ 1367 val_eq_z_samp1 = 1; 1369 /* value equal to the line with a value = zj in sample 2. 1370 * Here j=1, so the line is 1. 1371 */ 1373 val_eq_z_samp2 = 1; 1375 /* value equal to the last line with a value < zj 1376 * in sample 1. Here j=1, so the line is 0. 1377 */ 1379 val_st_z_samp1 = 0; 1381 /* value equal to the last line with a value < zj 1382 * in sample 1. Here j=1, so the line is 0. 1383 */ 1385 val_st_z_samp2 = 0; 1387 sum_adk_samp1 = 0; 1388 sum_adk_samp2 = 0; 1389 j = 1; 1391 // as mentioned above, j=1 1393 equal_z_both_samples = false; 1394 next_z_sample2 = false; 1396 //assuming the next z to be of sample 1 1398 stop_loop1 = n_sample1 + 1; 1400 // + 1 because vec[0] is a dummy, see n_sample1 declaration 1402 stop_loop2 = n_sample2 + 1; 1403 stop_loop3 = n_total + 1; 1405 /* The required z values are calculated until all values 1406 * of both samples have been taken into account. See the 1407 * lines above for the stoploop values. Construct required 1408 * to avoid a mathematical operation in the While condition 1409 */ 1411 while (((stop_loop1 > val_eq_z_samp1) 1412 || (stop_loop2 > val_eq_z_samp2)) && stop_loop3 > j) 1413 { 1414 if(val_eq_z_samp1 < n_sample1+1) 1415 { 1417 /* here, a preliminary zj value is set. 1418 * See below how to calculate the actual zj. 1419 */ 1421 z = (*vec1)[val_eq_z_samp1]; 1423 /* this while sequence calculates the number of values 1424 * equal to z. 1425 */ 1427 while ((val_eq_z_samp1+1 < n_sample1) 1428 && z == (*vec1)[val_eq_z_samp1+1] ) 1429 { 1430 val_eq_z_samp1++; 1431 } 1432 } 1433 else 1434 { 1435 val_eq_z_samp1 = 0; 1436 val_st_z_samp1 = n_sample1; 1438 // this should be val_eq_z_samp1 - 1 = n_sample1 1439 } 1441 if(val_eq_z_samp2 < n_sample2+1) 1442 { 1443 z_aux = (*vec2)[val_eq_z_samp2];; 1445 /* this while sequence calculates the number of values 1446 * equal to z_aux 1447 */ 1449 while ((val_eq_z_samp2+1 < n_sample2) 1450 && z_aux == (*vec2)[val_eq_z_samp2+1] ) 1451 { 1452 val_eq_z_samp2++; 1453 } 1455 /* the smaller of the two actual data values is picked 1456 * as the next zj. 1457 */ 1459 if(z > z_aux) 1460 { 1461 z = z_aux; 1462 next_z_sample2 = true; 1463 } 1464 else 1465 { 1466 if (z == z_aux) 1467 { 1468 equal_z_both_samples = true; 1469 } 1471 /* This is the case, if the last value of column1 is 1472 * smaller than the remaining values of column2. 1473 */ 1474 if (val_eq_z_samp1 == 0) 1475 { 1476 z = z_aux; 1477 next_z_sample2 = true; 1478 } 1479 } 1480 } 1481 else 1482 { 1483 val_eq_z_samp2 = 0; 1484 val_st_z_samp2 = n_sample2; 1486 // this should be val_eq_z_samp2 - 1 = n_sample2 1488 } 1490 /* in the following, sum j = 1 to L is calculated for 1491 * sample 1 and sample 2. 1492 */ 1494 if (equal_z_both_samples) 1495 { 1497 /* hj is the number of values in the combined sample 1498 * equal to zj 1499 */ 1500 hj = val_eq_z_samp1 - val_st_z_samp1 1501 + val_eq_z_samp2 - val_st_z_samp2; 1503 /* H_j is the number of values in the combined sample 1504 * smaller than zj plus one half the the number of 1505 * values in the combined sample equal to zj 1506 * (that's hj/2). 1507 */ 1509 H_j = val_st_z_samp1 + val_st_z_samp2 1510 + hj / 2; 1512 /* F1j is the number of values in the 1st sample 1513 * which are less than zj plus one half the number 1514 * of values in this sample which are equal to zj. 1515 */ 1517 F1j = val_st_z_samp1 + (double) 1518 (val_eq_z_samp1 - val_st_z_samp1) / 2; 1520 /* F2j is the number of values in the 1st sample 1521 * which are less than zj plus one half the number 1522 * of values in this sample which are equal to zj. 1523 */ 1524 F2j = val_st_z_samp2 + (double) 1525 (val_eq_z_samp2 - val_st_z_samp2) / 2; 1527 /* set the line of values equal to zj to the 1528 * actual line of the last value picked for zj. 1529 */ 1530 val_st_z_samp1 = val_eq_z_samp1; 1532 /* Set the line of values equal to zj to the actual 1533 * line of the last value picked for zjof each 1534 * sample. This is required as data smaller than zj 1535 * is accounted differently than values equal to zj. 1536 */ 1537 val_st_z_samp2 = val_eq_z_samp2; 1539 /* next the lines of the next values z, ie. zj+1 1540 * are addressed. 1541 */ 1543 val_eq_z_samp1++; 1545 /* next the lines of the next values z, ie. 1546 * zj+1 are addressed 1547 */ 1549 val_eq_z_samp2++; 1550 } 1551 else 1552 { 1554 /* the smaller z value was contained in sample 2, 1555 * hence this value is the zj to base the following 1556 * calculations on. 1557 */ 1558 if (next_z_sample2) 1559 { 1561 /* hj is the number of values in the combined 1562 * sample equal to zj, in this case these are 1563 * within sample 2 only. 1564 */ 1565 hj = val_eq_z_samp2 - val_st_z_samp2; 1567 /* H_j is the number of values in the combined sample 1568 * smaller than zj plus one half the the number of 1569 * values in the combined sample equal to zj 1570 * (that's hj/2). 1571 */ 1573 H_j = val_st_z_samp1 + val_st_z_samp2 1574 + hj / 2; 1576 /* F1j is the number of values in the 1st sample which 1577 * are less than zj plus one half the number of values in 1578 * this sample which are equal to zj. 1579 * As val_eq_z_samp2 < val_eq_z_samp1, these are the 1580 * val_st_z_samp1 only. 1581 */ 1582 F1j = val_st_z_samp1; 1584 /* F2j is the number of values in the 1st sample which 1585 * are less than zj plus one half the number of values in 1586 * this sample which are equal to zj. The latter are from 1587 * sample 2 only in this case. 1588 */ 1590 F2j = val_st_z_samp2 + (double) 1591 (val_eq_z_samp2 - val_st_z_samp2) / 2; 1593 /* Set the line of values equal to zj to the actual line 1594 * of the last value picked for zj of sample 2 only in 1595 * this case. 1596 */ 1597 val_st_z_samp2 = val_eq_z_samp2; 1599 /* next the line of the next value z, ie. zj+1 is 1600 * addressed. Here, only sample 2 must be addressed. 1601 */ 1603 val_eq_z_samp2++; 1604 if (val_eq_z_samp1 == 0) 1605 { 1606 val_eq_z_samp1 = stop_loop1; 1607 } 1608 } 1610 /* the smaller z value was contained in sample 2, 1611 * hence this value is the zj to base the following 1612 * calculations on. 1613 */ 1615 else 1616 { 1618 /* hj is the number of values in the combined 1619 * sample equal to zj, in this case these are 1620 * within sample 1 only. 1621 */ 1622 hj = val_eq_z_samp1 - val_st_z_samp1; 1624 /* H_j is the number of values in the combined 1625 * sample smaller than zj plus one half the the number 1626 * of values in the combined sample equal to zj 1627 * (that's hj/2). 1628 */ 1630 H_j = val_st_z_samp1 + val_st_z_samp2 1631 + hj / 2; 1633 /* F1j is the number of values in the 1st sample which 1634 * are less than zj plus, in this case these are within 1635 * sample 1 only one half the number of values in this 1636 * sample which are equal to zj. The latter are from 1637 * sample 1 only in this case. 1638 */ 1640 F1j = val_st_z_samp1 + (double) 1641 (val_eq_z_samp1 - val_st_z_samp1) / 2; 1643 /* F2j is the number of values in the 1st sample which 1644 * are less than zj plus one half the number of values 1645 * in this sample which are equal to zj. As 1646 * val_eq_z_samp1 < val_eq_z_samp2, these are the 1647 * val_st_z_samp2 only. 1648 */ 1650 F2j = val_st_z_samp2; 1652 /* Set the line of values equal to zj to the actual line 1653 * of the last value picked for zj of sample 1 only in 1654 * this case 1655 */ 1657 val_st_z_samp1 = val_eq_z_samp1; 1659 /* next the line of the next value z, ie. zj+1 is 1660 * addressed. Here, only sample 1 must be addressed. 1661 */ 1662 val_eq_z_samp1++; 1664 if (val_eq_z_samp2 == 0) 1665 { 1666 val_eq_z_samp2 = stop_loop2; 1667 } 1668 } 1669 } 1671 denom_1_aux = n_total * F1j - n_sample1 * H_j; 1672 denom_2_aux = n_total * F2j - n_sample2 * H_j; 1674 sum_adk_samp1 = sum_adk_samp1 + hj 1675 * (denom_1_aux * denom_1_aux) / 1676 (H_j * (n_total - H_j) 1677 - n_total * hj / 4); 1678 sum_adk_samp2 = sum_adk_samp2 + hj 1679 * (denom_2_aux * denom_2_aux) / 1680 (H_j * (n_total - H_j) 1682 - n_total * hj / 4); 1684 next_z_sample2 = false; 1685 equal_z_both_samples = false; 1687 /* index to count the z. It is only required to prevent 1688 * the while slope to execute endless 1689 */ 1690 j++; 1691 } 1693 // calculating the adk value is the final step. 1695 adk_result = (double) (n_total - 1) / (n_total 1696 * n_total * (k - 1)) 1697 * (sum_adk_samp1 / n_sample1 1698 + sum_adk_samp2 / n_sample2); 1700 /* if(adk_result <= adk_criterium) 1701 * adk_2_sample test is passed 1702 */ 1704 Figure 5 1706 Appendix C. Glossary 1708 +-------------+-----------------------------------------------------+ 1709 | ADK | Anderson-Darling K-Sample test, a test used to | 1710 | | check whether two samples have the same statistical | 1711 | | distribution. | 1712 | ECMP | Equal Cost Multipath, a load balancing mechanism | 1713 | | evaluating MPLS labels stacks, IP addresses and | 1714 | | ports. | 1715 | EDF | The "Empirical Distribution Function" of a set of | 1716 | | scalar measurements is a function F(x) which for | 1717 | | any x gives the fractional proportion of the total | 1718 | | measurements that were smaller than or equal as x. | 1719 | Metric | A measured quantity related to the performance and | 1720 | | reliability of the Internet, expressed by a value. | 1721 | | This could be a singleton (single value), a sample | 1722 | | of single values or a statistic based on a sample | 1723 | | of singletons. | 1724 | OWAMP | One-way Active Measurement Protocol, a protocol for | 1725 | | communication between IPPM measurement systems | 1726 | | specified by IPPM. | 1727 | OWD | One-Way Delay, a performance metric specified by | 1728 | | IPPM. | 1729 | Sample | A sample metric is derived from a given singleton | 1730 | metric | metric by evaluating a number of distinct instances | 1731 | | together. | 1732 | Singleton | A singleton metric is, in a sense, one atomic | 1733 | metric | measurement of this metric. | 1734 | Statistical | A 'statistical' metric is derived from a given | 1735 | metric | sample metric by computing some statistic of the | 1736 | | values defined by the singleton metric on the | 1737 | | sample. | 1738 | TWAMP | Two-way Active Measurement Protocol, a protocol for | 1739 | | communication between IPPM measurement systems | 1740 | | specified by IPPM. | 1741 +-------------+-----------------------------------------------------+ 1743 Table 2 1745 Authors' Addresses 1747 Ruediger Geib (editor) 1748 Deutsche Telekom 1749 Heinrich Hertz Str. 3-7 1750 Darmstadt, 64295 1751 Germany 1753 Phone: +49 6151 58 12747 1754 Email: Ruediger.Geib@telekom.de 1756 Al Morton 1757 AT&T Labs 1758 200 Laurel Avenue South 1759 Middletown, NJ 07748 1760 USA 1762 Phone: +1 732 420 1571 1763 Fax: +1 732 368 1192 1764 Email: acmorton@att.com 1765 URI: http://home.comcast.net/~acmacm/ 1766 Reza Fardid 1767 Cariden Technologies 1768 888 Villa Street, Suite 500 1769 Mountain View, CA 94041 1770 USA 1772 Phone: 1773 Email: rfardid@cariden.com 1775 Alexander Steinmitz 1776 Deutsche Telekom 1777 Memmelsdorfer Str. 209b 1778 Bamberg, 96052 1779 Germany 1781 Phone: 1782 Email: Alexander.Steinmitz@telekom.de