idnits 2.17.00 (12 Aug 2021) /tmp/idnits55038/draft-ietf-ippm-delay-var-as-02.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- ** The document seems to lack a License Notice according IETF Trust Provisions of 28 Dec 2009, Section 6.b.i or Provisions of 12 Sep 2009 Section 6.b -- however, there's a paragraph with a matching beginning. Boilerplate error? -- It seems you're using the 'non-IETF stream' Licence Notice instead Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- No issues found here. Miscellaneous warnings: ---------------------------------------------------------------------------- == The copyright year in the IETF Trust and authors Copyright Line does not match the current year == The document doesn't use any RFC 2119 keywords, yet seems to have RFC 2119 boilerplate text. -- The document seems to lack a disclaimer for pre-RFC5378 work, but may have content which was first submitted before 10 November 2008. If you have contacted all the original authors and they are all willing to grant the BCP78 rights to the IETF Trust, then this is fine, and you can ignore this comment. If not, you may need to add the pre-RFC5378 disclaimer. (See the Legal Provisions document at https://trustee.ietf.org/license-info for more information.) -- The document date (January 8, 2009) is 4874 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- ** Obsolete normative reference: RFC 2679 (Obsoleted by RFC 7679) ** Obsolete normative reference: RFC 2680 (Obsoleted by RFC 7680) == Outdated reference: draft-ietf-ippm-framework-compagg has been published as RFC 5835 == Outdated reference: draft-ietf-ippm-spatial-composition has been published as RFC 6049 == Outdated reference: A later version (-07) exists of draft-morton-ippm-reporting-metrics-06 -- Obsolete informational reference (is this intentional?): RFC 1305 (Obsoleted by RFC 5905) Summary: 3 errors (**), 0 flaws (~~), 5 warnings (==), 4 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Network Working Group A. Morton 3 Internet-Draft AT&T Labs 4 Intended status: Informational B. Claise 5 Expires: July 12, 2009 Cisco Systems, Inc. 6 January 8, 2009 8 Packet Delay Variation Applicability Statement 9 draft-ietf-ippm-delay-var-as-02 11 Status of this Memo 13 This Internet-Draft is submitted to IETF in full conformance with the 14 provisions of BCP 78 and BCP 79. 16 Internet-Drafts are working documents of the Internet Engineering 17 Task Force (IETF), its areas, and its working groups. Note that 18 other groups may also distribute working documents as Internet- 19 Drafts. 21 Internet-Drafts are draft documents valid for a maximum of six months 22 and may be updated, replaced, or obsoleted by other documents at any 23 time. It is inappropriate to use Internet-Drafts as reference 24 material or to cite them other than as "work in progress." 26 The list of current Internet-Drafts can be accessed at 27 http://www.ietf.org/ietf/1id-abstracts.txt. 29 The list of Internet-Draft Shadow Directories can be accessed at 30 http://www.ietf.org/shadow.html. 32 This Internet-Draft will expire on July 12, 2009. 34 Copyright Notice 36 Copyright (c) 2009 IETF Trust and the persons identified as the 37 document authors. All rights reserved. 39 This document is subject to BCP 78 and the IETF Trust's Legal 40 Provisions Relating to IETF Documents 41 (http://trustee.ietf.org/license-info) in effect on the date of 42 publication of this document. Please review these documents 43 carefully, as they describe your rights and restrictions with respect 44 to this document. 46 Abstract 48 Packet delay variation metrics appear in many different standards 49 documents. The metric definition in RFC 3393 has considerable 50 flexibility, and it allows multiple formulations of delay variation 51 through the specification of different packet selection functions. 53 Although flexibility provides wide coverage and room for new ideas, 54 it can make comparisons of independent implementations more 55 difficult. Two different formulations of delay variation have come 56 into wide use in the context of active measurements. This memo 57 examines a range of circumstances for active measurements of delay 58 variation and their uses, and recommends which of the two forms is 59 best matched to particular conditions and tasks. 61 Requirements Language 63 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 64 "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this 65 document are to be interpreted as described in RFC 2119 [RFC2119]. 67 Table of Contents 69 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 70 1.1. Background Literature in IPPM and Elsewhere . . . . . . . 6 71 1.2. Organization of the Memo . . . . . . . . . . . . . . . . . 7 72 2. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 7 73 3. Brief Descriptions of Delay Variation Uses . . . . . . . . . . 8 74 3.1. Inferring Queue Occupation on a Path . . . . . . . . . . . 8 75 3.2. Determining De-jitter Buffer Size . . . . . . . . . . . . 9 76 3.3. Spatial Composition . . . . . . . . . . . . . . . . . . . 10 77 3.4. Service Level Comparison . . . . . . . . . . . . . . . . . 11 78 3.5. Application-Layer FEC Design . . . . . . . . . . . . . . . 11 79 4. Formulations of IPDV and PDV . . . . . . . . . . . . . . . . . 11 80 4.1. IPDV: Inter-Packet Delay Variation . . . . . . . . . . . . 11 81 4.2. PDV: Packet Delay Variation . . . . . . . . . . . . . . . 12 82 4.3. A "Point" about Measurement Points . . . . . . . . . . . . 12 83 4.4. Examples and Initial Comparisons . . . . . . . . . . . . . 13 84 5. Survey of Earlier Comparisons . . . . . . . . . . . . . . . . 14 85 5.1. Demichelis' Comparison . . . . . . . . . . . . . . . . . . 14 86 5.2. Ciavattone et al. . . . . . . . . . . . . . . . . . . . . 15 87 5.3. IPPM List Discussion from 2000 . . . . . . . . . . . . . . 16 88 5.4. Y.1540 Appendix II . . . . . . . . . . . . . . . . . . . . 18 89 5.5. Clark's ITU-T SG 12 Contribution . . . . . . . . . . . . . 18 90 6. Additional Properties and Comparisons . . . . . . . . . . . . 18 91 6.1. Packet Loss . . . . . . . . . . . . . . . . . . . . . . . 18 92 6.2. Path Changes . . . . . . . . . . . . . . . . . . . . . . . 19 93 6.2.1. Lossless Path Change . . . . . . . . . . . . . . . . . 20 94 6.2.2. Path Change with Loss . . . . . . . . . . . . . . . . 21 95 6.3. Clock Stability and Error . . . . . . . . . . . . . . . . 22 96 6.4. Spatial Composition . . . . . . . . . . . . . . . . . . . 24 97 6.5. Reporting a Single Number (SLA) . . . . . . . . . . . . . 24 98 6.6. Jitter in RTCP Reports . . . . . . . . . . . . . . . . . . 25 99 6.7. MAPDV2 . . . . . . . . . . . . . . . . . . . . . . . . . . 25 100 6.8. Load Balancing . . . . . . . . . . . . . . . . . . . . . . 26 101 7. Applicability of the Delay Variation Forms and 102 Recommendations . . . . . . . . . . . . . . . . . . . . . . . 27 103 7.1. Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 104 7.1.1. Inferring Queue Occupancy . . . . . . . . . . . . . . 27 105 7.1.2. Determining De-jitter Buffer Size (and FEC Design) . . 27 106 7.1.3. Spatial Composition . . . . . . . . . . . . . . . . . 28 107 7.1.4. Service Level Specification: Reporting a Single 108 Number . . . . . . . . . . . . . . . . . . . . . . . . 28 109 7.2. Challenging Circumstances . . . . . . . . . . . . . . . . 28 110 7.2.1. Clock and Storage Issues . . . . . . . . . . . . . . . 28 111 7.2.2. Frequent Path Changes . . . . . . . . . . . . . . . . 29 112 7.2.3. Frequent Loss . . . . . . . . . . . . . . . . . . . . 29 113 7.2.4. Load Balancing . . . . . . . . . . . . . . . . . . . . 29 114 7.3. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 29 116 8. Measurement Considerations . . . . . . . . . . . . . . . . . . 30 117 8.1. Measurement Stream Characteristics . . . . . . . . . . . . 31 118 8.2. Measurement Devices . . . . . . . . . . . . . . . . . . . 32 119 8.3. Units of Measurement . . . . . . . . . . . . . . . . . . . 32 120 8.4. Test Duration . . . . . . . . . . . . . . . . . . . . . . 33 121 8.5. Clock Sync Options . . . . . . . . . . . . . . . . . . . . 33 122 8.6. Distinguishing Long Delay from Loss . . . . . . . . . . . 33 123 8.7. Accounting for Packet Reordering . . . . . . . . . . . . . 34 124 8.8. Results Representation and Reporting . . . . . . . . . . . 34 125 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 35 126 10. Security Considerations . . . . . . . . . . . . . . . . . . . 35 127 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 35 128 12. Appendix on Calculating the D(min) in PDV . . . . . . . . . . 35 129 13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 36 130 13.1. Normative References . . . . . . . . . . . . . . . . . . . 36 131 13.2. Informative References . . . . . . . . . . . . . . . . . . 37 132 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 38 134 1. Introduction 136 There are many ways to formulate packet delay variation metrics for 137 the Internet and other packet-based networks. The IETF itself has 138 several specifications for delay variation [RFC3393], sometimes 139 called jitter [RFC3550] or even inter-arrival jitter [RFC3550], and 140 these have achieved wide adoption. The International 141 Telecommunication Union - Telecommunication Standardization Sector 142 (ITU-T) has also recommended several delay variation metrics (called 143 parameters in their terminology) [Y.1540] [G.1020], and some of these 144 are widely cited and used. Most of the standards above specify more 145 than one way to quantify delay variation, so one can conclude that 146 standardization efforts have tended to be inclusive rather than 147 selective. 149 This memo uses the term "delay variation" for metrics that quantify a 150 path's ability to transfer packets with consistent delay. [RFC3393] 151 and [Y.1540] both prefer this term. Some refer to this phenomenon as 152 "jitter" (and the buffers that attempt to smooth the variations as 153 de-jitter buffers). Applications of the term "jitter" are much 154 broader than packet transfer performance, with "unwanted signal 155 variation" as a general definition. "Jitter" has been used to 156 describe frequency or phase variations, such as data stream rate 157 variations or carrier signal phase noise. The phrase "delay 158 variation" is almost self-defining and more precise, so it is 159 preferred in this memo. 161 Most (if not all) delay variation metrics are derived metrics, in 162 that their definitions rely on another fundamental metric. In this 163 case, the fundamental metric is one-way delay, and variation is 164 assessed by computing the difference between two individual one-way 165 delay measurements, or a pair of singletons. One of the delay 166 singletons is taken as a reference, and the result is the variation 167 with respect to the reference. The variation is usually summarized 168 for all packets in a stream using statistics. 170 The industry has predominantly implemented two specific formulations 171 of delay variation (for one survey of the situation, 172 see[Krzanowski]): 174 1. Inter-Packet Delay Variation, IPDV, where the reference is the 175 previous packet in the stream (according to sending sequence), 176 and the reference changes for each packet in the stream. 177 Properties of variation are coupled with packet sequence in this 178 formulation. This form was called Instantaneous Packet Delay 179 Variation in early IETF contributions, and is similar to the 180 packet spacing difference metric used for interarrival jitter 181 calculations in [RFC3550]. 183 2. Packet Delay Variation, PDV, where a single reference is chosen 184 from the stream based on specific criteria. The most common 185 criterion for the reference is the packet with the minimum delay 186 in the sample. This term derives its name from a similar 187 definition for Cell Delay Variation, an ATM performance metric 188 [I.356]. 190 It is important to note that the authors of relevant standards for 191 delay variation recognized there are many different users with 192 varying needs, and allowed sufficient flexibility to formulate 193 several metrics with different properties. Therefore, the comparison 194 is not so much between standards bodies or their specifications as it 195 is between specific formulations of delay variation. Both Inter- 196 Packet Delay Variation and Packet Delay Variation are compliant with 197 [RFC3393], because different packet selection functions will produce 198 either form. 200 1.1. Background Literature in IPPM and Elsewhere 202 With more people joining the measurement community every day, it is 203 possible this memo is the first from the IP Performance Metrics 204 (IPPM) Working Group that the reader has consulted. This section 205 provides a brief roadmap and background on the IPPM literature, and 206 the published specifications of other relevant standards 207 organizations. 209 The IPPM framework [RFC2330] provides a background for this memo and 210 other IPPM RFCs. Key terms such as singleton, sample, and statistic 211 are defined there, along with methods of collecting samples (Poisson 212 streams), time related issues, and the "packet of Type-P" convention. 214 There are two fundamental and related metrics that can be applied to 215 every packet transfer attempt: one-way loss [RFC2680] and one-way 216 delay [RFC2679]. The metrics use a waiting time threshold to 217 distinguish between lost and delayed packets. Packets that arrive at 218 the measurement destination within their waiting time have finite 219 delay and are not lost. Otherwise, packets are designated lost and 220 their delay is undefined. Guidance on setting the waiting time 221 threshold may be found in [RFC2680] and 222 [I-D.morton-ippm-reporting-metrics]. 224 Another fundamental metric is packet reordering as specified in 225 [RFC4737]. The reordering metric was defined to be "orthogonal" to 226 packet loss. In other words, the gap in a packet sequence caused by 227 loss does not result in reordered packets, but a re-arrangement of 228 packet arrivals from their sending order constitutes reordering. 230 Derived metrics are based on the fundamental metrics. The metric of 231 primary interest here is delay variation [RFC3393], a metric which is 232 derived from one-way delay [RFC2680]. Another derived metric is the 233 loss patterns metric [RFC3357], which is derived from loss. 235 The measured values of all metrics (both fundamental and derived) 236 depend to great extent on the stream characteristics used to collect 237 them. Both Poisson streams [RFC3393] and Periodic streams [RFC3432] 238 have been used with the IPDV and PDV metrics. The choice of stream 239 specifications for active measurement will depend on the purpose of 240 the characterization and the constraints of the testing environment. 241 Periodic streams are frequently chosen for use with IPDV and PDV, 242 because the application streams that are most sensitive to delay 243 variation exhibit periodicity. Additional details that are method- 244 specific are discussed the section on Measurement Considerations. 246 In the ITU-T, the framework, fundamental metrics and derived metrics 247 for IP performance are specified in Recommendation Y.1540 [Y.1540]. 248 [G.1020] defines additional delay variation metrics, analyses the 249 operation of fixed and adaptive de-jitter buffers, and describes an 250 example adaptive de-jitter buffer emulator. Appendix II of [G.1050] 251 describes the models for network impairments (including delay 252 variation) that are part of standardized IP network emulator which 253 may be useful when evaluating measurement techniques. 255 1.2. Organization of the Memo 257 The Purpose and Scope follows in Section 2. We then give a summary 258 of the main tasks for delay variation metrics in section 3. Section 259 4 defines the two primary forms of delay variation, and section 5 260 presents summaries of four earlier comparisons. Section 6 adds new 261 comparisons to the analysis, and section 7 reviews the applicability 262 and recommendations for each form of delay variation. Section 8 then 263 looks at many important delay variation measurement considerations. 264 Following the IANA and Security Considerations, there is an Appendix 265 on the calculation of the minimum delay for the PDV form. 267 2. Purpose and Scope 269 The IPDV and PDV formulations have certain features that make them 270 more suitable for one circumstance and less so for another. The 271 purpose of this memo is to compare two forms of delay variation, so 272 that it will be evident which of the two is better suited for each of 273 many possible uses and their related circumstances. 275 The scope of this memo is limited to the two forms of delay variation 276 briefly described above (Inter-Packet Delay Variation and Packet 277 Delay Variation), circumstances related to active measurement, and 278 uses that are deemed relevant and worthy of inclusion here through 279 IPPM Working Group consensus. 281 It is entirely possible that the analysis and conclusions drawn here 282 are applicable beyond the intended scope, but the reader is cautioned 283 to fully appreciate the circumstances of active measurement on IP 284 networks before doing so. 286 The scope excludes assessment of delay variation for packets with 287 undefined delay. This is accomplished by conditioning the delay 288 distribution on arrival within a reasonable waiting time based on an 289 understanding of the path under test and packet lifetimes. The 290 waiting time is sometimes called the loss threshold [RFC2680]: if a 291 packet arrives beyond this threshold, it may as well have been lost 292 because it is no longer useful. This is consistent with [RFC3393], 293 where the Type-P-One-way-ipdv is undefined when the destination fails 294 to receive one or both packets in the selected pair. Furthermore, it 295 is consistent with application performance analysis to consider only 296 arriving packets, because a finite waiting time-out is a feature of 297 many protocols. 299 3. Brief Descriptions of Delay Variation Uses 301 This section presents a set of tasks that call for delay variation 302 measurements. Here, the memo provides several answers to the 303 question, "How will the results be used?" for the delay variation 304 metric. 306 3.1. Inferring Queue Occupation on a Path 308 As packets travel along the path from source to destination, they 309 pass through many network elements, including a series of router 310 queues. Some types of the delay sources along the path are constant, 311 such as links between two locations. But the latency encountered in 312 each queue varies, depending on the number of packets in the queue 313 when a particular packet arrives. If one assumes that at least one 314 of the packets in a test stream encounters virtually empty queues all 315 along the path (and the path is stable), then the additional delay 316 observed on other packets can be attributed to the time spent in one 317 or more queues. Otherwise, the delay variation observed is the 318 variation in queue time experienced by the test stream. 320 It is worth noting that delay variation can occur beyond IP router 321 queues, in other communication components. Examples include media 322 contention: DOCSIS, IEEE 802.11 and some mobile radio technologies. 323 However, delay variation from all sources at the IP layer and below 324 will be quantified using the two formulations discussed here. 326 3.2. Determining De-jitter Buffer Size 328 Note - while this memo and other IPPM literature prefer the term 329 delay variation, the terms "jitter buffer" and the more accurate "de- 330 jitter buffer" are widely adopted names for a component of packet 331 communication systems, and they will be used here to designate that 332 system component. 334 Most Isochronous applications (a.k.a. real-time applications) employ 335 a buffer to smooth out delay variation encountered on the path from 336 source to destination. The buffer must be big enough to accommodate 337 the expected variation of delay, or packet loss will result. 338 However, if the buffer is too large, then some of the desired 339 spontaneity of communication will be lost and conversational dynamics 340 will be affected. Therefore, application designers need to know the 341 range of delay variation they must accommodate, whether they are 342 designing fixed or adaptive buffer systems. 344 Network service providers also attempt to constrain delay variation 345 to ensure the quality of real-time applications, and monitor this 346 metric (possibly to compare with a numerical objective or Service 347 Level Agreement). 349 De-jitter buffer size can be expressed in units of octets of storage 350 space for the packet stream, or in units of time that the packets are 351 stored. It is relatively simple to convert between octets and time 352 when the buffer read rate (in octets per second) is constant: 354 read_rate * storage_time = storage_octets 356 Units of time are used in the discussion below. 358 The objective of a de-jitter buffer is to compensate for all prior 359 sources of delay variation and produce a packet stream with constant 360 delay. Thus, a packet experiencing the minimum transit delay from 361 source to destination, D_min, should spend the maximum time in a de- 362 jitter buffer, B_max. The sum of D_min and B_max should equal the 363 sum of the maximum transit delay (D_max) and the minimum buffer time 364 (B_min). We have 366 Constant = D_min + B_max = D_max + B_min, 368 after rearranging terms, 370 B_max - B_min = D_max - D_min = range(B) = range(D) 372 where range(B) is the range of packet buffering times, and range(D) 373 is the range of packet transit delays from source to destination. 375 Packets with transit delay between the max and min spend a 376 complimentary time in the buffer and also see the constant delay. 378 In practice, the minimum buffer time, B_min, may not be zero, and the 379 maximum transit delay, D_max may be a high percentile (99.9%-ile) 380 instead of the maximum. 382 Note that B_max - B_min = range(B) is the range of buffering times 383 needed to compensate for delay variation. The actual size of the 384 buffer may be larger (where B_min > 0) or smaller than range(B). 386 There must be a process to align the de-jitter buffer time with 387 packet transit delay. This is a process to identify the packets with 388 minimum delay and schedule their play-out time so that they spend the 389 maximum time in the buffer. The error in the alignment process can 390 be accounted for by a variable, A. In the equation below, the range 391 of buffering times *available* to the packet stream, range(b), 392 depends on buffer alignment with the actual arrival times of D_min 393 and D_max. 395 range(b) = b_max - b_min = D_max - D_min + A 397 where variable b represents the *available* buffer in a system with a 398 specific alignment, A, and b_max and b_min represent the limits of 399 the available buffer. 401 When A is positive, the de-jitter buffer applies more delay than 402 necessary (where Constant = D_max+b_min+A represents one possible 403 alignment). When A is negative, there is insufficient buffer time 404 available to compensate for range(D) because of mis-alignment. 405 Packets with D_min may be arriving too early and encountering a full 406 buffer, or packets with D_max may be arriving too late, and in either 407 case the packets would be discarded. 409 In summary, the range of transit delay variation is a critical factor 410 in the determination of de-jitter buffer size. 412 3.3. Spatial Composition 414 In Spatial Composition, the tasks are similar to those described 415 above, but with the additional complexity of a multiple network path 416 where several sub-paths are measured separately and no source to 417 destination measurements are available. In this case, the source to 418 destination performance must be estimated, using Composed Metrics as 419 described in [I-D.ietf-ippm-framework-compagg] and [Y.1541]. Note 420 that determining the composite delay variation is not trivial: simply 421 summing the sub-path variations is not accurate. 423 3.4. Service Level Comparison 425 IP performance measurements are often used as the basis for 426 agreements (or contracts) between service providers and their 427 customers. The measurement results must compare favorably with the 428 performance levels specified in the agreement. 430 Packet delay variation is usually one of the metrics specified in 431 these agreements. In principle, any formulation could be specified 432 in the Service Level Agreement (SLA). However, the SLA is most 433 useful when the measured quantities can be related to ways in which 434 the communication service will be utilized by the customer, and this 435 can usually be derived from one of the tasks described above. 437 3.5. Application-Layer FEC Design 439 The design of application-layer Forward Error Correction (FEC) 440 components is closely related to the design of a de-jitter buffer in 441 several ways. The FEC designer must choose a protection interval 442 (time to send/receive a block of packets in a constant packet rate 443 system) consistent with the packet loss characteristics, but also 444 mindful of the extent of delay variation expected. Further, the 445 system designer must decide how long to wait for "late" packets to 446 arrive. Again, the range of delay variation is the relevant 447 expression delay variation for these tasks. 449 4. Formulations of IPDV and PDV 451 This section presents the formulations of IPDV and PDV, and provides 452 some illustrative examples. We use the basic singleton definition in 453 [RFC3393] (which itself is based on [RFC2679]): 455 "Type-P-One-way-ipdv is defined for two packets from Src to Dst 456 selected by the selection function F, as the difference between the 457 value of the Type-P-One-way-delay from Src to Dst at T2 and the value 458 of the Type-P-One-Way-Delay from Src to Dst at T1." 460 4.1. IPDV: Inter-Packet Delay Variation 462 If we have packets in a stream consecutively numbered i = 1,2,3,... 463 falling within the test interval, then IPDV(i) = D(i)-D(i-1) where 464 D(i) denotes the one-way-delay of the ith packet of a stream. 466 One-way delays are the difference between timestamps applied at the 467 ends of the path, or the receiver time minus the transmission time. 468 So D(2) = R2-T2. With this timestamp notation, it can be shown that 469 IPDV also represents the change in inter-packet spacing between 470 transmission and reception: 472 IPDV(2) = D(2) - D(1) = (R2-T2) - (R1-T1) = (R2-R1) - (T2-T1) 474 An example selection function given in [RFC3393] is "Consecutive 475 Type-P packets within the specified interval." This is exactly the 476 function needed for IPDV. The reference packet in the pair is always 477 the previous packet in the sending sequence. 479 Note that IPDV can take on positive and negative values (and zero). 480 One way to analyze the IPDV results is to concentrate on the positive 481 excursions. However, this approach has limitations that are 482 discussed in more detail below (see section 5.3). 484 The mean of all IPDV(i) for a stream is usually zero. However, a 485 slow delay change over the life of the stream, or a frequency error 486 between the measurement system clocks, can result in a non-zero mean. 488 4.2. PDV: Packet Delay Variation 490 The name Packet Delay Variation is used in [Y.1540] and its 491 predecessors, and refers to a performance parameter equivalent to the 492 metric described below. 494 The Selection Function for PDV requires two specific roles for the 495 packets in the pair. The first packet is any Type-P packet within 496 the specified interval. The second, or reference packet is the 497 Type-P packet within the specified interval with the minimum one-way- 498 delay. 500 Therefore, PDV(i) = D(i)-D(min) (using the nomenclature introduced in 501 the IPDV section). D(min) is the delay of the packet with the lowest 502 value for delay (minimum) over the current test interval. Values of 503 PDV may be zero or positive, and quantiles of the PDV distribution 504 are direct indications of delay variation. 506 PDV is a version of the one-way delay distribution, shifted to the 507 origin by normalizing to the minimum delay. 509 4.3. A "Point" about Measurement Points 511 Both IPDV and PDV are derived from the one-way delay metric. One way 512 delay requires knowledge of time at two points, e.g., the source and 513 destination of an IP network path in end-to-end measurement. 514 Therefore, both IPDV and PDV can be categorized as 2-point metrics 515 because they are derived from one-way delay. Specific methods of 516 measurement may make assumptions or have a priori knowledge about one 517 of the measurement points, but the metric definitions themselves are 518 based on information collected at two measurement points. 520 4.4. Examples and Initial Comparisons 522 Note: This material originally presented in slides 2 and 3 of 523 [Morton06]. 525 The Figure below gives a sample of packet delays and calculates IPDV 526 and PDV values and depicts a histogram for each one. 528 Packet # 1 2 3 4 5 529 ------------------------------- 530 Delay, ms 20 10 20 25 20 532 IPDV U -10 10 5 -5 534 PDV 10 0 10 15 10 536 | | 537 4| 4| 538 | | 539 3| 3| H 540 | | H 541 2| 2| H 542 | | H 543 H H 1| H H 1|H H H 544 H H | H H |H H H 545 ---------+-------- +--------------- 546 -10 -5 0 5 10 0 5 10 15 548 IPDV Histogram PDV Histogram 550 Figure 1: IPDV and PDV Comparison 552 The sample of packets contains three packets with "typical" delays of 553 20ms, one packet with a low delay of 10ms (the minimum of the sample) 554 and one packet with 25ms delay. 556 As noted above, this example illustrates that IPDV may take on 557 positive and negative values, while the PDV values are greater than 558 or equal to zero. The Histograms of IPDV and PDV are quite different 559 in general shape, and the ranges are different, too (IPDV range = 560 20ms, PDV range = 15 ms). Note that the IPDV histogram will change 561 if the sequence of delays is modified, but the PDV histogram will 562 stay the same. PDV normalizes the one-way delay distribution to the 563 minimum delay and emphasizes the variation independent from the 564 sequence of delays. 566 5. Survey of Earlier Comparisons 568 This section summarizes previous work to compare these two forms of 569 delay variation. 571 5.1. Demichelis' Comparison 573 In [Demichelis], Demichelis compared the early draft versions of two 574 forms of delay variation. Although the IPDV form would eventually 575 see widespread use, the ITU-T work-in-progress he cited did not 576 utilize the same reference packets as PDV. Demichelis compared IPDV 577 with the alternatives of using the delay of the first packet in the 578 stream and the mean delay of the stream as the PDV reference packet. 579 Neither of these alternative references were used in practice, and 580 they are now deprecated in favor of the minimum delay of the stream 581 [Y.1540]. 583 Active measurements of a transcontinental path (Torino to Tokyo) 584 provided the data for the comparison. The Poisson test stream had 585 0.764 second average inter-packet interval, with more than 58 586 thousand packets over 13.5 hours. Among Demichelis' observations 587 about IPDV are the following: 589 1. IPDV is a measure of the network's ability to preserve the 590 spacing between packets. 592 2. The distribution of IPDV is usually symmetrical about the origin, 593 having a balance of negative and positive values (for the most 594 part). The mean is usually zero, unless some long-term delay 595 trend is present. 597 3. IPDV singletons distinguish quick delay variations (short-term, 598 on the order of the interval between packets) from longer term 599 variations. 601 4. IPDV places reduced demands on the stability and skew of 602 measurement clocks. 604 He also notes these features of PDV: 606 1. The PDV distribution does not distinguish short-term variation 607 from variation over the complete test interval. (Comment: PDV 608 can be determined over any sub-intervals when the singletons are 609 stored.) 611 2. The location of the distribution is very sensitive to the delay 612 of the first packet, IF this packet is used as the reference. 613 This would be a new formulation that differs from the PDV 614 definition in this memo (PDV references the packet with minimum 615 delay, so it does not have this drawback). 617 3. The shape of the PDV distribution is identical to the delay 618 distribution, but shifted by the reference delay. 620 4. Use of a common reference over measurement intervals that are 621 longer than a typical session length may indicate more PDV than 622 would be experienced by streams that support such sessions. 623 (Ideally, the measurement interval should be aligned with the 624 session length of interest, and this influences determination of 625 the reference delay, D(min).) 627 5. The PDV distribution characterizes the range of queue occupancies 628 along the measurement path (assuming the path is fixed), but the 629 range says nothing about how the variation took place. 631 The summary metrics used in this comparison were the number of values 632 exceeding a +/-50ms range around the mean, the Inverse Percentiles, 633 and the Inter-Quartile Range. 635 5.2. Ciavattone et al. 637 In [Cia03], the authors compared IPDV and PDV (referred to as delta) 638 using a periodic packet stream conforming to [RFC3432] with inter- 639 packet interval of 20 ms. 641 One of the comparisons between IPDV and PDV involves a laboratory 642 set-up where a queue was temporarily congested by a competing packet 643 burst. The additional queuing delay was 85ms to 95ms, much larger 644 than the inter-packet interval. The first packet in the stream that 645 follows the competing burst spends the longest time queued, and 646 others experience less and less queuing time until the queue is 647 drained. 649 The authors observed that PDV reflects the additional queuing time of 650 the packets affected by the burst, with values of 85, 65, 45, 25, and 651 5ms. Also, it is easy to determine (by looking at the PDV range) 652 that a de-jitter buffer of >85 ms would have been sufficient to 653 accommodate the delay variation. Again, the measurement interval is 654 a key factor in the validity of such observations (it should have 655 similar length to the session interval of interest). 657 The IPDV values in the congested queue example are very different: 658 85, -20, -20, -20, -20, -5ms. Only the positive excursion of IPDV 659 gives an indication of the de-jitter buffer size needed. Although 660 the variation exceeds the inter-packet interval, the extent of 661 negative IPDV values is limited by that sending interval. This 662 preference for information from the positive IPDV values has prompted 663 some to ignore the negative values, or to take the absolute value of 664 each IPDV measurement (sacrificing key properties of IPDV in the 665 process, such as its ability to distinguish delay trends). 667 Note that this example illustrates a case where the IPDV distribution 668 is asymmetrical, because the delay variation range (85ms) exceeds the 669 inter-packet spacing (20ms). We see that the IPDV values 85, -20, 670 -20, -20, -20, -5ms have zero mean, but the left side of the 671 distribution is truncated at -20ms. 673 Elsewhere, the authors considered the range as a summary statistic 674 for IPDV, and the 99.9%-ile minus the minimum delay as a summary 675 statistic for delay variation, or PDV. 677 5.3. IPPM List Discussion from 2000 679 Mike Pierce made many comments in the context of the 05 version of 680 draft-ietf-ippm-ipdv. One of his main points was that a delay 681 histogram is a useful approach to quantifying variation. Another 682 point was that the time duration of evaluation is a critical aspect. 684 Carlo Demichelis then mailed his comparison paper to the IPPM list, 685 [Demichelis] as discussed in more detail above. 687 Ruediger Geib observed that both IPDV and the delay histogram (PDV) 688 are useful, and suggested that they might be applied to different 689 variation time scales. He pointed out that loss has a significant 690 effect on IPDV, and encouraged that the loss information be retained 691 in the arrival sequence. 693 Several example delay variation scenarios were discussed, including: 695 Packet # 1 2 3 4 5 6 7 8 9 10 11 696 ------------------------------------------------------- 697 Ex. A 698 Lost 700 Delay, ms 100 110 120 130 140 150 140 130 120 110 100 702 IPDV U 10 10 10 10 10 -10 -10 -10 -10 -10 704 PDV 0 10 20 30 40 50 40 30 20 10 0 706 ------------------------------------------------------- 707 Ex. B 708 Lost L 710 Delay, ms 100 110 150 U 120 100 110 150 130 120 100 712 IPDV U 10 40 U U -10 10 40 -20 -10 -20 714 PDV 0 10 50 U 20 0 10 50 30 20 0 716 Figure 2: Delay Examples 718 Clearly, the range of PDV values is 50 ms in both cases above, and 719 this is the statistic that determines the size of a de-jitter buffer. 720 The IPDV range is minimal in response to the smooth variation in 721 Example A (20 ms). However, IPDV responds to the faster variations 722 in Example B (60 ms range from 40 to -20). Here the IPDV range is 723 larger than the PDV range, and over-estimates the buffer size 724 requirements. 726 A heuristic method to estimate buffer size using IPDV is to sum the 727 consecutive positive or zero values as an estimate of PDV range. 728 However, this is more complicated to assess than the PDV range, and 729 has strong dependence on the actual sequence of IPDV values (any 730 negative IPDV value stops the summation, and again causes an 731 underestimate). 733 IPDV values can be viewed as the adjustments that an adaptive de- 734 jitter buffer would make, IF it could make adjustments on a packet- 735 by-packet basis. However, adaptive de-jitter buffers don't make 736 adjustments this frequently, so the value of this information is 737 unknown. The short-term variations may be useful to know in some 738 other cases. 740 5.4. Y.1540 Appendix II 742 Appendix II of [Y.1540] describes a secondary terminology for delay 743 variation. It compares IPDV, PDV (referred to as 2-point PDV), and 744 1-point packet delay variation (which assumes a periodic stream and 745 assesses variation against an ideal arrival schedule constructed at a 746 single measurement point). This early comparison discusses some of 747 the same considerations raised in section 6 below. 749 5.5. Clark's ITU-T SG 12 Contribution 751 Alan Clark's contribution to ITU-T Study Group 12 in January 2003, 752 provided an analysis of the root causes of delay variation and 753 investigated different techniques for measurement and modeling of 754 "jitter" [COM12.D98]. Clark compared a metric closely related to 755 IPDV, Mean Packet-to-Packet Delay Variation, MPPDV = mean(abs(D(i)- 756 D(i-1))) to the newly proposed Mean Absolute Packet Delay Variation 757 (MAPDV2, see [G.1020]). One of the tasks for this study was to 758 estimate the number of packet discards in a de-jitter buffer. Clark 759 concluded that MPPDV did not track the ramp delay variation he 760 associated access link congestion (similar to Figure 2, Example A 761 above), but MAPDV2 did. 763 Clark also briefly looked at PDV (as described in the 2002 version 764 of[Y.1541]). He concluded that if PDV was applied to a series of 765 very short measurement intervals (e.g., 200ms), it could be used to 766 determine the fraction of intervals with high packet discard rates. 768 6. Additional Properties and Comparisons 770 This section treats some of the earlier comparison areas in more 771 detail, and introduces new areas for comparison. 773 6.1. Packet Loss 775 The measurement packet loss is of great influence for the delay 776 variation results, as displayed in the figures 3 and 4 (L means Lost 777 and U means undefined). Figure 3 shows that in the extreme case of 778 every other packet loss, the IPDV doesn't produce any results, while 779 the PDV produces results for all arriving packets. 781 Packet # 1 2 3 4 5 6 7 8 9 10 782 Lost L L L L L 783 --------------------------------------- 784 Delay, ms 3 U 5 U 4 U 3 U 4 U 786 IPDV U U U U U U U U U U 788 PDV 0 U 2 U 1 U 0 U 1 U 790 Figure 3: Path Loss Every Other Packet 792 In case of a burst of packet loss, as displayed in figure 3, both the 793 IPDV and PDV produces some results. Note that PDV still produces 794 more values than IPDV. 796 Packet # 1 2 3 4 5 6 7 8 9 10 797 Lost L L L L L 798 --------------------------------------- 799 Delay, ms 3 4 U U U U U 5 4 3 801 IPDV U 1 U U U U U U -1 -1 803 PDV 0 1 U U U U U 2 1 0 805 Figure 4: Burst of Packet Loss 807 In conclusion, the PDV results are affected by the packet loss ratio. 808 The IPDV results are affected by both the packet loss ratio and the 809 packet loss distribution. In the extreme case of loss of every other 810 packet, IPDV doesn't provide any results. 812 6.2. Path Changes 814 When there is little or no stability in the network under test, then 815 the devices that attempt to characterize the network are equally 816 stressed, especially if the results displayed are used to make 817 inferences which may not be valid. 819 Sometimes the path characteristics change during a measurement 820 interval. The change may be due to link or router failure, 821 administrative changes prior to maintenance (e.g., link cost change), 822 or re-optimization of routing using new information. All these 823 causes are usually infrequent, and network providers take appropriate 824 measures to ensure this. Automatic restoration to a back-up path is 825 seen as a desirable feature of IP networks. 827 Frequent path changes and prolonged congestion with substantial 828 packet loss clearly make delay variation measurements challenging. 830 Path changes are usually accompanied by a sudden, persistent increase 831 or decrease in one-way-delay. [Cia03] gives one such example. We 832 assume that a restoration path either accepts a stream of packets, or 833 is not used for that particular stream (e.g., no multi-path for 834 flows). 836 In any case, a change in the TTL (or Hop Limit) of the received 837 packets indicates that the path is no longer the same. Transient 838 packet reordering may also be observed with path changes, due to use 839 of non-optimal routing while updates propagate through the network 840 (see [Casner] and [Cia03] ) 842 Many, if not all, packet streams experience packet loss in 843 conjunction with a path change. However, it is certainly possible 844 that the active measurement stream does not experience loss. This 845 may be due to use of a long inter-packet sending interval with 846 respect to the restoration time, and it becomes more likely as "fast 847 restoration" techniques see wider deployment (e.g., [RFC4090]. 849 Thus, there are two main cases to consider, path changes accompanied 850 by loss, and those that are lossless from the point of view of the 851 active measurement stream. The subsections below examine each of 852 these cases. 854 6.2.1. Lossless Path Change 856 In the lossless case, a path change will typically affect only one 857 IPDV singleton. For example, the delay sequence in the Figure below 858 always produces IPDV=0 except in the one case where the value is 5 859 (U, 0, 0, 0, 5, 0, 0, 0, 0). 861 Packet # 1 2 3 4 5 6 7 8 9 862 Lost 863 ------------------------------------ 864 Delay, ms 4 4 4 4 9 9 9 9 9 866 IPDV U 0 0 0 5 0 0 0 0 868 PDV 0 0 0 0 5 5 5 5 5 870 Figure 5: Lossless Path Change 872 However, if the change in delay is negative and larger than the 873 inter-packet sending interval, then more than one IPDV singleton may 874 be affected because packet reordering is also likely to occur. 876 The use of the new path and its delay variation can be quantified by 877 treating the PDV distribution as bi-modal, and characterizing each 878 mode separately. This would involve declaring a new path within the 879 sample, and using a new local minimum delay as the PDV reference 880 delay for the sub-sample (or time interval) where the new path is 881 present. 883 The process of detecting a bi-modal delay distribution is made 884 difficult if the typical delay variation is larger than the delay 885 change associated with the new path. However, information on TTL (or 886 Hop Limit) change or the presence of transient reordering can assist 887 in an automated decision. 889 The effect of path changes may also be reduced by making PDV 890 measurements over short intervals (minutes, as opposed to hours). 891 This way, a path change will affect one sample and its PDV values. 892 Assuming that the mean or median one-way-delay changes appreciably on 893 the new path, then subsequent measurements can confirm a path change 894 and trigger special processing on the interval to revise the PDV 895 result. 897 Alternatively, if the path change is detected, by monitoring the test 898 packets TTL or Hop Limit, or monitoring the change in the IGP link- 899 state database, the results of measurement before and after the path 900 change could be kept separated, presenting two different 901 distributions. This avoids the difficult task of determining the 902 different modes of a multi-modal distribution. 904 6.2.2. Path Change with Loss 906 If the path change is accompanied by loss, such that there are no 907 consecutive packet pairs that span the change, then no IPDV 908 singletons will reflect the change. This may or may not be 909 desirable, depending on the ultimate use of the delay variation 910 measurement. Figure 6, in which L means Lost and U means undefined, 911 illustrates this case. 913 Packet # 1 2 3 4 5 6 7 8 9 914 Lost L L 915 ------------------------------------ 916 Delay, ms 3 4 3 3 U U 8 9 8 918 IPDV U 1 -1 0 U U U 1 -1 920 PDV 0 1 0 0 U U 5 6 5 922 Figure 6: Path Change with Loss 924 PDV will again produce a bimodal distribution. But here, the 925 decision process to define sub-intervals associated with each path is 926 further assisted by the presence of loss, in addition to TTL, 927 reordering information, and use of short measurement intervals 928 consistent with the duration of user sessions. It is reasonable to 929 assume that at least loss and delay will be measured simultaneously 930 with PDV and/or IPDV. 932 IPDV does not help to detect path changes when accompanied by loss, 933 and this is a disadvantage for those who rely solely on IPDV 934 measurements. 936 6.3. Clock Stability and Error 938 Low cost or low complexity measurement systems may be embedded in 939 communication devices that do not have access to high stability 940 clocks, and time errors will almost certainly be present. However, 941 larger time-related errors (~1ms) may offer an acceptable trade-off 942 for monitoring performance over a large population (the accuracy 943 needed to detect problems may be much less than required for a 944 scientific study, ~0.01ms for example). 946 Maintaining time accuracy <<1ms has typically required access to 947 dedicated time receivers at all measurement points. Global 948 positioning system (GPS) receivers have often been installed to 949 support measurements. The GPS installation conditions are fairly 950 restrictive, and many prospective measurement efforts have found the 951 deployment complexity and system maintenance too difficult. 953 As mentioned above, [Demichelis] observed that PDV places greater 954 demands on clock synchronization than for IPDV. This observation 955 deserves more discussion. Synchronization errors have two 956 components: time of day errors and clock frequency errors (resulting 957 in skew). 959 Both IPDV and PDV are sensitive to time-of-day errors when attempting 960 to align measurement intervals at the source and destination. Gross 961 mis-alignment of the measurement intervals can lead to lost packets, 962 for example if the receiver is not ready when the first test packet 963 arrives. However, both IPDV and PDV assess delay differences, so the 964 error present in any two one-way-delay singletons will cancel as long 965 as the error is constant. So, the demand for NTP or GPS 966 synchronization comes primarily from one-way delay measurement time- 967 of-day accuracy requirements. Delay variation and measurement 968 interval alignment are relatively less demanding. 970 Skew is a measure of the change in clock time over an interval w.r.t. 971 a reference clock. Both IPDV and PDV are affected by skew, but the 972 error sensitivity in IPDV singletons is less because the intervals 973 between consecutive packets are rather small, especially when 974 compared to the overall measurement interval. Since PDV computes the 975 difference between a single reference delay (the sample minimum) and 976 all other delays in the measurement interval, the constraint on skew 977 error is greater to attain the same accuracy as IPDV. Again, use of 978 short PDV measurement intervals (on the order of minutes, not hours) 979 provides some relief from the effects of skew error. Thus, the 980 additional accuracy demand of PDV can be expressed as a ratio of the 981 measurement interval to the inter-packet spacing. 983 A practical example is a measurement between two hosts, one with a 984 synchronized clock and the other with a free-running clock having 50 985 part per million (ppm) long term accuracy. 987 o If IPDV measurements are made on packets with a 1 second spacing, 988 the maximum singleton error will be 1 x 5 x 10^-5 seconds, or 989 0.05ms. 991 o If PDV measurements are made on the same packets over a 60 second 992 measurement interval, then the delay variation due to the max 993 free-running clock error will be 60 x 5 x 10-5 seconds, or 3ms 994 delay variation error from the first packet to the last. 996 Therefore, the additional accuracy required for equivalent PDV error 997 under these conditions is a factor of 60 more than for IPDV. This is 998 a rather extreme scenario, because time-of-day error of 1 second 999 would accumulate in ~5.5 hours, potentially causing the measurement 1000 interval alignment issue described above. 1002 If skew is present in a sample of one-way-delays, its symptom is 1003 typically a nearly linear growth or decline over all the one-way- 1004 delay values. As a practical matter, if the same slope appears 1005 consistently in the measurements, then it may be possible to fit the 1006 slope and compensate for the skew in the one-way-delay measurements, 1007 thereby avoiding the issue in the PDV calculations that follow. See 1008 [RFC3393] for additional information on compensating for skew. 1010 Values for IPDV may have non-zero mean over a sample when clock skew 1011 is present. This tends to complicate IPDV analysis when using the 1012 assumptions of a zero mean and a symmetric distribution. 1014 There is a third factor related to clock error and stability: this is 1015 the presence of a clock synchronization protocol (e.g., NTP) and the 1016 time adjustment operations that result. When a time error is 1017 detected (typically on the order of a few milliseconds), the host 1018 clock frequency is continuously adjusted to reduce the time error. 1019 If these adjustments take place during a measurement interval, they 1020 may appear as delay variation when none was present, and therefore 1021 are a source of error (regardless of the DV form considered). 1023 6.4. Spatial Composition 1025 ITU-T Recommendation [Y.1541] gives a provisional method to compose a 1026 PDV metric using PDV measurement results from two or more sub-paths. 1027 Additional methods are considered in 1028 [I-D.ietf-ippm-spatial-composition]. 1030 PDV has a clear advantage at this time, since there is no validated 1031 method to compose an IPDV metric. In addition, IPDV results depend 1032 greatly on the exact sequence of packets and may not lend themselves 1033 easily to the composition problem, where segments must be assumed to 1034 have independent delay distributions. 1036 6.5. Reporting a Single Number (SLA) 1038 Despite the risk of over-summarization, measurements must often be 1039 displayed for easy consumption. If the right summary report is 1040 prepared, then the "dashboard" view correctly indicates whether there 1041 is something different and worth investigating further, or that the 1042 status has not changed. The dashboard model restricts every 1043 instrument display to a single number. The packet network dashboard 1044 could have different instruments for loss, delay, delay variation, 1045 reordering, etc., and each must be summarized as a single number for 1046 each measurement interval. The single number summary statistic is a 1047 key component of SLAs, where a threshold on that number must be met 1048 x% of the time. 1050 The simplicity of the PDV distribution lends itself to this 1051 summarization process (including use of the percentiles, median or 1052 mean). An SLA of the form "no more than x% of packets in a 1053 measurement interval shall have PDV >= y ms, for no less than z% of 1054 time" is relatively straightforward to specify and implement. 1055 [Y.1541] introduced the notion of a pseudo-range when setting an 1056 objective for the 99.9%-ile of PDV. The conventional range (max-min) 1057 was avoided for several reasons, including stability of the maximum 1058 delay. The 99.9%-ile of PDV is helpful to performance planners 1059 (seeking to meet some user-to-user objective for delay) and in design 1060 of de-jitter buffer sizes, even those with adaptive capabilities. 1062 IPDV does not lend itself to summarization so easily. The mean IPDV 1063 is typically zero. As the IPDV distribution will have two tails 1064 (positive and negative) the range or pseudo-range would not match the 1065 needed de-jitter buffer size. Additional complexity may be 1066 introduced when the variation exceeds the inter-packet sending 1067 interval, as discussed above (in sections 5.2 and 6.2.1). Should the 1068 Inter-Quartile Range be used? Should the singletons beyond some 1069 threshold be counted (e.g., mean +/- 50ms)? A strong rationale for 1070 one of these summary statistics has yet to emerge. 1072 When summarizing IPDV, some prefer the simplicity of the single-sided 1073 distribution created by taking the absolute value of each singleton 1074 result, abs(D(i)-D(i-1)). This approach sacrifices the two-sided 1075 inter-arrival spread information in the distribution. It also makes 1076 the evaluation using percentiles more confusing, because a single 1077 late packet that exceeds the variation threshold will cause two pairs 1078 of singletons to fail the criteria (one positive, the other negative 1079 converted to positive). The single-sided PDV distribution is an 1080 advantage in this category. 1082 6.6. Jitter in RTCP Reports 1084 [RFC3550] gives the calculation of the inter-arrival Jitter field for 1085 the RTCP report, with a sample implementation in an Appendix. 1087 The RTCP Jitter value can be calculated using IPDV singletons. If 1088 there is packet reordering, as defined in [RFC4737], then estimates 1089 of Jitter based on IPDV may vary slightly, because [RFC3550] 1090 specifies the use of receive packet order. 1092 Just as there is no simple way to convert PDV singletons to IPDV 1093 singletons without returning to the original sample of delay 1094 singletons, there is no clear relationship between PDV and [RFC3550] 1095 Jitter. 1097 6.7. MAPDV2 1099 MAPDV2 stands for Mean Absolute Packet Delay Variation (version) 2, 1100 and is specified in [G.1020]. The MAPDV2 algorithm computes a 1101 smoothed running estimate of the mean delay using the one-way delays 1102 of 16 previous packets. It compares the current one-way-delay to the 1103 estimated mean, separately computes the means of positive and 1104 negative deviations, and sums these deviation means to produce 1105 MAPVDV2. In effect, there is a MAPDV2 singleton for every arriving 1106 packet, so further summarization is usually warranted. 1108 Neither IPDV or PDV forms assist in the computation of MAPDV2. 1110 6.8. Load Balancing 1112 Network traffic load balancing is a process to divide packet traffic 1113 in order to provide a more even distribution over two or more equally 1114 viable paths. The paths chosen are based on the IGP cost metrics, 1115 while the delay depends on the path's physical layout. Usually, the 1116 balancing process is performed on a per-flow basis to avoid delay 1117 variation experienced when packets traverse different physical paths. 1119 If the sample includes test packets with different characteristics 1120 such as IP addresses/ports, there could be multi-modal delay 1121 distributions present. The PDV form makes the identification of 1122 multiple modes possible. IPDV may also reveal that multiple paths 1123 are in use with a mixed flow sample, but the different delay modes 1124 are not easily divided and analyzed separately. 1126 Should the delay singletons using multiple addresses/ports be 1127 combined in the same sample? Should we characterize each mode 1128 separately? (This question also applies to the Path Change case.) 1129 It depends on the task to be addressed by the measurement. 1131 For the task of de-jitter buffer sizing or assessing queue 1132 occupation, the modes should be characterized separately because 1133 flows will experience only one mode on a stable path. Use of a 1134 single flow description (address/port combination) in each sample 1135 simplifies this analysis. Multiple modes may be identified by 1136 collecting samples with different flow attributes, and 1137 characterization of multiple paths can proceed with comparison of the 1138 delay distributions from each sample. 1140 For the task of capacity planning and routing optimization, 1141 characterizing the modes separately could offer an advantage. 1142 Network wide capacity planning (as opposed to link capacity planning) 1143 takes as input the core traffic matrix, which corresponds to a matrix 1144 of traffic transferred from every source to every destination in the 1145 network. Applying the core traffic matrix along with the routing 1146 information (typically the link state database of a routing protocol) 1147 in a capacity planning tool offers the possibility to visualize the 1148 paths where the traffic flows and to optimize the routing based on 1149 the link utilization. In the case where equal cost multiple paths 1150 (ECMP) are used, the traffic will be load balanced onto multiple 1151 paths. If each mode of the IP delay multi-modal distribution can be 1152 associated with a specific path, the delay performance offers an 1153 extra optimization parameter, i.e. the routing optimization based on 1154 the IP delay variation metric. As an example, the load balancing 1155 across ECMPs could be suppressed so that the VoIP calls would only be 1156 routed via the path with the lower IP delay variation. Clearly, any 1157 modifications can result in new delay performance measurements, so 1158 there must be a verification step to ensure the desired outcome. 1160 7. Applicability of the Delay Variation Forms and Recommendations 1162 Based on the comparisons of IPDV and PDV presented above, this 1163 section matches the attributes of each form with the tasks described 1164 earlier. We discuss the more general circumstances first. 1166 7.1. Uses 1168 7.1.1. Inferring Queue Occupancy 1170 The PDV distribution is anchored at the minimum delay observed in the 1171 measurement interval. When the sample minimum coincides with the 1172 true minimum delay of the path, then the PDV distribution is 1173 equivalent to the queuing time distribution experienced by the test 1174 stream. If the minimum delay is not the true minimum, then the PDV 1175 distribution captures the variation in queuing time and some 1176 additional amount of queuing time is experienced, but unknown. One 1177 can summarize the PDV distribution with the mean, median, and other 1178 statistics. 1180 IPDV can capture the difference in queuing time from one packet to 1181 the next, but this is a different distribution from the queue 1182 occupancy revealed by PDV. 1184 7.1.2. Determining De-jitter Buffer Size (and FEC Design) 1186 This task is complimentary to the problem of inferring queue 1187 occupancy through measurement. Again, use of the sample minimum as 1188 the reference delay for PDV yields a distribution that is very 1189 relevant to de-jitter buffer size. This is because the minimum delay 1190 is an alignment point for the smoothing operation of de-jitter 1191 buffers. A de-jitter buffer that is ideally aligned with the delay 1192 variation adds zero buffer time to packets with the longest 1193 accommodated network delay (any packets with longer delays are 1194 discarded). Thus, a packet experiencing minimum network delay should 1195 be aligned to wait the maximum length of the de-jitter buffer. With 1196 this alignment, the stream is smoothed with no unnecessary delay 1197 added. [G.1020] illustrates the ideal relationship between network 1198 delay variation and buffer time. 1200 The PDV distribution is also useful for this task, but different 1201 statistics are preferred. The range (max-min) or the 99.9%-ile of 1202 PDV (pseudo-range) are closely related to the buffer size needed to 1203 accommodate the observed network delay variation. 1205 The PDV distribution directly addresses the FEC waiting time 1206 question. When the PDV distribution has a 99th percentile of 10ms, 1207 then waiting 10ms longer than the FEC protection interval will allow 1208 99% of late packets to arrive and be used in the FEC block. 1210 In some cases, the positive excursions (or series of positive 1211 excursions) of IPDV may help to approximate the de-jitter buffer 1212 size, but there is no guarantee that a good buffer estimate will 1213 emerge, especially when the delay varies as a positive trend over 1214 several test packets. 1216 7.1.3. Spatial Composition 1218 PDV has a clear advantage at this time, since there is no validated 1219 method to compose an IPDV metric. 1221 7.1.4. Service Level Specification: Reporting a Single Number 1223 The one-sided PDV distribution can be constrained with a single 1224 statistic, such as an upper percentile, so it is preferred. The IPDV 1225 distribution is two-sided, usually has zero mean, and no universal 1226 summary statistic that relates to a physical quantity has emerged in 1227 years of experience. 1229 7.2. Challenging Circumstances 1231 Note that measurement of delay variation may not be the primary 1232 concern under unstable and unreliable circumstances. 1234 7.2.1. Clock and Storage Issues 1236 When appreciable skew is present between measurement system clocks, 1237 then IPDV has an advantage because PDV would require processing over 1238 the entire sample to remove the skew error. However, significant 1239 skew can invalidate IPDV analysis assumptions, such as the zero mean 1240 and symmetric distribution characteristics. Small skew may well be 1241 within the error tolerance, and both PDV and IPDV results will be 1242 usable. There may be a portion of the skew, measurement interval, 1243 and required accuracy 3-D space where IPDV has an advantage, 1244 depending on the specific measurement specifications. 1246 Neither form of delay variation is more suited than the other to on- 1247 the-fly summarization without memory, and this may be one of the 1248 reasons that [RFC3550] RTCP Jitter and MAPDV2 in [G.1020] have 1249 attained deployment in low-cost systems. 1251 7.2.2. Frequent Path Changes 1253 If the network under test exhibits frequent path changes, on the 1254 order of several new routes per minute, then IPDV appears to isolate 1255 the delay variation on each path from the transient effect of path 1256 change (especially if there is packet loss at the time of path 1257 change). However, if one intends to use IPDV to indicate path 1258 changes, it cannot do this when the change is accompanied by loss. 1259 It is possible to make meaningful PDV measurements when paths are 1260 unstable, but great importance would be placed on the algorithms that 1261 infer path change and attempt to divide the sample on path change 1262 boundaries. 1264 When path changes are frequent and cause packet loss, delay variation 1265 is probably less important than the loss episodes and attention 1266 should be turned to the loss metric instead. 1268 7.2.3. Frequent Loss 1270 If the network under test exhibits frequent loss, then PDV may 1271 produce a larger set of singletons for the sample than IPDV. This is 1272 due to IPDV requiring consecutive packet arrivals to assess delay 1273 variation, compared to PDV where any packet arrival is useful. The 1274 worst case is when no consecutive packets arrive, and the entire IPDV 1275 sample would be undefined. PDV would successfully produce a sample 1276 based on the arriving packets. 1278 7.2.4. Load Balancing 1280 PDV distributions offer the most straightforward way to identify that 1281 a sample of packets have traversed multiple paths. The tasks of de- 1282 jitter buffer sizing or assessing queue occupation with PDV should be 1283 use a sample with a single flow because flows will experience only 1284 one mode on a stable path, and it simplifies the analysis. 1286 7.3. Summary 1287 +---------------+----------------------+----------------------------+ 1288 | Comparison | PDV | IPDV | 1289 | Area | | | 1290 +---------------+----------------------+----------------------------+ 1291 | Challenging | Less sensitive to | Preferred when path | 1292 | Circumstances | packet loss, and | changes are frequent or | 1293 | | simplifies analysis | when measurement clocks | 1294 | | when load balancing | exhibit some skew | 1295 | | or multiple paths | | 1296 | | are present | | 1297 |---------------|----------------------|----------------------------| 1298 | Spatial | All validated | Has sensitivity to | 1299 | Composition | methods use this | sequence and spacing | 1300 | of DV metric | form | changes, which tends to | 1301 | | | break the requirement for | 1302 | | | independent distributions | 1303 | | | between path segments | 1304 |---------------|----------------------|----------------------------| 1305 | Determine | "Pseudo-range" | No reliable relationship, | 1306 | De-Jitter | reveals this | but some heuristics | 1307 | Buffer Size | property by | | 1308 | Required | anchoring the | | 1309 | | distribution at the | | 1310 | | minimum delay | | 1311 |---------------|----------------------|----------------------------| 1312 | Estimate of | Distribution has | No reliable relationship | 1313 | Queuing Time | one-to-one | | 1314 | and Variation | relationship on a | | 1315 | | stable path, | | 1316 | | especially when | | 1317 | | sample min = true | | 1318 | | min | | 1319 |---------------|----------------------|----------------------------| 1320 | Specification | One constraint | Distribution is two-sided, | 1321 | Simplicity: | needed for | usually has zero mean, and | 1322 | Single Number | single-sided | no universal summary | 1323 | SLA | distribution, and | statistic that relates to | 1324 | | easily related to | a physical quantity | 1325 | | quantities above | | 1326 +---------------+----------------------+----------------------------+ 1328 Summary of Comparisons 1330 8. Measurement Considerations 1332 This section discusses the practical aspects of delay variation 1333 measurement, with special attention to the two formulations compared 1334 in this memo. 1336 8.1. Measurement Stream Characteristics 1338 As stated in the background section, there is a strong dependency 1339 between the active measurement stream characteristics and the 1340 results. The IPPM literature includes two primary methods for 1341 collecting samples: Poisson sampling described in [RFC2330], and 1342 Periodic sampling in[RFC3432]. The Poisson method was intended to 1343 collect an unbiased sample of performance, while the Periodic method 1344 addresses a "known bias of interest". Periodic streams are required 1345 to have random start times and limited stream duration, in order to 1346 avoid unwanted synchronization with some other periodic process, or 1347 cause congestion-aware senders to synchronize with the stream and 1348 produce atypical results. The random start time should be different 1349 for each new stream. 1351 It is worth noting that [RFC3393] was developed in parallel with 1352 [RFC3432]. As a result, all the stream metrics defined in [RFC3393] 1353 specify the Poisson sampling method. 1355 Periodic sampling is frequently used in measurements of delay 1356 variation. Several factors foster this choice: 1358 1. Many application streams that are sensitive to delay variation 1359 also exhibit periodicity, and so exemplify the bias of interest. 1360 If the application has a constant packet spacing, this constant 1361 spacing can be the inter-packet gap for the test stream. VoIP 1362 streams often use 20ms spacing, so this is an obvious choice for 1363 an Active stream. This applies to both IPDV and PDV forms. 1365 2. The spacing between packets in the stream will influence whether 1366 the stream experiences short-range dependency, or only long-range 1367 dependency, as investigated in [Li.Mills]. The packet spacing 1368 also influences the IPDV distribution and the stream's 1369 sensitivity to reordering. For example, with a 20 ms spacing the 1370 IPDV distribution cannot go below -20ms without packet 1371 reordering. 1373 3. The measurement process may make several simplifying assumptions 1374 when the send spacing and send rate are constant. For example, 1375 the inter-arrival times at the destination can be compared with 1376 an ideal sending schedule, and allowing a one-point measurement 1377 of delay variation (described in [Y.1540]) that approximates the 1378 IPDV form. Simplified methods that approximate PDV are possible 1379 as well (some are discussed in Appendix II of [Y.1541]). 1381 4. Analysis of truncated, or non-symmetrical IPDV distributions is 1382 simplified. Delay variations in excess of the periodic sending 1383 interval can cause multiple singleton values at the negative 1384 limit of the packet spacing (see section 5.2 and [Cia03]). Only 1385 packet reordering can cause the negative spacing limit to be 1386 exceeded. 1388 Despite the emphasis on inter-packet delay differences with IPDV, 1389 both Poisson [Demichelis] and Periodic [Li.Mills] streams have been 1390 used, and these references illustrate the different analyses that are 1391 possible. 1393 The advantages of using a Poisson distribution are discussed in 1394 [RFC2330]. The main properties are to avoid predicting the sample 1395 times, avoid synchronization with periodic events that are present in 1396 networks, and avoid inducing synchronization with congestion-aware 1397 senders. When a Poisson stream is used with IPDV, the distribution 1398 will reflect inter-packet delay variation on many different time 1399 scales (or packet spacings). The unbiased Poisson sampling brings a 1400 new layer of complexity in the analysis of IPDV distributions. 1402 8.2. Measurement Devices 1404 One key aspect of measurement devices is their ability to store 1405 singletons (or individual measurements). This feature usually is 1406 closely related to local calculation capabilities. For example, an 1407 embedded measurement device with limited storage will like provide 1408 only a few statistics on the delay variation distribution, while 1409 dedicated measurement systems store all the singletons and allow 1410 detailed analysis (later calculation of either form of delay 1411 variation is possible with the original singletons). 1413 Therefore, systems with limited storage must choose their metrics and 1414 summary statistics in advance. If both IPDV and PDV statistics are 1415 desired, the supporting information must be collected as packets 1416 arrive. For example, the PDV range and high percentiles can be 1417 determined later if the minimum and several of the largest delays are 1418 stored while the measurement is in-progress. 1420 8.3. Units of Measurement 1422 Both IPDV and PDV can be summarized as a range in milliseconds. 1424 With IPDV, it is interesting to report on a positive percentile, and 1425 an inter-quantile range is appropriate to reflect both positive and 1426 negative tails (e.g., 5% to 95%). If the IPDV distribution is 1427 symmetric around a mean of zero, then it is sufficient to report on 1428 the positive side of the distribution. 1430 With PDV, it is sufficient to specify the upper percentile (e.g., 1431 99.9%). 1433 8.4. Test Duration 1435 At several points in this memo, we have recommended use of test 1436 intervals on the order of minutes. In their paper examining the 1437 stability of Internet path properties[Zhang.Duff], Zhang et al. 1438 concluded that consistency was present on the order of minutes for 1439 the performance metrics considered (loss, delay, and throughput) for 1440 the paths they measured. 1442 The topic of temporal aggregation of performance measured in small 1443 intervals to estimate some larger interval is described in the Metric 1444 Composition Framework [I-D.ietf-ippm-framework-compagg]. 1446 The primary recommendation here is to test using durations that are 1447 similar in length to the session time of interest. This applies to 1448 both IPDV and PDV, but is possibly more relevant for PDV since the 1449 duration determines how often the D_min will be determined, and the 1450 size of the associated sample. 1452 8.5. Clock Sync Options 1454 As with one-way delay measurements, local clock synchronization is an 1455 important matter for delay variation measurements. 1457 There are several options available: 1459 1. Global Positioning System receivers 1461 2. In some parts of the world, Cellular Code Division Multiple 1462 Access (CDMA) systems distribute timing signals that are derived 1463 from GPS and traceable to UTC. 1465 3. Network Time Protocol [RFC1305] is a convenient choice in many 1466 cases, but usually offers lower accuracy than the options above. 1468 When clock synchronization is inconvenient or subject to appreciable 1469 errors, then round-trip measurements may give a cumulative indication 1470 of the delay variation present on both directions of the path. 1471 However, delay distributions are rarely symmetrical, so it is 1472 difficult to infer much about the one-way delay variation from round- 1473 trip measurements. Also, measurements on asymmetrical paths add 1474 complications for the one-way delay metric. 1476 8.6. Distinguishing Long Delay from Loss 1478 Lost and delayed packets are separated by a waiting time threshold. 1479 Packets that arrive at the measurement destination within their 1480 waiting time have finite delay and are not lost. Otherwise, packets 1481 are designated lost and their delay is undefined. Guidance on 1482 setting the waiting time threshold may be found in [RFC2680] and 1483 [I-D.morton-ippm-reporting-metrics]. 1485 In essence, [I-D.morton-ippm-reporting-metrics] suggests to use a 1486 long waiting time to serve network characterization and revise 1487 results for specific application delay thresholds as needed. 1489 8.7. Accounting for Packet Reordering 1491 Packet reordering, defined in [RFC4737], is essentially an extreme 1492 form of delay variation where the packet stream arrival order differs 1493 from the sending order. 1495 PDV results are not sensitive to packet arrival order, and are not 1496 affected by reordering other than to reflect the more extreme 1497 variation. 1499 IPDV results will change if reordering is present because they are 1500 sensitive to the sequence of delays of arriving packets. The main 1501 example of this sensitivity is in the truncation of the negative tail 1502 of the distribution. 1504 o When there is no reordering, the negative tail is limited by the 1505 sending time spacing between packets. 1507 o If reordering occurs (and the reordered packets are not 1508 discarded), the negative tail can take on any value (in 1509 principal). 1511 In general, measurement systems should have the capability to detect 1512 when sequence has changed. If IPDV measurements are made without 1513 regard to packet arrival order, the IPDV will be under-reported when 1514 reordering occurs. 1516 8.8. Results Representation and Reporting 1518 All of the references that discuss or define delay variation suggest 1519 ways to represent or report the results, and interested readers 1520 should review the various possibilities. 1522 For example, [I-D.morton-ippm-reporting-metrics] suggests to report a 1523 pseudo range of delay variation based on calculating the difference 1524 between a high percentile of delay and the minimum delay. The 99.9%- 1525 ile minus the minimum will give a value that can be compared with 1526 objectives in [Y.1541]. 1528 9. IANA Considerations 1530 This document makes no request of IANA. 1532 Note to RFC Editor: this section may be removed on publication as an 1533 RFC. 1535 10. Security Considerations 1537 The security considerations that apply to any active measurement of 1538 live networks are relevant here as well. See the security 1539 considerations sections in [RFC2330], [RFC2679], [RFC3393], 1540 [RFC3432], and[RFC4656]. 1542 Security considerations do not contribute to the selection of PDV or 1543 IPDV forms of delay variation. 1545 11. Acknowledgements 1547 The authors would like to thank Phil Chimento for his suggestion to 1548 employ the convention of conditional distributions of Delay to deal 1549 with packet loss, and his encouragement to "write the memo" after 1550 hearing "the talk" on this topic at IETF-65. We also acknowledge 1551 constructive comments from Alan Clark, Loki Jorgenson, Carsten 1552 Schmoll, and Robert Holley. 1554 12. Appendix on Calculating the D(min) in PDV 1556 Practitioners have raised questions several questions that this 1557 section intends to answer: 1559 - how is this D_min calculated? Is it DV(99%) as mentioned in 1560 [Krzanowski]? 1562 - do we need to keep all the values from the interval, then take the 1563 minimum? Or do we keep the minimum from previous intervals? 1565 The value of D_min used as the reference delay for PDV calculations 1566 is simply the minimum delay of all packets in the current sample. 1567 The usual single value summary of the PDV distribution is D_99.9%-ile 1568 minus D_min. 1570 It may be appropriate to segregate sub-sets and revise the minimum 1571 value during a sample. For example, if it can be determined with 1572 certainty that the path has changed by monitoring the Time to Live or 1573 Hop Count of arriving packets, this may be sufficient justification 1574 to reset the minimum for packets on the new path. There is also a 1575 simpler approach to solving this problem: use samples collected over 1576 short evaluation intervals (on the order of minutes). Intervals with 1577 path changes may be more interesting from the loss or one-way delay 1578 perspective (possibly failing to meet one or more SLAs), and it may 1579 not be necessary to conduct delay variation analysis. Short 1580 evaluation intervals are preferred for measurements that serve as a 1581 basis for troubleshooting, since the results are available to report 1582 soon after collection. 1584 It is not necessary to store all delay values in a sample when 1585 storage is a major concern. D_min can be found by comparing each new 1586 singleton value with the current value and replacing it when 1587 required. In a sample with 5000 packets, evaluation of the 99.9%-ile 1588 can also be achieved with limited storage. One method calls for 1589 storing the top 50 delay singletons and revising the top value list 1590 each time 50 more packets arrive. 1592 13. References 1594 13.1. Normative References 1596 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 1597 Requirement Levels", BCP 14, RFC 2119, March 1997. 1599 [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, 1600 "Framework for IP Performance Metrics", RFC 2330, 1601 May 1998. 1603 [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1604 Delay Metric for IPPM", RFC 2679, September 1999. 1606 [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way 1607 Packet Loss Metric for IPPM", RFC 2680, September 1999. 1609 [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation 1610 Metric for IP Performance Metrics (IPPM)", RFC 3393, 1611 November 2002. 1613 [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network 1614 performance measurement with periodic streams", RFC 3432, 1615 November 2002. 1617 [RFC4090] Pan, P., Swallow, G., and A. Atlas, "Fast Reroute 1618 Extensions to RSVP-TE for LSP Tunnels", RFC 4090, 1619 May 2005. 1621 [RFC4656] Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. 1622 Zekauskas, "A One-way Active Measurement Protocol 1623 (OWAMP)", RFC 4656, September 2006. 1625 [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov, 1626 S., and J. Perser, "Packet Reordering Metrics", RFC 4737, 1627 November 2006. 1629 13.2. Informative References 1631 [COM12.D98] 1632 Clark, Alan., "ITU-T Delayed Contribution COM 12 - D98, 1633 "Analysis, measurement and modelling of Jitter"", 1634 January 2003. 1636 [Casner] "A Fine-Grained View of High Performance Networking, NANOG 1637 22 Conf.; http://www.nanog.org/mtg-0105/agenda.html", May 1638 20-22 2001. 1640 [Cia03] "Standardized Active Measurements on a Tier 1 IP Backbone, 1641 IEEE Communications Mag., pp 90-97.", June 2003. 1643 [Demichelis] 1644 http://www.advanced.org/ippm/archive.3/att-0075/ 1645 01-pap02.doc, "Packet Delay Variation Comparison between 1646 ITU-T and IETF Draft Definitions", November 2000. 1648 [G.1020] ITU-T Recommendation G.1020, ""Performance parameter 1649 definitions for the quality of speech and other voiceband 1650 applications utilizing IP networks"", 2006. 1652 [G.1050] ITU-T Recommendation G.1050, ""Network model for 1653 evaluating multimedia transmission performance over 1654 Internet Protocol"", November 2005. 1656 [I-D.ietf-ippm-framework-compagg] 1657 Morton, A., "Framework for Metric Composition", 1658 draft-ietf-ippm-framework-compagg-07 (work in progress), 1659 October 2008. 1661 [I-D.ietf-ippm-spatial-composition] 1662 Morton, A. and E. Stephan, "Spatial Composition of 1663 Metrics", draft-ietf-ippm-spatial-composition-07 (work in 1664 progress), July 2008. 1666 [I-D.morton-ippm-reporting-metrics] 1667 Morton, A., Ramachandran, G., and G. Maguluri, "Reporting 1668 Metrics: Different Points of View", 1669 draft-morton-ippm-reporting-metrics-06 (work in progress), 1670 January 2009. 1672 [I.356] ITU-T Recommendation Y.1540, "B-ISDN ATM layer cell 1673 transfer performance", March 2000. 1675 [Krzanowski] 1676 Presentation at IPPM, IETF-64, "Jitter Definitions: What 1677 is What?", November 2005. 1679 [Li.Mills] 1680 Li, Quong. and David. Mills, ""The Implications of Short- 1681 Range Dependency on Delay Variation Measurement", Second 1682 IEEE Symposium on Network Computing and Applications", 1683 2003. 1685 [Morton06] 1686 Morton, A., ""A Brief Jitter Metrics Comparison, and not 1687 the last word, by any means...", Slide Presentation at 1688 IETF-65, IPPM Session,", March 2006. 1690 [RFC1305] Mills, D., "Network Time Protocol (Version 3) 1691 Specification, Implementation", RFC 1305, March 1992. 1693 [RFC3357] Koodli, R. and R. Ravikanth, "One-way Loss Pattern Sample 1694 Metrics", RFC 3357, August 2002. 1696 [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. 1697 Jacobson, "RTP: A Transport Protocol for Real-Time 1698 Applications", STD 64, RFC 3550, July 2003. 1700 [Y.1540] ITU-T Recommendation Y.1540, "Internet protocol data 1701 communication service - IP packet transfer and 1702 availability performance parameters", November 2007. 1704 [Y.1541] ITU-T Recommendation Y.1541, "Network Performance 1705 Objectives for IP-Based Services", February 2006. 1707 [Zhang.Duff] 1708 Zhang, Yin., Duffield, Nick., Paxson, Vern., and Scott. 1709 Shenker, ""On the Constancy of Internet Path Properties", 1710 Proceedings of ACM SIGCOMM Internet Measurement 1711 Workshop,", November 2001. 1713 Authors' Addresses 1715 Al Morton 1716 AT&T Labs 1717 200 Laurel Avenue South 1718 Middletown,, NJ 07748 1719 USA 1721 Phone: +1 732 420 1571 1722 Fax: +1 732 368 1192 1723 Email: acmorton@att.com 1724 URI: http://home.comcast.net/~acmacm/ 1726 Benoit Claise 1727 Cisco Systems, Inc. 1728 De Kleetlaan 6a b1 1729 Diegem, 1831 1730 Belgium 1732 Phone: +32 2 704 5622 1733 Fax: 1734 Email: bclaise@cisco.com 1735 URI: