idnits 2.17.00 (12 Aug 2021) /tmp/idnits15332/draft-lencse-bmwg-benchmarking-stateful-03.txt: Checking boilerplate required by RFC 5378 and the IETF Trust (see https://trustee.ietf.org/license-info): ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/1id-guidelines.txt: ---------------------------------------------------------------------------- No issues found here. Checking nits according to https://www.ietf.org/id-info/checklist : ---------------------------------------------------------------------------- == There are 2 instances of lines with private range IPv4 addresses in the document. If these are generic example addresses, they should be changed to use any of the ranges defined in RFC 6890 (or successor): 192.0.2.x, 198.51.100.x or 203.0.113.x. Miscellaneous warnings: ---------------------------------------------------------------------------- == The document seems to lack the recommended RFC 2119 boilerplate, even if it appears to use RFC 2119 keywords -- however, there's a paragraph with a matching beginning. Boilerplate error? (The document does seem to have the reference to RFC 2119 which the ID-Checklist requires). -- The document date (4 March 2022) is 71 days in the past. Is this intentional? Checking references for intended status: Informational ---------------------------------------------------------------------------- == Outdated reference: A later version (-02) exists of draft-lencse-v6ops-transition-scalability-01 -- Duplicate reference: RFC4814, mentioned in 'LEN2020', was also mentioned in 'RFC4814'. -- Duplicate reference: RFC8219, mentioned in 'SIITPERF', was also mentioned in 'RFC8219'. Summary: 0 errors (**), 0 flaws (~~), 3 warnings (==), 3 comments (--). Run idnits with the --verbose option for more detailed information about the items above. -------------------------------------------------------------------------------- 2 Benchmarking Methodology Working Group G. Lencse 3 Internet-Draft Szechenyi Istvan University 4 Intended status: Informational K. Shima 5 Expires: 5 September 2022 IIJ Innovation Institute 6 4 March 2022 8 Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 9 Pseudorandom Port Numbers 10 draft-lencse-bmwg-benchmarking-stateful-03 12 Abstract 14 RFC 2544 has defined a benchmarking methodology for network 15 interconnect devices. RFC 5180 addressed IPv6 specificities and it 16 also provided a technology update, but excluded IPv6 transition 17 technologies. RFC 8219 addressed IPv6 transition technologies, 18 including stateful NAT64. However, none of them discussed how to 19 apply RFC 4814 pseudorandom port numbers to any stateful NAT (NAT44, 20 NAT64, NAT66) technologies. We discuss why using pseudorandom port 21 numbers with stateful NAT gateways is a hard problem and recommend a 22 solution. 24 Status of This Memo 26 This Internet-Draft is submitted in full conformance with the 27 provisions of BCP 78 and BCP 79. 29 Internet-Drafts are working documents of the Internet Engineering 30 Task Force (IETF). Note that other groups may also distribute 31 working documents as Internet-Drafts. The list of current Internet- 32 Drafts is at https://datatracker.ietf.org/drafts/current/. 34 Internet-Drafts are draft documents valid for a maximum of six months 35 and may be updated, replaced, or obsoleted by other documents at any 36 time. It is inappropriate to use Internet-Drafts as reference 37 material or to cite them other than as "work in progress." 39 This Internet-Draft will expire on 5 September 2022. 41 Copyright Notice 43 Copyright (c) 2022 IETF Trust and the persons identified as the 44 document authors. All rights reserved. 46 This document is subject to BCP 78 and the IETF Trust's Legal 47 Provisions Relating to IETF Documents (https://trustee.ietf.org/ 48 license-info) in effect on the date of publication of this document. 49 Please review these documents carefully, as they describe your rights 50 and restrictions with respect to this document. Code Components 51 extracted from this document must include Revised BSD License text as 52 described in Section 4.e of the Trust Legal Provisions and are 53 provided without warranty as described in the Revised BSD License. 55 Table of Contents 57 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 58 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3 59 2. Pseudorandom Port Numbers and Stateful Translation . . . . . 3 60 3. Test Setup and Terminology . . . . . . . . . . . . . . . . . 4 61 4. Recommended Benchmarking Method . . . . . . . . . . . . . . . 5 62 4.1. Restricted Port Number Ranges . . . . . . . . . . . . . . 5 63 4.2. Preliminary Test Phase . . . . . . . . . . . . . . . . . 6 64 4.3. Consideration of the Cases of Stateful Operation . . . . 7 65 4.4. Control of the Connection Tracking Table Entries . . . . 8 66 4.5. Measurement of the Maximum Connection Establishment 67 Rate . . . . . . . . . . . . . . . . . . . . . . . . . . 9 68 4.6. Real Test Phase . . . . . . . . . . . . . . . . . . . . . 10 69 4.7. Measurement of the Connection Tear Down Rate . . . . . . 11 70 4.8. Writing and Reading Order of the State Table . . . . . . 11 71 5. Implementation and Experience . . . . . . . . . . . . . . . . 12 72 6. Limitations of using UDP as Transport Layer Protocol . . . . 12 73 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12 74 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 75 9. Security Considerations . . . . . . . . . . . . . . . . . . . 13 76 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 13 77 10.1. Normative References . . . . . . . . . . . . . . . . . . 13 78 10.2. Informative References . . . . . . . . . . . . . . . . . 13 79 Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 15 80 A.1. 00 . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 81 A.2. 01 . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 82 A.3. 02 . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 83 A.4. 03 . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 84 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 15 86 1. Introduction 88 [RFC2544] has defined a comprehensive benchmarking methodology for 89 network interconnect devices, which is still in use. It was mainly 90 IP version independent, but it used IPv4 in its examples. [RFC5180] 91 addressed IPv6 specificities and also added technology updates, but 92 declared IPv6 transition technologies out of its scope. [RFC8219] 93 addressed the IPv6 transition technologies, including stateful NAT64. 94 It has reused several benchmarking procedures from [RFC2544] (e.g. 95 throughput, frame loss rate), it has redefined the latency 96 measurement, and added further ones, e.g. the PDV (packet delay 97 variation) measurement. 99 However, none of them discussed, how to apply [RFC4814] pseudorandom 100 port numbers, when benchmarking stateful NATxy (NAT44, NAT64, NAT66) 101 gateways. We are not aware of any other RFCs that address this 102 question. 104 First, we discuss why using pseudorandom port numbers with stateful 105 NATxy gateways is a hard problem. 107 Then we recommend a solution. 109 1.1. Requirements Language 111 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", 112 "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 113 "OPTIONAL" in this document are to be interpreted as described in 114 BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all 115 capitals, as shown here. 117 2. Pseudorandom Port Numbers and Stateful Translation 119 In its appendix, [RFC2544] has defined a frame format for test frames 120 including specific source and destination port numbers. [RFC4814] 121 recommends to use pseudorandom and uniformly distributed values for 122 both source and destination port numbers. However, stateful NATxy 123 (NAT44, NAT64, NAT66) solutions use the port numbers to identify 124 connections. The usage of pseudorandom port numbers causes different 125 problems depending on the direction. 127 * As for the private to public direction, pseudorandom source and 128 destination port numbers could be used, however, this approach 129 would be a denial of service attack against the stateful NATxy 130 gateway, because it would exhaust its connection tracking table. 131 To that end, let us see some calculations using the 132 recommendations of RFC 4814: 134 - The recommended source port range is: 1024-65535, thus its size 135 is: 64512. 137 - The recommended destination port range is: 1-49151, thus its 138 size is: 49151. 140 - The number of source and destination port number combinations 141 is: 3,170,829,312. 143 We note that section 12 of [RFC2544] also requires testing with 144 256 destination networks, which further increases the number of 145 connection tracking table entries. 147 * As for the public to private direction, the stateful DUT (Device 148 Under Test) would drop any packets that do not belong to an 149 existing connection, therefore, the direct usage of pseudorandom 150 port numbers from the above-mentioned ranges is not feasible. 152 3. Test Setup and Terminology 154 Our methodology works with any IP version. We use IPv4 in the Test 155 Setup shown in Figure 1 to facilitate its easy understanding based on 156 the well-known stateful NAT44 (also called NAPT: Network Address and 157 Port Translation) solution. 159 +--------------------------------------+ 160 10.0.0.2 |Initiator Responder| 198.19.0.2 161 +-------------| Tester |<------------+ 162 | private IPv4| [state table]| public IPv4 | 163 | +--------------------------------------+ | 164 | | 165 | +--------------------------------------+ | 166 | 10.0.0.1 | DUT: | 198.19.0.1 | 167 +------------>| Sateful NATxy gateway |-------------+ 168 private IPv4| [connection tracking table] | public IPv4 169 +--------------------------------------+ 171 Figure 1: Test Setup for benchmarking stateful NATxy gateways 173 As for transport layer protocol, [RFC2544] recommended testing with 174 UDP, and it was kept also in [RFC8219]. For the general 175 recommendation, we also keep UDP, thus the port numbers in the 176 following text are to be understood as UDP port numbers. We discuss 177 the limitation of this approach in Section 6. 179 We define the most important elements of our proposed benchmarking 180 system as follows. 182 * Connection tracking table: The stateful NATxy gateway uses a 183 connection tracking table to be able to perform the stateful 184 translation in the public to private direction. Its size, policy 185 and content are unknown for the Tester. 187 * Four tuple: The four numbers that identify a connection are source 188 IP address, source port number, destination IP address, 189 destination port number. 191 * State table: The Responder of the Tester extracts the four tuple 192 from each received test frame and stores it in its state table. 193 Recommendation is given for writing and reading order of the state 194 table in Section 4.8. 196 * Initiator: The port of the Tester that may initiate a connection 197 through the stateful DUT in the private to public direction. 198 Theoretically, it can use any source and destination port numbers 199 from the ranges recommended by [RFC4814]: if the used four tuple 200 does not belong to an existing connection, the DUT will register a 201 new connection into its connection tracking table. 203 * Responder: The port of the Tester that may not initiate a 204 connection through the stateful DUT in the public to private 205 direction. It may send only frames that belong to an existing 206 connection. To that end, it uses four tuples that have been 207 previously extracted from the received test frames and stored in 208 its state table. 210 * Preliminary test phase: Test frames are sent only by the Initiator 211 to the Responder through the DUT to fill both the connection 212 tracking table of the DUT and the state table of the Responder. 213 This is a newly introduced operation phase for stateful NATxy 214 benchmarking. The necessity of this phase is explained in 215 Section 4.2. 217 * Real test phase: The actual test (e.g. throughput, latency, etc.) 218 is performed in this phase after the completion of the preliminary 219 test phase. Test frames are sent as required (e.g. bidirectional 220 test or unidirectional test in any of the two directions). 222 4. Recommended Benchmarking Method 224 4.1. Restricted Port Number Ranges 226 The Initiator SHOULD use restricted ranges for source and destination 227 port numbers to avoid the denial of service attack like event against 228 the connection tracking table of the DUT described in Section 2. The 229 size of the source port number range SHOULD be larger (e.g. in the 230 order of a few times ten thousand), whereas the size of the 231 destination port number range SHOULD be smaller (may vary from a few 232 to several hundreds or thousands as needed). The rationale is that 233 source and destination port numbers that can be observed in the 234 Internet traffic are not symmetrical. Whereas source port numbers 235 may be random, there are a few very popular destination port numbers 236 (e.g. 443, 80, etc., see [IIR2020]) and others hardly occur. And we 237 have found that their role is also asymmetric in the Linux kernel 238 routing hash function [LEN2020]. 240 The product of the sizes of the two ranges can be used as a 241 parameter. The performance of the stateful NATxy gateway MAY be 242 examined as a function of this parameter. 244 4.2. Preliminary Test Phase 246 The preliminary phase serves two purposes: 248 1. The connection tracking table of the DUT is filled. It is 249 important, because its maximum connection establishment rate may 250 be lower than its maximum frame forwarding rate (that is 251 throughput). 253 2. The state table of the Responder is filled with valid four 254 tuples. It is a precondition for the Responder to be able to 255 transmit frames that belong to connections exist in the 256 connection tracking table of the DUT. 258 Whereas the above two things are always necessary before the real 259 test phase, the preliminary phase can be used without the real test 260 phase. It is done so, when the maximum connection establishment rate 261 is measured (as described in Section 4.5). 263 A preliminary test phase MUST be performed before all tests performed 264 in the real test phase. In this phase, the following things happen: 266 1. The Initiator sends test frames to the Responder through the DUT 267 at a specific frame rate. 269 2. The DUT performs the stateful translation of the test frames and 270 it also stores the new combinations in its connection tracking 271 table. 273 3. The Responder receives the translated test frames and updates its 274 state table with the received four tuples. The responder 275 transmits no test frames during the preliminary phase. 277 When the preliminary test phase is performed in preparation to the 278 real test phase, the applied frame rate and the duration of the 279 preliminary phase SHOULD be carefully selected so that: 281 * The applied frame rate be safely lower than the maximum connection 282 establishment rate. 284 * Enough four tuples be stored in the state table of the Responder 285 so that it can generate frames with the proper distribution of the 286 four tuples. 288 Please refer to Section 4.4 for further conditions regarding timeout 289 and port number combinations. 291 4.3. Consideration of the Cases of Stateful Operation 293 We consider the most important Events that may happen during the 294 operation of a stateful NATxy gateway, and the Actions of the gateway 295 as follows. 297 1. EVENT: A packet not belonging to an existing connection arrives 298 in the private to public direction. ACTION: A new connection is 299 registered into the connection tracking table and the packet is 300 translated and forwarded. 302 2. EVENT: A packet not belonging to an existing connection arrives 303 in the public to private direction. ACTION: The packet is 304 discarded. 306 3. EVENT: A packet belonging to an existing connection arrives (in 307 any dicection). ACTION: The packet is translated and forwarded 308 and the timeout counter of the corresponding connection tracking 309 table entry is reset. 311 4. EVENT: A connection tracking table entry times out. ACTION: The 312 entry is deleted from the connection tracking table. 314 Due to "black box" testing, the Tester is not able to directly 315 examine (or delete) the entries of the connection tracking table. 316 But the entires can be and MUST be controlled by setting an 317 appropriate timeout value and carefully selecting the port numbers of 318 the packets (as described in Section 4.4) to be able to produce 319 meaningful and repeatable measurement results. 321 We aim to support the measurement of the following performance 322 characteristics of a stateful NATxy gateway: 324 1. maximum connection establishment rate 326 2. all "classic" performance metrics like throughput, frame loss 327 rate, latency, etc. 329 3. connection tear down rate. 331 4.4. Control of the Connection Tracking Table Entries 333 It is necessary to control the connection tracking table entries of 334 the DUT in order to achieve clear conditions for the measurements. 335 We can simply achieve the following two extreme situations: 337 1. All frames create a new entry in the connection tracking table of 338 the DUT and no old entries are deleted during the test. This is 339 required for measuring the maximum connection establishment rate. 341 2. No new entries are created in the connection tracking table of 342 the DUT and no old ones are deleted during the test. This is 343 ideal for the real test phase measurements, like throughput, 344 latency, etc. 346 From this point we use the following three assumptions: 348 1. A single source address destination address pair is used for all 349 tests. We make this assumption for simplicity. Of course, we 350 are aware that [RFC2544] requires testing also with 256 different 351 destination networks. 353 2. The connection tracking table of the stateful NATxy is large 354 enough to store all connections defined by the different source 355 port number destination port number combinations. 357 3. Each experiment is started with an empty connection tracking 358 table. (It can be ensured by deleting its content before the 359 experiment.) 361 The first extreme situation can be achieved by 363 * using all different source port number destination port number 364 combinations in the preliminary phase and 366 * setting the UDP timeout of the NATxy gateway to a value higher 367 than the length of the preliminary phase. 369 The second extreme situation can be achieved by 371 * using all different source port number destination port number 372 combinations in the preliminary phase and 374 * enumerating all the possible source port number destination port 375 number combinations in the preliminary phase and 377 * setting the UDP timeout of the NATxy gateway to a value higher 378 than the length of the preliminary phase plus the gap between the 379 two phases plus the length of the real test phase. 381 [RFC4814] REQUIRES pseudorandom port numbers, which we believe is a 382 good approximation of the distribution of the source port numbers a 383 NATxy gateway on the Internet may face with. 385 We note that pseudorandom all different source port number 386 destination port number combinations may be computing efficiently 387 generated by preparing a random permutation of the previously 388 enumerated all possible source port number destination port number 389 combinations using Dustenfeld's random shuffle algorithm [DUST1964]. 390 This method also satisfies the criterion for the second case that all 391 possible source port number destination port number combinations must 392 be enumerated during the preliminary phase. 394 Important warning: in normal (non-NAT) router testing, the port 395 number selection algorithm, whether it is pseudo-random or enumerated 396 in increasing (or decreasing) order does not affect final results. 397 However, our experience with iptables shows that if the connection 398 tracking table is filled using port number enumeration in increasing 399 order, then the maximum connection establishment rate of iptables 400 degrades significantly compared to its performance using pseudorandom 401 port numbers [LEN2021]. 403 The enumeration of the source port number destination port number 404 combinations in increasing or decreasing order (or in any other 405 specific order) MAY be used as an additional measurement. 407 4.5. Measurement of the Maximum Connection Establishment Rate 409 The maximum connection establishment rate is an important 410 characteristic of the stateful NATxy gateway and its determination is 411 necessary for the safe execution of the preliminary test phase 412 (without frame loss) before the real test phase. 414 The measurement procedure of the maximum connection establishment 415 rate is very similar to the throughput measurement procedure defined 416 in [RFC2544]. 418 Procedure: The Initiator sends a specific number of test frames using 419 all different source port number destination port number combinations 420 at a specific rate through the DUT. The Responder counts the frames 421 that are successfully translated by the DUT. If the count of offered 422 frames is equal to the count of received frames, the rate of the 423 offered stream is raised and the test is rerun. If fewer frames are 424 received than were transmitted, the rate of the offered stream is 425 reduced and the test is rerun. 427 The maximum connection establishment rate is the fastest rate at 428 which the count of test frames successfully translated by the DUT is 429 equal to the number of test frames sent to it by the Initiator. 431 Notes: 433 1. In practice, we RECOMMEND the usage of binary search. 435 2. As for the successful translation, the Responder MAY (or SHOULD?) 436 check that the source IP address is different than the original 437 source IP address set by the Initiator. 439 4.6. Real Test Phase 441 As for the traffic direction, there are three possible cases during 442 the real test phase: 444 * bidirectional traffic: The Initiator sends test frames to the 445 Responder and the Responder sends test frames to the Initiator. 447 * unidirectional traffic from the Initiator to the Responder: The 448 Initiator sends test frames to the Responder but the Responder 449 does not send test frames to the Initiator. 451 * unidirectional traffic from the Responder to the Initiator: The 452 Responder sends test frames to the Initiator but the Initiator 453 does not send test frames to the Responder. 455 If the Initiator sends test frames, then it uses pseudorandom source 456 port numbers and destination port numbers from the restricted port 457 number ranges. The responder receives the test frames, updates its 458 state table and processes the test frames as required by the given 459 measurement procedure (e.g. only counts them for throughput test, 460 handles timestamps for latency or PDV tests, etc.). 462 If the Responder sends test frames, then it uses the four tuples from 463 its state table. The reading order of the state table may follow 464 different policies (discussed in Section 4.8). The Initiator 465 receives the test frames, and processes them as required by the given 466 measurement procedure. 468 As for the actual measurement procedures, we RECOMMEND to use the 469 updated ones from Section 7 of [RFC8219]. 471 4.7. Measurement of the Connection Tear Down Rate 473 Connection tear down can cause significant load for the NATxy 474 gateway. The connection tear down performance can be measured as 475 follows: 477 1. Load a certain number of connections (N) into the connection 478 tracking table of the DUT (in the same way as done to measure the 479 maximum connection establishment rate). 481 2. Record TimestampA. 483 3. Delete the content of the connection tracking table of the DUT. 485 4. Record TimestampB. 487 The connection tear down rate can be computed as: 489 connection tear down rate = N / ( TimestampB - TimestampA) 491 The connection tear down rate SHOULD be measured for various 492 (important) values of N. 494 We assume that the content of the connection tracking table may be 495 deleted by an out-of-band control mechanism specific to the given 496 NATxy gateway implementation. (E.g. by removing the appropriate 497 kernel module under Linux.) 499 We are aware that the performance of removing the entire content of 500 the connection tracking table at one time may be different from 501 removing all the entries one by one. 503 4.8. Writing and Reading Order of the State Table 505 As for writing policy of the state table of the Responder, we 506 RECOMMEND round robin, because it ensures that its entries are 507 automatically kept fresh and consistent with that of the connection 508 tracking table of the DUT. 510 The Responder can read its state table in various orders, for 511 example: 513 * pseudorandom 515 * round robin 517 We RECOMMEND pseudorandom to follow the spirit of [RFC4814]. Round 518 robin may be used as a computationally cheaper alternative. 520 5. Implementation and Experience 522 The "stateful" branch of siitperf [SIITPERF] is an implementation of 523 this concept. It is documented in a paper currently under second 524 review [LEN2022]. 526 Our experience with this methodology using siitperf for measuring the 527 scalability of the iptables stateful NAT44 and Jool stateful NAT64 528 implementations is described in 529 [I-D.lencse-v6ops-transition-scalability]. 531 6. Limitations of using UDP as Transport Layer Protocol 533 Stateful NATxy solutions handle TCP and UDP differently, e.g. 534 iptables uses 30s timeout for UDP and 60s timeout for TCP. Thus 535 benchmarking results produced using UDP do not necessarily 536 characterize the performance of a NATxy gateway well enough, when 537 they are used for forwarding Internet traffic. As for the given 538 example, timeout values of the DUT may be adjusted, but it requires 539 extra consideration. 541 Other differences in handling UDP or TCP are also possible. Thus we 542 recommend that further investigations are to be performed in this 543 field. 545 As a mitigation of this problem, we recommend that testing with 546 protocols usig TCP (like HTTP and HTTPS) can be performed as 547 described in [I-D.ietf-bmwg-ngfw-performance]. This approach also 548 solves the potential problem of protocol helpers may be present in 549 the stateful DUT. 551 7. Acknowledgements 553 The authors would like to thank Al Morton, Sarah Banks, Edwin 554 Cordeiro, Lukasz Bromirski and Sandor Repas for their comments. 556 8. IANA Considerations 558 This document does not make any request to IANA. 560 9. Security Considerations 562 We have no further security considerations beyond that of [RFC8219]. 563 Perhaps they should be cited here so that they be applied not only 564 for the benchmarking of IPv6 transition technologies, but also for 565 the benchmarking of stateful NATxy gateways. 567 10. References 569 10.1. Normative References 571 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate 572 Requirement Levels", BCP 14, RFC 2119, 573 DOI 10.17487/RFC2119, March 1997, 574 . 576 [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for 577 Network Interconnect Devices", RFC 2544, 578 DOI 10.17487/RFC2544, March 1999, 579 . 581 [RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked 582 Factors in Network Device Benchmarking", RFC 4814, 583 DOI 10.17487/RFC4814, March 2007, 584 . 586 [RFC5180] Popoviciu, C., Hamza, A., Van de Velde, G., and D. 587 Dugatkin, "IPv6 Benchmarking Methodology for Network 588 Interconnect Devices", RFC 5180, DOI 10.17487/RFC5180, May 589 2008, . 591 [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 592 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, 593 May 2017, . 595 [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking 596 Methodology for IPv6 Transition Technologies", RFC 8219, 597 DOI 10.17487/RFC8219, August 2017, 598 . 600 10.2. Informative References 602 [DUST1964] Durstenfeld, R., "Algorithm 235: Random 603 permutation", Communications of the ACM, vol. 7, no. 7, 604 p.420., DOI 10.1145/364520.364540, July 1964, 605 . 607 [I-D.ietf-bmwg-ngfw-performance] 608 Balarajah, B., Rossenhoevel, C., and B. Monkman, 609 "Benchmarking Methodology for Network Security Device 610 Performance", Work in Progress, Internet-Draft, draft- 611 ietf-bmwg-ngfw-performance-13, 12 January 2022, 612 . 615 [I-D.lencse-v6ops-transition-scalability] 616 Lencse, G., "Scalability of IPv6 Transition Technologies 617 for IPv4aaS", Work in Progress, Internet-Draft, draft- 618 lencse-v6ops-transition-scalability-01, 21 February 2022, 619 . 622 [IIR2020] Kurahashi, T., Matsuzaki, Y., Sasaki, T., Saito, T., and 623 F. Tsutsuji, "Periodic observation report: Internet trends 624 as seen from IIJ infrastructure - 2020", Internet 625 Infrastructure Review, vol. 49, December 2020, 626 . 629 [LEN2020] Lencse, G., "Adding RFC 4814 Random Port Feature to 630 Siitperf: Design, Implementation and Performance 631 Estimation", International Journal of Advances in 632 Telecommunications, Electrotechnics, Signals and Systems, 633 vol 9, no 3, pp. 18-26., DOI 10.11601/ijates.v9i3.291, 634 2020, . 637 [LEN2021] Lencse, G., "Design and Implementation of a Software 638 Tester for Benchmarking Stateful NAT64 Gateways: Theory 639 and Practice of Extending Siitperf for Stateful 640 Tests", it was under review in Computer 641 Communications, then it was significantly rewritten, 642 2021, . 645 [LEN2022] Lencse, G., "Design and Implementation of a Software 646 Tester for Benchmarking Stateful NAT64xy Gateways: Theory 647 and Practice of Extending Siitperf for Stateful 648 Tests", revised version, under second review in Computer 649 Communications, may be revised or removed without notice, 650 2022, . 653 [SIITPERF] Lencse, G., "Siitperf: An RFC 8219 compliant SIIT 654 (stateless NAT64) tester written in C++ using 655 DPDK", source code, available from GitHub, 2019-2022, 656 . 658 Appendix A. Change Log 660 A.1. 00 662 Initial version. 664 A.2. 01 666 Updates based on the comments received on the BMWG mailing list and 667 minor corrections. 669 A.3. 02 671 Section 4.4 was completely re-written. As a consequence, the 672 occurrences of the now undefined "mostly different" source port 673 number destination port number combinations were deleted from 674 Section 4.5, too. 676 A.4. 03 678 Added Section 4.3 about the consideration of the cases of stateful 679 operation. 681 Consistency checking. Removal of some parts obsolated by the 682 previous re-writing of Section 4.4. 684 Added Section 4.7 about the method for measuring connection tear down 685 rate. 687 Updates for Section 5 about the implementation and experience. 689 Authors' Addresses 690 Gabor Lencse 691 Szechenyi Istvan University 692 Gyor 693 Egyetem ter 1. 694 H-9026 695 Hungary 696 Email: lencse@sze.hu 698 Keiichi Shima 699 IIJ Innovation Institute 700 Iidabashi Grand Bloom, 2-10-2 Fujimi, Tokyo 701 102-0071 702 Japan 703 Email: keiichi@iijlab.net