Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port Numbers

has defined a comprehensive benchmarking methodology for network interconnect devices, which is still in use. It was mainly IP version independent, but it used IPv4 in its examples. addressed IPv6 specificities and also added technology updates, but declared IPv6 transition technologies out of its scope. addressed the IPv6 transition technologies, including stateful NAT64. It has reused several benchmarking procedures from (e.g. throughput, frame loss rate), it has redefined the latency measurement, and added further ones, e.g. the PDV (packet delay variation) measurement. However, none of them discussed, how to apply pseudorandom port numbers, when benchmarking stateful NATxy (NAT44, NAT64, NAT66) gateways. We are not aware of any other RFCs that address this question. First, we discuss why using pseudorandom port numbers with stateful NATxy gateways is a hard problem. Then we recommend a solution.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP14 when, and only when, they appear in all capitals, as shown here.

In its appendix, has defined a frame format for test frames including specific source and destination port numbers. recommends to use pseudorandom and uniformly distributed values for both source and destination port numbers. However, stateful NATxy (NAT44, NAT64, NAT66) solutions use the port numbers to identify connections. The usage of pseudorandom port numbers causes different problems depending on the direction. As for the private to public direction, pseudorandom source and destination port numbers could be used, however, this approach would be a denial of service attack against the stateful NATxy gateway, because it would exhaust its connection tracking table. To that end, let us see some calculations using the recommendations of RFC 4814: The recommended source port range is: 1024-65535, thus its size is: 64512. The recommended destination port range is: 1-49151, thus its size is: 49151. The number of source and destination port number combinations is: 3,170,829,312. We note that section 12 of also requires testing with 256 destination networks, which further increases the number of connection tracking table entries. As for the public to private direction, the stateful DUT would drop any packets that do not belong to an existing connection, therefore, the direct usage of pseudorandom port numbers from the above-mentioned ranges is not feasible.

Our methodology works with any IP version. We use IPv4 in the Test Setup shown in to facilitate its easy understanding based on the well-known stateful NAT44 (also called NAPT: Network Address and Port Translation) solution.

| Sateful NATxy gateway |-------------+ private IPv4| [connection tracking table] | public IPv4 +--------------------------------------+ ]]> As for transport layer protocol, recommended testing with UDP, and it was kept also in . For the general recommendation, we also keep UDP, thus the port numbers in the following text are to be understood as UDP port numbers. We discuss the limitation of this approach in . We define the most important elements of our proposed benchmarking system as follows. Connection tracking table: The stateful NATxy gateway uses a connection tracking table to be able to perform the stateful translation in the public to private direction. Its size, policy and content is unknown for the Tester. Four tuple: The four numbers that identify a connection are source IP address, source port number, destination IP address, destination port number. Initiator: The port of the Tester that may initiate a connection through the stateful DUT in the private to public direction. Theoretically, it can use any source and destination port numbers: if they do not belong to an existing connection, the DUT will register a new connection into its connection tracking table. Responder: The port of the Tester that may not initiate a connection through the stateful DUT in the public to private direction. It may send only frames that belong to an existing connection. To that end, it uses four tuples that have been previously extracted from the received test frames and stored in its state table. State table: The Responder of the Tester extracts the four tuple from each received test frame and stores it in its state table. Recommendation is given for writing and reading order of the state table in . Preliminary test phase: Test frames are sent only by the Initiator to the Responder through the DUT to fill both the connection tracking table of the DUT and the state table of the Responder. This is a newly introduced operation phase for stateful NATxy benchmarking. The necessity of this phase is explained in . Real test phase: The actual test (e.g. throughput, latency, etc.) is performed in this phase after the completion of the preliminary test phase. Test frames are sent as required (e.g. bidirectional test or unidirectional test in any of the two directions).

The Initiator SHOULD use restricted ranges for source and destination port numbers to avoid the denial of service attack against the connection tracking table of the DUT described in . The size of the source port number range SHOULD be larger (e.g. in the order of a few times ten thousand), whereas the size of the destination port number range SHOULD be smaller (may vary from a few to several hundreds or thousands as needed). The rationale is that source and destination port numbers that can be observed in the Internet traffic are not symmetrical. Whereas source port numbers may be random, there are a few very popular destination port numbers (e.g. 443, 80, etc., see ) and others hardly occur. And we have found that their role is also asymmetric in the Linux kernel routing hash function . The product of the sizes of the two ranges can be used as a parameter. The performance of the stateful NATxy gateway MAY be examined as function of this parameter.

The preliminary phase serves two purposes: The connection tracking table of the DUT is filled. It is important, because its maximum connection establishment rate may be lower than its maximum frame forwarding rate (that is throughput). The state table of the Responder is filled with valid four tuples. It is a precondition for the Responder to be able to transmit frames that belong to connections exist in the connection tracking table of the DUT. Whereas the above two things are always necessary before the real test phase, the preliminary phase can be used without the real test phase. It is done so, when the maximum connection establishment rate is measured (as described in ). A preliminary test phase MUST be performed before all tests performed in the real test phase. In this phase, the following things happen: The Initiator sends test frames to the Responder through the DUT at a specific frame rate. The DUT performs the stateful translation of the test frames and it also stores the new combinations in its connection tracking table. The Responder receives the translated test frames and updates its state table with the received four tuples. The responder transmits no test frames during the preliminary phase. When the preliminary test phase is performed in preparation to the real test phase, the applied frame rate and the duration of the preliminary phase SHOULD be carefully selected so that: The applied frame rate be safely lower than the maximum connection establishment rate. The initial transient of the filling of the connection tracking table of the DUT be finished. Enough four tuples be stored in the state table of the Responder so that it can generate frames with the proper distribution of the four tuples. The connections do not time out in the DUT even during the beginning of the real test phase.

The number of the entries in the connection tracking table of the DUT MAY be controlled by using all different source port number destination port number combinations. Let NF and NC denote the number of test frames to send and the number of all possible source port number destination port number combinations, respectively. If NF and NC are in the same order of magnitude, then the all different source port number destination port number combinations may be computing efficiently generated by preparing a random permutation of the previously enumerated all possible source port number destination port number combinations using Dustenfeld's random shuffle algorithm . If NF is at least an order of magnitude less than NC, then a simpler solution may be used: the Initiator registers in a table and then it checks if a given source port number destination port number combination was already used (and if yes, then it MUST generate a new one). If NF is at least two orders of magnitude less than NC, then mostly different source port number destination port number combinations can be generated without any specific provision. Important warning: in normal router testing, the port number selection algorithm (whether it is pseudo-random or enumerated) does not affect final results. However, our experience with iptables shows that if the connection tracking table is filled using port number enumeration in increasing order, then the maximum connection establishment rate of iptables degrades significantly compared to its performance using pseudorandom port numbers . REQUIRES pseudorandom port numbers, which we believe is a good approximation of the distribution of the source port numbers a NATxy gateway on the Internet may face with. The enumeration of the source port number destination port number combinations in increasing order MAY be used as an additional measurement to discover worst case performance.

The maximum connection establishment rate is an important characteristic of the stateful NATxy gateway and its determination is necessary for the safe execution of the preliminary test phase (without frame loss) before the real test phase. The measurement procedure of the maximum connection establishment rate is very similar to the throughput measurement procedure defined in . Procedure: The Initiator sends a specific number of test frames using all (or mostly) different source port number destination port number combinations at a specific rate through the DUT. The Responder counts the frames that are successfully translated by the DUT. If the count of offered frames is equal to the count of received frames, the rate of the offered stream is raised and the test is rerun. If fewer frames are received than were transmitted, the rate of the offered stream is reduced and the test is rerun. The maximum connection establishment rate is the fastest rate at which the count of test frames successfully translated by the DUT is equal to the number of test frames sent to it by the Initiator. Notes: All different source port number destination port number combinations SHOULD be used, if the results of the measurement are published. Mostly different source port number destination port number combinations MAY be used, if the results of the measurement are not published, but they are used only to determine a good enough frame rate for the preliminary test phase preparing the test system for the real test phase. In practice, we RECOMMEND the usage of binary search. As for the successful translation, the Responder MAY (or SHOULD?) check that the source IP address is different than the original source IP address set by the Initiator.

As for the traffic direction, there are three possible cases during the real test phase: bidirectional traffic: The Initiator sends test frames to the Responder and the Responder sends test frames to the Initiator. unidirectional traffic from the Initiator to the Responder: The Initiator sends test frames to the Responder but the Responder does not send test frames to the Initiator. unidirectional traffic from the Responder to the Initiator: The Responder sends test frames to the Initiator but the Initiator does not send test frames to the Responder. If the Initiator sends test frames, then it uses pseudorandom source port numbers and destination port numbers from the restricted port number ranges. The responder receives the test frames, updates its state table and processes the test frames as required by the given measurement procedure (e.g. only counts them for throughput test, handles timestamps for latency or PDV tests, etc.). If the Responder sends test frames, then it uses the four tuples from its state table. The reading order of the state table may follow different policies (discussed in ). The Initiator receives the test frames, and processes them as required by the given measurement procedure. As for the actual measurement procedures, we RECOMMEND to use the updated ones from Section 7 of .

As for writing policy of the state table of the Responder, we RECOMMEND round robin, because it ensures that its entries are automatically kept fresh and thus there is no need to handle timeout. The Responder can read its state table in various orders. We RECOMMEND one of the following ones: round robin pseudorandom (with restriction!) random permutation (no position is repeated until all positions are used). Pseudorandom reading order of the state table MAY NOT be used with unidirectional traffic from the Responder to the Initiator, because if a four tuple is not used until timeout time, then its connection is deleted from the connection tracking table of the DUT and a later use of the given four tuple will cause frame loss. There is no such problem, when bidirectional traffic is used, because then the state table of the Responder is periodically refreshed. We do not see any problem in the round robin reading order, because the state table is filled using pseudorandom port numbers.

Stateful testing involves some issues not present in stateless testing.

Even though we do black box testing, one MUST consider timeout and carefully manage timeout budget. For example, if the frame rate is high enough, then every single entry of the state table of the Responder is refreshed within timeout time and it prevents frame sending with a stale four tuple. If the entries of the state table are not refreshed (due to testing with single directional traffic from the Responder to the Initiator) then using all four tuples within timeout time can keep all connection tracking table entries of the DUT alive. Special care should be taken for the lower frame rate in the preliminary phase. If the binary search (or the decreasing of the applied frame rates during the frame loss rate test) results in a frame rate that is too low to keep alive the connection tracking table entries, then it results in the failure of the consecutive tests (the binary search of the throughput test counts down to zero).

Several network performance tester vendors include a parameter called "Loss Tolerance" (or similar) for the throughput test and several benchmarking professionals actually use nonzero values . If frames are lost during stateful testing (especially if it happens during a test with unidirectional traffic from the Responder to the Initiator) the refreshing of the corresponding connection tracking table element is not ensured and it may result in the loss of further frames (not due to the low performance of the DUT, but due to using a stale four tuple).

The "stateful" branch of siitperf is a partial implementation of this concept. Our experience is documented in a paper currently under review .

Stateful NATxy solutions handle TCP and UDP differently, e.g. iptables uses 30s timeout for UDP and 60s timeout for TCP. Thus benchmarking results produced using UDP do not necessarily characterize the performance of a NATxy gateway well enough, when they are used for forwarding Internet traffic. As for the given example, timeout values of the DUT may be adjusted, but it requires extra consideration. Other differences in handling UDP or TCP are also possible. Thus we recommend that further investigations are to be performed in this field.

The authors would like to thank ... (TBD)

This document does not make any request to IANA.

We have no further security considerations beyond that of . Perhaps they should be cited here so that they be applied not only for the benchmarking of IPv6 transition technologies, but also for the benchmarking of stateful NATxy.