Mozilla

daniel@haxx.se http://daniel.haxx.se

General httpbis Internet-Draft This document records current best practice for using all versions of HTTP over TCP.

HTTP version 1.1 as well as HTTP version 2 are defined to use TCP , and their performance can depend greatly upon how TCP is configured. This document records the best current practice for using HTTP over TCP, with a focus on improving end-user perceived performance. These practices are generally applicable to HTTP/1 as well as HTTP/2, although some may note particular impact or nuance regarding a particular protocol version. There are countless scenarios, roles and setups where HTTP is being using so there can be no single specific “Right Answer” to most TCP questions. This document intends only to cover the most important areas of concern and suggest possible actions.

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in .

Your HTTP server or intermediary may need configuration changes to some system tunables and timeout periods to perform optimally. Actual values will depend on how you are scaling the platform, horizontally or vertically, and other connection semantics. Changing system limits and altering thresholds will change the behavior of your web service and its dependencies. These dependencies are usually common to other services running on the same system, so good planning and testing is advised. This is a list of values to consider and some general advice on how those values can be modified on Linux systems.

A modern HTTP server will serve a large number of TCP connections and in most systems each open socket equals an open file. Make sure that limit isn’t a bottle neck.

Raise the number of packets allowed to get queued when a particular interface receives packets faster than the kernel can process them.

The number of new connection requests that are allowed to queue up in the kernel. These can be connections that are in SYN RECEIVED or ESTABLISHED states. Historically, operating systems used a single backlog queue for both of these states. Newer implemntations use two separate queues: one for connections in SYN RECEIVED and one for those which are ESTABLISHED state (better known as the accept queue).

To make sure the TCP stack can take full advantage of the entire set of possible sockets, give it a larger range of local port numbers to use.

High connection completion rates will consume ephemeral ports quickly. Lower the time during which connections are in FIN-WAIT-2/TIME_WAIT states so that they can be purged faster and thus maintain a maximal number of available sockets. The primitives for the assignment of these values were described in , however significantly lower values are commonly used.

When running backend servers on a managed, low latency network you might allow the reuse of sockets in TIME_WAIT state for new connections when a protocol complete termination has occurred. There is no RFC that covers this behaviour.

Systems meant to handle and serve a huge number of TCP connections at high speeds can require significant amounts of memory for TCP socket buffers to maintain performance. On some systems you can tell the TCP stack what default buffer sizes to use and how much they are allowed to dynamically grow and shrink. Window Scaling is typically linked to socket buffer sizes. The minimum and default values for socket buffers tend to require less proactive amendment than the maximum value. When deriving maximum values for use, you should consider the BDP (Bandwidth Delay Product) of the target environment and client paths. Consider also that ‘read’ and ‘write’ values do not require to be synchronised, as the BDP for a load balancer or middle-box might be very different when acting as a sender or receiver due to the network charateristics in either context. e.g. A cache with fast, low latency network to an origin serving high latency clients. Allowing needlessly high values beyond the expected limitations of the platform will not improve performance however can cause buffer induced delays within the path or excessive retransmissions during congestion events. Extensions such as ECN coupled with AQM can help mitigate this undesirable behaviour .

Window Scaling is provided as a function of the congestion control algorithm used on a platform. Initial and maximal values can usually be configured. The window size used at connection startup is a calculated value using the MSS discovered during the 3WHS and the Initial Window (IW) in both send and receive contexts (initcwnd and initrwnd). covers Window Scaling in greater detail. You may have to increase the largest allowed window size from the system default to increase the throuput for high latency clients. Window scaling must be accommodated within the maximal values, however it is not uncommon to see the maximum definable higher than the scalable limit; these values can be statically defined within socket parameters (SO_RCVBUF,SO_SNDBUF). Changes to the size of the window or incomplete window usage are common secondary symptoms of ‘slow transfer rates’ on a loss free path. Locating the root cause of these symptoms, usually on the client or server system, is an important step commonly overlooked in favour of blaming the path.

On a modern shared platform it can be common to plan for both long and short lived connections on the same implementation. However, the delivery of static assets and a ‘web push’ or ‘long poll’ service provide very different quality of service promises. Fail ‘fast’: TCP resources can be highly contended. For fault tolerance reasons a server needs to be able to determine within a reasonable time frame whether a connection is still active or required. e.g. If static assets typically return in 100s of milliseconds, and users ‘switch off’ after <10s keeping timeouts of >30s make little sense and defining a ‘quality of service’ appropriate to the target platform is encouraged. On a shared platform with mixed session lifetimes, applications that require longer render times have various options to ensure the underlying service and upstream servers in the path can identify the session as not failed: HTTP continuations, Redirects, 202s or sending data. Clients and servers typically have many timeout options, a few notable options are: Connect(client), time to request(server), time to first byte(client), between bytes(server/client), total connection time(server/client). Some implementations merge these values into a single ‘timeout’ definition even when statistics are reported individually. All should be considered as the defaults in many implementations are highly underiable, even infinite timeouts have been observed.

TCP Fast Open (a.k.a. TFO, ) allows data to be sent on the TCP handshake, thereby allowing a request to be sent without any delay if a connection is not open. TFO requires both client and server support, and additionally requires application knowledge, because the data sent on the SYN needs to be idempotent. Therefore, TFO can only be used on idempotent, safe HTTP methods (e.g., GET and HEAD), or with intervening negotiation (e.g, using TLS). It should be noted that TFO requires a secret to be defined on the server to mitigate security vulnerabilities it introduces. TFO therefore requires more server side deployment planning than other enhancements. Support for TFO is growing in client platforms, especially mobile, due to the significant performance advantage it gives.

proposes a new IW of 10*MSS, and is now fairly widely deployed server-side. Many implementations allow you to tune both initcwnd and initrwnd values. Some implementations allow these values to be applied to specific routes which allows a greater degree of control over known paths. There has been experimentation with larger initial windows in combination with packet pacing, however IW10 has been reported to perform fairly well even in both general and high volume use cases.

TCP SYN Flood mitigations are necessary and there will be thresholds to tweak.

TBD cubic codel pacing

Apple deploying in iOS and OSX.

Nagle’s Algorithm is the mechanism that makes the TCP stack hold (small) outgoing packets for a short period of time so that it can potentially merge that packet with the next outgoing one. It is optimized for throughput at the expense of latency. HTTP/2 in particular requires that the client can send a packet back fast even during transfers that are perceived as single direction transfers. Even small delays in those sends can cause a significant performance loss. HTTP/1.1 is also affected, especially when sending off a full request in a single write() system call. In POSIX systems you switch it off like this:

Delayed ACK is a mechanism enabled in most TCP stacks that causes the stack to delay sending acknowledgement packets in response to data. The ACK is delayed up until a certain threshold, or until the peer has some data to send, in which case the ACK will be sent along with that data. Depending on the traffic flow and TCP stack this delay can be as long as 500ms. This interacts poorly with peers that have Nagle’s Algorithm enabled. Because Nagle’s Algorithm delays sending until either one MSS of data is provided or until an ACK is received for all sent data, delaying ACKs can force Nagle’s Algorithm to buffer packets when it doesn’t need to (that is, when the other peer has already processed the outstanding data). Delayed ACKs can be useful in situations where it is reasonable to assume that a data packet will almost immediately (within 500ms) cause data to be sent in the other direction. In general in both HTTP/1.1 and HTTP/2 this is unlikely: therefore, disabling Delayed ACKs can provide an improvement in latency. However, the TLS handshake is a clear exception to this case. For the duration of the TLS handshake it is likely to be useful to keep Delayed ACKs enabled. Additionally, for low-latency servers that can guarantee responses to requests within 500ms, on long-running connections (such as HTTP/2), and when requests are small enough to fit within a small packet, leaving delayed ACKs turned on may provide minor performance benefits. Effective use of switching off delayed ACKs requires extensive profiling.

TCP keep-alive is likely disabled - at least on mobile clients for energy saving purposes. App-level keep-alive is then required for long-lived requests to detect failed peers or connections reset by stateful firewalls etc.

Slow-start is one of the algorithms that TCP uses to control congestion inside the network. It is also known as the exponential growth phase. Each TCP connection will start off in slow-start but will also go back to slow-start after a certain amount of idle time.

There are several HTTP authentication mechanisms in use today that are used or can be used to authenticate a connection rather than a single HTTP request. Two popular ones are NTLM and Negotiate. If such an authentication has been negotiated on a TCP connection, that connection can remain authenticated throughout the rest of its lifetime. This discrepancy with how other HTTP authentications work makes it important to handle these connections with care.

The client or server is free to half-close after a request or response has been completed; or when there is no pending stream in HTTP/2. Half-closing is sometimes the only way for a server to make sure it closes down connections cleanly so that it doesn’t accept more requests while still allowing clients to receive the ongoing responses.

No client abort for HTTP/1.1 after the request body has been sent. Delayed full close is expected following an error response to avoid RST on the client.

Keeping open connections around for subsequent connection reuse is key for many HTTP clients’ performance. The value of an existing connection quickly degrades and after only a few minutes the chance that a connection will successfully get reused by a web browser is slim.

draft

This document does not require action from IANA.

TBD

In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document provides an overview of HTTP architecture and its associated terminology, defines the "http" and "https" Uniform Resource Identifier (URI) schemes, defines the HTTP/1.1 message syntax and parsing requirements, and describes related security concerns for implementations. This specification describes an optimized expression of the semantics of the Hypertext Transfer Protocol (HTTP), referred to as HTTP version 2 (HTTP/2). HTTP/2 enables a more efficient use of network resources and a reduced perception of latency by introducing header field compression and allowing multiple concurrent exchanges on the same connection. It also introduces unsolicited push of representations from servers to clients.This specification is an alternative to, but does not obsolete, the HTTP/1.1 message syntax. HTTP's existing semantics remain unchanged. This memo discusses some aspects of congestion control in IP/TCP Internetworks. It is intended to stimulate thought and further discussion of this topic. While some specific suggestions are made for improved congestion control implementation, this memo does not specify any standards. This RFC is an official specification for the Internet community. It incorporates by reference, amends, corrects, and supplements the primary protocol standards documents relating to hosts. [STANDARDS-TRACK] This document describes TCP SYN flooding attacks, which have been well-known to the community for several years. Various countermeasures against these attacks, and the trade-offs of each, are described. This document archives explanations of the attack and common defense techniques for the benefit of TCP implementers and administrators of TCP servers or networks, but does not make any standards-level recommendations. This memo provides information for the Internet community. This document proposes an experiment to increase the permitted TCP initial window (IW) from between 2 and 4 segments, as specified in RFC 3390, to 10 segments with a fallback to the existing recommendation when performance issues are detected. It discusses the motivation behind the increase, the advantages and disadvantages of the higher initial window, and presents results from several large-scale experiments showing that the higher initial window improves the overall performance of many web services without resulting in a congestion collapse. The document closes with a discussion of usage and deployment for further experimental purposes recommended by the IETF TCP Maintenance and Minor Extensions (TCPM) working group. This document provides recommendations of best current practice for dropping or marking packets using any active queue management (AQM) algorithm, including Random Early Detection (RED), BLUE, Pre- Congestion Notification (PCN), and newer schemes such as CoDel (Controlled Delay) and PIE (Proportional Integral controller Enhanced). We give three strong recommendations: (1) packet size should be taken into account when transports detect and respond to congestion indications, (2) packet size should not be taken into account when network equipment creates congestion signals (marking, dropping), and therefore (3) in the specific case of RED, the byte- mode packet drop variant that drops fewer small packets should not be used. This memo updates RFC 2309 to deprecate deliberate preferential treatment of small packets in AQM algorithms. This document specifies a set of TCP extensions to improve performance over paths with a large bandwidth * delay product and to provide reliable operation over very high-speed paths. It defines the TCP Window Scale (WS) option and the TCP Timestamps (TS) option and their semantics. The Window Scale option is used to support larger receive windows, while the Timestamps option can be used for at least two distinct mechanisms, Protection Against Wrapped Sequences (PAWS) and Round-Trip Time Measurement (RTTM), that are also described herein.This document obsoletes RFC 1323 and describes changes from it. This document describes an experimental TCP mechanism called TCP Fast Open (TFO). TFO allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, and saves up to one full round-trip time (RTT) compared to the standard TCP, which requires a three-way handshake (3WHS) to complete before data can be exchanged. However, TFO deviates from the standard TCP semantics, since the data in the SYN could be replayed to an application in some rare circumstances. Applications should not use TFO unless they can tolerate this issue, as detailed in the Applicability section.

This specification builds upon previous work and help from Mark Nottingham, Craig Taylor

Here are some sample operating system settings for the Linux operating system, along with the section it refers to.

]]>

]]>

]]>

]]>

]]>

]]>

]]>

]]>

Turning off Nagle’s Algorithm:

On recent Linux kernels (since Linux 2.4.4), Delayed ACKs can be disabled like this:

Unlike disabling Nagle’s Algorithm, disabling Delayed ACKs on Linux is not a one-time operation: processing within the TCP stack can cause Delayed ACKs to be re-enabled. As a result, to use TCP_QUICKACK effectively requires setting and unsetting the socket option during the life of the connection.