ICAP Trailers

ICAP specification says that the Trailer header field is "defined in ICAP the same as [...] in HTTP". Unfortunately, that phrase alone is not enough for trailer-related interoperability in the ICAP context because of the following conflicting interpretations, requirements, and needs: Both ICAP and HTTP message headers might contain a Trailer field.HTTP messages might contain HTTP trailers (that ICAP servers could be interested in receiving or even sending). An HTTP trailer can be present with or without an HTTP Trailer header field.ICAP agents need to distinguish an HTTP trailer from an ICAP trailer.HTTP uses Chunked Transfer Coding to transmit trailers. The chunked coding is applied to the entire HTTP message body. This choice places an HTTP trailer inside an HTTP message body.It is possible to interpret the ICAP specification as placing the ICAP trailer either inside the HTTP message body or inside the ICAP message body.ICAP does not chunk-encode ICAP message bodies. Instead, ICAP message bodies contain a combination of zero, one, or two HTTP headers possibly followed by a chunked-encoded HTTP message body.Chunked coding does not support multiple trailers: Chunked HTTP messages always contain exactly one (but possibly empty) trailer part.HTTP effectively restricts trailers usage to messages with bodies, presumably because, without a body, the information in the trailer can usually be placed in the message header. In ICAP context, it is not obvious whether trailers ought to be restricted to messages with HTTP bodies (embedded in ICAP bodies) or to messages with ICAP bodies (that might only contain HTTP headers and no HTTP body). These problems led to a ban on ICAP trailers. Several designs were considered for introducing proper ICAP trailers support: Extend chunked coding to support multiple trailers (one for HTTP and one for ICAP). This option was rejected because many ICAP agents use existing HTTP-focused libraries to parse embedded HTTP bodies. As anecdotal evidence related to the ICAP-only "ieof" chunk extension support shows, it would be difficult to extend those libraries to handle a complicated ICAP-only extension. Also, this design would make it difficult to send an ICAP trailer when processing large HTTP messages without bodies.Embed ICAP trailer fields inside the chunked HTTP message body trailer, using an ICAP-specific field name prefix (e.g., "ICAP-"). This option was rejected because it would either allow malicious HTTP messages to inject ICAP trailers or require ICAP clients to hide conflicting HTTP trailer fields from the ICAP server. This design also badly violates layering boundaries by mixing HTTP- and ICAP-level information in the same protocol structure.Extend Encapsulated header with a "trailer" token. This option was rejected because the Encapsulated header describes embedded HTTP message parts and an ICAP trailer is not a part of any HTTP message. In other words, ICAP trailers do not get encapsulated.Clarify the ICAP Trailer semantics (and transfer mechanism) without introducing any new trailer support-negotiation mechanism. This option was rejected because trailers affect message framing and many existing ICAP agent implementations cannot parse any form of trailers.Add a new trailer support-negotiation mechanism (e.g., "Allow: trailers") and a new trailer presence-signaling mechanism (e.g., Trailer2) while leaving the poorly defined Trailer header semantics as is. This option was narrowly rejected because a new trailer support-negotiation mechanism alone was deemed sufficient to resolve conflicts between this specification and any reasonable existing implementation of the poorly defined Trailer semantics.Add a new trailer support-negotiation mechanism and only clarify Trailer header semantics (and transfer mechanism) upon successful negotiation, while reusing the well-known Trailer header field name as the trailer presence-signaling mechanism (requiring successful support negotiation). This specification documents this design.

Trailers allow an ICAP agent to transmit metadata after the message body. Such delayed transmission is useful when the same information was not available at the start of the message transmission. For example: A client uses an ICAP trailer to relay the current HTTP/1.1 connection or HTTP/2 stream status after transmitting a large HTTP message. A server uses that information to optimize message analysis (e.g., skip or abort analysis of HTTP requests sent by already disconnected HTTP clients).A server uses an ICAP trailer to relay audit information about viruses present at the end of a large HTTP message.A server uses an ICAP trailer to relay prefetching information about HTML parts referenced from a large HTTP message.

This section informally describes the overall feature operation. This description is deliberately imprecise and cannot be used to build compliant implementations. The following sections contain actual protocol requirements. To announce feature support, ICAP agents exchange "Allow: trailers" settings during an OPTIONS transaction. To send a trailer at the end of a particular REQMOD or RESPMOD transaction, the agent first sends both "Allow: trailers" and "Trailer" header fields. The Trailer field lists header field names expected in the message trailer section. After sending the entire ICAP message body, the agent sends the trailer section (a.k.a. "trailer"). The trailer section is syntactically equivalent to the ICAP message header section. The trailer section does not have to contain any of the promised fields and might even have no fields at all. A trailer makes sense only in an ICAP message with a body. However, a trailer could be sent if the ICAP message body encapsulates just HTTP headers. If a trailer is sent, its bytes are always the last bytes sent during the entire ICAP transaction. Thus, a client never sends a trailer at the end of Preview unless it sent the "ieof" chunk extension as well. Similarly, a server never a sends trailer with a 100 (Continue) control message.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in . Conformance criteria and error handling considerations are defined in Section 2.5 of . This specification uses the Augmented Backus-Naur Form (ABNF) notation of with a list extension defined in Section 7 of . All syntax rules not explicitly defined in this specification (e.g., header-field and CRLF) are defined in (or included by reference from) . The "Allow/X" notation is defined in .

The use of the Allow header field defined in sections 4.6 and 4.10.2 of is extended while preserving its original syntax and general semantics (i.e., a comma-separated list of tokens, each identifying an ICAP feature supported by the sending agent):

Allow = 1#token defines Allow usage in OPTIONS requests and responses. This specification extends Allow usage to other messages and adds "trailers" to the list of possible Allow value tokens. An agent MUST treat multiple Allow header fields as one Allow field with a comma-separated list of individual field value tokens, concatenated in the order of their appearance in the ICAP header. An ICAP agent MUST ignore Allow header tokens it does not understand. This document does not specify the significance of Allow tokens order and impact of repeated tokens. An agent MAY send an Allow header in any message. Such header contains a list of ICAP features supported by the sending agent. For example, a client could send an "Allow: trailers" in a REQMOD request and receive an "Allow: 204, trailers, 206" header field in response. The exact meaning of each Allow token in a context of its message is defined by the corresponding feature specification. By sending Allow, the agent indicates compliance with each listed feature's specification but does not necessarily commit to offer or use any of the listed features during future transactions. To keep this specification succinct, we introduce "Allow/X" notation to mean an ICAP Allow header field which value contains an "X" token, possibly among other tokens. For example, "Allow/trailers" stands for an Allow header field with a "trailers" token and possibly other tokens. This notation covers multi-field Allow headers as well because they are equivalent to a combined single-field Allow header. The same approach to extending Allow header usage was successfully applied to the ICAP 206 extension (XXX: reference our expired ICAP icap-ext-partial-content draft). This specification is compatible with the ICAP 206 extension.

An ICAP message with a trailer is a concatenation of a regular ICAP message and an trailer section. The trailer section syntax is identical to the ICAP header syntax:

ICAP-message-without-trailer = <see [RFC3507]> ICAP-message-with-trailer = ICAP-message-without-trailer trailer-section trailer-section = *( trailer-field CRLF ) CRLF trailer-field = header-field Note that any trailer, even a trailer without fields, ends with CRLF. That terminating sequence is essential for proper message framing on persistent ICAP connections. A sender MUST NOT generate a trailer section that contains a field necessary for message framing (e.g., Encapsulated, Preview, and Trailer), routing (e.g., Host), or authentication.

The ICAP Trailer header value syntax is identical to the HTTP Trailer syntax:

Trailer = 1#field-name A Trailer header field sender SHOULD enumerate the names of all expected trailer fields. This a priori knowledge of trailer fields might help the recipient with trailer processing (e.g., certain message annotation actions could be delayed in anticipation of the trailer section). However, enumerating all expected trailer fields can be impractical or even impossible in some environments. A Trailer sender MAY send a trailer section with a set of field names that differs from the set of field names listed in the Trailer header field. This specification does not place any restriction on the order of field names in the Trailer header field. Senders SHOULD NOT generate duplicate names for the Trailer header field.

A client compliant with this specification SHOULD send Allow/trailers in each OPTIONS request. A non-authenticating server cannot be expected to mark an Allow/trailers-sending client specially, but this support announcement requirement is meant to minimize interoperability problems associated with servers sending Allow/trailers in OPTIONS responses. Some servers that do not support trailers might not be able to ignore Allow/trailers in OPTIONS requests; therefore, a client SHOULD offer a configuration option or other means of disabling sending Allow/trailers in OPTIONS requests. An ICAP service sending Allow/trailers in OPTIONS response is called a trailers-supporting service. That service designation, maintained by the client, starts upon receiving the OPTIONS service response carrying Allow/trailers and lasts until OPTIONS expiration or a new OPTIONS response from that service. A client compliant with this specification SHOULD send Allow/trailers in each request to a trailers-supporting service. Doing so allows the service to respond with a trailer (and is also necessary for sending a client trailer, as detailed further below). A client receiving both a Trailer header field and Allow/trailers in the response MUST expect a trailer section in that response. In all other cases, a client MUST use the usual trailer-free ICAP response syntax. A client receiving a Trailer header field without Allow/trailers in a response MAY treat the response as syntactically malformed and MUST NOT reuse the connection for any other messages (including pending pipelined requests, if any). A client MAY send a trailer in any request that satisfies all of these conditions: the request is sent to a trailers-supporting service;the request has a body. A client MUST NOT send a trailer in any other request. To send a trailer, the client MUST send Allow/trailers and a Trailer header field in the same request. A client MUST NOT send a combination of those two header fields without sending a trailer.

A server compliant with this specification SHOULD send Allow/trailers in each successful response to an OPTIONS request carrying Allow/trailers. Although ICAP/1.0 allows a list of features in the Allow header, some ICAP clients might not be able to handle an Allow header other than "Allow: 204"; therefore, a server SHOULD NOT send Allow/trailers in a response to a request without Allow/trailers. A server compliant with this specification MAY send Allow/trailers in a response without a trailer to a request with Allow/trailers. The client receiving a no-trailer REQMOD or RESPMOD response with Allow/trailers ought to ignore Allow/trailers. Nevertheless, the server is allowed to respond with Allow/trailers in this context because doing so might simplify server implementation and configuration. A server receiving both a Trailer header field and Allow/trailers in the request MUST expect a trailer in that request. In all other cases, a server MUST use the usual trailer-free ICAP request syntax. A server receiving a Trailer header field without Allow/trailers in a request MAY treat the request as syntactically malformed and MUST NOT reuse the connection for any future requests. A server MAY send a trailer in any response that satisfies all of these conditions: the response is for a request containing Allow/trailers;the response has a body. A server MUST NOT send a trailer in any other response. To send a trailer, the server MUST send Allow/trailers and a Trailer header field in the same response. A server MUST NOT send a combination of those two header fields without sending a trailer.

The following examples illustrate trailer exchanges between ICAP agents compliant with this specification. To show important details, all CRLF sequences after major message parts are shown as "\r\n" lines. CRLF sequences at the end of other lines are implied. All REQMOD and RESPMOD messages in these examples share the context of the following OPTIONS handshake.

shows an OPTIONS request and response when both the client and the server are compliant with this specification. The client also supports the ICAP 206 extension, but the server does not. OPTIONS icap://example.net/sample-service ICAP/1.0 Host: example.net User-Agent: Example-ICAP-Client/1.2 Allow: 204, trailers, 206 \r\n ICAP/1.0 200 OK Date: Sat, 13 Aug 2016 12:17:21 GMT Methods: RESPMOD Service: FOO Tech Server 1.0 ISTag: "W3E4R7U9-L2E4-2" Encapsulated: null-body=0 Max-Connections: 1000 Options-TTL: 7200 Allow: 204 Allow: trailers Preview: 0 Transfer-Complete: asp, bat, exe, com Transfer-Ignore: html Transfer-Preview: * \r\n The OPTIONS response in contains two Allow headers to illustrate one of several possible implementations. A compliant server can also send a single Allow header with a list of values, just like the client does in the above example.

is a RESPMOD request with an ICAP trailer that the client can send after receiving an OPTIONS response with Allow/trailers shown in . Note that Trailer field value does not match the actual fields in the trailer. RESPMOD icap://example.net/sample-service ICAP/1.0 Host: example.net User-Agent: Example-ICAP-Client/1.2 Allow: 204, trailers Trailer: TBD Encapsulated: req-hdr=0, res-hdr=131, res-body=292 \r\n GET /origin-resource HTTP/1.1 Host: www.example.com Accept: text/html, text/plain, image/gif Accept-Encoding: gzip, compress \r\n HTTP/1.1 200 OK Date: Sat, 13 Aug 2016 12:17:22 GMT Server: Testserver/1.0 (Unix) ETag: "63840-1ab7-378d415b" Content-Type: text/html Content-Length: 24 \r\n 18 Origin server sent this. 0 \r\n X-Client-Log-Lineno: 15612570 X-Client-Status: disconnected (at 1470262108) \r\n

is a RESPMOD response with an HTTP and ICAP trailers. The ICAP trailer contains X-Threat-Found and Connection field. To respond with an ICAP trailer, the server ought to receive Allow/trailers in the corresponding ICAP request. ICAP/1.0 200 OK Date: Sat, 13 Aug 2016 12:17:23 GMT Server: ICAP-Server-Software/1.0 Connection: keep-alive ISTag: "W3E4R7U9-L2E4-2" Encapsulated: res-hdr=0, res-body=276 Allow: trailers Trailer: X-ICAP-Log-Lineno, X-Threat-Found \r\n HTTP/1.1 200 OK Date: Sat, 13 Aug 2016 12:17:22 GMT Server: Testserver/1.0 (Unix) ETag: "63840-1ab7-378d415b" Content-Type: text/html Content-Length: 24 Trailer: X-Content-Checksum \r\n 18 Origin server sent this. 0 X-Content-Checksum: sha-short=366bc24373b \r\n X-Threat-Found: Type=0; Resolution=1; Threat=plain-text Connection: close \r\n XXX: Polish and validate offsets, lengths, etc. TODO: Figure out how to keep figure descriptions and figures on the same page without inserting empty space into HTML rendering of the draft.

Proper trailer support reduces old ICAP security concerns because implementations unaware of trailer complexities are arguably more likely to misbehave when receiving HTTP or ICAP trailers. Sending request headers with multiple Allow tokens could crash poor-quality ICAP servers unaware of this specification. Trailer support negotiation rules partially mitigate that risk by restricting unaware implementations exposure; such implementations are exposed only during OPTIONS exchanges. Since OPTIONS transaction has to precede any HTTP message processing, and since virtually all ICAP client-server relationships are stable, most poor-quality implementations would be detected early and reliably. A naive implementation of trailer interpretation logic might update an already "frozen" or "committed" (at header parsing time) state of the ICAP transaction or connection, resulting in crashes and other problems. For example, such an implementation could panic after discovering a Connection trailer field value that contradicts the Connection header field value that has already been received and processed at the beginning of the same transaction.