Swiss Federal Institute of Technology Zurich

Gloriastrasse 35 8092 Zurich Switzerland +41 44 632 70 13 ietf@trammell.ch

Operations IPFIX Working Group This document defines UTF-8 representations for IPFIX abstract data types, to support interoperable usage of the IPFIX Information Elements with protocols based on textual encodings.

The IPFIX Information Model provides a set of abstract data types for the IANA IPFIX Information Element Registry, which contains a rich set of Information Elements for description of information about network entities and network traffic data, and abstract data types for these Information Elements. The IPFIX Protocol Specification, in turn, defines a big-endian binary encoding for these abstract data types suitable for use with the IPFIX Protocol. However, present and future operations and management protocols and applications may use textual encodings, and generic framing and structure, as in JSON or XML. A definition of canonical textual encodings for the IPFIX abstract data types would allow this set of Information Elements to be used for such applications, and for these applications to interoperate with IPFIX applications at the Information Element definition level. Note that templating or other mechanisms for data description for such applications and protocols are application-specific, and therefore out of scope for this document: only Information Element identification and value representation are defined here. In most cases where a textual representation will be used, an explicit tradeoff is made for human readability or manipulability over compactness; this assumption is used in defining standard representations of IPFIX ADTs.

Capitalized terms defined in the IPFIX Protocol Specification and the IPFIX Information Model are used in this document as defined in those documents. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in . In addition, this document defines the following terminology for its own use: A textual representation of Information Element values is applied to use the IPFIX Information Model within some existing textual format (e.g. XML, JSON). This outer format is referred to as the Enclosing Context within this document. Enclosing Contexts define escaping and quoting rules for represented values.

The IPFIX Information Element Registry defines a set of Information Elements numbered by Information Element Identifiers and named for human-readability. These Information Element Identifiers are meant for use with the IPFIX protocol, and have little meaning when applying the IPFIX Information Element Registry to textual representations. Instead, applications using textual representations of Information Elements use Information Element names to identify them; see for examples illustrating this principle.

Each subsection of this section defines a textual encoding for the abstract data types defined in . This section uses ABNF, including the Core Rules in Appendix B of , to describe the format of textual representations of IPFIX abstract data types. If future documents update to add new Abstract Data Types to the IPFIX Information Model, and those Abstract Data Types are generally useful, this document will also need to be updated in order to define textual encodings for those Abstract Data Types.

If the Enclosing Context defines a representation for binary objects, that representation SHOULD be used. Otherwise, since the goal of textual representation of Information Elements is human-readability over compactness, the values of Information Elements of the octetArray data type are represented as a string of pairs of hexadecimal digits, one pair per byte, in the order the bytes would appear on the wire were the octetArray encoded directly in IPFIX per . Whitespace may occur between any pair of digits to assist in human readability of the string, but is not necessary. In ABNF: hex-octet = 2HEXDIG octetarray = hex-octet *([WSP] hex-octet)

If the Enclosing Context defines a representation for unsigned integers, that representation SHOULD be used. In the special case that the unsigned Information Element has identifier semantics, and refers to a set of codepoints, either in an external registry, a sub-registry, or directly in the description of the Information Element, then the name or short description for that codepoint as a string MAY be used to improve readability. Otherwise, the values of Information Elements of an unsigned integer type may be represented either as unprefixed base-10 (decimal) strings, as base-16 (hexadecimal) strings prefixed by "0x", or as base-2 (binary) strings prefixed by "0b". In ABNF: unsigned = 1*DIGIT / "0x" 1*HEXDIG / "0b" 1*BIT Leading zeroes are allowed in any representation, and do not signify base-8 (octal) representation. Binary representation is intended for use with Information Elements with flag semantics, but can be used in any case. The encoded value MUST be in range for the corresponding abstract data type or Information Element. Out of range values are interpreted as clipped to the implicit range for the Information Element as defined by the abstract data type, or to the explicit range of the Information Element if defined. Minimum and maximum values for abstract data types are shown in below. type minimum maximum unsigned8 0 255 unsigned16 0 65536 unsigned32 0 4294967295 unsigned64 0 18446744073709551615

If the Enclosing Context defines a representation for signed integers, that representation SHOULD be used. Otherwise, the values of Information Elements of signed integer types are represented as optionally-prefixed base-10 (decimal) strings. In ABNF: sign = "+" / "-" signed = [sign] 1*DIGIT If the sign is omitted, it is assumed to be positive. Leading zeroes are allowed, and do not signify base-8 (octal) encoding. The representation "-0" is explicitly allowed, and is equal to zero. The encoded value MUST be in range for the corresponding abstract data type or Information Element. Out of range values are to be interpreted as clipped to the implicit range for the Information Element as defined by the abstract data type, or to the explicit range of the Information Element if defined. Minimum and maximum values for abstract data types are shown in below. type minimum maximum signed8 -128 +127 signed16 -32768 +32767 signed32 -2147483648 +2147483647 signed64 -9223372036854775808 +9223372036854775807

If the Enclosing Context defines a representation for floating point numbers, that representation SHOULD be used. Otherwise, the values of Information Elements of float32 or float64 types are represented as optionally sign-prefixed, optionally base-10 exponent-suffixed, floating point decimal numbers, as in . The special strings "NaN", "+inf", and "-inf" represent not a number, positive infinity and negative infinity, respectively. In ABNF: sign = "+" / "-" exponent = "e" [sign] 1*3DIGIT right-decimal = "." 1*DIGIT mantissa = 1*DIGIT [right-decimal] num = [sign] mantissa [exponent] naninf = "NaN" / (sign "inf") float = num / naninf The expressed value is ( mantissa * 10 ^ exponent ). If the sign is omitted, it is assumed to be positive. If the exponent is omitted, it is assumed to be zero. Leading zeroes may appear in the mantissa and/or the exponent. Values MUST be within range for single or double precision numbers; finite values outside the approprate range are to be interpreted as clamped to be within the range. Note that no more than three digits are required or allowed for exponents in this encoding due to these ranges. Note that, since this representation is meant for human readability, writers MAY sacrifice precision to use a more human-readable representation of a given value, at the expense of the ability to recover the exact bit pattern at the reader. Therefore, decoders MUST NOT assume that the represented values are exactly compararable for equality.

If the Enclosing Context defines a representation for boolean values, that representation SHOULD be represented. Otherwise, a true boolean value is represented by the literal string "true", and a false boolean value with the literal string "false". In ABNF: boolean-true = "true" boolean-false = "false" boolean = boolean-true / boolean-false

MAC addresses are represented as IEEE 802 MAC-48 addresses, hexadecimal bytes, most significant byte first, separated by colons. In ABNF: hex-octet = 2HEXDIG macaddress = hex-octet 5( ":" hex-octet )

As Information Elements of the string type are simply Unicode strings (encoded as UTF-8 when appearing in Data Sets in IPFIX Messages ), they are represented directly, using the Unicode encoding rules and quoting and escaping rules of the Enclosing Context. If the Enclosing Context cannot natively represent Unicode characters, the escaping facility provided by the Enclosing Context MUST be used for non- representable characters. Additionally, strings containing characters reserved in the Enclosing Context (e.g. control characters, markup characters, quotes) MUST be escaped or quoted according to the rules of the Enclosing Context. It is presumed that the Enclosing Context has sufficient restrictions on the use of Unicode to prevent the unsafe use of non-printing and control characters. As there is no accepted solution for the processing and safe display of mixed-direction strings, mixed-direction strings should be avoided using this encoding. Note also that, since this draft presents no additional requirements for the normalization of Unicode strings, care must be taken when comparing strings using this encoding; direct byte-pattern comparisons are not sufficient for determining whether two strings are equivalent. See and for more on possible unexpected results and related risks in comparing Unicode strings.

Timestamp abstract data types are represented generally as in , with two important differences. First, all IPFIX timestamps are expressed in terms of UTC, so textual representations of these Information Elements are explicitly in UTC as well. Time zone offsets are therefore not required or supported. Second, there are four timestamp abstract data types, separated by the precision which they can express. Fractional seconds are omitted in dateTimeSeconds, expressed in milliseconds in dateTimeMilliseconds, and so on. In ABNF, taken from and modified:

date-fullyear = 4DIGIT date-month = 2DIGIT ; 01-12 date-mday = 2DIGIT ; 01-28, 01-29, 01-30, 01-31 time-hour = 2DIGIT ; 00-23 time-minute = 2DIGIT ; 00-59 time-second = 2DIGIT ; 00-58, 00-59, 00-60 time-msec = "." 3DIGIT time-usec = "." 6DIGIT time-nsec = "." 9DIGIT full-date = date-fullyear "-" date-month "-" date-mday integer-time = time-hour ":" time-minute ":" time-second datetimeseconds = full-date "T" integer-time datetimemilliseconds = full-date "T" integer-time "." time-msec datetimemicroseconds = full-date "T" integer-time "." time-usec datetimenanoseconds = full-date "T" integer-time "." time-nsec

IP version 4 addresses are represented in dotted-quad format, most-significant-byte first, as it would in a Uniform Resource Identifier ; the ABNF for an IPv4 address is taken from and reproduced below:

dec-octet = DIGIT ; 0-9 / %x31-39 DIGIT ; 10-99 / "1" 2DIGIT ; 100-199 / "2" %x30-34 DIGIT ; 200-249 / "25" %x30-35 ; 250-255 ipv4address = dec-octet 3( "." dec-octet )

IP version 6 addresses are represented as in section 2.2 of , as updated by section 4 of . The ABNF for an IPv6 address is taken from and reproduced below, using the ipv4address production from the previous section:

ls32 = ( h16 ":" h16 ) / ipv4address ; least-significant 32 bits of address h16 = 1*4HEXDIG ; 16 bits of address represented in hexadecimal ; zeroes to suppressed as in RFC 5952 ipv6address = 6( h16 ":" ) ls32 / "::" 5( h16 ":" ) ls32 / [ h16 ] "::" 4( h16 ":" ) ls32 / [ h16 ":" h16 ] "::" 3( h16 ":" ) ls32 / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32 / [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32 / [ *4( h16 ":" ) h16 ] "::" ls32 / [ *5( h16 ":" ) h16 ] "::" h16 / [ *6( h16 ":" ) h16 ] "::"

These abstract data types, defined for IPFIX Structured Data, do not represent actual data types; they are instead designed to provide a mechanism by which complex structure can be represented in IPFIX below the template level. It is assumed that protocols using textual Information Element representation will provide their own structure. Therefore, Information Elements of these Data Types MUST NOT be used in textual representations.

The security considerations for the IPFIX Protocol apply. Implementations of decoders of Information Element values using these representations must take care to correctly handle invalid input, but the encodings presented here are not special in that respect. The encoding specified in this document, and representations that may be built upon it, are specifically not intended for the storage of data. However, since storage of data in the format in which it is exchanged is a very common practice, and the ubiquity of tools for indexing and searching text significantly increases the ease of searching and the risk of privacy-sensitive data being accidentally indexed or searched, the privacy considerations in Section 11.8 of are especially important to observe when storing data using the encoding specified in this document that was derived from the measurement of network traffic. When using representations based on this encoding to transmit or store network traffic data, consider omitting especially privacy-sensitive values by not representing the columns or keys containing those values, as in black-marker anonymization as discussed in Section 4 of . Other anonymization techinques described in may also be useful in these situations. The encodings for all Abstract Data Types other than 'string' are defined in such a way as to be representable in the US-ASCII character set, and therefore should be unproblematic for all Enclosing Contexts. However, the 'string' Abstract Data Type may be vulnerable to problems with ill-formed UTF-8 strings as discussed in section 6.1.6 of ; see for background.

This document has no considerations for IANA.

Thanks to Paul Aitken, Benoit Claise, Andrew Feren, Juergen Quittek, David Black, and the IESG for the review and comments. Thanks to Dave Thaler and Stephan Neuhaus for discussions which improved the floating-point representation section. This work is materially supported by the European Union Seventh Framework Programme under grant agreement 318627 mPlane.

In this section, we examine an IPFIX Template and a Data Record defined by that Template, and show how that Data Record would be represented in JSON according to the specification in this document. Note that this is specifically NOT a recommendation for a particular representation, merely an illustration of the encodings in this document; the quoting and formatting in the example are JSON-specific. shows a Template in IESpec format as defined in section 10.1 of ; a corresponding JSON Object representing a record defined by this template in the text format specified in this document is shown in .

[8] flowEndMilliseconds(153)[8] octetDeltaCount(1)[4] packetDeltaCount(2)[4] sourceIPv6Address(27)[16]{key} destinationIPv6Address(28)[16]{key} sourceTransportPort(7)[2]{key} destinationTransportPort(11)[2]{key} protocolIdentifier(4)[1]{key} tcpControlBits(6)[1] flowEndReason(136)[1] ]]>