DetNet Data Plane Protocol and Solution Alternatives

This section provides criteria to help to evaluate potential options. Each deterministic networking data plane solution alternative is described and evaluated using the criteria described in this section. The used criteria enumerated in this section are selected so that they highlight the existence or lack of features that are expected or seen important to a solution alternative for the data plane solution. The criteria for the DetNet Service layer: Encapsulation and overhead Flow identification (Service ID part of the DetNet flows) Packet sequencing and duplicate elimination Flow duplication and merging Operations, Administration and Maintenance (capabilities) Class and quality of service capabilities (DetNet Service specific) Technical maturity The criteria for the DetNet Transport layer: Encapsulation and overhead Flow identification Explicit routes (network path) Flow duplication and merging (sometimes, flow duplication and merging is also doable at the transport layer, not just at the service layer) Operations, Administration and Maintenance (capabilities, performance management, packet traceability) Class and quality of service capabilities (DetNet Transport specific) Packet traceability (can be part of OAM) Technical maturity [Editor's Note: numbering is off because #7 is removed.] [Editor's Note: #9 should(?) be integrated into #6.] Most of the criteria is relevant for both the DetNet Service and DetNet Transport layers. However, different aspects of the same criteria may relevant for different layers, for example, as it is the case with criteria #5 Packet replication and elimination.

Encapsulation and overhead is related to how the DetNet data plane carries DetNet flow. In several cases a DetNet flow has to be encapsulated inside other protocols, for example, when transporting a layer-2 Ethernet frame over an IP transport network. In some cases a tunneling like encapsulation can be avoided by underlying transport protocol translation, for example, translating layer-2 Ethernet frame including addressing and flow identification into native IP traffic. Last it is possible that sources and destinations handle deterministic flows natively in layer-3. This criteria concerns what is the encapsulation method the solution alternative support: tunneling like encapsulation, protocol translation or native layer-3 transport. In addition to the encapsulation mechanism this criteria is also concerned of the processing and specifically the encapsulate header overhead.

The solution alternative has to provide means to identify specific deterministic flows. The flow identification can, for example, be explicit field in the data plane encapsulation header or implicitly encoded into the addressing scheme of the used data plane protocol or their combination. This criteria concerns the availability and details of deterministic flow identification the data plane protocol alternative has.

The solution alternative has to provide means for end systems to number packets sequentially and transport that sequencing information along with the sent packets. In addition to possible reordering packets other important uses for sequencing are detecting duplicates and lost packets. In a case of intentional packet duplication a combination of flow identification and packet sequencing allows for detecting and eliminating duplicates at the destination (see for more details).

The solution alternative has to provide a mechanism(s) for establishing explicit routes that all packets belonging to a deterministic flow will follow. The explicit route can be seen as a form of source routing or a pre-reserved path e.g., using some network management procedure. It should be noted that the explicit route does not need to be detailed to a level where every possible intermediate node along the path is part of the named explicit route. RSVP-TE supports explicit routes, and typically provides pinned data paths for established LSPs. At Layer-2, the IEEE 802.1Qca specification defines how to do explicit path control in a bridged network and its IETF counter part is defined in . This criteria concerns the available mechanisms for explicit routes for the data plane protocol alternative.

Flow duplication and flow merging are methods being considered to provide DetNet service protection. The objective for supporting flow duplication and flow merging is to enable hitless (or lossless) 1+1 protection. Other methods, if so identified, are also permissible. The solution alternative has to provide means for end systems, relay and edge nodes to be able to duplicate packets into duplicate flows, and later merge the flows into one for duplicate elimination. The duplication and merging may take place at multiple points in the network in order to ensure that one (or more) equipment failure event(s) still leave at least one path intact for a deterministic networking flow. The goal is again to enable hitless 1+1 protection in a way that no packet gets lost or there is no ramp up time when either one of the paths fails for one reason or another. Another concern regarding packet duplication is how to enforce duplicated packets to take different route or path while the final destination still remains the same. With strict source routing, all the intermediate hops are listed and paths can be guaranteed to be non-overlapping. Loose source routing only signals some of the intermediate hops and it takes additional knowledge to ensure that there is no single point of failure. The IEEE 802.1CB (seamless redundancy) is an example of Ethernet-based solution that defines packet sequence numbering, flow duplication, flow merging, duplicate packet identification and elimination. The deterministic networking data plane solution alternative at layer-3 has to provide equivalent functionality. This criteria concerns the available mechanisms for packet replication and duplicate deletion the data plane protocol alternative has.

The solution alternative should demonstrate an availability of appropriate standardized OAM tools that can be extended for deterministic networking purposes with a reasonable effort, when required. The OAM tools do not necessarily need to be specific to the data plane protocol as it could be the case, for example, with MPLS-based data planes. But any OAM-related implications or requirements on data plane hardware must be considered. The OAM includes but is not limited to tools listed in the requirements for overlay networks . Specifically, the performance management requirements are of interest at both service and transport layers.

Class and quality of service, i.e., CoS and QoS, are terms that are often used interchangeably and confused. In the context of DetNet, CoS is used to refer to mechanisms that provide traffic forwarding treatment based on aggregate group basis and QoS is used to refer to mechanisms that provide traffic forwarding treatment based on a specific DetNet flow basis. Examples of CoS mechanisms include DiffServ which is enabled by IP header differentiated services code point (DSCP) field and MPLS label traffic class field , and at Layer-2, by IEEE 802.1p priority code point (PCP). Quality of Service (QoS) mechanisms for flow specific traffic treatment typically includes a guarantee/agreement for the service, and allocation of resources to support the service. Example QoS mechanisms include discrete resource allocation, admission control, flow identification and isolation, and sometimes path control, traffic protection, shaping, policing and remarking. Example protocols that support QoS control include Resource ReSerVation Protocol (RSVP) (RSVP) and RSVP-TE and . A critical DetNet service enabled by QoS (and perhaps CoS) is delivering zero congestion loss. There are different mechanisms that maybe used separately or in combination to deliver a zero congestion loss service. The key aspect of this objective is that DetNet packets are not discarded due to congestion at any point in a DetNet aware network. In the context of the data plane solution there should be means for flow identification, which then can be used to map a flow against specific resources and treatment in a node enforcing the QoS. Hereto, certain aspects of CoS and QoS may be provided by the underlying sub-net technology, e.g., actual queuing or IEEE 802.3x priority flow control (PFC).

For the network management and specifically for tracing implementation or network configuration errors any means to find out whether a packet is a replica, which node performed replication, and which path was intended for the replica, can be very useful. This criteria concerns the availability of solutions for tracing packets in the context of data plane protocol alternative. Packet traceability can also be part of OAM.

The technical maturity of the data plane solution alternative is crucial, since it basically defines the effort, time line and risks involved for the use of the solution in deployments. For example, the maturity level can be categorized as available immediately, available with small extensions, available with re-purposing/redefining portions of the protocol or its header fields. Yet another important measure for maturity is the deployment experience. This criteria concerns the maturity of the data plane protocol alternative as the solution alternative. This criteria is particularly important given, as previously noted, that the DetNet data plane solution is expected to impact, i.e., be supported in, hardware.

The following sections describe and rate deterministic data plane solution alternatives. In "Analysis and Discussion" section each alternative is evaluated against the criteria given in and rated using the following: (M)eets the criteria, (W)ork needed, and (N)ot suitable or too much work envisioned.

This section investigates the application of native IPv6 as the data plane for deterministic networking along the criteria collected in . The application of higher OSI layer headers, i.e., headers deeper in the packet, can be considered. Two aspects have to be taken into account for such solutions. (i) Those header fields can be encrypted. (ii) Those header fields are deeper in the packet, therefore, routers have to apply deep packet inspection. See further details in .

IPv6 can encapsulate DetNet Service layer headers (and associated DetNet flow payload) like any other upper-layer header indicated by the Next Header. The fixed header of an IPv6 packet is 40 bytes . This overhead is bigger if any Extension Header is used, and a generic behaviour for host and forwarding nodes is specified in . However, the exact overhead () depends on what solution is actually used to provide DetNet features, e.g., explicit routing or DetNet service protection if any of these is applied. IPv6 has two types of Extension Headers that are processed by intermediate routers between the source and the final destination and may be of interest for the data plane signaling, the Routing Header that is used to direct the traffic via intermediate routers in a strict or loose source routing way, and the Hop-by-Hop Options Header that carries optional information that must be examined by every node along a packet's delivery path. The Hop-by-Hop Options Header, when present, must immediately follow the IPv6 Header and it is not possible to limit its processing to the end points of Source Routed segments. IPv6 also provides a Destination Options Header that is used to carry optional information to be examined only by a packet's destination node(s). The encoding of the options used in the Hop-by-Hop and in the Destination Options Header indicates the expected behavior when a processing IPv6 node does not recognize the Option Type, e.g. skip or drop; it should be noted that due to performance restrictions nodes may ignore the Hop-by-Hop Option Header, drop packets containing a Hop-by-Hop Option Header, or assign packets containing a Hop-by-Hop Option Header to a slow processing path (e.g. punt packets from hardware to software forwarding which is highly detrimental to the performance). The creation of new Extension Headers that would need to be processed by intermediate nodes is strongly discouraged. In particular, new Extension Header(s) having hop-by-hop behavior must not be created or specified. New options for the existing Hop-by-Hop Header should not be created or specified unless no alternative solution is feasible . The 20-bit flow label field of the fixed IPv6 header is suitable to distinguish different deterministic flows. But guidance on the use of the flow label provided by places restrictions on how the flow label can be used. In particular, labels should be chosen from an approximation to a discrete uniform distribution. Additionally, existing implementations generally do not open APIs to control the flow label from the upper layers. Alternatively, the Flow identification could be transported in a new option in the Hop-by-Hop Options Header. One possibility is for a Software-Defined Networking (SDN) based approach to be applied to compute, establish and manage the explicit routes, leveraging Traffic Engineering (TE) extensions to routing protocols and evolving to the Path Computation Element (PCE) Architecture , though a number of issues remain to be solved . Segment Routing (SR) is a new initiative to equip IPv6 with explicit routing capabilities. The idea for the DetNet data plane would be to apply SR to IPv6 with the addition of a new type of routing extension header to explicitly signal the path in the data plane between the source and the destination, and/or between replication points and elimination points if this functionality is used. The functionality of replicating a packet exists in IPv6 but is limited to multicast flows. In order to enforce replicated packets to take different routes and eventually again merge flow (bring them to a specific merging point), IP-in-IP encapsulation and Segment Routing could be leveraged to signal a segment in a packet. A replication point would insert a different routing header in each copy it makes, the routing header providing explicitly the hops to the merging point for that particular replica of the packet, in a strict or in a loose source routing fashion. A flow merging point would pop the routing headers from the various copies it gets and do the rest of the required processing for merging the two flows into one flow. IPv6 enjoys the existing toolbox for generic IP network management. However, IPv6 specific management features are still not at the level comparable to that of IPv4. Particular areas of concerns are those that are IPv6 specific, for example, related to neighbor discovery protocol (ND), stateless address autoconfiguration (SLAAC), subscriber identification, and security. While the standards are already mostly in place the implementations in deployed equipment can be lacking or inadequate for commercial deployments. This is larger issue with older existing equipment. IPv6 provides support for CoS and QoS. CoS is provided by DiffServ which is enabled by IP header differentiated services code point (DSCP) and QoS is defined as part of RSVP. DiffServ support is widely available, while RSVP for IP packets is generally not supported. The traceability of replicated packets involves the capability to resolve which replication point issued a particular copy of a packet, which segment was intended for that replica, and which particular packet of which particular flow this is. Sequence also depends on the sequencing mechanism. As an example, the replication point may be indicated as the source of the packet if IP-in-IP encapsulation is used to forward along segments. Another alternate to IP-in-IP tunneling along segments would be to protect the original source address in a destination option similar to the Home Address option and then use the address of the replication point as source in the IP header. The traceability also involves the capability to determine if a particular segment is operational. While IPv6 as such has no support for reversing a path, it appears source route extensions such as the one defined for segment routing could be used for tracing purposes. Though it is not a usual practice, IPv6 expects that a Source Route path may be reversed, and the standard insists that a node must not include the reverse of a Routing Header in the response unless the received Routing Header was authenticated. IPv6 has been around about 20 years. However, large scale global and commercial IPv6 deployments are rather new dating only few years back to around 2012. While IPv6 has proven itself for best effort traffic, DiffServ usage is less common and QoS capabilities are not currently present. Additional, there are number of small issues to work on as they show up once operations experience grows. The Cisco 6Lab site provides information on IPv6 deployment per country, indicating figures for prefixes, transit AS, content and users. Per this site, many countries, including Canada, Brazil, the USA, Germany, France, Japan, Portugal, Sweden, Finland, Norway, Greece, and Ecuador, achieve a deployment ratio above 30 percent, and the overall adoption reported by Google Statistics is now above 10 percent.

IPv6 supports a significant portion of the identified DetNet data plane criteria today. There are aspects of the DetNet data plane that are not fully supported, notably QoS, but these can be incrementally added or supplemented by the underlying sub-network layer. IPv6 may be a choice as the DetNet Transport layer in networks where other technologies such as MPLS are not deployed.

IPv4 is in principle the same as IPv6, except that it has a smaller address space. However, IPv6 was designed around the fact that extension headers are an integral part of the protocol and operation from the beginning, although the practice may some times prove differently . IPv4 does support header options, but these have historically not been supported on in hardware-based forwarding so are generally blocked or handled at a much slower rate. In either case, the use of IP header options is generally avoided. In the context of deterministic networking data plane solutions the major difference between IPv4 and IPv6 seems to be the practical support for header extensibility. Anything below and above the IP header independent of the version is practically the same.

The fixed header of an IPv4 packet is 20 bytes . IP options add overhead, but are not generally used and are not considered as part of this document. The IPv4 header has a 16-bit identification field that was originally intended for assisting fragmentation and reassembly of IPv4 packets as described in . The identification field has also been proposed to be used for actually identifying flows between two IP addresses and a given protocol for detecting and removing duplicate packets . However, recent update to both and restricts the use of IPv4 identification field only to fragmentation purposes. The IPv4 also has a stream identifier option , which contains a 16-bit SATNET stream identifier. However, the option has been deprecated . The conclusion is that stream identification does not work nicely with IPv4 header alone and a traditional 5-tuple identification might not also be enough in a case of a flow duplication or encrypted flows. For a working solution, upper layer protocol headers such as RTP or PWs may be required for unambiguous flow identification. There is also emerging work within the IETF that may provide new flow identification alternatives. IPv4 has two source routing option specified: the loose source and record route option (LSRR), and the strict source and record route option (SSRR) . The support of these options in the Internet is questionable but within a closed network the support may be assumed. But as both these options use IP header options, which are generally not supported in hardware, use of these options are questionable. Of course, the same options of SDN and SR approaches discussed above for IPv6 may be equally applicable to IPv4. The functionality of replicating a packet exists in IPv4 but is limited to multicast flows. In general the issue regarding the IPv6 packet replication also applies to IPv4. Duplicate packet detection for IPv4 is studied in to a great detail in the context of simplified multicast forwarding. In general there is no good way to detect duplicated packets for IPv4 without additional upper layer protocol support. IPv4 enjoys the extensive and "complete" existing toolbox for generic IP network management. IPv4 provides support for CoS and QoS. CoS is provided by DiffServ which is enabled by IP header differentiated services code point (DSCP) and QoS is defined as part of RSVP. DiffServ support is widely available, while RSVP for IP packets is generally not supported. The IPv4 has similar needs and requirements for traceability as IPv6 (see ). The IPv4 has a traceroute option that could be used to record the route the packet took. However, the option has been deprecated . IPv4 can be considered mature technology with over 30 years of implementation, deployment and operations experience. As with IPv6, today's commercial implementations and deployments of IPv4 generally lack any support for QoS.

The IPv4 has specifications to support most of the identified DetNet data plane criteria today. However, several of those have already been deprecated or their wide support is not guaranteed. The DetNet data plane criteria that are not fully supported could be incrementally added or supplemented by the underlying sub-network layer. Unfortunately, the IPv4 has had limited success getting its extensions deployed at large. However, introducing new extensions might have a better success in closed networks (like DetNet) than in Internet. Due to the popularity of the IPv4, it should be considered as a potential choice for the DetNet Transport layer.

Multiprotocol Label Switching Architecture (MPLS) and its variants, MPLS with Traffic Engineering (MPLS-TE) and , and MPLS Transport Profile (MPLS-TP) is a widely deployed technology that switches traffic based on MPLS label stacks and . MPLS is the foundation for Pseudowire-based services and emerging technologies such as Bit-Indexed Explicit Replication (BIER) and Source Packet Routing. MPLS supports the equivalent of both the DetNet Service and DetNet Transport layers, and provides a very rich set of mechanisms that can be reused directly, and perhaps augmented in certain cases, to deliver DetNet services. At the DetNet Transport layer, MPLS provides forwarding, protection and OAM services. At the DetNet Service Layer it provides client service adaption, directly, via Pseudowires and via other label-like mechanisms such as EPVN . A representation of these options are shown in .

DetNet Transport layer boundary ]]> MPLS can be controlled in a number of ways including via a control plane, via the management plane, or via centralized controller (SDN) based approaches. MPLS also provides standard control plane reference points. Additional information on MPLS architecture and control can be found in . A summary of MPLS control plane related functions can be found in . The remainder of this section will focus . The remainder of this section will focus on the MPLS transport data plane, additional information on the MPLS service data plane can be found below in .

The following draws heavily from . Encapsulation and forwarding of packets traversing MPLS LSPs follows standard MPLS packet encapsulation and forwarding as defined in , , , and . Data plane Quality of Service capabilities are included in the MPLS in the form of Traffic Engineered (TE) LSPs and the MPLS Differentiated Services (DiffServ) architecture . Both E-LSP and L-LSP MPLS DiffServ modes are defined. The Traffic Class field (formerly the EXP field) of an MPLS label follows the definition of and . Except for transient packet reordering that may occur, for example, during fault conditions, packets are delivered in order on L-LSPs, and on E-LSPs within a specific ordered aggregate. The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL processing models are described in and and may be used for MPLS LSPs. Equal-Cost Multi-Path (ECMP) load-balancing is possible with MPLS LSPs and can be avoided using a number of techniques. The same holds for Penultimate Hop Popping (PHP). MPLS includes the following LSP types: Point-to-point unidirectional Point-to-point associated bidirectional Point-to-point co-routed bidirectional Point-to-multipoint unidirectional Point-to-point unidirectional LSPs are supported by the basic MPLS architecture . A point-to-point associated bidirectional LSP between LSRs A and B consists of two unidirectional point-to-point LSPs, one from A to B and the other from B to A, which are regarded as a pair providing a single logical bidirectional transport path. A point-to-point co-routed bidirectional LSP is a point-to-point associated bidirectional LSP with the additional constraint that its two unidirectional component LSPs in each direction follow the same path (in terms of both nodes and links). An important property of co-routed bidirectional LSPs is that their unidirectional component LSPs share fate. A point-to-multipoint unidirectional LSP functions in the same manner in the data plane, with respect to basic label processing and packet-switching operations, as a point-to-point unidirectional LSP, with one difference: an LSR may have more than one (egress interface, outgoing label) pair associated with the LSP, and any packet it transmits on the LSP is transmitted out all associated egress interfaces. Point-to-multipoint LSPs are described in and . TTL processing and exception handling for point-to- multipoint LSPs is the same as for point-to-point LSPs. Additional data plane capabilities include Linear Protection, and . And the in progress work on MPLS support for time synchronization .

There are two perspectives to consider when looking at encapsulation. The first is encapsulation to support services. These considerations are part of the DetNet service layer and are covered below, see Sections and . The second perspective relates to encapsulation, if any, is needed to transport packets across network. In this case, the MPLS label stack, is used to identify flows across a network. MPLS labels are compact and highly flexible. They can be stacked to support client adaptation, protection, network layering, source routing, etc. The number of DetNet Transport layer specific labels is flexible and support a wide range of applicable functions and MPLS domain characteristics (e.g., TE-tunnels, Hierarchical-LSPs, etc.). MPLS label stacks provide highly flexible ways to identify flows. Basically, they enable the complete separation of traffic classification from traffic treatment and thereby enable arbitrary combinations of both. For the DetNet flow identification the MPLS label stack can be used to support n-layers of DetNet flow identification. For example, using dedicated LSP per DetNet flow would simplify flow identification for intermediate transport nodes, and additional hierarchical LSPs could be used to facilitate scaling. MPLS supports explicit routes based on how LSPs are established, e.g., via TE explicit routes . Additional, but not required, capabilities are being defined as part of Segment Routing (SR) . MPLS as DetNet Transport layer supports the replication via point-to-multipoint LSPs. At the MPLS LSP level, there are mechanisms defined to provide 1+1 protection, which could help realizing the flow merging function. The current definitions and use OAM mechanisms to support and coordinate protection switching and packet loss is possible during a switch. While such this level of protection may be sufficient for many DetNet applications, when truly hitless (i.e., zero loss) switching is required, additional mechanisms will be needed. It is expected that these additional mechanisms will be defined at a DetNet layer. MPLS already includes a rich set of OAM functions at both the Service and Transport Layers. This includes LSP ping [ref] and those enabled via the MPLS Generic Associated Channel and registered by IANA. As previously mentioned, Data plane Quality of Service capabilities are included in the MPLS in the form of Traffic Engineered (TE) LSPs and the MPLS Differentiated Services (DiffServ) architecture . Both E-LSP and L-LSP MPLS DiffServ modes are defined. The Traffic Class field (formerly the EXP field) of an MPLS label follows the definition of and . One potential open area of work is synchronized, time based scheduling. Another is shaping, which is generally not supported in shipping MPLS hardware. MPLS supports multiple tracing mechanisms. A control based one is defined in . An OAM based mechanism is defined in MPLS On-Demand Connectivity Verification and Route Tracing . MPLS as a mature technology that has been widely deployed in many networks for many years. Numerous vendor products and multiple generations of MPLS hardware have been built and deployed.

MPLS is a mature technology that has been widely deployed. Numerous vendor products and multiple generations of MPLS hardware have been built and deployed. MPLS LSPs support a significant portion of the identified DetNet data plane criteria today. Aspects of the DetNet data plane that are not fully supported can be incrementally added. It's worth noting that a number of limitations are in shipping hardware, versus at the protocol specification level, e.g., shaping.

Bit Indexed Explicit Replication (BIER) is a network plane replication technique that was initially intended as a new method for multicast distribution. In a nutshell, a BIER header includes a bitmap that explicitly signals the destinations that are intended for a particular packet, which means that 1) the source is aware of the individual destinations and 2) the BIER control plane is a simple extension of the unicast routing as opposed to a dedicated multicast data plane, which represents a considerable reduction in OPEX. For this reason, the technology faces a lot of traction from Service Providers. discusses the applicability of BIER for replication in the DetNet. The simplicity of the BIER technology makes it very versatile as a network plane signaling protocol. Already, a new Traffic Engineering variation is emerging that uses bits to signal segments along a TE path. While the more classical BIER is mainly a multicast technology that typically leverages a unicast distributed control plane through IGP extensions, BIER-TE is mainly a unicast technology that leverages a central computation to setup path, compute segments and install the mapping in the intermediate nodes. discusses the applicability of BIER-TE for replication, traceability and OAM operations in DetNet. Bit-Indexed Explicit Replication (BIER) layer may be considered to be included into Deterministic Networking data plane solution. Encapsulation of a BIER packet in MPLS network presented in

The DetNet may be presented in BIER as distinctive payload type with its own Proto(col) ID. Then it is likely that DetNet will have the header that would identify: Version; Sequence Number; Timestamp; Payload type, e.g. data vs. OAM. DetNet node, collocated with BFIR, may use multiple BIER sub-domains to create replicated flows. Downstream DetNet nodes, collocated with BFER, would terminate redundant flows based on Sequence Number and/or Timestamp information. Such DetNet may be BFER in one BIER sub-domain and BFIR in another. Thus DetNet flow would traverse several BIER sub-domains.

Consider DetNet flow that must traverse BIER enabled domain from A to G and H. DetNet may use three BIER subdomains: A-B-D-E-G (dash-dot): A is BFIR, E and G are BFERs, A-C-E-F-H (dash-double-dot): A is BFIR, E and H are BFERs, E-G-H (dotted): E is BFIR, G and H are BFERs. DetNet node A sends DetNet into red and purple BIER sub-domains. DetNet node E receives DetNet packet and sends into green sub-domain while terminating duplicates and those that deemed too-late. DetNet nodes G and H receive DetNet flows, terminate duplicates and those that are too-late.

BIER over MPLS network encapsulation (will refer as "BIER over MPLS" further for short), , is being defined [I-D. ietf-bier-mpls-encapsulation] within the BIER working group. Flow identification and separation can be achieved through use of BIER domains and/or Entropy value in the BIER over MPLS, . Explicit routes may be used as underlay for BIER domain. BIER underlay may be calculated using PCE and instantiated using any southbound mechanism. Packet replication, as indicated by its name, is core function of the Bit-Indexed Explicit Replication. Elimination of the duplicates and/or too-late packets cannot be done within BIER sub-domain but may be done at DetNet overlay at the edge of the BIER sub-domain. [Editor's note: how about the flow merging?] BIER over MPLS guarantees that OAM is fate-sharing, i.e. in-band with a data flow being monitored or measured. Additionally, BIER over MPLS enables passive performance measurement, e.g. with the marking method . Some OAM protocols, e.g. can be applied and used in BIER over MPLS as demonstrated , while new protocols being worked on, e.g. ping/traceroute or Path MTU Discovery . Class of Service can be inherited from the underlay of the particular BIER sub-domain. Quality of Service, i.e. scheduling and bandwidth reservations can be used among other constrains in calculating explicit path for the BIER sub-domain's underlay. Ability to do passive performance measurement by using OAM field of the BIER over MPLS, , is unmatched and significantly simplifies truly passive tracing of selected flows and packets within them. The BIER over MPLS is nearing finalization within the BIER WG and several experimental implementations are expected soon.

BIER over MPLS supports a significant portion of the identified DetNet data plane requirements, including controlled packet replication, traffic engineering, while some requirements, e.g. duplicate and too-late packet elimination may be realized as function of the DetNet overlay. BIER over MPLS is a viable candidate as the DetNet Transport layer in MPLS networks.

An alternate use of Bit-Indexed Explicit Replication (BIER) uses bits in the BitString to represent adjacencies as opposed to destinations, as discussed in BIER Traffic Engineering (TE). The proposed function of BIER-TE in the DetNet data plane is to control the process of replication and elimination, as opposed to the identification of the flows or and the sequencing of packets within a flow. At the path ingress, BIER-TE identifies the adjacencies that are activated for this packet (under the rule of the controller). At the egress, BIER-TE is used to identify the adjacencies where transmission failed. This information is passed to the controller, which in turn can modify the active adjacencies for the next packets. The value is that the replication can be controlled and monitored in a loop that may involve an external controller, with the granularity of a packet and an adjacency .

BIER-TE enables to activate the replication and elimination functions in a manner that is abstract to the data plane forwarding information. An adjacency, which is represented by a bit in the BIER header, can correspond in the data plane to an Ethernet hop, a Label Switched Path, or it can correspond to an IPv6 loose or strict source routed path. In a nutshell, BIER-TE is used as follows: A controller computes a complex path, sometimes called a track, which takes the general form of a ladder. The steps and the side rails between them are the adjacencies that can be activated on demand on a per-packet basis using bits in the BIER header.

(A) ====> (C) ==== // ^ | ^ | \\ ingress (I) | | | | (E) egress \\ | v | v // ===> (B) ====> (D) ==== ]]> The controller assigns a BIER domain, and inside that domain, assigns bits to the adjacencies. The controller assigns each bit to a replication node that sends towards the adjacency, for instance the ingress router into a segment that will insert a routing header in the packet. A single bit may be used for a step in the ladder, indicating the other end of the step in both directions.

(A) ====> (C) ==== // 1 ^ | 4 ^ | 7 \\ ingress (I) |2| |6| (E) egress \\ 3 | v 5 | v 8 // ===> (B) ====> (D) ==== ]]> The controller activates the replication by deciding the setting of the bits associated with the adjacencies. This decision can be modified at any time, but takes the latency of a controller round trip to effectively take place. Below is an example that uses replication and elimination to protect the A->C adjacency. Bit # Adjacency Owner Example Bit Setting 1 I->A I 1 2 A->B A 1 B->A B 3 I->C I 0 4 A->C A 1 5 B->D B 1 6 C->D C 1 D->C D 7 C->E C 1 8 D->E D 0 replication and elimination Protecting A->C The BIER header with the controlling BitString is injected in the packet by the ingress node of the deterministic path. That node may act as a replication point, in which case it may issue multiple copies of the packet

Repl ===> Elim ==== // | ^ \\ ingress | | egress v | Fwd ====> Fwd ]]> For each of its bits that is set in the BIER header, the owner replication point resets the bit and transmits towards the associated adjacency; to achieve this, the replication point copies the packet and inserts the relevant data plane information, such as a source route header, towards the adjacency that corresponds to the bit Adjacency BIER BitString I->A 01011110 A->B 00011110 B->D 00010110 D->C 00010010 A->C 01001110 BitString in BIER Header as Packet Progresses Adversely, an elimination node on the way strips the data plane information and performs a bitwise AND on the BitStrings from the various copies of the packet that it has received, before it forwards the packet with the resulting BitString. Operation BIER BitString D->C 00010010 A->C 01001110 -------- AND in C 00000010 C->E 00000000 BitString Processing at Elimination Point C In this example, all the transmissions succeeded and the BitString at arrival has all the bits reset - note that the egress may be an Elimination Point in which case this is evaluated after this node has performed its AND operation on the received BitStrings). Failing Adjacency Egress BIER BitString I->A Frame Lost I->B Not Tried A->C 00010000 A->B 01001100 B->D 01001100 D->C 01001100 C->E Frame Lost D->E Not Tried BitString indicating failures But if a transmission failed along the way, one (or more) bit is never cleared. provides the possible outcomes of a transmission. If the frame is lost, then it is probably due to a failure in either I->A or C->E, and the controller should enable I->B and D->E to find out. A BitString of 00010000 indicates unequivocally a transmission error on the A->C adjacency, and a BitString of 01001100 indicates a loss in either A->B, B->D or D->C; enabling D->E on the next packets may provide more information to sort things out. In more details: The BIER header is of variable size, and a DetNet network of a limited size can use a model with 64 bits if 64 adjacencies are enough, whereas a larger deployment may be able to signal up to 256 adjacencies for use in very complex paths. illustrates a BIER header as encapsulated within MPLS. The format of this header is common to BIER and BIER-TE. For the DetNet data plane, a replication point is an ingress point for more than one adjacency, and an elimination point is an egress point for more than one adjacency. A pre-populated state in a replication node indicates which bits are served by this node and to which adjacency each of these bits corresponds. With DetNet, the state is typically installed by a controller entity such as a PCE. The way the adjacency is signaled in the packet is fully abstracted in the bit representation and must be provisioned to the replication nodes and maintained as a local state, together with the timing or shaping information for the associated flow. The DetNet data plane uses BIER-TE to control which adjacencies are used for a given packet. This is signaled from the path ingress, which sets the appropriate bits in the BIER BitString to indicate which replication must happen. The replication point clears the bit associated to the adjacency where the replica is placed, and the elimination points perform a logical AND of the BitStrings of the copies that it gets before forwarding. As is apparent in the examples above, clearing the bits enables to trace a packet to the replication points that made any particular copy. BIER-TE also enables to detect the failing adjacencies or sequences of adjacencies along a path and to activate additional replications to counter balance the failures. Finally, using the same BIER-TE bit for both directions of the steps of the ladder enables to avoid replication in both directions along the crossing adjacencies. At the time of sending along the step of the ladder, the bit may have been already reset by performing the AND operation with the copy from the other side, in which case the transmission is not needed and does not occur (since the control bit is now off).

The size of the BIER header depends on the number of segments in the particular path. It is very concise considering the amount of information that is carried (control of replication, traceability, and measurement of the reliability of the segments). Some fields in the BIER header could be used to identify the flows but they are not the primary purpose, so it's probably not a good idea. A separate procedure must be used to set up the paths and allocate the bits for the adjacencies. The bits should be distributed as a form of tag by the route setup protocol. This procedure requires more work and is separate from the data plane method that is described here. The bitmap expresses in a very concise fashion which replication and merging (and elimination) should take place for a given packet. It also enables to control that process on a per packet basis, depending on the loss that it enables to measure. The net result is that a complex path may be installed with all the possibilities and that the decision of which possibilities are used is controlled in the data plane. The setting of the bits at arrival enables to determine which adjacencies worked and which did not, enabling a dynamic control of the replication and elimination process. This is a form of OAM that is in-band with the data stream as opposed to leveraging separate packets, which is a more accurate information on the reliability of the link for the user. BIER-TE does not signal that explicitly. This is a strong point of the solution. The solution enables to determine which is the current segment that a given packet is expected to traverse, which node performed the replication and which should perform the elimination if any Some components of the technology are more mature, e.g. segment routing and BIER. Yet, the overall solution has never been deployed as is not fully defined.

BIER-TE occupies a particular position in the DetNet data plane. In the one hand it is optional, and only useful if replication and elimination is taking place. In the other hand, it has unique capabilities to: control which replication take place on a per packet basis, so that replication points can be configured but not actually utilized trace the replication activity and determine which node replicated a particular packet measure the quality of transmission of the actual data packet along the replication segments and use that in a control loop to adapt the setting of the bits and maintain the reliability.

Generic Routing Encapsulation (GRE) provides an encapsulation of an arbitrary network layer protocol over another arbitrary network layer protocol. The encapsulation of a GRE packet can be found in .

Based on RFC2784, further includes sequencing number and Key in optional fields of the GRE header, which may help to transport DetNet traffic flows over IP networks. The format of a GRE header is presented in .

GRE can provide encapsulation at the service layer over the transport layer. A new protocol type for DetNet traffic should be allocated as an "Ether Type" in and in IANA Ethernet Numbers. The fixed header of a GRE packet is 4 octets while the maximum header is 16 octets with optional fields in . There is no flow identification field in GRE header. However, it can rely on the flow identification mechanism applied in the delivery protocols, such as flow identification stated in IP Sections and when the delivery protocols are IPv6 and IPv4 respectively. Alternatively, the Key field can also be extended to carry the flow identification. The size of Key field is 4 octets. As stated in , GRE provides an optional sequencing number in its header to provide sequencing services for packets. The size of the sequencing number is 32 bits. The GRE header could be extended to indicate the duplicated packets by defining a flag in reserved fields or using the sequencing number of a flow. GRE has no flow/packet replication and merging support in its header. It can use the transport IPv4/IPv6 protocols at the transport layer to replicate the packets and take the different routes as discussed in and . GRE uses the network management provided by the IP protocols as transport layer. For the class of service capability, an optional code point field to indicate CoS of the traffic could be added into the GRE header. Otherwise, GRE can reuse the class and quality of service of delivery protocols at transport layer such as IPv6 and IPv4 stated in and . GRE has been developed over 20 years. The delivery protocol mostly used is IPv4, while the IPv6 support for GRE is to be standardized now in IETF as . Due to its good extensibility, GRE has also been extended to support network virtualization in Data Center, which is NVGRE .

As a tunneling protocol, GRE can encapsulate a wide variety of network layer protocols over another network layer, which can naturally serve as the service layer protocol for DetNet. Currently, it supports a portion of the Detnet service layer criteria, and still some are not fully supported but can be incrementally added or supported by delivery protocols at as the transport layer. In general, GRE can be a choice as the DetNet service layer and can work with IPv6 and IPv4 as the DetNet Transport layer.

MPLS based technologies supports both the DetNet Service and DetNet Transport layers. This, as well as a general overview of MPLS, is covered above in . These sections focus on the DetNet Service Layer it provides client service adaption, via Pseudowires and via native and other label-like mechanisms such as EPVN in . A representation of these options was previously discussed and is shown in . The following text is adapted from : The MPLS native service adaptation functions interface the client layer network service to MPLS. For Pseudowires, these adaptation functions are the payload encapsulation described in Section 4.4 of and Section 6 of . For network layer client services, the adaptation function uses the MPLS encapsulation format as defined in . The purpose of this encapsulation is to abstract the data plane of the client layer network from the MPLS data plane, thus contributing to the independent operation of the MPLS network. MPLS may itself be a client of an underlying server layer. MPLS can thus also bounded by a set of adaptation functions to this server layer network, which may itself be MPLS. These adaptation functions provide encapsulation of the MPLS frames and for the transparent transport of those frames over the server layer network. While MPLS service can provided on and true end-system to end-system basis, it's more likely that DetNet service will be provided over Pseudowires as described in or via an EPVN-based service described in . MPLS labels in the label stack may be used to identify transport paths, see , or as service identifiers. Typically a single label is used for service identification. Packet sequencing mechanisms are added in client-related adaptation processing, see Sections and . The MPLS client inherits its Quality of Service (QoS) from the MPLS transport layer, which in turn inherits its QoS from the server (sub-network) layer. The server layer therefore needs to provide the necessary QoS to ensure that the MPLS client QoS commitments can be satisfied.

Pseudo Wire Emulation Edge-to-Edge (PWE3) or simply PseudoWires (PW) provide means of emulating the essential attributes and behaviour of a telecommunications service over a packet switched network (PSN) using IP or MPLS transport. In addition to traditional telecommunications services such as T1 line or Frame Relay, PWs also provide transport for Ethernet service and for generic packet service . illustrate the reference PWE3 stack model.

|(e.g., Eth, ...)| +----------------+ +----------------+ | Payload | | Payload | CW, | Encapsulation |<=== Pseudo Wire ====>| Encapsulation | Timing, | | | | Seq., .. +----------------+ +----------------+ |PW Demultiplexer| |PW Demultiplexer| | PSN Tunnel, |<==== PSN Tunnel ====>| PSN Tunnel, | MPLS. | PSN & Physical | | PSN & Physical | L2TP, | Layers | | Layers | IP, .. +-------+--------+ ___________ +---------+------+ | / \ | +============/ PSN \==============+ \ / \___________/ ]]> PWs appear as a good data plane solution alternative for a number of reasons. PWs are a proven and deployed technology with a rich OAM control plane , and enjoy the toolbox developed for MPLS networks in a case of MPLS-based PSN. Furthermore, PWs may have an optional Control Word (CW) as part of the payload encapsulation between the PSN and the emulated service that is, for example, capable of frame sequencing and duplicate detection. The encapsulation layer may also provide timing using RTP as described in Sections 5.2.2, 5.4.1 and 5.4.2 of and utilized by . Furthermore, advanced DetNet node functions are conceptually already supported by PW framework (with some added functional required), such as the DetNet Relay node modeled after the Multi-Segment PWE3 . PWs can be also used if the PSN is IP, which enables the application of PWs in networks that do not have MPLS enabled in their core routers. One approach to provide PWs over IP is to provide MPLS over IP in some way and then leverage what is available for PWs over MPLS. The following standard solutions are available both for IPv4 and IPv6 to follow this approach. The different solutions have different overhead as discussed in the following subsection. The MPLS-in-IP encapsulation is specified by . The IPv4 Protocol Number field or the IPv6 Next Header field is set to 137, which indicates an MPLS unicast packet. (The use of the MPLS-in-IP encapsulation for MPLS multicast packets is not supported.) The MPLS-in-GRE encapsulation is specified in , where the IP header (either IPv4 or IPv6) is followed by a GRE header, which is followed by an MPLS label stack. The protocol type field in the GRE header is set to MPLS Unicast (0x8847) or Multicast (0x8848). MPLS over L2TPv3 over IP encapsulation is specified by . The MPLS-in-UDP encapsulation is specified by , where the UDP Destination Port indicates tunneled MPLS packet and the UDP Source Port is an entropy value that is generated by the encapsulator to uniquely identify a flow. MPLS-in-UDP encapsulation can be applied to enable UDP-based ECMP (Equal-Cost Multipath) or Link Aggregation. All these solutions can be secured with IPsec .

PWs offer encapsulation services practically for any types of payloads over any PSN. New PW types need a code point allocation and in some cases an emulated service specific document. Specifically in the case of the MPLS PSN the PW encapsulation overhead is minimal. Typically minimum two labels and a CW is needed, which totals to 12 octets. PW type specific handling might, however, allow optimizations on the emulated service in the provider edge (PE) device's native service processing (NSP) / forwarder function. These optimizations could be used, for example, to reduce header overhead. Ethernet PWs already have rather low overhead . Without a CW and VLAN tags the Ethernet header gets reduced to 14 octets (minimum Ethernet header overhead is 26). The overhead is somewhat bigger in case of IP PSN if an MPLS over IP solution is applied to provide PWs. IP adds at least 20 (IPv4) or 40 (IPv6) bytes overhead to the PW over MPLS overhead; furthermore, the GRE, L2TPv3, or UDP header has to be taken into account if any of these further encapsulations is used. PWs provide multiple layers of flow identification, especially in the case of the MPLS PSN. The PWs are typically prepended with an endpoint specific PW label that can be used to identify a specific PW per endpoint. Furthermore, the MPLS PSN also uses one or more labels to transport packets over a specific label switched paths (that then would carry PWs). So, a DetNet flow can be identified in this example by the service and transport layer labels. IP (and other) PSNs may need other mechanisms, such as, UDP port numbers, upper layer protocol header (like RTP) or some IP extension header to provide required flow identification. As mentioned earlier PWs may contain an optional CW that is able to provide sequencing services. The size of the sequence number in the generic CW is 16 bits, which might be, depending on the used link and DetNet flow speed be too little. The PW duplicate detection mechanism is already conceptually specified but no emulated service makes use of it currently. PWs could use a (extended) version of existing transport layer provided protection mechanisms (e.g., hitless 1+1 protection) for both flow duplication and flow merging. The service layer has to provide the functionality to map DetNet flows into appropriate transport leyer connection, though. PWs have rich control plane for OAM and in a case of the MPLS PSN enjoy the full control plane toolbox developed for MPLS network OAM likewise IP PSN have the full toolbox of IP network OAM tools. There could be, however, need for deterministic networking specific extensions for the mentioned control planes. In a case of IP PSN the 6-bit differentiated services code point (DSCP) field can be used for indicating the class of service and 2-bit field reserved for the explicit congestion notification (ECN) . Similarly, in a case of MPLS PSN, there are 3-bit traffic class field (TC) in the label reserved for for both Explicitly TC-encoded-PSC LSPs (E-LSP) and ECN . Due to the limited number of bits in the TC field, their use for QoS and ECN functions restricted and intended to be flexible. Although the QoS/CoS mechanism is already in place some clarifications may be required in the context of deterministic networking flows, for example, if some specific mapping between bit fields have to be done. When PWs are used over MPLS, MPLS LSPs can be used to provide both CoS (E-LSPs and L-LSPs) and QoS (dedicated TE LSPS). PWs, IP and MPLS are proven technologies with wide variety of deployments and years of operational experience. Furthermore, the estimated work for missing functionality (packet replication and elimination) does not appear to be extensive, since the existing protection mechanism already get close to what is needed from the deterministic networking data plane solution.

PseudoWires appear to be a strong candidate as the deterministic networking data plane solution alternative for the DetNet Service layer. The strong points are the technical maturity and the extensive control plane for OAM. This holds specifically for MPLS-based PSN. Extensions are required to realize the packet replication and duplicate detection features of the deterministic networking data plane.

MPLS-Based Ethernet VPN (EVPN), in the form documented in and , is an increasingly popular approach to delivering MPLS-based Ethernet services and is designed to be the successor to Virtual Private LAN Service (VPLS), . EVPN provides client adaptation and reuses the MPLS data plane discussed above in . While not required, the PW Control Word is also used. EVPN control is via BGP, , and may use TE-LSPs, e.g., controlled via for MPLS transport. Additional EVPN related RFCs and in progress drafts are being developed by the BGP Enabled Services Working Group.

EVPN generally uses a single MPLS label stack entry to support its client adaptation service. The optional addition of a second label is also supported. In certain cases PW Control Word may also be used. EVPN currently uses labels to identify flows per {Ethernet Segment Identifier, VLAN} or per MAC level. Additional definition will be needed to standardize identification of finer granularity DetNet flows as well as mapping of TSN services to DetNet Services. Like MPLS, EVPN generally orders packets similar to Ethernet. Reordering is possible primarily during path changes and protection switching. In order to avoid misordering due to ECMP, EVPN uses the "Preferred PW MPLS Control Word" (in which case EVPN inherits this function from PWs) or the entropy labels . If additional ordering mechanisms are required, such mechanisms will need to be defined. EVPN relies on the MPLS layer for all protection functions. See and . Some extensions, either at the EVPN or MPLS levels, will be need to support those DetNet applications which require true hitless (i.e., zero loss) 1+1 protection switching. (Network coding may be an interesting alternative to investigate to delivering such hitless loss protection capability.) Nodes supporting EVPN may participate in either or both Ethernet level and MPLS level OAM. It is likely that it may make sense to map or adapt the OAM functions at the different levels, but such has yet to be defined. provides some useful background on this topic. EVPN is largely silent on the topics of CoS and QoS, but the 802.1 TSN Ethernet and existing MPLS TE mechanisms can be directly used. The inter-working of such is new work and within the scope of DetNet. The existing MPLS mechanisms include both CoS (E-LSPs and L-LSPs) and QoS (dedicated TE LSPs). EVPN is a second (or third) generation MPLS-based L2VPN service standard. From a data plane standpoint it makes uses of existing MPLS data plane mechanisms. The mechanisms have been widely implemented and deployed.

EVPN is the emerging successor to VPLS. EVPN is standardized, implemented and deployed. It makes use of the mature MPLS data plane. While offering a mature and very comprehensive set of features, certain DetNet required features are not fully/directly supported and additional standardization in these areas are needed. Examples include: mapping CoS and QoS; use of labels per DetNet flow, and hitless 1+1 protection.

Fields of headers belonging to higher OSI layers can be used to implement functionality that is not provided e.g., by the IPv6 or IPv4 header fields. However, this approach cannot be always applied, e.g., due to encryption. Furthermore, even if this approach is applicable, it requires deep packet inspection from the routers and switches. There are implementation dependent limits how far into the packet the lookup can be done efficiently in the fast path. When encryption is not used, a safe bet is generally between 128 and 256 octets for the maximum lookup depth. Various higher layer protocols can be applied. Some examples are provided here for the sequence numbering feature ().

The TCP header includes a sequence number parameter, which can be applied to detect and eliminate duplicate packets if DetNet service protection is used. As the TCP header is right after the IP header, it does not require very deep packet inspection; the 4-byte sequence number is conveyed by bits 32 through 63 of the TCP header. In addition to sequencing, the TCP header also contain source and destination port information that can be used for assisting the flow identification.

Real-time Transport Protocol (RTP) is often used to deliver time critical traffic in IP networks. RTP is typically carried on top of UDP/IP. However, as noted earlier in PseudoWires also have a well-defined way of embedding and transposting RTP header as part of its payload encapsulation headers/sub-layer. RTP is also augmented by its own control protocol RTCP, which monitors of the data delivery and provides minimal control and identification functionality. RTCP packets do not carry "media payload". Although both RTP and RTCP are typically used with UDP/IP transport they are designed to be independent of the underlying transport and network layers. The RTP header includes a 2-byte sequence number, which can be used to detect and eliminate duplicate packets if DetNet service protection is used. The sequence number is conveyed by bits 16 through 31 of the RTP header. In addition to the sequence number the RTP header has also timestamp field (bits 32 through 63) that can be useful for time synchronization purposes. Furthermore, the RTP header has also one or more synchronization sources (bits starting from 64) that can potentially be useful for flow identification purposes.

RTP adds minimum 12 octets of header overhead. Typically 8 octets overhead of UDP header has to be also added, at least in a case when RTP is transported over IP. Although RTCP packets do not contribute to the media payload transport they still consume overall network capacity, since all participants to an RTP session including sourcess and multicast session destinations are expected to send RTCP reports. The RTP header contains a synchronization source (SSRC) identifier. The intent is that no two synchronization sources within the same RTP session has the same SSRC identifier. The RTP header contains a 16 bit sequence number. The sequence number can be also used to detect duplicate packets. RTP has precedence of being used for hitless protection switching , which essentially is equivalent to DetNet service protection. Furthermore, recent work in IETF for RTP stream duplication as a mechanism to protect media flows from packet loss is again equivalent to Detnet service protection. RTP has its own control protocol RTCP for (minimal) management and stream monitoring purposes. Existing IP OAM tools can directly leveraged when RTP is deployed over IP transport. TBD. [Editor's note: relies on lower layers to provide CoS/QoS] RTP has been deployed and used in large commercial systems for over ten years and can be considered a mature technology.

RTP appears to be a good candidate as the deterministic networking data plane solution alternative for the DetNet Service layer. The strong points are the technical maturity and the fact it was designed for transporting time-sensitive payload from the beginning. RTP is specifically well suited to be used with (UDP)/IP transport. Extensions may be required to realize the packet replication and duplicate detection features of the deterministic networking data plane. However, there is already precedence of similar solutions that could potentially be leveraged .