]> Infinera Corp

169 Java Drive Sunnyvale CA 94089 USA +1-408-572-5200 IHussain@infinera.com

Infinera Corp

169 Java Drive Sunnyvale CA 94089 USA +1-408-572-5200 rvaliveti@infinera.com

Infinera Corp

169 Java Drive Sunnyvale CA 94089 USA +1-408-572-5200 kpithewan@infinera.com

General Internet Engineering Task Force template Traditionally, Ethernet MAC rates were constrained to match the rates of the Ethernet PHY(s). OIF's implementation agreement was the first step in allowing MAC rates to be different than the PHY rates. OIF has recently approved another implementation agreement which allows complete decoupling of the MAC data rates and the Ethernet PHY(s) that support them. This includes support for (a) MAC rates which are greater than the rate of a single PHY (satisfied by bonding of multiple PHY(s)), (b) MAC rates which are less than the rate of a PHY (sub-rate), (c) support of multiple FlexE client signals carried over a single PHY, or over a collection of bonded PHY(s). This draft catalogs the usecases that are encountered when these Flexible rate Ethernet client signals are transported over OTN networks.

Traditionally, Ethernet MAC rates were constrained to match the rates of the Ethernet PHY(s). OIF's implementation agreement was the first step in allowing MAC rates to be different than the PHY rates standardized by IEEE. OIF has recently approved another implementation agreement which allows complete decoupling of the MAC data rates and the Ethernet PHY(s) that support them. This includes support for (a) MAC rates which are greater than the rate of a single PHY (satisfied by bonding of multiple PHY(s)), (b) MAC rates which are less than the rate of a PHY (sub-rate), (c) support of multiple FlexE client signals carried over a single PHY, or over a collection of bonded PHY(s). The capabilities supported by the OIF FlexE implementation agreement version 1.0 are: Support a large rate Ethernet MAC over bonded Ethernet PHYs, e.g. supporting a 200G MAC over 2 bonded 100GBASE-R PHY(s) Support a sub-rate Ethernet MAC over a single Ethernet PHY, e.g. supportnig a 50G MAC over a 100GBASE-R PHY Support a collection of flexible Ethernet clients over a single Ethernet PHY, e.g. supporting two MACs with the rates 25G, 50G over a single 100GBASE-R PHY Support a sub-rate Ethernet MAC over bonded PHYs, e.g. supporting a 150G Ethernet client over 2 bonded 100GBASE-R PHY(s) Support a collection of Ethernet MAC clients over bonded Ethernet PHYs, e.g. supporting a 50G, and 150G MAC over 2 bonded Ethernet PHY(s) Optical Transport Networks (defined by and ) have, until recently, only dealt with bit (or codeword) transparent transport of Ethernet client signals. The introduction of the FlexE capabilities at the OTN client interfaces requires the OTNs to examine the various usecases. This Internet-Draft examines the various usecases that arise when transporting the Flexible Rate Ethernet signals in Optical transport networks. This list of usecases will help identify the Control Plane (i.e. Routing and Signaling) extensions that may be required).

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Ethernet PHY: an entity representing 100G-R Physical Coding Sublayer (PCS), Physical Media Attachment (PMA), and Physical Media Dependent (PMD) layers. FlexE Group: a group of from 1 to 254 bonded Ethernet PHYs. FlexE Client: an Ethernet flow based on a MAC data rate that may or may not correspond to any Ethernet PHY rate (e.g., 10, 40, m x 25 Gb/s). FlexE Shim: the layer that maps or demaps the FlexE clients carried over a FlexE group. FlexE Calendar: Representation of a FlexE group of n PHYs as a calendar of 20n slots logical length with 20 slots per PHY for scheduling of slots (i.e., a PHY bandwidth) among the FlexE clients.

The FlexE shim layer in a router maps the FlexE client(s) over the FlexE group. The transport network is unware of the FlexE. Each of the FlexE group PHY is carried independently across the transport network over the same fiber route. The FlexE shim in the router tolerates end-to-end skew across the network. This usecase allows to utilize Network Processor Unit (NPU) and router port rate full capacities with legacy transport equipment that provides PCS-codeword transparent transport of 100GbE. It allows striping of PHYs in the FlexE group over multiple transport line cards.

================================================================== ==================================================================

This scenario represents an optimization of the FlexE unaware transport presented in , and illustrated in . In this application (see ), the devices at the edge of the transport network do not terminate the FlexE shim layer, but are aware of the format of the FlexE overhead. They "snoop" the FlexE overhead to determine the subset of the set of all calendar slots that are available for use (i.e. these calendar slots may be used, or unused). The transport network edge removes the unavailable calendar slots at the ingress to the network, and adds the same unavailable calendar slots back when exiting the network. The result is that the FlexE Shim layers at both routers see exactly the same input that they saw in the FlexE unware scenario -- with the added benefit that the line (or DWDM) side bandwidth has been optimized to be sufficient to carry only the available calendar slots in all of the Ethernet PHY(s) in the FlexE group. This mode may be used in cases where the bandwidth of the Ethernet PHY is greater than the bit rate supported by a wavelength (and it is known that that all calendar slots in the PHY are not "available"). The transport network edge device could learn of the set of unavailable calendar slots in a variety of ways; a few examples are listed below: The set of unavailable calendar slots could be configured against each Ethernet PHY in the FlexE group. The FlexE demux function in the transport network edge device (A) compares the information about calendar slots which are expected to be unavailable (as per user supplied configuration), with the corresponding information encoded by the customer edge device in the FlexE overhead (as specified in ). If there is a mismatch between the unavailable calendar slots in any of the PHYs within a FlexE group, the transport edge node software could raise an alarm to report the inconsistency between the provisioning information at the transport network edge, and the customer edge device. The Transport network edge could be configured to act in a "slave" mode. In this mode, the FlexE demux function at the Transport network edge (A) receives the information about the available/unavailable calendar slots by observing the FlexE overhead (as specified in ) and uses this information to select (a) the set of wavelengths (with appropriate capacities) or (b) the bandwidth of the ODUflex (or fixed rate ODUs) that could carry the FlexE PCS end-to-end. In the basic FlexE aware mode, the transport network edge does not expect the number of unavailable calendar slots to change dynamically. Note that the process of removing unavailable calendar slots from a FlexE PHY is called "crunching" (see ). The following additional notes apply to : The crunched FlexE PHYs are independently transported through the transport network. The number of used (and unused) calendar slots can be different across the FlexE group. In particular, if all the calendar slots in a FlexE PHY are in use, the crunching operation leaves the original signal intact. In this illustration, the different FlexE PHY(s) are transported using ODUflex containers in the transport network. These ODUflex connections can be of different rates. When the crunched FlexE PHY(s) have a rate that is identical to that of a standard Ethernet PHY, it is possible that the transport network may utilize standard ODU containers such as ODU2e, ODU4 etc.

================================================================ ===============================================================

These usecases build upon the basic router-transport equipment connectivity illustrated in . The FlexE shim layer at the router maps to the set of FlexE clients over the FlexE group, as usual. This section considers various usecases in which the equipment located at the edge of the transport network is fully aware of the FlexE OH, and FlexE Shim layers on the transport network edge, and the customer edge are peers. In the router to network direction, the transport edge node terminates the FlexE shim layer, and extracts one or more FlexE client signals, and transports them through the network. That is, these usecases are distinguished from the FlexE unaware cases in that the FlexE group, and the FlexE shim layer end at the transport network edge, and only the extracted FlexE client signals transit the optical network. In the network to router direction, the transport edge node maps a set of FlexE clients to the FlexE group (i.e. performing the same functions as the router which connects to the transport network).The various usecases differ in the combination of service endpoints in the transport network. In the FlexE termination scenarios, the distance between the FlexE Shims is limited the normal Ethernet link distance. The FlexE shims in the router, and the equipment need to support a small amount skew.

In this scenario, service consists of transporting a FlexE client through the transport network, and possibly combining this FlexE client with other FlexE clients into a FlexE group at the endpoints. The FlexE client signal can be transported in two manners within the OTN: (i) directly over one or more wavelengths (ii) mapped into an ODUflex (of the appropriate rate) and then switched across the OTN. illustrates the scenario involving the mapping of a FlexE client to an ODUflex envelope; this figure only shows the signal "stack" at the service endpoints, and doesn't illustrate the switching of the ODUflex entity through the OTN. The ODUflex mapping will be beneficial in scenarios where the rate of the FlexE client is less than the capacity of a single wavelength deployed on the DWDM side of the OTN network, and allows the network operators to packet multiple FlexE client signals into the same wavelength -- thereby improving the network efficiency. Although illustrates the scenario in which one FlexE client is transported within the OTN, the following points should be noted: When the FlexE Shim termination function recovers multiple FlexE client signals (at node A), the FlexE signals can be transported independently. In other words, it is not a requirement that all the FlexE client signals be co-routed. Conversely, at the egress node, FlexE clients from different endpoints can be combined via the FlexE shim, eventually exiting the transport edge node over an Ethernet group.

================================================================== =================================================================

The OIF implementation agreement currently supports FlexE client signals carried over one or more 100GBASE-R PHY(s). There is a calendar of 5G timeslots associated with each PHY, and each FlexE client can make use of a number of timeslots (possibly distributed across the members of the FlexE group. This implies that the FlexE client rates are multiples of 5Gbps. When the rates of the FlexE client signals matches the MAC rates corresponding to existing Ethernet PHYs, i.e. 10GBASE-R/40GBASE-R/100GBASE-R, there is a need for the FlexE client signal to interwork with the native Ethernet client received from a single (non-FlexE capable) Ethernet PHY. This capability is expected to be extended to any future Ethernet PHY rates that the IEEE may define in future (e.g. 25G, 50G, 200G etc.). In these cases, although the bit rate of the FlexE client matches the MAC rate of other endpoint, the 64B66B PCS codewords for the FlexE client need to be transformed (via ordered set translation) to match the specification for the specific Ethernt PHY. These details are described in Section 7.2.2 of and are not eloborated any further in this document. illustrates a scenario involving the interworking of a 10G FlexE client with a 10GBASE-R native Ethernet signal. In this example, the network wrapper is ODU2e.

================================================================== =================================================================

As explained in the Introduction section ( OIFMLG3 introduced support for carrying 10GE and 40GE client signals over a group of 100GBASE-R Ethernet PHY(s). While the most recent implementation agreement doesn't call it out explicitly, it is expected that the FlexE clients (as defined in ), and 10GBASE-R/40GBASE-R clients supported by OIFMLG3 ) will interoperate. illustrates a scenario involving the interworking of a 10G FlexE client with a 10GBASE-R client supported by an OIFMLG3 interface. In this example, the network wrapper is ODU2e.

================================================================== =================================================================

This section covers an extension of the scenario presented in . Each FlexE client signal defined in has a rate which is a multiple of 5G, and occupies the required number of calendar slots (5G granularity) in the FlexE group (possibly distributed among the PHY(s) which have been bonded. The OIF implementation agreement defines two calendars, one currently active, and the future calendar to which the sender wants to transition to. This capability can be used to coordinate a synchronized switchover of calendars between the two FlexE Shim functions -- one located in the customer eddge device (typically a router), and the transport network edge. In this scenario, there are three independent resizing domains which must be coordinated (see ). Between the router 1 and Transport edge node A-end Between the transport edge nodes A, Z Between the transport edge node Z, and router 2 It is possible to coordinate the resize operations in these domains in such a manner that the FlexE clients get the benefit of an end-to-end bandwidth change (increase/decrease), without involving any additional provisioning steps in the provider network. Note for the FlexE unaware use case (), the client BW can be resized by FlexE shim coordination between router 1 and router 2.

This section covers a degenerate FlexE aware scenario where router1, router2, and router3 are interconnected through back-to-back FlexE groups without an intermediate transport network (see ).

================================================================== =================================================================

The list of aforementioned FlexE usecases can also be supported by mapping FlexE directly over one or more wavelengths. An example for the FlexE unaware transport over wavelength is depicted in . Equivalent network diagrams for the other usecases can be obtained by replacing an OTN container with an Optical Channel (OCh).

================================================================== ==================================================================

This section summarizes solution requirements for the usecases described in this document to help identify the Control Plane (i.e. Routing and Signaling) extensions that may be required. The solution SHALL support a FlexE group to address abovementioned usecases including FlexE unaware (where FlexE mux and demux can be separated by longer distances), FlexE aware (where FlexE mux and demux can be separated by shorter distances), and FlexE partially aware. The solution SHALL support a flexible mechanism for configuring a FlexE group -- such as a signaling protocol or a SDN controller/management system with network access to the FlexE mux/demux at each end of the FlexE group. The solution SHALL support the ability to add/remove Ethernet PHYs to/from a FlexE group. The solution SHOULD allow decoupling of FlexE group's initial configuration and bring up operation from an addition (or removal) of FlexE clients to the FlexE group. For instance, it SHOULD be possible to configure and bring up a FlexE group without any FlexE client (e.g., with all calendar slots set to unused or unavailable). The solution SHALL allow adding or removing a FlexE client to a FlexE group without affecting traffic on other clients. The solution SHALL allow resizing of FlexE client BW through coordination of calendar updates within a single FlexE group. There SHOULD be no expectation that FlexE client BW resizing be hitless in all network scenarios. For the FlexE unaware case, each of the 100GBASE-R PHYs in the FlexE group SHALL be carried independently across transport network using a PCS codeword transparent mapping. All PHYs of the FlexE group SHALL be interconnected between the same two FlexE shims. The Ethernet PHYs SHOULD be carried over the same fiber route across the transport network (i.e., co-routed) For the FlexE partially aware case, each of the 100GBASE-R PHYs in the FlexE group SHALL be carried independently across transport network. All PHYs of the FlexE group SHALL be interconnected between the same two FlexE shims. The Ethernet PHYs SHOULD be carried over the same fiber route across the transport network. In the transport network, in mux direction, the OTN mapper SHALL be able to discard unavailable slots (e.g., this can be based on static configuration as the rate of a wavelength is not expected to change in-service). In the transport network, in the demux direction, the OTN mapper SHALL be able to restore unavailable slots to match the original PHY rate. For the FlexE aware case, the FlexE group SHALL be terminated at the transport network edge. It SHOULD be possible to carry (switch) each FlexE client extracted from the FlexE group independently across transport network using OTN mapping (e.g., ODUflex).

This memo includes no request to IANA.

None.

&RFC2119; OIF ITU ITU OIF

This becomes an Appendix.