Usage and Applicability of BGP Link-State Shortest Path Routing (BGP-SPF) in Data Centers

With the BGP-SPF extensions , the BGP best-path computation and route computation are replaced with link-state algorithms such as those used by OSPF , both to determine whether an BGP-LS SPF NLRI has changed and needs to be re-advertised and to compute the BGP routes. These modifications will significantly improve convergence of the underlay while affording the operational benefits of a single routing protocol . Data center controllers typically require visibility to the BGP topology to compute traffic-engineered paths. These controllers learn the topology and other relevant information via the BGP-LS address family which is totally independent of the underlay address families (usually IPv4/IPv6 unicast). Furthermore, in traditional BGP underlays, all the BGP routers will need to advertise their BGP-LS information independently. With the BGP-SPF extensions, controllers can learn the topology using the same BGP advertisements used to compute the underlay routes. Furthermore, these data center controllers can avail the convergence advantages of the BGP-SPF extensions. The placement of controllers can be outside of the forwarding path or within the forwarding path. Alternatively, as each and every router in the BGP-SPF domain will have a complete view of the topology, the operator can also choose to configure BGP sessions in the hop-by-hop peering model described in along with BFD . In doing so, while the hop-by-hop peering model lacks the inherent benefits of the controller-based model, BGP updates need not be serialized by the BGP best-path algorithm in either of these models. This helps overall network convergence.

Section 5.1 of defines a new BGP-LS-SPF SAFI for announcement of the BGP-SPF link-state. The NLRI format and its associated attributes follow the format of BGP-LS for node, link, and prefix announcements. Whether the peering model within a Clos follows hop-by-hop peering described in or any controller-based or route-reflector peering, an operator can exchange BGP-LS SPF SAFI routes over the BGP peering by simply configuring BGP-LS SPF SAFI between the necessary BGP speakers. The BGP-LS SPF SAFI can also co-exist with BGP IP Unicast SAFI which could exchange overlapping IP routes. One use case for this is where BGP-LS SPF routes are used for the underlay and BGP IP Unicast routes for VPNs are advertised in the overlay as described in . The routes received by these SAFIs are evaluated, stored, and announced independently according to the rules of . The tie-breaking of route installation is a matter of the local policies and preferences of the network operator. Finally, as the BGP-SPF peering is done following the procedures described in , all the existing transport security mechanisms including are available for the BGP-LS SPF SAFI.

Normally, the BGP-LS SPF AFI/SAFI is used solely to compute the underlay and is given preference over other AFI/SAFIs. Other BGP SAFIs, e.g., IPv6/IPv6 Unicast VPN would use the BGP-SPF computed routes for next hop resolution.

As previously stated, BGP-SPF can be deployed using the existing peering model where there is a single-hop BGP session on each and every link in the data center fabric . This provides for both the advertisement of routes and the determination of link and neighboring switch availability. With BGP-SPF, the underlay will converge faster due to changes to the decision process that will allow NLRI changes to be advertised faster after detecting a change.

Alternately, BFD can be used to swiftly determine the availability of links and the BGP peering model can be significantly sparser than the data center fabric. BGP-SPF sessions only need to be established with enough peers to provide a bi-connected graph. If Internal BGP (IBGP) is used, then the BGP routers at tier N-1 will act as route-reflectors for the routers at tier N. The obvious usage of sparse peering is to avoid parallel BGP sessions on links between the same two switches in the data center fabric. However, this use case is not very useful since parallel L3 links between the same two BGP routers are rare in Clos or Fat-Tree topologies. Additionally, when there are multiple links, they are often aggregated at the link layer rather than the IP layer. Two more interesting scenarios are described below. In current data center topologies, there is often a very dense mesh of links between levels, e.g., leaf and spine, providing 32-way, 64-way, or more Equal-Cost Multi-Path (ECMP) paths. In these topologies, it is desirable not to have a BGP session on every link and techniques such as the one described in can be used to establish sessions on some subset of northbound links. For example, in a Spine-Leaf topology, each leaf switch would only peer with a subset of the spines dependent on the flooding redundancy required to be reasonably certain that every node within the BGP-LS SPF routing domain has the complete topology. Alternately, controller-based data center topologies are envisioned where BGP speakers within the data center only establish BGP sessions with two or more controllers. In these topologies, fabric nodes below the first tier, as shown in Figure 1 of , will establish BGP multi-hop sessions with the controllers. For the multi-hop sessions, determining the route to the controllers without depending on BGP would need to be through some other means beyond the scope of this document. However, the BGP discovery mechanisms described in would be one possibility.

With this heuristic, discovery of BGP SPF peers is assumed, e.g., as described in . In this context, "bi-connected" refers to the fact that there must be an adverised link NLRI for both BGP SPF peers associated with the link before the link can be used in the BGP SPF route calcuation. Additionally, it assumed that the direction of the peering can be ascertained. In the context of a data center fabric, the direction is either northbound (toward the spine), southbound (toward the Top-Of-Rack (ToR) switches) or east-west (same level in the hierarchy). The determination of the direction is beyond the scope of this document. However, it would be reasonable to assume a technique where the ToR switches can be identified and the number of hops to the ToR is used to determine the direction. In this heuristic, BGP speakers allow passive session establishment for southbound BGP sessions. For northbound sessions, BGP speakers will attempt to maintain two northbound BGP sessions with different switches (in data center fabrics there is normally a single layer-3 connection anyway). For east-west sessions, passive BGP session establishment is allowed. However, a BGP speaker will never actively establish an east-west BGP session unless it cannot establish two northbound BGP sessions. BGP SPF sparse peering deployments not using this hueristic are possible but are not described herein and are considered out of scope.

One of the advantages of using BGP-SPF as the underlay protocol is that BGP policy can be applied at any level. For example, depending on the topology, it may be possible to aggregate prefix advertisements using existing BGP policy. In Spine/Leaf topologies, it is not necessary to advertise BGP-LS Prefix NLRI received by leaves northbound to the spine nodes. An aggregate route or a default route could suffice. If a common AS is used for the spine nodes, this can easily be accomplished with EBGP and a simple policy to filter advertisements from the leaves to the spine if the first AS in the AS path is the spine AS. In the figure below, the leaves would not advertise any NLRI with AS 64512 as the first AS in the AS path.

Spine/Leaf Topology Policy

The basic requirement is to be able to discover the address of a single-hop peer in case where the peer address is not pre-configured. This is being accomplished today by using IPv6 Router Advertisements (RA) and assuming that a BGP session is desired with any discovered peer. Beyond the basic requirement, it is useful to have to following information relating to the BGP session: Autonomous System (AS) and BGP Identifier of a potential peer. The latter can be used for debugging and to decrease the likelihood of BGP session establishment collisions. Security capabilities supported and for cryptographic authentication, the security capabilities and possibly a key-chain to be used. Session Policy Identifier - A group number or name used to associate common session parameters with the peer. For example, in a data center, BGP sessions with a ToR device could have different parameters than BGP sessions between leaf and spine. In a data center fabric, it is often useful to know whether a peer is southbound (towards the servers) or northbound (towards the spine or super-spine), e.g., . A potential requirement would be to determine this dynamically. One mechanism, without specifying all the details, might be for the ToR switches to be identified when installed and for the others switches in the fabric to determine their level based on the distance from the closest ToR switch. If there are multiple links between BGP speakers or the links between BGP speakers are unnumbered, it is also useful to be able to establish multi-hop sessions using the loopback addresses. This will often require the discovery protocol to install route(s) toward the potential peer loopback addresses prior to BGP session establishment. Finally, a simple BGP discovery protocol could also be used to establish a multi-hop session with one or more controllers by advertising connectivity to one or more controllers. However, once the multi-hop session traverses multiple nodes, it is bordering a distance-vector routing protocol and possibly this is not a good requirement for the discovery protocol.

To conserve IPv4 address space and simplify operations, BGP-LS SPF routers in Clos/Fat Tree deployments can use IPv6 addresses as peer address. For IPv4 address families, IPv6 peering as specified in can be deployed to avoid configuring IPv4 addresses on BGP-LS SPF router interfaces. When this is done, dynamic discovery mechanisms, as described in , can be used to learn the global or link-local IPv6 peer addresses and IPv4 addresses need not be configured on these interfaces. If IPv6 link-local peering is used, then configuration of IPv6 global addresses is also not required and these IPv6 link-local addresses must then be advertised in the BGP-LS Link Descriptor IPv6 Address TLV (262) .

Irrespective of whether or not BGP-LS SPF is used for route calculation, the BGP-LS SPF route advertisements can be used to periodically construct the Clos/Fat Tree topology. This is especially useful in deployments where an Interior Gateway Protocol (IGP) is not used and the base BGP-LS routes are not available. The resultant topology visibility can then be used for troubleshooting and consistency checking. This would normally be done on a central controller or other management tool which could also be used for fabric data path verification. The precise algorithms and heuristics, as well as the complete set of management applications is beyond the scope of this document.

Since BGP-SPF is to be used for the routing underlay and DCI gateway boxes typically have direct or very simple connectivity, BGP external sessions would typically not include the BGP-LS SPF SAFI.