<?xml version='1.0' encoding='ascii'?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>
<?rfc strict="yes" ?>
<?rfc toc="yes"?>
<?rfc tocdepth="3"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes" ?>
<?rfc compact="yes" ?>
<?rfc subcompact="no" ?>
<rfc category="std" docName="draft-ietf-nfsv4-mv0-trunking-update-04" updates="7530" submissionType="IETF" xml:lang="en" obsoletes="">
  <!--RFC Editor: Please correct the usage of "this document" and "the current document" to use the current approved RFC style.  -->
  <front>
    <title abbrev="NFSv4.0 Trunking Update">NFS version 4.0 Trunking Update </title>
    <author initials="C.L." surname="Lever" fullname="Charles Lever" role="editor">
      <organization abbrev="Oracle">Oracle Corporation </organization>
      <address>
        <postal>
          <street/>
          <city/>
          <region/>
          <code/>
          <country>United States of America</country>
        </postal>
        <email>chuck.lever@oracle.com</email>
      </address>
    </author>
    <author initials="D.N." surname="Noveck" fullname="David Noveck">
      <organization>NetApp </organization>
      <address>
        <postal>
          <street/>
          <city/>
          <region/>
          <code/>
          <country>United States of America</country>
        </postal>
        <email>davenoveck@gmail.com</email>
      </address>
    </author>
    <date/>
    <area>Transport</area>
    <workgroup>Network File System Version 4</workgroup>
    <abstract>
      <t>The file system location-related attribute in NFS version 4.0, fs_locations, informs clients about alternate locations of file systems.  An NFS version 4.0 client can use this information to handle migration and replication of server filesystems.  This document describes how an NFS version 4.0 client can additionally use this information to discover an NFS version 4.0 server's trunking capabilities.  This document updates RFC 7530.  </t>
    </abstract>
  </front>
  <middle>
    <section title="Introduction" anchor="section:987FB02C-C86C-43AB-B174-8F8BE4B2AD1E" toc="default">
      <t>The NFS version 4.0 specification <xref target="RFC7530" pageno="false" format="default"/> defines a migration feature that enables the transfer of a file system from one server to another without disruption of client activity.  There were a number of issues with the original definition of this feature, now resolved with the publication of <xref target="RFC7931" pageno="false" format="default"/>.  </t>
      <t>After a migration event, a client must determine whether state recovery is necessary.  To do this, it needs to determine whether the source and destination server addresses represent the same server instance, if the client has already established a lease on the destination server for other file systems, and if the destination server instance has lock state for the migrated file system.  </t>
      <t>As part of addressing this need, <xref target="RFC7931" pageno="false" format="default"/> introduces trunking into NFS version 4.0 along with a trunking detection mechanism.  A trunking detection mechanism enables a client to determine whether two distinct network addresses are connected to the same NFS version 4.0 server instance.  Without this knowledge, a client unaware of a trunking relationship between paths it is using simultaneously is likely to become confused in ways described in <xref target="RFC7530" pageno="false" format="default"/>.  </t>
      <t>NFSv4.1 was defined with an integral means of trunking detection, described in <xref target="RFC5661" pageno="false" format="default"/>.  NFSv4.0 initially did not have one, with it being added by <xref target="RFC7931" pageno="false" format="default"/>.  Nevertheless, the use of the concept of server-trunkability is the same in both protocol versions.  </t>
      <t>File system migration, replication, and referrals are distinct protocol features.  However, it is not appropriate to treat each of these features in isolation.  For example, client migration recovery processing needs to deal with the possibility of multiple server addresses in a returned fs_locations attribute.  In addition, the contents of the fs_locations attribute, which provides both trunking-related and replication information, may change over repeated retrievals, requiring an integrated description of how clients are to deal with such changes.  The issues discussed in the current document relate to the interpretation of the fs_locations attribute and to the proper client and server handling of changes in fs_locations attribute values.  </t>
      <t>Therefore the goals of the current document are: <list style="symbols"><t>To provide NFS version 4.0 with a means of finding addresses trunkable with a given address; i.e., trunking discovery, compatible with the means of trunking detection introduced by <xref target="RFC7931" pageno="false" format="default"/>.  For an explanation of trunking detection and discovery, see <xref target="section:3B92F813-F92B-46BE-B39D-C68BC628149F" pageno="false" format="default"/>.  </t><t>To describe how NFS version 4.0 clients are to handle the presence of multiple network addresses associated to the same server, when recovering from a replication and migration event.  </t><t>To describe how NFS version 4.0 clients are to handle changes in the contents of returned fs_locations attributes, including those that indicate changes in the responding NFS version 4.0 server's trunking configuration.  </t></list> </t>
      <t>The current document pursues these goals by presenting a set of updates to <xref target="RFC7530" pageno="false" format="default"/> as summarized in Sections <xref target="section:F1FF10EB-76C8-4098-A3C4-7AF19483ADFD" format="counter" pageno="false"/> and <xref target="section:E6A08F19-9B5A-48D6-9B8A-D5945E749644" format="counter" pageno="false"/> below.  </t>
    </section>
    <section title="Requirements Language" anchor="section:54A4DD23-8FEE-4171-AF02-3A075F336BEB" toc="default">
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 <xref target="RFC2119" pageno="false" format="default"/> <xref target="RFC8174" pageno="false" format="default"/> when, and only when, they appear in all capitals, as shown here.  </t>
    </section>
    <section title="Terminology" anchor="section:3B92F813-F92B-46BE-B39D-C68BC628149F" toc="default">
      <t>Most of the terms related to handling the fs_locations attribute are appropriately defined in <xref target="section:D521608C-EFBB-46E8-A8AE-46E47DAD7EF8" pageno="false" format="default"/> below.  However, there are a few terms used outside that context that are explained in this section.  </t>
      <t>Trunking refers to a situation in which a client uses multiple network addresses communicate with the same server.  Trunking was first introduced to NFSv4.0 by <xref target="RFC7931" pageno="false" format="default"/>.  Regarding network addresses and the handling of trunking we use the following terminology: <list style="symbols"><t>Each NFSv4 server is assumed to have a set of IP addresses to which NFSv4 requests may be sent by clients.  These are referred to as the server's network addresses.  Access to a specific server network address might involve the use of multiple network ports, since the ports to be used for particular types of connections might be required to be different.  </t><t>Clients may establish connections to NFSv4 servers via one of several connection types, supporting the NFSv4 protocol layered on top of an RPC stream transport, as described in <xref target="RFC5531" pageno="false" format="default"/>, or on top of RPC-over-RDMA, as described in <xref target="RFC8166" pageno="false" format="default"/>.  The combination of a server network address and a particular connection type is referred to as a "server endpoint".  </t><t>Each network address, when combined with a pathname providing the location of a file system root directory relative to the associated server root file handle, defines a file system network access path.  </t><t>Two network addresses connected to the same server are said to be server-trunkable.  Unlike subsequent NFSv4 minor versions, NFSv4.0 recognizes only a single type of trunking relationship between addresses.  </t></list> </t>
      <t>Particularly important is the distinction between trunking detection and trunking discovery.  The definitions we present are applicable to all minor versions of NFSv4, but we put particular emphasis on how these terms apply to NFS version 4.0.  <list style="symbols"><t>Trunking detection refers to ways of confirming that two unique network addresses are associated with the same NFSv4 server instance.  The means available to make this determination depends on the protocol version and, in some cases, on the client implementation.  <vspace blankLines="1"/> In the case of NFS version 4.0, the means to be used are described in <xref target="RFC7931" pageno="false" format="default"/> and require use of the Uniform Client String approach to be effective.  This is in contrast to later minor versions for which the means of trunking detection are described by <xref target="RFC5661" pageno="false" format="default"/>.  </t><t>Trunking discovery is a process by which an NFSv4 client, accessing one server network address, can obtain other addresses that might be associated with the same server instance.  Typically a client builds on a trunking detection facility by providing one or more methods by which candidate addresses are made available to the client, who then uses trunking detection to appropriately filter them.  <vspace blankLines="1"/> Trunking discovery is not discussed in <xref target="RFC7530" pageno="false" format="default"/> and no description of it is provided in <xref target="RFC7931" pageno="false" format="default"/>.  </t></list> </t>
      <t>Discussion of the term "replica" is complicated for a number of reasons.  Even though the term is used in explaining the issues in <xref target="RFC7530" pageno="false" format="default"/> that need to be addressed in the current document, a full explanation of this term requires explanation of related terms connected to the fs_locations attribute, which is provided in <xref target="section:D521608C-EFBB-46E8-A8AE-46E47DAD7EF8" pageno="false" format="default"/> of the current document.  </t>
      <t>The term is also used in previous documents about NFSv4.0 (i.e., <xref target="RFC7530" pageno="false" format="default"/> and <xref target="RFC7931" pageno="false" format="default"/>) with a meaning different from that in the current document.  In these documents each replica is identified by a single network access path.  However, in the current document a set of network access paths which have server-trunkable network addresses and the same root-relative file system pathname are considered to be a single replica with multiple network access paths.  Although <xref target="RFC7931" pageno="false" format="default"/> enables an NFSv4.0 client to determine whether two network addresses were server-trunkable, it never described these as connected to a single replica, leaving in effect the approach established in <xref target="RFC7530" pageno="false" format="default"/>.  </t>
    </section>
    <section title="Document Organization" anchor="section:773E739E-065E-4858-9FB4-1F06A806E69A" toc="default">
      <t>The sections of the current document are divided into four types based on how they relate to the eventual updating of the NFS verion 4.0 specification.  Once this update is published, NFS version 4.0 will be specified by multiple documents that need to be read together, until such time as a consolidated replacement specification is produced.  <list style="symbols"><t>The base specification <xref target="RFC7530" pageno="false" format="default"/>.  </t><t>The migration-related update <xref target="RFC7931" pageno="false" format="default"/>.  </t><t>This document [RFC-TBD].  </t></list> <!--RFC Editor: Please replace RFC-TBD with the RFC number assigned to this document.  --> </t>
      <t>The section types are as follows.  See <xref target="section:4EB27D51-743A-4C6A-97C2-19DA646AC319" pageno="false" format="default"/> for a classification of each section of the current document.  <list style="symbols"><t>An explanatory section does not contain any material that is meant to update the specification of NFS version 4.0.  Such sections may contain explanation about why and how changes are to be done, but do not include any text that is to update <xref target="RFC7530" pageno="false" format="default"/> or appear in an eventual consolidated document.  </t><t>A replacement section contains text that is to replace and thus supersede text within <xref target="RFC7530" pageno="false" format="default"/> and then appear in an eventual consolidated document.  The titles of replacement sections indicate what section of <xref target="RFC7530" pageno="false" format="default"/> is to be replaced.  </t><t>An additional section contains text which, although not replacing anything in <xref target="RFC7530" pageno="false" format="default"/>, will be part of the specification of NFS version 4.0 and will be expected to be part of an eventual consolidated document.  The titles of additional sections provide an indication of where in an updated <xref target="RFC7530" pageno="false" format="default"/>, the new section would appear.  </t><t>An editing section contains some text that replaces text within <xref target="RFC7530" pageno="false" format="default"/>, although the entire section will not consist of such text and will include other text as well.  Such sections make relatively minor adjustments in the existing NFS version 4.0 specification which are expected to be reflected in an eventual consolidated document.  Generally such replacement text appears as a quotation, possibly taking the form of an indented set of paragraphs.  </t></list> </t>
    </section>
    <section title="Changes Within Section 8 of [RFC7530]" anchor="section:F1FF10EB-76C8-4098-A3C4-7AF19483ADFD" toc="default">
      <t>Most of the updates to <xref target="RFC7530" pageno="false" format="default"/> to provide support for trunking using the fs_locations attribute apply to Section 8 of that document, entitled "Multi-Server Namespace".  <list style="symbols"><t><xref target="section:D521608C-EFBB-46E8-A8AE-46E47DAD7EF8" pageno="false" format="default"/> replaces Section 8.1 of <xref target="RFC7530" pageno="false" format="default"/>, entitled "Location Attributes".  This section has been reorganized and extended to explicitly allow the use of fs_locations to provide trunking-related information that appropriately interacts with the migration, replication and referral features of fs_locations.  Terminology used to describe the interactions is added.  </t><t><xref target="section:52768EEE-5B10-463C-AF8C-391D213769A5" pageno="false" format="default"/> updates Section 8.4 of <xref target="RFC7530" pageno="false" format="default"/>, entitled "Uses of Location Information".  This section comprises the bulk of the updates.  Each paragraph of Section 8.4 and its sub-sections has been reviewed to clarify the provision of trunking-related information using the fs_locations attribute.  <list style="symbols"><t><xref target="section:229B16B4-662D-4A1A-A162-EF1886F3DFD0" pageno="false" format="default"/> replaces the introductory material within Section 8.4 of <xref target="RFC7530" pageno="false" format="default"/>; i.e., the material within Section 8.4 exclusive of subsections.  </t><t><xref target="section:7E2CC373-4CE4-4D60-A0FB-8235047E46C9" pageno="false" format="default"/> is to be added as a new sub-section of Section 8.4 before the updated Section 8.4.1 of <xref target="RFC7530" pageno="false" format="default"/>.  In a consolidated document, it would appear as Section 8.4.1.  </t><t><xref target="section:75D48BA4-7400-4069-A5E8-47E4BAF8B669" pageno="false" format="default"/> is to be added as a new sub-section of Section 8.4 before the updated Section 8.4.1 of <xref target="RFC7530" pageno="false" format="default"/>.  In a consolidated document, it would appear as Section 8.4.2.  </t><t><xref target="section:AB6A7002-268E-4059-ABC0-9955D4F70180" pageno="false" format="default"/> replaces Section 8.4.1 of <xref target="RFC7530" pageno="false" format="default"/>, entitled "File System Replication".  In a consolidated document, it would appear as Section 8.4.3.  </t><t><xref target="section:C303CB29-5B44-4F63-9DE6-F6FF4D52B145" pageno="false" format="default"/> replaces Section 8.4.2 of <xref target="RFC7530" pageno="false" format="default"/>, entitled "File System Migration".  In a consolidated document, it would appear as Section 8.4.4.  </t><t><xref target="section:BF08A0E3-8AD6-48E1-AFBA-FE08D77DBDB5" pageno="false" format="default"/> is to be added as a new sub-section of Section 8.4 before Section 8.4.3 of <xref target="RFC7530" pageno="false" format="default"/>.  In a consolidated document, it would appear as Section 8.4.5, while the existing Section 8.3 would appear as Section 8.4.6.  </t></list> </t><t><xref target="section:CC288662-B4E9-41A0-A1A3-07E92849733A" pageno="false" format="default"/> replaces Section 8.5 of <xref target="RFC7530" pageno="false" format="default"/>, entitled "Location Entries and Server Identity".  The last paragraph of the existing section has been removed.  </t></list> </t>
      <section title="Updated Section 8.1 of [RFC7530],         entitled &quot;Location Attributes&quot;" anchor="section:D521608C-EFBB-46E8-A8AE-46E47DAD7EF8" toc="default">
        <t>The fs_locations attribute allows specification of file system locations where the data corresponding to a given file system may be accessed.  This attribute represents such file system instances as a server address target (as either a DNS host name representing one or more network addresses, or a single literal network address) together with the path of that file system within the associated single-server namespace.  Individual fs_locations entries can express trunkable addresses, locations of file system replicas on other servers, migration targets, or pure referrals.  </t>
        <t>We introduce the following terminology: <list style="symbols"><t>Trunking is a situation in which multiple network addresses are connected to the same NFS server.  Network addresses connected to the same NFS server instance are said to be server-trunkable.  </t><t>Trunking detection refers to ways of confirming that two distinct network addresses are connected to the same NFSv4 server instance.  </t><t>Trunking discovery is a process by which a client using one network address can obtain other candidate addresses that are server-trunkable with it.  </t></list> </t>
        <t>Regarding terminology relating to GETATTR attributes used in trunking discovery and other multi-server namespace features: <list style="symbols"><t>Location attributes include only the fs_locations GETATTR attribute.  </t><t>Location entries (fs_location4, defined in <xref target="RFC7530" pageno="false" format="default"/> Section 2.2.6) are the individual file system locations in the fs_locations attribute (defined in <xref target="RFC7530" pageno="false" format="default"/> Section 2.2.7).  A file system location entry designates a set of network addresses to which clients may establish connections.  The entry may designate multiple such addresses because the server host name may map to multiple network addresses, and because multiple connection types may be used to communicate with each specified network address.  Such addresses provide multiple ways of connecting to a single server.  <vspace blankLines="1"/> Clients use the NFSv4.0 trunking detection mechanism <xref target="RFC7931" pageno="false" format="default"/> to confirm that such addresses are connected to the same server.  The client can ignore non-confirmed trunking relationships and treat the corresponding addresses as connected to different servers.  </t><t>File system location elements are derived from file system location entries.  If a file system location entry specifies a network address, there is only a single corresponding location element.  When a file system location entry contains a host name, the client resolves the hostname, producing one file system location element for each of the resulting network addresses.  Issues regarding the trustworthiness of hostname resolutions are further discussed in <xref target="section:D3A9F015-D315-4D23-996F-F0D8DFAA5662" pageno="false" format="default"/> of the current document.  </t><t>All file system location elements consist of a file system location address, which is the network address of an interface to a server, and an fs_name, which is the location of the file system within the server's pseudo-fs.  </t><t>If the server has no pseudo-fs and only a single exported file system at the root filehandle, the fs_name may be empty.  </t></list> </t>
      </section>
      <section title="Updates to Section 8.4 of [RFC7530],         entitled &quot;Uses of Location Information&quot;" anchor="section:52768EEE-5B10-463C-AF8C-391D213769A5" toc="default">
        <t>The subsections below provide replacement sections for existing sections within Section 8.4 of <xref target="RFC7530" pageno="false" format="default"/> or new sub-sections to be added to that section.  </t>
        <section title="Updated Introduction to Section 8.4 of [RFC7530],         entitled &quot;Uses of Location Information&quot;" anchor="section:229B16B4-662D-4A1A-A162-EF1886F3DFD0" toc="default">
          <t>Together with the possibility of absent file systems, the file system location-bearing attribute fs_locations provides a number of important facilities that enable reliable, manageable, and scalable data access.  </t>
          <t>When a file system is present on the queried server, this attribute can provide a set of locations that clients may use to access the file system.  In the event that server failure, communications problems, or other difficulties make continued access to the file system impossible or otherwise impractical, the returned information provides alternate locations that enable continued access to the file system.  Provision of such alternative file system locations is referred to as "replication".  </t>
          <t>When alternative file system locations are provided, they may represent distinct physical copies of the same file system data or separate NFS server instances that provide access to the same physical file system.  Another possible use of the provision of multiple file system location entries is trunking, wherein the file system location entries do not in fact represent different servers but rather are distinct network paths to the same server.  </t>
          <t>A client may use file system location elements simultaneously to provide higher-performance access to the target file system.  This can be done using trunking, although the use of multiple replicas simultaneously is possible.  To enable simultaneous access, the client utilizes trunking detection and/or discovery, further described in <xref target="section:7E2CC373-4CE4-4D60-A0FB-8235047E46C9" pageno="false" format="default"/> of the current document, to determine a set of network paths that are server-trunkable with the one currently being used to access the file system.  Once this determination is made, requests may be routed across multiple paths, using the existing state management mechanism.  </t>
          <t>Multiple replicas may also be used simultaneously, typically used when accessing read-only datasets.  In this case, each replica requires its own state management.  The client performs multiple file opens to read the same file content from multiple replicas.  </t>
          <t>When a file system is present and subsequently becomes absent, clients can be given the opportunity to have continued access to their data at an alternative file system location.  Transfer of the file system contents to the new file system location is referred to as "migration".  The client's responsibilities in dealing with this transition depend on the specific nature of the new access path as well as how and whether data was in fact migrated.  See Sections <xref target="section:C303CB29-5B44-4F63-9DE6-F6FF4D52B145" format="counter" pageno="false"/> and <xref target="section:BF08A0E3-8AD6-48E1-AFBA-FE08D77DBDB5" format="counter" pageno="false"/> of the current document for details.  </t>
          <t>The fs_locations attribute can designate one or more remote file system locations in place of an absent file system.  This is known as a "referral".  A particularly important case is that of a "pure referral", in which the absent file system has never been present on the NFS server.  Such a referral is a means by which a file system located on one server can redirect clients to file systems located on other servers, thus enabling the creation of a multi-server namespace.  </t>
          <t>Because client support for the fs_locations attribute is OPTIONAL, a server may (but is not required to) take action to hide migration and referral events from such clients by acting as a proxy, for example.  </t>
        </section>
        <section title="New Sub-section of Section 8.4 of [RFC7530],         to be entitled &quot;Trunking Discovery and Detection&quot;" anchor="section:7E2CC373-4CE4-4D60-A0FB-8235047E46C9" toc="default">
          <t>Trunking is a situation in which multiple distinct network addresses are associated with the same NFS server instance.  As a matter of convenience, we say that two network addresses connected to the same NFS server instance are server-trunkable.  Section 5.4 of <xref target="RFC7931" pageno="false" format="default"/> explains why NFSv4 clients need to be aware of NFS server identity to manage lease and lock state effectively when multiple connections to the same server exist.  </t>
          <t>Trunking detection refers to a way for an NFSv4 client to confirm that two independently acquired network addresses are connected to the same NFSv4 server.  Section 5.8 of <xref target="RFC7931" pageno="false" format="default"/> describes an OPTIONAL means by which it can be determined if two network addresses correspond to the same NFSv4.0 server instance.  Without trunking detection, an NFSv4.0 client has no other way to confirm that two network addresses are server-trunkable.  </t>
          <t>In the particular context of NFS version 4.0, trunking detection requires that the client support the Uniform Client ID String approach (UCS), described in Section 5.6 of <xref target="RFC7931" pageno="false" format="default"/>.  Any NFSv4.0 client that supports migration or trunking detection needs to present a Uniform Client ID String to all NFSv4.0 servers.  If it does not do so, it will be unable to perform trunking detection.  </t>
          <t>Trunking discovery is the process by which an NFSv4 client using a host name or one of an NFSv4 server's network addresses can obtain other candidate network addresses that are trunkable with it; i.e., a set of addresses that might be connected to the same NFSv4 server instance.  An NFSv4.0 client can discover server-trunkable network addresses in a number of ways: <list style="symbols"><t>An NFS server's host name is provided either at mount time or in a returned file system location entry.  A DNS query of this host name can return more than one network address.  The returned network addresses are candidates for trunking.  </t><t>Location entries returned in an fs_locations attribute can specify network addresses.  These network addresses are candidates for trunking.  </t></list> When there is a means of trunking detection available, an NFSv4.0 client can confirm that a set of network addresses correspond to the same NFSv4.0 server instance and thus any of them can be used to access that server.  </t>
        </section>
        <section title="New Sub-section of Section 8.4 of [RFC7530],         to be entitled &quot;Location Attributes and Connection Type Selection&quot;" anchor="section:75D48BA4-7400-4069-A5E8-47E4BAF8B669" toc="default">
          <t>NFS version 4.0 may be implemented using a number of different types of connections: <list style="bullets"><t>Stream connections may be used to provide RPC service, as described in <xref target="RFC5531" pageno="false" format="default"/>.  </t><t>RDMA-capable connections may be used to provide RPC service, as	described in <xref target="RFC8166" pageno="false" format="default"/>.  </t></list> </t>
          <t>Because of the need to support multiple connections, clients face the issue of determining the proper connection type to use when establishing a connection to a server network address.  The fs_locations attribute provides no information to support connection type selection.  As a result, clients supporting multiple connection types need to attempt to establish a connection on various connection types allowing it to determine, via a trial-and-error approach, which connection types are supported.  </t>
          <t>If a client strongly prefers one connection type, it can perform these attempts serially in order of declining preference.  Once there is a successful attempt, the established connection can be used.  Note that with this approach, network partitions can result in a sequence of long waits for a successful connection.  </t>
          <t>To avoid waiting when there is at least one viable network path available, simultaneous attempts to establish multiple connection types are possible.  Once a viable connection is established, the client discards less-preferred connections.  </t>
        </section>
        <section title="Updated Section 8.4.1 of [RFC7530],         entitled &quot;File System Replication and Trunking&quot;" anchor="section:AB6A7002-268E-4059-ABC0-9955D4F70180" toc="default">
          <t>On first access to a file system, the client should obtain the value of the set of alternative file system locations by interrogating the fs_locations attribute.  Trunking discovery and/or detection can then be applied to the file system location entries to separate the candidate server-trunkable addresses from the replica addresses that provide alternative locations of the file system.  Server-trunkable addresses may be used simultaneously to provide higher performance through the exploitation of multiple paths between client and target file system.  </t>
          <t>In the event that server failures, communications problems, or other difficulties make continued access to the current file system impossible or otherwise impractical, the client can use the alternative file system locations as a way to maintain continued access to the file system.  See <xref target="section:BF08A0E3-8AD6-48E1-AFBA-FE08D77DBDB5" pageno="false" format="default"/> of the current document for more detail.  </t>
        </section>
        <section title="Updated Section 8.4.2 of [RFC7530],         entitled &quot;File System Migration&quot;" anchor="section:C303CB29-5B44-4F63-9DE6-F6FF4D52B145" toc="default">
          <t>When a file system is present and becomes absent, clients can be given the opportunity to have continued access to their data at an alternative file system location specified by the fs_locations attribute.  Typically, a client will be accessing the file system in question, get an NFS4ERR_MOVED error, and then use the fs_locations attribute to determine the new location of the data.  See <xref target="section:BF08A0E3-8AD6-48E1-AFBA-FE08D77DBDB5" pageno="false" format="default"/> of the current document for more detail.  </t>
          <t>Such migration can help provide load balancing or general resource reallocation.  The protocol does not specify how the file system will be moved between servers.  It is anticipated that a number of different server-to-server transfer mechanisms might be used, with the choice left to the server implementer.  The NFSv4 protocol specifies the method used to communicate the migration event between client and server.  </t>
          <t>When the client receives indication of a migration event via an NFS4ERR_MOVED error, data propagation to the destination server must have already occurred.  Once the client proceeds to access the alternate file system location, it must see the same data.  Where file systems are writable, a change made on the original file system must be visible on all migration targets.  Where a file system is not writable but represents a read-only copy (possibly periodically updated) of a writable file system, similar requirements apply to the propagation of updates.  Any change visible in the original file system must already be effected on all migration targets, to avoid any possibility that a client, in effecting a transition to the migration target, will see any reversion in file system state.  </t>
        </section>
        <section title="New Sub-section of Section 8.4 of [RFC7530],         to be entitled &quot;Interaction of Trunking, Migration, and Replication&quot;" anchor="section:BF08A0E3-8AD6-48E1-AFBA-FE08D77DBDB5" toc="default">
          <t>When the set of network addresses on a server change in a way that would affect a file system location attribute, there are several possible outcomes for clients currently accessing that file system.  NFS4ERR_MOVED is returned only when the server cannot satisfy a request from the client, whether because the file system has been migrated to a different server, is only accessible at a different trunked address on the same server, or some other reason.  In the cases 1 and 2 below, NFS4ERR_MOVED is not returned.  <list style="numbers"><t>When the list of network addresses is a superset of that previously in effect, there is no need for migration or any other sort of client adjustment.  Nevertheless, the client is free to use an additional address in the replacement list if that address provides another path to the same server.  Or, the client may treat that address as it does a replica, to be used if current server addresses become unavailable.  </t><t>When the list of network addresses is a subset of that previously in effect, immediate action is not needed if an address missing in the replacement list is not currently in use by the client.  The client should avoid using that address to access that file system in the future, whether the address is for a replica or an additional path to the server being used.  </t><t>When an address being removed is one of a number of paths to the current server, the client may continue to use it until NFS4ERR_MOVED is received.  This is not considered a migration event unless the last available path to the server has become unusable.  </t></list> </t>
          <t>When migration does occur, multiple addresses may be in use on the server previous to migration and multiple addresses may be available for use on the destination server.  </t>
          <t>With regard to the server in use, it may be that return of NFS4ERR_MOVED indicates that a particular network address is no longer to be used, without implying that migration of the file system to a different server is needed.  Clients should not conclude that migration has occurred until confirming that all network addresses known to be associated with that server are not usable.  </t>
          <t>It should be noted that the need to defer this determination is not absolute.  If a client is not aware of all network addresses for any reason, it may conclude that migration has occurred when it has not and treat a switch to a different server address as if it were a migration event.  This is harmless since the use of the same server via a new address will appear as a successful instance of Transparent State Migration.  </t>
          <t>Although significant harm cannot arise from this misapprehension, it can give rise to disconcerting situations.  For example, if a lock has been revoked during the address shift, it will appear to the client as if the lock has been lost during migration.  When such a lock is lost, it is the responsibility of the destination server to provide for its recovery via the use of an fs-specific grace period.  </t>
          <t>With regard to the destination server, it is desirable for the client to be aware of all valid network addresses that can be used to access the destination server.  However, there is no need for this to be done immediately.  Implementations can process the additional file system location elements in parallel with normal use of the first valid file system location entry found to access the destination.  </t>
          <t>Because a file system location attribute may include entries relating to the current server, the migration destination, and possible replicas to use, scanning for available network addresses that might be trunkable with addresses the client has already seen could potentially be a long process.  To keep this process as short as possible, servers that provide information about trunkable network paths are REQUIRED to place file system location entries that represent addresses usable with the current server or a migration target before those associated with replicas.  </t>
          <t>This ordering allows a client to cease scanning for trunkable file system location entries once it encounters a file system location element whose fs_name differs from the current fs_name, or whose address is not server-trunkable with the one it is currently using.  Although the possibility exists that a client might prematurely cease scanning for trunkable addresses when receiving a location attribute from an older server that does not follow the ordering constraint above, the harm is expected to be limited since such servers would not be expected to present information about trunkable server access paths.  </t>
        </section>
      </section>
      <section title="Updated Section 8.5 of [RFC7530],         entitled &quot;Location Entries and Server Identity Update&quot;" anchor="section:CC288662-B4E9-41A0-A1A3-07E92849733A" toc="default">
        <t>As mentioned above, a single file system location entry may have a server address target in the form of a DNS host name that resolves to multiple network addresses; it is also possible for multiple file system location entries to have their own server address targets that reference the same server.  </t>
        <t>When server-trunkable addresses for a server exist, the client may assume that for each file system in the namespace of a given server network address, there exist file systems at corresponding namespace locations for each of the other server-trunkable network addresses.  It may do this even in the absence of explicit listing in fs_locations.  Such corresponding file system locations can be used as alternative locations, just as those explicitly specified via the fs_locations attribute.  </t>
        <t>If a single file system location entry designates multiple server IP addresses, the client should choose a single one to use.  When two server addresses are designated by a single file system location entry and they correspond to different servers, this normally indicates some sort of misconfiguration.  The client should avoid using such file system location entries when alternatives are available.  When they are not, the client should pick one of the IP addresses and use it, without using others that are not directed to the same server.  </t>
      </section>
    </section>
    <section title="Updates to [RFC7530] Outside Section Eight" anchor="section:E6A08F19-9B5A-48D6-9B8A-D5945E749644" toc="default">
      <t>Since the existing description of NFS4ERR_MOVED in Section 13.1.2.4 of <xref target="RFC7530" pageno="false" format="default"/> does not take proper account of trunking, it needs to be modified by replacing the first two sentences of the description with the following material: <list style="none"><t>The file system that contains the current filehandle object cannot be accessed using the current network address.  It may be accessible using other network addresses connected to the same server, it may have been relocated to another server, or it may never have been present.  </t></list> </t>
    </section>
    <section title="Security Considerations" anchor="section:D3A9F015-D315-4D23-996F-F0D8DFAA5662" toc="default">
      <t>The Security Considerations section of <xref target="RFC7530" pageno="false" format="default"/> needs the additions below to properly address some aspects of trunking discovery, referral, migration, and replication.  <list style="none"><t>The possibility that requests to determine the set of network addresses corresponding to a given server might be interfered with or have their responses corrupted needs to be taken into account.  <list style="format o  "><t>When DNS is used to convert NFS server host names to network addresses and DNSSEC <xref target="RFC4033" pageno="false" format="default"/> is not available, the validity of the network addresses returned cannot be relied upon.  However, when the client uses RPCSEC_GSS <xref target="RFC7861" pageno="false" format="default"/> to access NFS servers, it is possible for mutual authentication to detect invalid server addresses.  Other forms of transport layer security (e.g., <xref target="RFC8446" pageno="false" format="default"/>) can also offer strong authentication of NFS servers.  </t><t>Fetching file system location information SHOULD be performed using RPCSEC_GSS with integrity protection, as previously explained in the Security Considerations section of <xref target="RFC7530" pageno="false" format="default"/>.  Making a request of this sort without using strong integrity protection permits corruption during transit of returned file system location information.  The client implementer needs to recognize that using such information to access an NFS server without use of RPCSEC_GSS (e.g., by using AUTH_SYS as defined in <xref target="RFC5531" pageno="false" format="default"/>) can result in the client interacting with an unverified network address that is posing as an NFSv4 server.  </t><t>Despite the fact that it is a REQUIREMENT of <xref target="RFC7530" pageno="false" format="default"/> that "implementations" provide "support" for the use of RPCSEC_GSS, it cannot be assumed that use of RPCSEC_GSS is always possible between any particular client-server pair.  </t><t>Returning only network addresses to a client that has no trusted DNS resolution service can hamper its ability to use RPCSEC_GSS.  </t></list> </t><t>Therefore an NFSv4 server SHOULD present file system location entries that correspond to file systems on other servers using only host names.  This enables the client to interrogate the fs_locations on the destination server to obtain trunking information (as well as replica information) using RPCSEC_GSS with integrity, validating the host name provided while assuring that the response has not been corrupted.  </t><t>When RPCSEC_GSS is not available on an NFS server, returned file system location information is subject to corruption during transit and cannot be relied upon.  In the case of a client being directed to another server after NFS4ERR_MOVED, this could vitiate the authentication provided by the use of RPCSEC_GSS on the destination.  Even when RPCSEC_GSS authentication is available on the destination, this server might validly represent itself as the server to which the client was erroneously directed.  Without a way to decide wether the server is a valid one, the client can only determine, using RPCSEC_GSS, that the server corresponds to the host name provided, with no basis for trusting that server.  The client should not use such unverified file system location entries as a basis for migration, even though RPCSEC_GSS might be available on the destination server.  </t><t>When a file system location attribute is fetched upon connecting with an NFSv4 server, it SHOULD, as stated above, be done using RPCSEC_GSS with integrity protection.  </t><t>When file system location information cannot be protected in transit, the client can subject it to additional filtering to prevent the client from being inappropriately directed.  For example, if a range of network addresses can be determined that assure that the servers and clients using AUTH_SYS are subject to appropriate constraints (such as physical network isolation and the use of administrative controls within the operating systems), then network adresses in this range can be used, with others discarded or restricted in their use of AUTH_SYS.  </t><t>When neither integrity protection nor filtering is possible, it is best for the client to ignore trunking and replica information or simply not fetch the file system location information for these purposes.  </t><t>To summarize considerations regarding the use of RPCSEC_GSS in fetching file system location information, consider the following recommendations for requests to interrogate location information, with interrogation approaches on the referring and destination servers arrived at separately: <list style="format o  "><t>The use of RPCSEC_GSS with integrity protection is RECOMMENDED in all cases, since the absence of integrity protection exposes the client to the possibility of the results being modified in transit.  </t><t>The use of requests issued without RPCSEC_GSS (e.g., using AUTH_SYS), while undesirable, might be unavoidable in some cases.  Where the use of returned file system location information cannot be avoided, it should be subject to filtering to eliminate untrusted network addresses.  The specifics will vary depending on the degree of network isolation and whether the request is to the referring or destination servers.  </t></list> Privacy considerations relating to uniform client strings (UCS) vs.  non-uniform client strings (non-UCS), discussed in Section 5.6 of <xref target="RFC7931" pageno="false" format="default"/>, are also applicable to their usage for trunking detection in NFS version 4.0.  </t></list> </t>
    </section>
    <section title="IANA Considerations" anchor="section:FA9F75CC-36FA-4F40-B2F1-8731EE27E300" toc="default">
      <t>This document does not require actions by IANA.  </t>
    </section>
  </middle>
  <back>
    <references title="Normative References">
      <reference anchor="RFC2119" target="https://www.rfc-editor.org/info/rfc2119">
        <front>
          <title>Key words for use in RFCs to Indicate Requirement Levels</title>
          <author initials="S." surname="Bradner" fullname="S. Bradner">
            <organization/>
          </author>
          <date year="1997" month="March"/>
          <abstract>
            <t>In many standards track documents several words are used to signify the requirements in the specification.  These words are often capitalized. This document defines these words as they should be interpreted in IETF documents.  This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.</t>
          </abstract>
        </front>
        <seriesInfo name="BCP" value="14"/>
        <seriesInfo name="RFC" value="2119"/>
        <seriesInfo name="DOI" value="10.17487/RFC2119"/>
      </reference>
      <reference anchor="RFC5531" target="https://www.rfc-editor.org/info/rfc5531">
        <front>
          <title>RPC: Remote Procedure Call Protocol Specification Version 2</title>
          <author initials="R." surname="Thurlow" fullname="R. Thurlow">
            <organization/>
          </author>
          <date year="2009" month="May"/>
          <abstract>
            <t>This document describes the Open Network Computing (ONC) Remote Procedure Call (RPC) version 2 protocol as it is currently deployed and accepted.  This document obsoletes RFC 1831.   [STANDARDS-TRACK]</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="5531"/>
        <seriesInfo name="DOI" value="10.17487/RFC5531"/>
      </reference>
      <reference anchor="RFC7530" target="https://www.rfc-editor.org/info/rfc7530">
        <front>
          <title>Network File System (NFS) Version 4 Protocol</title>
          <author initials="T." surname="Haynes" fullname="T. Haynes" role="editor">
            <organization/>
          </author>
          <author initials="D." surname="Noveck" fullname="D. Noveck" role="editor">
            <organization/>
          </author>
          <date year="2015" month="March"/>
          <abstract>
            <t>The Network File System (NFS) version 4 protocol is a distributed file system protocol that builds on the heritage of NFS protocol version 2 (RFC 1094) and version 3 (RFC 1813).  Unlike earlier versions, the NFS version 4 protocol supports traditional file access while integrating support for file locking and the MOUNT protocol. In addition, support for strong security (and its negotiation), COMPOUND operations, client caching, and internationalization has been added.  Of course, attention has been applied to making NFS version 4 operate well in an Internet environment.</t>
            <t>This document, together with the companion External Data Representation (XDR) description document, RFC 7531, obsoletes RFC 3530 as the definition of the NFS version 4 protocol.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="7530"/>
        <seriesInfo name="DOI" value="10.17487/RFC7530"/>
      </reference>
      <reference anchor="RFC7931" target="https://www.rfc-editor.org/info/rfc7931">
        <front>
          <title>NFSv4.0 Migration: Specification Update</title>
          <author initials="D." surname="Noveck" fullname="D. Noveck" role="editor">
            <organization/>
          </author>
          <author initials="P." surname="Shivam" fullname="P. Shivam">
            <organization/>
          </author>
          <author initials="C." surname="Lever" fullname="C. Lever">
            <organization/>
          </author>
          <author initials="B." surname="Baker" fullname="B. Baker">
            <organization/>
          </author>
          <date year="2016" month="July"/>
          <abstract>
            <t>The migration feature of NFSv4 allows the transfer of responsibility for a single file system from one server to another without disruption to clients.  Recent implementation experience has shown problems in the existing specification for this feature in NFSv4.0. This document identifies the problem areas and provides revised specification text that updates the NFSv4.0 specification in RFC 7530.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="7931"/>
        <seriesInfo name="DOI" value="10.17487/RFC7931"/>
      </reference>
      <reference anchor="RFC8166" target="https://www.rfc-editor.org/info/rfc8166">
        <front>
          <title>Remote Direct Memory Access Transport for Remote Procedure Call Version 1</title>
          <author initials="C." surname="Lever" fullname="C. Lever" role="editor">
            <organization/>
          </author>
          <author initials="W." surname="Simpson" fullname="W. Simpson">
            <organization/>
          </author>
          <author initials="T." surname="Talpey" fullname="T. Talpey">
            <organization/>
          </author>
          <date year="2017" month="June"/>
          <abstract>
            <t>This document specifies a protocol for conveying Remote Procedure Call (RPC) messages on physical transports capable of Remote Direct Memory Access (RDMA).  This protocol is referred to as the RPC-over- RDMA version 1 protocol in this document.  It requires no revision to application RPC protocols or the RPC protocol itself.  This document obsoletes RFC 5666.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="8166"/>
        <seriesInfo name="DOI" value="10.17487/RFC8166"/>
      </reference>
      <reference anchor="RFC8174" target="https://www.rfc-editor.org/info/rfc8174">
        <front>
          <title>Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words</title>
          <author initials="B." surname="Leiba" fullname="B. Leiba">
            <organization/>
          </author>
          <date year="2017" month="May"/>
          <abstract>
            <t>RFC 2119 specifies common key words that may be used in protocol  specifications.  This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the  defined special meanings.</t>
          </abstract>
        </front>
        <seriesInfo name="BCP" value="14"/>
        <seriesInfo name="RFC" value="8174"/>
        <seriesInfo name="DOI" value="10.17487/RFC8174"/>
      </reference>
    </references>
    <references title="Informative References">
      <reference anchor="RFC4033" target="https://www.rfc-editor.org/info/rfc4033">
        <front>
          <title>DNS Security Introduction and Requirements</title>
          <author initials="R." surname="Arends" fullname="R. Arends">
            <organization/>
          </author>
          <author initials="R." surname="Austein" fullname="R. Austein">
            <organization/>
          </author>
          <author initials="M." surname="Larson" fullname="M. Larson">
            <organization/>
          </author>
          <author initials="D." surname="Massey" fullname="D. Massey">
            <organization/>
          </author>
          <author initials="S." surname="Rose" fullname="S. Rose">
            <organization/>
          </author>
          <date year="2005" month="March"/>
          <abstract>
            <t>The Domain Name System Security Extensions (DNSSEC) add data origin authentication and data integrity to the Domain Name System.  This document introduces these extensions and describes their capabilities and limitations.  This document also discusses the services that the DNS security extensions do and do not provide.  Last, this document describes the interrelationships between the documents that collectively describe DNSSEC.  [STANDARDS-TRACK]</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="4033"/>
        <seriesInfo name="DOI" value="10.17487/RFC4033"/>
      </reference>
      <reference anchor="RFC5661" target="https://www.rfc-editor.org/info/rfc5661">
        <front>
          <title>Network File System (NFS) Version 4 Minor Version 1 Protocol</title>
          <author initials="S." surname="Shepler" fullname="S. Shepler" role="editor">
            <organization/>
          </author>
          <author initials="M." surname="Eisler" fullname="M. Eisler" role="editor">
            <organization/>
          </author>
          <author initials="D." surname="Noveck" fullname="D. Noveck" role="editor">
            <organization/>
          </author>
          <date year="2010" month="January"/>
          <abstract>
            <t>This document describes the Network File System (NFS) version 4 minor version 1, including features retained from the base protocol (NFS version 4 minor version 0, which is specified in RFC 3530) and protocol extensions made subsequently.  Major extensions introduced in NFS version 4 minor version 1 include Sessions, Directory Delegations, and parallel NFS (pNFS).  NFS version 4 minor version 1 has no dependencies on NFS version 4 minor version 0, and it is considered a separate protocol.  Thus, this document neither updates nor obsoletes RFC 3530.  NFS minor version 1 is deemed superior to NFS minor version 0 with no loss of functionality, and its use is preferred over version 0.  Both NFS minor versions 0 and 1 can be used simultaneously on the same network, between the same client and server.  [STANDARDS-TRACK]</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="5661"/>
        <seriesInfo name="DOI" value="10.17487/RFC5661"/>
      </reference>
      <reference anchor="RFC7861" target="https://www.rfc-editor.org/info/rfc7861">
        <front>
          <title>Remote Procedure Call (RPC) Security Version 3</title>
          <author initials="A." surname="Adamson" fullname="A. Adamson">
            <organization/>
          </author>
          <author initials="N." surname="Williams" fullname="N. Williams">
            <organization/>
          </author>
          <date year="2016" month="November"/>
          <abstract>
            <t>This document specifies version 3 of the Remote Procedure Call (RPC) security protocol (RPCSEC_GSS).  This protocol provides support for multi-principal authentication of client hosts and user principals to a server (constructed by generic composition), security label assertions for multi-level security and type enforcement, structured privilege assertions, and channel bindings.  This document updates RFC 5403.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="7861"/>
        <seriesInfo name="DOI" value="10.17487/RFC7861"/>
      </reference>
      <reference anchor="RFC8446" target="https://www.rfc-editor.org/info/rfc8446">
        <front>
          <title>The Transport Layer Security (TLS) Protocol Version 1.3</title>
          <author initials="E." surname="Rescorla" fullname="E. Rescorla">
            <organization/>
          </author>
          <date year="2018" month="August"/>
          <abstract>
            <t>This document specifies version 1.3 of the Transport Layer Security (TLS) protocol.  TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery.</t>
            <t>This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961.  This document also specifies new requirements for TLS 1.2 implementations.</t>
          </abstract>
        </front>
        <seriesInfo name="RFC" value="8446"/>
        <seriesInfo name="DOI" value="10.17487/RFC8446"/>
      </reference>
    </references>
    <section title="Section Classification" anchor="section:4EB27D51-743A-4C6A-97C2-19DA646AC319" toc="default">
      <t>All sections of the current document are considered explanatory with the following exceptions.  <list style="symbols"><t>Sections <xref target="section:D521608C-EFBB-46E8-A8AE-46E47DAD7EF8" format="counter" pageno="false"/> and <xref target="section:229B16B4-662D-4A1A-A162-EF1886F3DFD0" format="counter" pageno="false"/> are replacement sections.  </t><t><xref target="section:7E2CC373-4CE4-4D60-A0FB-8235047E46C9" pageno="false" format="default"/> is an additional section.  </t><t>Sections <xref target="section:AB6A7002-268E-4059-ABC0-9955D4F70180" format="counter" pageno="false"/> and <xref target="section:C303CB29-5B44-4F63-9DE6-F6FF4D52B145" format="counter" pageno="false"/> are replacement sections.  </t><t><xref target="section:BF08A0E3-8AD6-48E1-AFBA-FE08D77DBDB5" pageno="false" format="default"/> is an additional section.  </t><t><xref target="section:CC288662-B4E9-41A0-A1A3-07E92849733A" pageno="false" format="default"/> is a replacement section.  </t><t><xref target="section:E6A08F19-9B5A-48D6-9B8A-D5945E749644" pageno="false" format="default"/> is an editing section.  </t><t><xref target="section:D3A9F015-D315-4D23-996F-F0D8DFAA5662" pageno="false" format="default"/> is an additional section.  </t></list> </t>
    </section>
    <section title="Acknowledgments" anchor="section:54E008DF-EBB3-483D-B9CD-4C5B296A2745" numbered="no" toc="default">
      <t>The authors wish to thank Andy Adamson, who wrote the original version of this document.  All the innovation in this document is the result of Andy's work, while mistakes are best ascribed to the current authors.  </t>
      <t>The editor wishes to thank Greg Marsden for his support of this work, and Robert Thurlow for his review and suggestions.  </t>
      <t>Special thanks go to Transport Area Director Spencer Dawkins, NFSV4 Working Group Chairs Spencer Shepler and Brian Pawlowski, and NFSV4 Working Group Secretary Thomas Haynes for their ongoing support.  We are also grateful for the thorough review of this document by Benjamin Kaduk and Ben Campbell.  </t>
    </section>
  </back>
</rfc>
