Reporting Errors in NFSv4.2 via LAYOUTRETURN

Layout State Recovery When an MDS restarts, clients are provided a grace recovery period where they are allowed to recover any state that they had established. With open files, the client can send an OPEN operation (see ) with a claim type of CLAIM_PREVIOUS (see ). The client uses the RECLAIM_COMPLETE operation (see ) to notify the MDS that it is done reclaiming state. The NFSv4 flexible file layout type allows for the client to mirror files (see ). With client-side mirroring, it is important for the client to inform the MDS of any I/O errors encountered with one of the mirrors. This is the only way for the MDS to determine if one or more of the mirrors are corrupt and then repair the mirrors via resilvering (see ). The client can use LAYOUTRETURN (see ) and the ff_ioerr4 structure (see ) to inform the MDS of I/O errors. A problem arises when the MDS restarts and the client has errors it needs to report but cannot do so. requires that the client MUST stop using layouts. While the intent there is that the client MUST stop doing I/O to the storage devices, it is also true that the layout stateids are no longer valid. The LAYOUTRETURN needs a layout stateid to proceed, and the client cannot get a layout during grace recovery (see ) to recover layout state. As such, clients have no choice but to not recover files with I/O errors. In turn, the MDS MUST assume that the mirrors are inconsistent and pick one for resilvering. It is a MUST because even if the MDS can determine that the client did modify data during the outage, it MUST NOT assume those modifications were consistent. To fix this issue, the MDS MUST accept the anonymous stateid of all zeros (see ) for the lrf_stateid in LAYOUTRETURN (see ). The client can use this anonymous stateid to inform the MDS of errors encountered. The MDS can then accurately resilver the file by picking the mirror(s) that does not have any associated errors. During the grace period, if the client sends an lrf_stateid in the LAYOUTRETURN with any value other than the anonymous stateid of all zeros, then the MDS MUST respond with an error of NFS4ERR_GRACE (see ). After the grace period, if the client sends an lrf_stateid in the LAYOUTRETURN with a value of the anonymous stateid of all zeros, then the MDS MUST respond with an error of NFS4ERR_NO_GRACE (see ). Also, when the MDS builds the reply to the LAYOUTRETURN with an lrf_stateid with the value of the anonymous stateid of all zeros, it MUST NOT bump the seqid of the lorr_stateid. If the MDS detects that the layout being returned in the LAYOUTRETURN does not match the current mirror instances found for the file, then it MUST ignore the LAYOUTRETURN and resilver the file in question. The MDS MUST resilver any files that are neither explicitly recovered with a CLAIM_PREVIOUS nor have a reported error via a LAYOUTRETURN. The client has most likely restarted and lost any state.

When to Resilver A write intent occurs when a client opens a file and gets a LAYOUTIOMODE4_RW from the MDS. The MDS MUST track outstanding write intents, and when it restarts, it MUST track recovery of those write intents. The method that the MDS uses to track write intents is implementation specific, i.e., outside the scope of this document. The decision to resilver a file depends on how the client recovers the file before the grace period ends. If the client reclaims the file and reports no errors, the MDS MUST NOT resilver the file. If the client reports an error on the file, then the file MUST be resilvered. If the client does not reclaim or report an error before the grace period ends, then under the old behavior, the MDS MUST resilver the file. The resilvering process is broadly to:

fence the file (see ),
record the need to resilver,
release the write intent, and
once there are no write intents on the file, start the resilvering process.

The MDS MUST NOT resilver a file if there are clients with outstanding write intents, i.e., multiple clients might have the file open with write intents. As the MDS MUST track write intents, it MUST also track the need to resilver, i.e., if the MDS restarts during the grace period, it MUST restart the file recovery if it replays the write intent, or else it MUST start the resilvering if it replays the resilvering intent. Whether the MDS prevents all I/O to the file until the resilvering is done, forces all I/O to go through the MDS, or allows a proxy server to update the new data file as it is being resilvered is all an implementation choice. The constraint is that the MDS is responsible for the reconstruction of the data file and for the consistency of the mirrors. If the MDS does allow the client access to the file during the resilvering, then the client MUST have the same layout (set of mirror instances) after the MDS as before. One way that such a resilvering can occur is for a proxy server to be inserted into the layout. That server will be copying a good mirror instance to a new instance. As it gets I/O via the layout, it will be responsible for updating the copy it is performing. This requirement is that the proxy server MUST stay in the layout until the grace period is finished.

Version Mismatch Considerations The MDS has no expectations for the client to use this new functionality. Therefore, if the client does not use it, the MDS will function normally. If the client does use the new functionality and the MDS does not support it, then the MDS MUST reply with a NFS4ERR_BAD_STATEID to the LAYOUTRETURN. If the client detects a NFS4ERR_BAD_STATEID error in this scenario, it should fall back to the old behavior of not reporting errors.