FTPEXT Working Group R. Elz Internet Draft University of Melbourne Expiration Date: December 1997 P. Hethmon Hethmon Brothers June 1997 Extended Directory Listing and Restart Mechanism for FTP draft-ietf-ftpext-mlst-01.txt Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." To learn the current status of any Internet-Draft, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Abstract In order to overcome the problems caused by the undefined format of the current FTP LIST command output, a new command is needed to transfer standardized listing information from Server-FTP to Client- FTP. Commands to enable this are defined in this document. This proposal also extends the FTP protocol to allow character sets other than US-ASCII[1] by allowing the transmission of 8-bit characters and the recommended use of UTF-8[2] encoding. Much implemented, but long undocumented, mechanisms to permit restarts of interrupted data transfers in STREAM mode, are also included here. Elz & Hethmon [Page 1] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 This version contains changes as agreed at the 38th IETF, Memphis, April 1997. In particular: the SIZE MDTM and modified REST draft has been folded into this doc, (and the FEAT and OPTS commands have been moved out to a separate draft). Almost everything that was case dependent no longer is, MLST/MLSD no longer send replies on the control connection but the ancient STAT command has been revitalised. Files of type "link" are no longer reported, and the 'x' permission indicator has gone to meet its maker. The draft has also be editorially updated in some significant ways. This paragraph will be deleted from the final version of this document. Elz & Hethmon [Page 2] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 Table of Contents Status of this Memo ..................................... 1 Abstract ................................................ 1 1 Introduction ............................................ 4 2 Document Conventions .................................... 4 2.1 Basic Tokens ............................................ 4 2.2 Pathnames ............................................... 5 2.3 Times ................................................... 6 2.4 Server Replies .......................................... 7 3 File Modification Time (MDTM) ........................... 7 3.1 Syntax .................................................. 8 3.2 Error responses ......................................... 8 3.3 FEAT response for MTDM .................................. 8 4 File SIZE ............................................... 9 4.1 Syntax .................................................. 9 4.2 Error responses ......................................... 9 4.3 FEAT response for SIZE .................................. 10 5 Restart of Interrupted Transfer (REST) .................. 10 5.1 Restarting in STREAM Mode ............................... 10 5.2 ERROR RECOVER AND RESTART ............................... 11 5.3 Syntax .................................................. 11 5.4 FEAT response for REST .................................. 12 6 Machine Parseable Listings (MLST and MLSD) .............. 13 6.1 Format of MLST Request .................................. 13 6.2 Format of MLST Response ................................. 14 6.3 Filename encoding ....................................... 15 6.4 Format of Facts ......................................... 16 6.5 Standard Facts .......................................... 16 6.6 FEAT response for MLST .................................. 23 6.7 OPTS parameters for MLST ................................ 23 7 Interpretation of STAT command output ................... 24 7.1 FEAT response for STAT .................................. 24 8 Impact On Other FTP Commands ............................ 24 8.1 Impact on Pathnames and Filenames ....................... 25 9 Character sets and Internationalisation ................. 25 10 Security ................................................ 25 11 References .............................................. 25 Acknowledgements ........................................ 26 Editors' Addresses ...................................... 27 Elz & Hethmon [Page 3] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 1. Introduction This document amends the File Transfer Protocol (FTP) [6]. Four new commands are added: "SIZE", "MDTM", "MLST", and "MLSD". Two existing commands are modified, those are "REST" and "STAT". Of those, the "SIZE" and "MDTM" commands, and the modifications to "REST" have been in wide use for many years. The others are new. These commands allow a client to restart an interrupted transfer in transfer modes not previously supported in any documented way, and to obtain a directory listing in a machine friendly, predictable, format. 2. Document Conventions This document makes use of the document conventions defined in RFC2119 [10]. That provides the interpretation of capitalized imperative words like MUST, SHOULD, etc. This document also uses notation defined in STD 9 [6]. In particular, the terms "reply", "user", "NVFS", "file", "pathname", "FTP commands", "DTP", "user-FTP process", "user-PI", "user-DTP", "server-FTP process", "server-PI", "server-DTP", "mode", "type", "NVT", "control connection", "data connection", and "ASCII", are all used here as defined there. Syntax required is defined using the Augmented BNF defined in [9]. Some general ABNF definitions are required throughout the document, those will be defined later in this section. At first reading, it may be wise to simply recall that these definitions exist here, and skip to the next section. 2.1. Basic Tokens This document imports the definitions given in Appendix A of [9]. There definitions will be found for basic ABNF elements like ALPHA, DIGIT, SP, etc. To that, the following terms are added for use in this document. PCHAR = %x21-7e ; a printing character, ! to ~ TCHAR = PCHAR / SP / %x08 ; printing plus white space ltext = ALPHA / DIGIT / "," / "." / ":" / "!" / "@" / "#" / "$" / "%" / "^" / "&" / "(" / ")" / "-" / "_" / "+" / "?" / "/" / "\" / "'" / %x22 ; <"> -- double quote character Elz & Hethmon [Page 4] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 The PCHAR, TCHAR, and ltext types give basic character types from varying sub-sets of the ASCII character set for use in various commands and responses. token = 1*ltext A "token" is a string whose precise meaning depends upon the context in which it is used. In some cases it will be a value from a set of possible values maintained elsewhere. In others it might be a string invented by one party to an FTP conversation from whatever sources it finds relevant. error-response = error-code SP *TCHAR CRLF error-code = ("4" / "5") 2DIGIT Note that in ABNF, string literals are case insensitive. That convention is preserved in this document. However note that ALPHA, in particular, is case sensitive. That implies that a "token" is a case sensitive value. That implication is correct. 2.2. Pathnames Various FTP commands take pathnames as arguments, or return pathnames in responses. When the MLST command is supported, as indicated in the response to the FEAT command [11], pathnames are to be transferred in one of the following two formats. utf-8-name = raw = Which format is used is at the option of the user-PI or server-PI sending the pathname. UTF-8 encodings contain enough internal structure that it is always, in practice, possible to determine whether a UTF-8 or raw encoding has been used, in the cases where it matters. Note that ASCII is a subset of UTF-8. Unless otherwise specified, the pathname is terminated by the CRLF that terminates the FTP command, or by the CRLF that ends a reply. Any trailing spaces preceding that CRLF form part of the name. Exactly one space will precede the pathname from the preceding syntax element, any additional spaces form part of the pathname. See [4] for a fuller explanation of the character encoding issues. All implementations supporting MLST MUST support [4]. Implementations should also beware that the control connection uses Telnet NVT conventions [12], and that the Telnet IAC character, if part of a pathname, MUST be correctly escaped as defined by the Telnet protocol. Elz & Hethmon [Page 5] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 2.3. Times The syntax of a time value is: time-val = 12DIGIT [ "." 1*DIGIT ] The leading, mandatory, twelve digits are to be interpreted as, in order from the leftmost, four digits giving the year, with a range of 1000-9999, two digits giving the month of the year, with a range of 01-12, two digits giving the day of the month, with a range of 01-31, two digits giving the hour of the day, with a range of 00-23, two digits giving minutes past the hour, with a range of 00-59, and finally, two digits giving seconds past the minute, with a range of 00-60 (with 60 being used only at a leap second). Years in the tenth century, and earlier, cannot be expressed. This is not considered a serious defect of the protocol. [ Ed-Note: Should we permit 12*DIGIT (or maybe 12*13DIGIT) so times in the 101st century and beyond can be represented? ] The optional digits, which must be preceded by a period, give decimal fractions of a second. These may be given to whatever precision is appropriate to the circumstance, however implementations MUST NOT add precision to time-vals where that precision does not exist in the underlying value being transmitted. Symbolically, a time-val may be viewed as YYYYMMDDHHMMSS.sss The "." and subsequent digits are optional. Time values are always represented in UTC (GMT), and in the Gregorian calendar regardless of what calendar may have been in use at the date and time indicated at the location of the server-PI. The technical differences between GMT, UTC, UT1, UT2, etc, are not considered here. A server-FTP process should always use the same time reference, so the times it returns will be consistent. Clients are not expected to be time synchronised with the server, so the possible difference in times that might be reported by the different time standards is not considered important. Elz & Hethmon [Page 6] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 2.4. Server Replies Section 4.2 of [6] defines the format and meaning of replies by the server-PI to FTP commands from the user-PI. Those reply conventions are used here without change. Implementors should note that the ABNF syntax (which was not used in [6]) in this document, and other FTP related documents, sometimes shows replies using the one line format. Unless otherwise explicitly stated, that is not intended to imply that multi-line responses are not permitted. Implementors should assume that, unless stated to the contrary, any reply to any FTP command (including QUIT) may be of the multiline format described in [6]. Throughout this document, replies will be identified by the three digit code that is their first element. Thus the term "500 reply" means a reply from the server-PI using the three digit code "500". 3. File Modification Time (MDTM) The FTP command, MODIFICATION TIME (MDTM), can be used to determine when a file in the server NVFS was last modified. This command has existed in many FTP servers for many years, as an adjunct to the REST command for STEAM mode, thus is widely available. However, where supported the "mtime" fact which can be provided in the result from the new MLST command is recommended as a superior alternative. When attempting to restart a RETRieve, if the User FTP makes use of the MDTM command, it can check and see if the modification time of the source file is more recent than the modification time of the partially transferred file. If it is, then most likely the source file has changed and it would be unsafe to restart in the middle of the file transfer. When attempting to restart a STORe, the User FTP can use the MDTM command to discover the modification time of the partially transferred file. If it is older than the modification time of the file that is about to be STORed, then most likely the source file has changed and it would be unsafe to restart in the middle of the file transfer. Using MLST can provide this information, and much more, thus giving an even better indication that a file has changed, and that restarting a transfer would not give valid results. Note that this is applicable to any RESTart attempt, regardless of the mode of the file transfer. Elz & Hethmon [Page 7] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 3.1. Syntax The syntax for the MDTM command is: mdtm = "MdTm" SP ( utf-8-name / raw ) CRLF The server-PI will respond to the MDTM command with a 213 reply giving the last modification time of the file whose pathname was supplied, or an error response if the file does not exist, the modification time is unavailable, or some other error has occurred. mdtm-response = "213" SP time-val CRLF / error-response 3.2. Error responses Where the command is correctly parsed, but the modification time is not available, either because the pathname identifies no existing entity, or because the information is not available for the entity named, then a 550 reply should be sent. Where the command cannot be correctly parsed, a 500 or 501 reply should be sent, as specified in [6]. 3.3. FEAT response for MTDM When replying to the FEAT command [11], a FTP server process that supports the MTDM command MUST include a line containing the single word "MDTM" (in upper case, and without the quotes). That is, the response MUST be C> FEAT S> 211- S> ... S> MDTM S> ... S> 211 End The ellipses indicate placeholders where other features may be included, and are not required. The one space indentation of the feature lines is mandatory. Elz & Hethmon [Page 8] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 4. File SIZE The FTP command, SIZE OF FILE (SIZE), is used to obtain the transfer size of a file from the server-FTP process. That is, the exact number of bytes (or octets) which would be transmitted over the data connection should that file be transmitted. This value will change depending on the current STRUcture, MODE and TYPE of the data connection, or a data connection which would be created were one created now. Thus, the result of the SIZE command is dependent on the currently established STRU, MODE and TYPE parameters. The SIZE command returns how many bytes would be transferred if the file were to be transferred using the current transfer structure, mode and type. This command is normally used in conjunction with the RESTART (REST) command. The server-PI might need to read the partially transferred file, do any appropriate conversion, and count the number of bytes that would be generated when sending the file in order to correctly respond to this command. Estimates of the file transfer size MUST NOT be returned, only precise information is acceptable. 4.1. Syntax The syntax of the SIZE command is: size = "Size" SP ( utf-8-name / raw ) CRLF The server-PI will respond to the SIZE command with a 213 reply giving the transfer size of the file whose pathname was supplied, or an error response if the file does not exist, the size is unavailable, or some other error has occurred. The value returned is in a format suitable for use with the RESTART (REST) command for mode STREAM, provided the transfer mode and type are not altered. size-response = "213" SP 1*DIGIT CRLF / error-response 4.2. Error responses Where the command is correctly parsed, but the size is not available, either because the pathname identifies no existing entity, or because the entity named cannot be transferred in the current MODE and TYPE (or at all), then a 550 reply should be sent. Where the command cannot be correctly parsed, a 500 or 501 reply should be sent, as specified in [6]. Elz & Hethmon [Page 9] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 4.3. FEAT response for SIZE When replying to the FEAT command [11], a FTP server process that supports the SIZE command MUST include a line containing the single word "SIZE" (in upper case, and without the quotes). That is, the response MUST be C> FEAT S> 211- S> ... S> SIZE S> ... S> 211 End The ellipses indicate placeholders where other features may be included, and are not required. The one space indentation of the feature lines is mandatory. 5. Restart of Interrupted Transfer (REST) To avoid having to resend the entire file if the file is only partially transferred, both sides need some way to be able to agree on where in the data stream to restart the data transfer. The FTP specification [6] includes three modes of data transfer, Stream, Block and Compressed. In Block and Compressed modes, the data stream that is transferred over the data connection is formatted, allowing the embedding of restart markers into the stream. The sending DTP can include a restart marker with whatever information it needs to be able to restart a file transfer at that point. The receiving DTP can keep a list of these restart markers, and correlate them with how the file is being saved. To restart the file transfer, the receiver just sends back that last restart marker, and both sides know how to resume the data transfer. Note that there are some flaws in the description of the restart mechanism in RFC 959 [6], see section 4.1.3.4 of RFC 1123 [7] for the corrections. 5.1. Restarting in STREAM Mode In Stream mode, the data connection contains just a stream of unformatted bytes of data. Explicit restart markers thus cannot be inserted into the data stream, they would be indistinguishable from data. For this reason, the FTP specification [6] did not provide the ability to do restarts in stream mode. However, there is not really a need to have explicit restart markers in this case, as restart markers can be implied by the byte offset into the data stream. Elz & Hethmon [Page 10] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 If the data representation TYPE is IMAGE, and the STRUcture is File, for many systems the file will be stored exactly in the same format as it is sent across the data connection. It is then very easy for the receiver to determine how much data was previously received, and notify the sender the byte offset where the transfer should be restarted. In other representation types and structures more effort will be required, but it remains always possible to determine the byte offset with finite, but perhaps non-negligible, effort. In the worst case an FTP process may need to open a data connection to itself, set the appropriate transfer type and structure, and actually transmit the file, counting the transmitted bytes. If the user-FTP process is intending to restart a retrieve, it will directly calculate the restart marker, and send that information in the RESTart command. However, if the user-FTP process is intending to restart sending the file, it needs to be able to determine how much data was previously sent, and correctly received and saved. A new FTP command is needed to get this information. This is the purpose of the SIZE command, as documented in section 4. 5.2. ERROR RECOVER AND RESTART STREAM MODE transfers with FILE STRUcture may be restarted even though no restart marker has been transferred in addition to the data itself. This is done by perhaps using the SIZE command in combination with the RESTART (REST) command. When using TYPE ASCII or IMAGE, the SIZE command will return the number of bytes that would actually be transferred if the file were to be sent between the two systems. I.e. with type IMAGE, the SIZE normally would be the number of octets in the file. With type ASCII, the SIZE would be the number of characters in the file INCLUDING any characters that would be inserted during the CR-LF expansion. 5.3. Syntax The syntax for the REST command when the current transfer mode is STREAM is: rest = "Rest" SP 1*DIGIT CRLF The numeric value gives the number of octets of the immediately following transfer to not actually send, effectively causing the transmission to be restarted at a later point. The server-PI will respond to the REST command with a 350 reply, indicating that the REST parameter has been saved, and that another command, which should be either RETR or STOR, should then follow to complete the restart. Elz & Hethmon [Page 11] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 rest-response = "350" SP *TCHAR CRLF / error-response Server-FTP processes may permit transfer commands other than RETR and STOR, such as APPE and STOU, to complete a restart, however, this is not recommended. STOU (store unique) is undefined in this usage, as storing the remainder of a file into a unique filename is rarely going to be useful. If APPE (append) is permitted, it MUST act identically to STOR when a restart marker has been set. That is, in both cases, octets from the data connection are placed into the file at the location indicated by the restart marker value. An error-response will follow a REST command only when the server does not implement the command, or the restart marker value is syntactically invalid for the current transfer mode. That is, in STREAM mode, if something other than one or more digits appears in the parameter to the REST command. Any other errors, including such problems as restart marker out of range, should be reported when the following transfer command is issued. 5.4. FEAT response for REST Where a server-FTP process supports RESTart in STREAM mode, as specified here, it MUST include in the response to the FEAT command [11], a line containing exactly the string "REST STREAM" (in upper case, and without the quotes). Where REST is not supported at all, or supported only in block or compressed modes, the REST command MUST NOT be included in the FEAT response. Where required, the response MUST be C> FEAT S> 211- S> ... S> REST STREAM S> ... S> 211 End The ellipses indicate placeholders where other features may be included, and are not required. The one space indentation of the feature lines is mandatory. Elz & Hethmon [Page 12] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 6. Machine Parseable Listings (MLST and MLSD) The MLST and MLSD commands are intended to standardize the file and directory information returned by the Server-FTP process. These commands differ from the LIST and NLST commands in that the format of the replies is strictly defined although extensible. Two commands are defined, MLST which provides data about exactly the object named on its command line, and no others. MLSD on the other hand will list the contents of a directory if a directory is named, otherwise it is identical to MLST. In either case, if no object is named, the current directory is assumed. That will cause MLST to send a one line response, and MLSD to list the contents of the current directory. [ Ed-Note: An alternative would be to have MLSD fail if the argument is not a directory. Which is better? ] In the sequel only MLST will be described, other than as previously mentioned, MLSD is identical. The MLST and MLSD commands also extend the FTP protocol as presented in RFC 959 [6] and RFC 1123 [7] to allow that transmission of 8-bit data. Note this is not specifying character sets which are 8-bit, but specifying that FTP implementations are to specifically allow the transmission and reception of 8-bit bytes over the control connection. That is, all 256 possible octet values are permitted. The MLST command allows both UTF-8/Unicode and "raw" forms as arguments, and in responses. 6.1. Format of MLST Request The MLST and MLSD commands each allow a single optional argument. This argument may be either a directory name or a filename. If a directory name is given then MLSD must return a listing of the contents of the named directory. If the argument is not a directory, and the command is MLSD, and in all cases for MLST, then only a single fact line containing the information about the named file or directory shall be returned. If no argument is given then MLSD must return a listing of the contents of the current working directory, and MLST must return a listing giving information about the current working directory itself. If the Client-FTP sends an invalid argument, the Server-FTP MUST reply with an error code of 501. Elz & Hethmon [Page 13] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 The syntax for the MLST command is: mlst = "MLst" [ SP ( utf-8-name / raw ) ] CRLF utf-8-name = raw = 6.2. Format of MLST Response The format of a response to the MLST command is as follows: mlst-response = initial-response final-response initial-response = "150" [ SP response-message ] CRLF / error-response response-message = *TCHAR final-response = "226" SP response-message CRLF data-response = *( entry CRLF ) entry = [ facts ] SP ( utf-8-name / raw ) facts = fact *( ";" fact ) fact = factname "=" value factname = 1*ltext value = 1*ltext end-token = "End" Upon receipt of a MLST or MLSD command, the server will verify the parameter, and if invalid return an error-response. If valid, the server will open a data connection as indicated in section 3.2 of RFC959. If that fails, the server will return an error-response. If all is OK, the server will return the initial-response, send the appropriate data-response over the new data connection, close that connection, and then send the final-response. The data connection opened for a MLST or MLSD response shall be connection as if the "TYPE L 8", "MODE S", and "STRU F" commands had been given, whatever FTP transfer type, mode and structure had actually been set, and without causing those settings to be altered for future commands. That is, this transfer type shall be set for the duration of the data connection established for this command only. While the content of the data sent can be viewed as a series of lines, implementations should note that there is no maximum line length defined. Implementations should be prepared to deal with arbitrarily long lines. The facts part of the specification would contain a series of "file facts" about the file/directory named on the same line. Typical information to be presented would include file size, last modification time, creation time, a unique identifier, and a file/directory flag. Elz & Hethmon [Page 14] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 The complete format for a successful reply to the MLST command would be: facts SP utf-8-name CRLF facts SP utf-8-name CRLF facts SP utf-8-name CRLF ... Note that the format is intended for machine processing, not human viewing, and as such the format is very rigid. Implementations must not vary the format by, for example, inserting extra spaces for readability, or inserting blank lines, or in any other way alter this format. Exactly one space is always required after the set of facts (which may be empty). More spaces may be present on a line if, and only if, the file name presented contains significant spaces. The set of facts must not contain any spaces anywhere inside it. 6.3. Filename encoding A FTP implementation using the MLST command must be 8-bit clean. This is necessary in order to transmit UTF-8 encoded filenames. This specification recommends the use of UTF-8 encoded filenames. FTP implementations SHOULD use UTF-8 whenever possible to encourage the maximum interoperability. Filenames are not restricted to UTF-8, however treatment of arbitrary character encodings is not specified by this standard. Applications are encouraged to treat non-UTF-8 encodings of filenames as octet sequences. Note that this encoding is unrelated to that of the contents of the file, even if the file contains character data. Further information about filename encoding for FTP may be found in "Internationalization of the File Transfer Protocol" [4]. 6.3.1. Notes about the Filename The filename returned in the MLST response should be the same name as was specified in the MLST command. Filenames returned in the output from an MLSD command should be unqualified names within the directory named. That is, the directory named in the MLSD command should not appear as a component of the filenames returned. If the server-FTP process is able, it MAY return in the MLSD response, an entry whose type is "cdir", which names the directory from which the contents of the listing were obtained. Where more than one name exists, multiple of these entries may be returned. The Elz & Hethmon [Page 15] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 server MUST return type "cdir" names in a format such that if the user-PI takes a name of type "cdir", and appends a name of type which is not "cdir", and which appeared in the same MLSD response as the type=cdir name, with no intervening separators, then a valid pathname will be produced, using which the user-PI can reference the file indicated from its current working directory. Alternatively, the user-PI can issue a CWD command ([6]) giving the name of type "cdir", and from that point reference the files returned in the MLSD response from which the cdir was obtained by using the filename components of the listing. Once having attempted any CWD command however, it is no longer guaranteed that a file can be referenced by the combination of type "cdir" and other names, whether using CWD or name concatenation. 6.3.2. Examples Once upon a (future) time, examples existed here. 6.4. Format of Facts The "facts" for a file in a reply to a MLST command consist of information about that file. The facts are a series of keyword=value pairs separated by a semi-colon (";") character. The complete series of facts may not contain the space character. A sample of a typical series of facts would be: (spread over two lines for presentation only) size=4161;lang=en-us;modify=19970214165800;create=19961001124534; type=file;x.myfact=foo,bar 6.5. Standard Facts This document defines a standard set of facts as follows: size -- Size in bytes modify -- Last modification time create -- Creation time type -- Entry type unique -- Unique id of file/directory perm -- File permissions, whether read, write, execute is allowed for the login id. lang -- Language of the filename per IANA[5] registry. media-type -- MIME media-type of file contents per IANA registry. charset -- Character set per IANA registry (if not UTF-8) Elz & Hethmon [Page 16] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 Fact names are case-insensitive. Size, size, SIZE, and SiZe are the same fact. Further operating system specific keywords could be specified by using the IANA operating system name as a prefix (examples only): OS/2.ea -- OS/2 extended attributes MACOS.rf -- MacIntosh resource forks UNIX.mode -- Unix file modes (permissions) Implementations may define keywords for experimental, or private use. All such keywords MUST begin with the two character sequence "x.". As type names are case independent, "x." and "X." are equivalent. For example: x.ver -- Version information x.desc -- File description x.type -- File type 6.5.1. The type Fact The type fact needs a special description. Part of the problem with current practices is deciding when a file is a directory. If it is a directory, is it the current directory, a regular directory, or a parent directory? The MLST specification makes this unambiguous using the type fact. The type fact given specifies information about the object listed on the same line of the MLST response. Five values are possible for the type fact: file -- a file entry cdir -- the current directory pdir -- the parent directory dir -- a directory or sub-directory OSname=type -- an OS or file system dependent file type The syntax is defined to be: type-fact = type-label "=" type-val type-label = "Type" type-val = "File" / "cdir" / "pdir" / "dir" / sys-depend-type sys-depend-type = "OS" 1*ltext "=" 1*ltext 6.5.1.1. type=file The presence of the type=file fact indicates the listed entry is a file containing non-system data. That is, it may be transferred from Elz & Hethmon [Page 17] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 one system to another of quite different characteristics, and perhaps still be meaningful. 6.5.1.2. type=cdir The type=cdir fact indicates the listed entry is the pathname of the directory whose contents are listed. The value of this entry (the filename part) plus the value of a type=file entry from the same MLSD listing together should represent a complete pathname suitable for a RETR command. The value for the type=cdir entry should include any necessary system delimiters used between path components. An example would be the forward slash "/" on a UNIX(TM) system, or a back slash "\" on an OS/2 or Windows system. 6.5.1.3. type=dir If present, the type=dir entry is the name of a directory. When executed with the current directory in the same place in the NVFS as it was when the MLST or MLSD command was issued, a CWD with its argument being the formed by appending the name with type=pdir to a name with type-cdir should succeed (assuming the user has the appropriate access rights). 6.5.1.4. type=pdir If present, which will occur only in the response to a MLSD command, the type=pdir entry represents a pathname of the parent directory of the listed directory. As well as having the properties of a type=dir, a CWD command with the appropriate value should change the user to the parent directory of the listed directory. User-FTP processes should note not all responses will include this information. 6.5.2. The unique Fact The unique fact is used to present a unique identifier for a file or directory in the NVFS accessed via a server-FTP process. The value of this fact should be the same for any number of filenames that refer to the same underlying file. The fact should have different values for names which reference distinct files. The mapping between files, and unique fact tokens should be maintained, and remain consistent, for at least the lifetime of the control connection from user-PI to server-PI. Elz & Hethmon [Page 18] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 unique-fact = "Unique" "=" token This fact would be expected to be used by Server-FTPs whose host system allows things such as symbolic links so that the same file may be represented in more than one directory on the server. The value of the unique fact should be considered an opaque string for comparison purposes. The only conclusion that should be drawn is that if two different names each have the same value for the unique fact, they refer to the same underlying object. 6.5.3. The modify Fact The modify fact is used to determine the last time the content of the file (or directory) indicated was modified. Any change of substance to the file should cause this value to alter. Changes which merely alter access rights, and similar attributes, which do not affect the content or meaning of the data contained in a file, should not cause this value to alter. For directories, this value should alter whenever a change occurs to the directory such that different filenames would (or might) be included in MLSD output of that directory. modify-fact = "Modify" "=" time-val 6.5.4. The create Fact The create fact indicates when a file, or directory, was first created. Exactly what "creation" is for this purpose is not specified here, and may vary from server to server. About all that can be said about the value returned is that it can never indicate a later time than the mtime fact. create-fact = "Create" "=" time-val Implementation Note: Implementors of this fact on UNIX(TM) systems should note that the unix "stat" "st_ctime" field does not give creation time, and that unix filesystems do not record creation time at all. Unix (and POSIX) implementations will normally not include this fact. 6.5.5. The perm Fact The perm fact is used to indicate access rights the current FTP user has over the object listed. Its value is always an unordered sequence of alphabetic characters. Elz & Hethmon [Page 19] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 perm-fact = "Perm" "=" pvals pvals = *ALPHA There are ten permission indicators currently defined. Many are meaningful only when used with a particular type of object. The ten characters currently defined are: a c d e f l m p r w [ Ed-Note: should these be case dependent or independent ? ] The "a" permission applies to objects of type=file, and indicates that the APPE (append) command may be applied to the file named. The "c" permission applies to objects of type=dir (and type=pdir, type=cdir). It indicates that files may be created in the directory named. That is, that a STOU command is likely to succeed, and that STOR and APPE commands might succeed if the file named did not previously exist, but is to be created in the directory object that has the "c" permission. It also indicates that the RNTO command is likely to succeed for names in the directory. The "d" permission applies to all types. For type=file it indicates that the file may be deleted, that is, that the DELE command may be applied to it. For the directory types it indicates that (some) files in the directory may be deleted. The "e" permission applies to the directory types. When set on an object of type=dir, type=cdir, or type=pdir it indicates that a CWD command naming the object should succeed, and the user should be able to enter the directory named. For type=pdir it also indicates that the CDUP command should succeed. The "f" permission for objects indicates that the object named may be renamed - that is, may be the object of an RNFR command. The "l" permission applies to the directory file types, and indicates that the listing commands, LIST, NLST, and MLSD may be applied to the directory in question, and that MLST, LIST, NLST, and STAT may be applied to objects in the directory. The "m" permission applies to directory types, and indicates that the MKD command may be used to create a new directory within the directory under consideration. The "p" permission applies to directory types, and indicates that the RMD command may be used to remove (purge) the directory named. Elz & Hethmon [Page 20] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 The "r" permission applies to type=file objects, and indicates that the RETR command may be applied to that object. The "w" permission applies to type=file objects, and indicates that the STOR command may be applied to the object named. Note: That a permission indicator is set can never imply that the appropriate command is guaranteed to work - just that it might. Other system specific limitations, such as limitations on available space for storing files, may cause an operation to fail, where the permission flags may have indicated that it was likely to succeed. The permissions are a guide only. Implementation note: The permissions are described here as they apply to FTP commands. They may not map easily into particular permissions available on the server's operating system. Servers are expected to synthesize these permission bits from the permission information available from operating system. For example, to correctly determine whether the "p" permission bit should be set on a directory for a server running on the UNIX(TM) operating system, the server should check that the directory named is empty, and that the user has write permission on both the directory under consideration, and its parent directory. Some systems may have more specific permissions than those listed here, such systems should map those to the flags defined as best they are able. Other systems may have only more broad access controls. They will generally have just a few possible permutations of permission flags, however they should attempt to correctly represent what is permitted. 6.5.6. The lang Fact The lang fact describes the natural language of the filename for use in display purposes. Values used here should be taken from the language registry of the IANA. lang-fact = "Lang" "=" token Server-FTP implementations MUST NOT guess language values. Language values must be determined in an unambiguous way such as filesystem tagging of language or by user configuration. Elz & Hethmon [Page 21] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 6.5.7. The size Fact The size should always reflect the approximate size of the file. This should be as accurate as the server can make it, without going to extraordinary lengths, such as reading the entire file. The size is expressed in units of octets. Given limitations in some systems, Client-FTP implementations must understand this size may not be precise and may change between the time of a MLST and RETR operation. Clients that need highly accurate size information for some particular reason should use the SIZE command as defined in section 4. The most common need for this accuracy is likely to be in conjunction with the REST command described in section 5. The size fact, on the other hand, should be used for purposes such as indicating to a human user the approximate size of the file to be transferred, and perhaps to give an idea of expected transfer completion time. size-fact = "Size" "=" 1*DIGIT 6.5.8. The media-type Fact The media-type fact represents the IANA media type of the file. The list of values used must follow the guidelines set by the IANA registry. media-type = "Media-Type" "=" Server-FTP implementations MUST NOT guess media type values. Media type values must be determined in an unambiguous way such as filesystem tagging of media-type or by user configuration. 6.5.9. The charset Fact The charset fact represents the IANA character set name for the encoded names in a MLST response. The default character set is UTF-8 unless specified otherwise. FTP implementations SHOULD use UTF-8 if possible to encourage maximum interoperability. charset-type = "Charset" "=" token Elz & Hethmon [Page 22] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 6.6. FEAT response for MLST When responding to the FEAT command, a server-FTP process that supports MLST, and related commands, MLSD, and the modified STAT, plus internationalisation of pathnames, MUST indicate that this support exists. It does this by including a MLST feature line. As well as indicating the basic support, the MLST feature line indicates which MLST facts are available from the server, and which of those will be returned if no subsequent "OPTS MLST" command is sent. mlst-feat = SP "MLST" [SP factlist] CRLF factlist = factname ["*"] *( ";" factname ["*"] ) The initial space shown in the mlst-feat response is that required by the FEAT command, two spaces are not permitted. If no factlist is given, then the server-FTP process is indicating that it supports MLST, but implements no facts. Only pathnames can be returned. This would be a minimal MLST implementation, and useless for most practical purposes. Where the factlist is present, the factnames included indicate the facts supported by the server. Where the optional asterisk appears after a factname, that fact will be included in MLST format responses, until an "OPTS MLST" is given to alter the list of facts returned. [ Ed-Note: Perhaps the sense of the "*" should be reversed? That is, make the asterisk indicate those facts not returned? ] 6.7. OPTS parameters for MLST For the MLST command, the Client-FTP may specify a list of facts it wishes to be returned in all subsequent MLST commands until another OPTS MLST command is sent. The format is specified by: mlst-opts = "OPTS" SP "MLST" [ SP factname *(";" factname) ] By sending the "OPTS MLST" command, the client requests the server to include only the facts listed as arguments to the command in subsequent output from MLST commands. Facts not included in the "OPTS MLST" command must not be returned by the server. Facts that are included should be returned for each entry returned from the MLST command where they apply. Facts requested that are not supported, or which are inappropriate to the file or directory being listed should simply be omitted from the MLST output. This is not an error. Note that where no factname arguments are present, the client is requesting that only the file names be returned. In this case, and in any other case where no facts are included in the result, the Elz & Hethmon [Page 23] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 space that separates the fact names and their values from the file name is still required. That is, the first character of the output line will be a space, and the file name will start immediately thereafter. Note, there is no "OPTS MLSD" command, the fact names set with the TS MLST" command apply to both MLST and MLSD commands, and to the STAT command when used with a file name argument and no transfer in progress. 7. Interpretation of STAT command output Where a server-FTP process supports the MLST and MLST commands, the format of the output from the STAT command ([6] section 4.1.3) when given with the optional pathname argument, permitted only when there is no file transfer operation in progress, is hereby specified to be the same as that of the MLST command, with just the following modifications, required because the output is sent over the control connection. More text comes here, eventually, setting out exactly what the STAT response will look like. Most of it will be borrowed from the original definition of MLST output, from the days when it was permitted over the control connection. 7.1. FEAT response for STAT There is no output in the FEAT command that specifically indicates that the STAT command behaves as described above. Implementations must infer this from support of the MLST command by the server, which is indicated in the FEAT output. 8. Impact On Other FTP Commands Along with the introduction of MLST, traditional FTP commands must be extended to allow for the use of more than US-ASCII or EBCDIC character sets. In general, the support of MLST requires support for arbitrary character sets wherever filenames and directory names are allowed. This applies equally to both arguments given to the following commands and to the replies from them, as appropriate. CWD RETR STOR STOU APPE RNFR RNTO Elz & Hethmon [Page 24] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 DELE RMD MKD PWD STAT The arguments to all of these commands should be processed the same way that MLST commands and responses are processed with respect to handling embedded CRs and NULs. See section 2.2. 8.1. Impact on Pathnames and Filenames The design of MLST requires the Server-FTP to allow concatenation of certain elements of a MLST response. Specifically, a typical response would include an element which indicates the current directory and one or more elements which are files in the indicated directory. A Server-FTP must be able to accept a simple concatenation of these two names even if the underlying operating system does not accept a simple concatenation. The Server-FTP must perform any translation of the concatenated name to local equivalents. 9. Character sets and Internationalisation This section will set out just what is going on with char sets, what data is part of the protocol, and always appears exactly as is specified (and could almost as easily be numbers, or any other kind of encoding), and what is text for users, which should be able to appear in their language of choice, or otherwise be handled in some kind of rational way. That is, it will once it is written. This is merely a placeholder. 10. Security This memo does not yet discuss security. It is possible that no new security concerns are raised in this memo above what already exists within the FTP protocol. However, the working group needs to consider this carefully. 11. References [1] Coded Character Set--7-bit American Standard Code for Information Interchange, ANSI X3.4-1986. [2] F. Yergeau, "UTF-8, a transformation format of Unicode and ISO 10646", RFC 2044, Alis Technologies, October 1996. Elz & Hethmon [Page 25] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 [3] D. Crocker, "Augmented BNF for Syntax Specifications: ABNF", Work In Progress , Internet Mail Consortium, March 1997. [4] W. Curtin, "Internationalization of the File Transfer Protocol", Work In Progress , Defense Information Systems Agency, November 1996. [5] Internet Assigned Numbers Authority. http://www.isi.edu/div7/iana/ Email: iana@iana.org. [6] J. Postel, J. Reynolds, "File Transfer Protocol (FTP)", STD 9 (RFC 959), ISI, October 1985 [7] R. Braden, "Requirements for Internet Hosts -- Application and Support", STD 3 (RFC 1123), October 1989 [8] ISO 3307 (need a citation for this please!) [9] D. Crocker, "Augmented BNF for Syntax Specifications: ABNF", Work In Progress , Internet Mail Consortium, March 1997. [10] S. Bradner, "Key words for use in RFCs to Indicate Requirement Levels", BCP 14 (RFC 2119), Harvard University, March 1997 [11] P. Hethmon, R. Elz, "Feature negotiation mechanism for the File Transfer Protocol", Work in progress, Hethmon Bros, University of Melbourne, June 1997. [12] J. Postel, J. Reynolds, "Telnet protocol Specification" STD 8 (RFC 854), ISI, May 1983 Acknowledgements The following people have contributed to this document: Alex Belits D. J. Berstein Martin J. Duerst Mark Harris Alun Jones James Matthews Keith Moore and the entire FTPEXT working group of the IETF. Elz & Hethmon [Page 26] Internet Draft draft-ietf-ftpext-mlst-01.txt June 1997 The description of the modifications to the REST command and the MDTM and SIZE commands comes from a set of modifications suggested for RFC959 by Rick Adams in 1989. A draft containing just those commands, edited by David Borman, has been merged with this document. Editors' Addresses Robert Elz University of Melbourne Department of Computer Science Parkville, Vic 3052 Australia Email: kre@munnari.OZ.AU Paul Hethmon Hethmon Brothers 2305 Chukar Road Knoxville, TN 37923 USA Phone: +1-423-690-8990 Email: phethmon@hethmon.com Elz & Hethmon [Page 27]