draft Charset policy June 97 IETF Policy on Character Sets and Languages Fri Aug 29 10:41:03 MET DST 1997 Harald Tveit Alvestrand UNINETT Harald.T.Alvestrand@uninett.no Status of this Memo This draft document is being circulated for comment. Please send comments to the author, or to the mailing list The following text is required by the Internet-draft rules: This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a "working draft" or "work in progress." Please check the I-D abstract listing contained in each Internet Draft directory to learn the current status of this or any other Internet Draft. The file name of this version is draft-alvestrand-charset- policy-01.txt Alvestrand Expires Dec 97 [Page 1] draft Charset policy June 97 1. Introduction The Internet is international. With the international Internet follows an absolute requirement to interchange data in a multiplicity of languages, which in turn utilize a bewildering number of characters. This document is (INTENDED TO BE) the current policies being applied by the Internet Engineering Steering Group towards the standardization efforts in the Internet Engineering Task Force in order to help Internet protocols fulfil these requirements. The document is very much based upon the recommendations of the IAB Character Set Workshop of February 29-March 1, 1996, which is documented in RFC 2130 [WR]. This document attempts to be concise, explicit and clear; people wanting more background are encouraged to read RFC 2130. The document uses the terms "MUST", "SHOULD" and "MAY", and their negatives, in the way described in [RFC 2119]. In this case, "the specification" as used by RFC 2119 refers to the processing of protocols being submitted to the IETF standards process. 2. Where to do internationalization Internationalization is for humans. This means that protocols are not subject to internationalization; text strings are. Where protocol elements look like text tokens, such as in many IETF application layer protocols, protocols MUST specify which parts are protocol and which are text. [WR 2.2.1.1] Names are a problem, because people feel strongly about them, many of them are mostly for local usage, and all of them tend to leak out of the local context at times. RFC 1958 [ARCH] recommends US- ASCII for all globally visible names. This document does not mandate a policy on name internationalization, but requires that all protocols describe whether names are internationalized or US-ASCII. NOTE: In the protocol stack for any given application, there is usually one or a few layers that need to address these problems. Alvestrand Expires Dec 97 [Page 2] draft Charset policy June 97 It would, for instance, not be appropriate to define language tags for Ethernet frames. But it is the responsibility of the WGs to ensure that whenever responsibility for internationalization is left to "another layer", those responsible for that layer are in fact aware that they HAVE that responsibility. 3. Definition of Terms This document uses the term "charset" to mean a set of rules for mapping from a sequence of octets to a sequence of characters, such as the combination of a coded character set and a character encoding scheme; this is also what is used as an identifier in MIME "charset=" parameters, and registered in the IANA charset registry [REG]. (Note that this is NOT a term used by other standards bodies, such as ISO). For a definition of the term "coded character set", refer to the workshop report. A "name" is an identifier such as a person's name, a hostname, a domainname, a filename or an E-mail address; it is often treated as an identifier rather than as a piece of text, and is often used in protocols as an identifier for entities, without surrounding text. 3.1. What charset to use All protocols MUST identify, for all character data, which charset is in use. Protocols MUST be able to use the UTF-8 charset, which consists of the ISO 10646 coded character set combined with the UTF-8 character encoding scheme, as defined in [10646] Annex R (published in Amendment 2), for all text. Protocols MAY specify, in addition, how to use other charsets or other character encoding schemes for ISO 10646, such as UTF-16, but lack of an ability to use UTF-8 is a violation of this policy; such a violation would need a variance procedure ([BCP9] section 9) with clear and solid justification in the protocol specification document before being entered into or advanced upon the standards track. Alvestrand Expires Dec 97 [Page 3] draft Charset policy June 97 For existing protocols or protocols that move data from existing datastores, support of other charsets, or even using a default other than UTF-8, may be a requirement. This is acceptable, but UTF-8 support MUST be possible. When using other charsets than UTF-8, these MUST be registered in the IANA charset registry, if necessary by registering them when the protocol is published. (Note: ISO 10646 calls the UTF-8 CES a "Transformation Format" rather than a "character encoding scheme", but it fits the charset workshop report definition of a character encoding scheme). 3.2. How to decide a charset In some cases, like HTTP, there is direct or semi-direct communication between the producer and the consumer of data containing text. In such cases, it may make sense to negotiate a charset before sending data. In other cases, like E-mail or stored data, there is no such communication, and the best one can do is to make sure the charset is clearly identified with the stored data, and choosing a charset that is as widely known as possible. Note that a charset is an absolute; text that is encoded in a charset cannot be rendered comprehensibly without supporting that charset. (This also applies to English texts; charsets like EBCDIC do NOT have ASCII as a proper subset) Negotiating a charset may be regarded as an interim mechanism that is to be supported until support for interchange of UTF-8 is prevalent; however, the timeframe of "interim" may be at least 50 years, so there is every reason to think of it as permanent in practice. Alvestrand Expires Dec 97 [Page 4] draft Charset policy June 97 4. Languages 4.1. The need for language information All human-readable text has a language. Many operations, including high quality formatting, text-to-speech synthesis, searching, hyphenation, spellchecking and so on benefit greatly from access to information about the language of a piece of text. [WC 3.1.1.4]. Humans have some tolerance for foreign languages, but are generally very unhappy with being presented text in a language they do not understand; this is why negotiation of language is needed. In most cases, machines will not be able to deduce the language of a transmitted text by themselves; the protocol must specify how to transfer the language information if it is to be available at all. The interaction between language and processing is complex; for instance, if I compare "name-of-thing(lang=en)" to "name-of- thing(lang=no)" for equality, I will generally expect a match, while the word "ask(no)" is a kind of tree, and is hardly useful as a command verb. 4.2. Requirement for language tagging Protocols that transfer text MUST provide for carrying information about the language of that text. Protocols SHOULD also provide for carrying information about the language of names, where appropriate. Note that this does NOT mean that such information must always be present; the requirement is that if the sender of information wishes to send information about the language of a text, the protocol provides a well-defined way to carry this information. Alvestrand Expires Dec 97 [Page 5] draft Charset policy June 97 4.3. How to identify a language The RFC 1766 language tag is at the moment the most flexible tool available for identifying a language; protocols SHOULD use this, or provide clear and solid justification for doing otherwise in the document. Note also that a language is distinct from a POSIX locale; a POSIX locale identifies a set of cultural conventions, which may imply a language (the POSIX or "C" locale of course do not), while a language tag as described in RFC 1766 identifies only a language. 4.4. Considerations for language negotiation Protocols where users have text presented to them in response to user actions MUST provide for support of multiple languages. How this is done will vary between protocols; for instance, in some cases, a negotiation where the client proposes a set of languages and the server replies with one is appropriate; in other cases, a server may choose to send multiple variants of a text and let the client pick which one to display. Negotiation is useful in the case where one side of the protocol exchange is able to present text in multiple languages to the other side, and the other side has a preference for one of these; the most common example is the text part of error responses, or Web pages that are available in multiple languages. Negotiating a language should be regarded as a permanent requirement of the protocol that will not go away at any time in the future. In many cases, it should be possible to include it as part of the connection establishment, together with authentication and other preferences negotiation. 4.5. Default Language When human-readable text must be presented in a context where the sender has no knowledge of the recipient's language preferences (such as login failures or E-mailed warnings, or prior to language Alvestrand Expires Dec 97 [Page 6] draft Charset policy June 97 negotiation), text SHOULD be presented in Default Language. The Default Language is English, since this is the language which the greatest number of people will be able to get adequate help in interpreting when working with computers. Note that negotiating English is NOT the same as Default Language; Default Language is an emergency measure in otherwise unmanageable situations. It may be appropriate for application designers to make sure that messages in Default Language are understandable to people with a limited understanding of the English language. 5. Locale The POSIX standard [POSIX] defines a concept called a "locale", which includes a lot of information about collating order for sorting, date format, currency format and so on. In some cases, and especially with text where the user is expected to do processing on the text, locale information may be usefully attached to the text; this would identify the sender's opinion about appropriate rules to follow when processing the document, which the recipient may choose to agree with or ignore. This document does not require the communication of locale information on all text, but encourages its inclusion when appropriate. Note that language and character set information will often be present as parts of a locale tag (such as no_NO.iso-8859-1; the language is before the underscore and the character set is after the dot); care must be taken to define precisely which specification of character set and language applies to any one text item. The default locale is the "POSIX" locale. 6. Documenting internationalization decisions In documents that deal with internationalization issues at all, a synopsis of the approaches chosen for internationalization SHOULD be collected into a section called "Internationalization Alvestrand Expires Dec 97 [Page 7] draft Charset policy June 97 considerations", and placed next to the Security Considerations section. This provides an easy reference for those who are looking for advice on these issues when implementing the protocol. 7. Security considerations Apart from the fact that security warnings in a foreign language may cause inappropriate behaviour from the user, and the fact that multilingual systems usually have problems with consistency between language variants, no security considerations relevant have been identified. 8. References [10646] ISO/IEC, Information Technology - Universal Multiple-Octet Coded Character Set (UCS) - Part 1: Architecture and Basic Multilingual Plane, May 1993, with amendments [RFC 2119] S. Bradner, "Key words for use in RFCs to Indicate Requirement Levels", 03/26/1997 - RFC 2119 [WR] C. Weider, C. Preston, K. Simonsen, H. Alvestrand, R. Atkinson, M. Crispin, P. Svanberg, "The Report of the IAB Character Set Workshop held 29 February - 1 March, 1996", 04/21/1997, RFC 2130 [ARCH] B. Carpenter, "Architectural Principles of the Internet", 06/06/1996, RFC 1958 [POSIX] ISO/IEC 9945-2:1993 Information technology -- Portable Operating System Interface (POSIX) -- Part 2: Shell and Utilities [REG] N. Freed, J. Postel: IANA Charset Registration Procedures, Alvestrand Expires Dec 97 [Page 8] draft Charset policy June 97 Work In Progress (draft-freed-charset-reg-02.txt) [UTF-8] F. Yergeau: UTF-8, a transformation format of Unicode and ISO 10646, Work In Progress (draft-yergeau-utf8-rev-00.txt, obsoletes RFC 2044) [BCP9] S. Bradner: The Internet Standards Process -- Revision 3. RFC 2026, BCP 9. 9. Author's address Harald Tveit Alvestrand UNINETT P.O.Box 6883 Elgeseter N-7002 TRONDHEIM NORWAY +47 73 59 70 94 Harald.T.Alvestrand@uninett.no Alvestrand Expires Dec 97 [Page 9]