Negotiating Human Language in Real-Time Communications

A mutually comprehensible language is helpful for human communication. This document addresses the real-time, interactive side of the issue. A companion document on language selection in email addresses the non-real-time side. When setting up interactive communication sessions (using SIP or other protocols), human (natural) language and media modality (spoken, signed, written) negotiation may be needed. Unless the caller and callee know each other or there is contextual or out-of-band information from which the language(s) and media modalities can be determined, there is a need for spoken, signed, or written languages to be negotiated based on the caller's needs and the callee's capabilities. This need applies to both emergency and non-emergency calls. For various reasons, including the ability to establish multiple streams using different media (e.g., voice, text, video), it makes sense to use a per-stream negotiation mechanism, in this case, SDP. This approach has a number of benefits, including that it is generic (applies to all interactive communications negotiated using SDP) and is not limited to emergency calls. In some cases such a facility isn't needed, because the language is known from the context (such as when a caller places a call to a sign language relay center, to a friend, or colleague). But it is clearly useful in many other cases. For example, it is helpful that someone calling a company call center or a Public Safety Answering Point (PSAP) be able to indicate preferred signed, written, and/or spoken languages, the callee be able to indicate its capabilities in this area, and the call proceed using the language(s) and media forms supported by both. Since this is a protocol mechanism, the user equipment (UE client) needs to know the user's preferred languages; a reasonable technique could include a configuration mechanism with a default of the language of the user interface. In some cases, a UE could tie language and media preferences, such as a preference for a video stream using a signed language and/or a text or audio stream using a written/spoken language. Including the user's human (natural) language preferences in the session establishment negotiation is independent of the use of a relay service and is transparent to a voice or other service provider. For example, assume a user within the United States who speaks Spanish but not English places a voice call. The call could be an emergency call or perhaps to an airline reservation desk. The language information is transparent to the voice service provider, but is part of the session negotiation between the UE and the terminating entity. In the case of a call to e.g., an airline, the call could be automatically handled by a Spanish-speaking agent. In the case of an emergency call, the Emergency Services IP network (ESInet) and the PSAP may choose to take the language and media preferences into account when determining how to process the call. By treating language as another attribute that is negotiated along with other aspects of a media stream, it becomes possible to accommodate a range of users' needs and called party facilities. For example, some users may be able to speak several languages, but have a preference. Some called parties may support some of those languages internally but require the use of a translation service for others, or may have a limited number of call takers able to use certain languages. Another example would be a user who is able to speak but is deaf or hard-of-hearing and requires a voice stream plus a text stream. Making language a media attribute allows the standard session negotiation mechanism to handle this by providing the information and mechanism for the endpoints to make appropriate decisions. Regarding relay services, in the case of an emergency call requiring sign language such as ASL, there are currently two common approaches: the caller initiates the call to a relay center, or the caller places the call to emergency services (e.g., 911 in the U.S. or 112 in Europe). (In a variant of the second case, the voice service provider invokes a relay service as well as emergency services.) In the former case, the language need is ancillary and supplemental. In the non-variant second case, the ESInet and/or PSAP may take the need for sign language into account and bridge in a relay center. In this case, the ESInet and PSAP have all the standard information available (such as location) but are able to bridge the relay sooner in the call processing. By making this facility part of the end-to-end negotiation, the question of which entity provides or engages the relay service becomes separate from the call processing mechanics; if the caller directs the call to a relay service then the human language negotiation facility provides extra information to the relay service but calls will still function without it; if the caller directs the call to emergency services, then the ESInet/PSAP are able to take the user's human language needs into account, e.g., by assigning to a specific queue or call taker or bridging in a relay service or translator. The term "negotiation" is used here rather than "indication" because human language (spoken/written/signed) is something that can be negotiated in the same way as which forms of media (audio/text/video) or which codecs. For example, if we think of non-emergency calls, such as a user calling an airline reservation center, the user may have a set of languages he or she speaks, with perhaps preferences for one or a few, while the airline reservation center will support a fixed set of languages. Negotiation should select the user's most preferred language that is supported by the call center. Both sides should be aware of which language was negotiated. This is conceptually similar to the way other aspects of each media stream are negotiated using SDP (e.g., media type and codecs). To reduce the complexity of the solution, this draft focuses on negotiating language per media; routing is out of scope.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

The desired solution is a media attribute (preferably per direction) that may be used within an offer to indicate the preferred language(s) of each (direction of a) media stream, and within an answer to indicate the accepted language. The semantics of including multiple values for a media stream within an offer is that the languages are listed in order of preference. (Negotiating multiple simultaneous languages within a media stream is out of scope of this document.)

RFC 4566 specifies an attribute 'lang' which appears similar to what is needed here, but is not sufficiently specific or flexible for the needs of this document. In addition, 'lang' is not mentioned in and there are no known implementations in SIP. Further, it is useful to be able to specify language per direction (sending and receiving). This document therefore defines two new attributes.

An SDP attribute (per direction) seems the natural choice to negotiate human (natural) language of an interactive media stream, using the language tags of BCP 47 .

The decision to base the proposal at the media negotiation level, and specifically to use SDP, came after significant debate and discussion. From an engineering standpoint, it is possible to meet the objectives using a variety of mechanisms, but none are perfect. None of the proposed alternatives was clearly better technically in enough ways to win over proponents of the others, and none were clearly so bad technically as to be easily rejected. As is often the case in engineering, choosing the solution is a matter of balancing trade-offs, and ultimately more a matter of taste than technical merit. The two main proposals were to use SDP and SIP. SDP has the advantage that the language is negotiated with the media to which it applies, while SIP has the issue that the languages expressed may not match the SDP media negotiated (for example, a session could negotiate a signed language at the SIP level but fail to negotiate a video media stream at the SDP layer). The mechanism described here for SDP can be adapted to media negotiation protocols other than SDP.

This document defines two media-level attributes starting with 'hlang' (short for "human interactive language") to negotiate which human language is used in each interactive media stream. There are two attributes, one ending in "-send" and the other in "-recv", registered in . Each can appear in an offer for a media stream. In an offer, the 'hlang-send' value is a list of one or more language(s) the offerer is willing to use when sending using the media, and the 'hlang-recv' value is a list of one or more language(s) the offerer is willing to use when receiving using the media. The list of languages is in preference order (first is most preferred). When a media is intended for interactive communication using a language in one direction only (such as a user sending using text and receiving using audio), either hlang-send or hlang-recv MAY be omitted. When a media is not primarily intended for language (for example, a video or audio stream intended for background only) both SHOULD be omitted. Otherwise, both SHOULD have the same value. The two SHOULD NOT be set to languages which are difficult to match together (e.g., specifying a desire to send audio in Hungarian and receive audio in Portuguese will make it difficult to successfully complete the call). In an answer, 'hlang-send' is the language the answerer will send when using the media (which in most cases is one of the languages in the offer's 'hlang-recv'), and 'hlang-recv' is the language the answerer expects to receive in the media (which in most cases is one of the languages in the offer's 'hlang-send'). Each value MUST be a list of one or more language tags per BCP 47 , separated by white space. BCP 47 describes mechanisms for matching language tags. Note that Section 4.1 advises to "tag content wisely" and not include unnecessary subtags. In an offer, each value MAY have an asterisk appended as the final value. An asterisk appended to either value in an offer indicates a request by the caller to the callee to not fail the call if there is no language in common. See for more information and discussion. When placing an emergency call, and in any other case where the language cannot be inferred from context, in an offer each media stream primarily intended for human language communication SHOULD specify both (or for asymmetrical language use, one of) the 'hlang-send' and 'hlang-recv' attributes. Note that while signed language tags are used with a video stream to indicate sign language, a spoken language tag for a video stream in parallel with an audio stream with the same spoken language tag indicates a request for a supplemental video stream to see the speaker. Clients acting on behalf of end users are expected to set one or both 'hlang-send' and 'hlang-recv' attributes on each media stream primarily intended for human communication in an offer when placing an outgoing session, and either ignore or take into consideration the attributes when receiving incoming calls, based on local configuration and capabilities. Systems acting on behalf of call centers and PSAPs are expected to take into account the attributes when processing inbound calls. Note that media and language negotiation might result in more media streams being accepted than are needed by the users (e.g., if more preferred and less preferred combinations of media and language are all accepted). This is not a problem.

A consideration with the ability to negotiate language is if the call proceeds or fails if the callee does not support any of the languages requested by the caller. This document does not mandate either behavior, although it does provide a way for the caller to indicate a preference for the call succeeding when there is no language in common. It is OPTIONAL for the callee to honor this preference. For example, a PSAP is likely to attempt the call even without an indicated preference when there is no language in common, while a call center might choose to fail the call. The mechanism for indicating this preference is that, in an offer, if the last value of either 'hlang-recv' or 'hlang-send' is an asterisk, this indicates a request to not fail the call. The called party MAY ignore the indication, e.g., for the emergency services use case, regardless of the absence of an asterisk, a PSAP will likely not fail the call; some call centers might reject a call even if the offer contains an asterisk. If the call is rejected due to lack of any languages in common, it is suggested to use SIP response code 488 (Not Acceptable Here) or 606 (Not Acceptable) and include a Warning header field in the SIP response. The Warning header field contains a warning code of [TBD: IANA VALUE, e.g., 308] and a warning text indicating that there are no mutually-supported languages; the text SHOULD also contain the supported languages and media. Example: [TBD: IANA VALUE, e.g., 308] proxy.example.com "Incompatible language specification: Requested languages not supported. Supported languages are: es, en; supported media are: audio, text."

With the exception of the case mentioned in (a spoken language tag for a video stream in parallel with an audio stream with the same spoken language tag), the behavior when specifying a spoken/written language tag for a video media stream, or a signed language tag for an audio or text media stream, is not defined.

Some examples are shown below. For clarity, only the most directly relevant portions of the SDP block are shown. An offer or answer indicating spoken English both ways: m=audio 49170 RTP/AVP 0 a=hlang-send:en a=hlang-recv:en An offer or answer indicating American Sign Language both ways, and requesting that the call proceed even if the callee does not support the language: m=video 51372 RTP/AVP 31 32 a=hlang-send:ase* a=hlang-recv:ase* An offer requesting spoken Spanish both ways (most preferred), spoken Basque both ways (second preference), or spoken English both ways (third preference). The offer further requests that the call proceed even if the callee does not support any of the languages: m=audio 49250 RTP/AVP 20 a=hlang-send:es eu en * a=hlang-recv:es eu en * An answer to the above offer indicating spoken Spanish both ways: m=audio 49250 RTP/AVP 20 a=hlang-send:es a=hlang-recv:es An alternative answer to the above offer indicating spoken Italian both ways (as the callee does not support any of the requested languages but chose to proceed with the call): m=audio 49250 RTP/AVP 20 a=hlang-send:it a=hlang-recv:it An offer of answer indicating written Greek both ways: m=text 45020 RTP/AVP 103 104 a=hlang-send:gr a=hlang-recv:gr An offer requesting the following media streams: video for the caller to send using Argentine Sign Language, text for the caller to send using written Spanish (most preferred) or written Portuguese, audio for the caller to receive spoken Spanish (most preferred) or spoken Portuguese. The offer also requests that the call proceed even if the callee does not support any of the languages: m=video 51372 RTP/AVP 31 32 a=hlang-send:aed * m=text 45020 RTP/AVP 103 104 a=hlang-send:sp pt * m=audio 49250 RTP/AVP 20 a=hlang-recv:sp pt * An answer for the above offer, indicating text in which the callee will receive written Spanish, and audio in which the callee will send spoken Spanish: m=text 45020 RTP/AVP 103 104 a=hlang-recv:sp m=audio 49250 RTP/AVP 20 a=hlang-send:sp An offer requesting the following media streams: text for the caller to send using written English (most preferred) or written Spanish, audio for the caller to receive spoken English (most preferred) or spoken Spanish, supplemental video. The offer also requests that the call proceed even if the callee does not support any of the languages: m=text 45020 RTP/AVP 103 104 a=hlang-send:en sp m=audio 49250 RTP/AVP 20 a=hlang-recv:en sp * m=video 51372 RTP/AVP 31 32 An answer for the above offer, indicating text in which the callee will receive written Spanish, audio in which the callee will send spoken Spanish, and supplemental video: m=text 45020 RTP/AVP 103 104 a=hlang-recv:sp m=audio 49250 RTP/AVP 20 a=hlang-send:sp m=video 51372 RTP/AVP 31 32

IANA is kindly requested to add two entries to the 'att-field (media level only)' table of the SDP parameters registry: hlang-recv Randall Gellens rg+ietf@randy.pensive.org Language-Tag *( SP Language-tag ) [ SP asterisk ] ; Language-Tag as defined in BCP 47 "*" ; an asterisk (ASCII %2A) character 1*" " ; one or more ASCII space (%20) characters Described in of TBD: THIS DOCUMENT media NORMAL No See of TBD: THIS DOCUMENT See of TBD: THIS DOCUMENT TBD: THIS DOCUMENT hlang-send Randall Gellens rg+ietf@randy.pensive.org hlang-value Described in of TBD: THIS DOCUMENT media NORMAL No See of TBD: THIS DOCUMENT See of TBD: THIS DOCUMENT TBD: THIS DOCUMENT

IANA is requested to add a new value in the warn-codes sub-registry of SIP parameters in the 300 through 329 range that is allocated for indicating problems with keywords in the session description. The reference is to this document. The warn text is "Incompatible language specification: Requested languages not supported. Supported languages and media are: [list of supported languages and media]."

The Security Considerations of BCP 47 apply here. In addition, if the 'hlang-send' or 'hlang-recv' values are altered or deleted en route, the session could fail or languages incomprehensible to the caller could be selected; however, this is also a risk if any SDP parameters are modified en route.

Language and media information can suggest a user's nationality, background, abilities, disabilities, etc.

RFC EDITOR: Please remove this section prior to publication.

Deleted Section 3 ("Expected Use") Reworded modalities in Introduction from "voice, video, text" to "spoken, signed, written" Reworded text about "increasingly fine-grained distinctions" to instead merely point to BCP 47 Section 4.1's advice to "tag content wisely" and not include unnecessary subtags Changed IANA registration of new SDP attributes to follow RFC 4566 template with extra fields suggested in 4566-bis (expired draft) Deleted "(known as voice carry over)" Changed textual instanced of RFC 5646 to BCP 47, although actual reference remains RFC due to xml2rfc limitations

Added Examples Added Privacy Considerations section Other editorial changes for clarity

Deleted most of and replaced with a very short summary Replaced "wishes to" with "is willing to" in Reworded description of attribute usage to clarify when to set both, only one, or neither Deleted all uses of "IMS" Other editorial changes for clarity

Editorial changes to wording in Section 5.

Updated title to reflect WG adoption

Removed Use Cases section, per face-to-face discussion at IETF 93 Removed discussion of routing, per face-to-face discussion at IETF 93

Updated NENA usage mention Removed background text reference to draft-saintandre-sip-xmpp-chat-04 since that draft expired

Revision to keep draft from expiring

Changed name from -mmusic- to -slim- to reflect proposed WG name As a result of the face-to-face discussion in Toronto, the SDP vs SIP issue was resolved by going back to SDP, taking out the SIP hint, and converting what had been a set of alternate proposals for various ways of doing it within SIP into an informative annex section which includes background on why SDP is the proposal Added mention that enabling a mutually comprehensible language is a general problem of which this document addresses the real-time side, with reference to which addresses the non-real-time side.

Added clarifying text on leaving attributes unset for media not primarily intended for human language communication (e.g., background audio or video). Added new section ("Alternative Proposal: Caller-prefs") discussing use of SIP-level Caller-prefs instead of SDP-level.

Relaxed language on setting -send and -receive to same values; added text on leaving on empty to indicate asymmetric usage. Added text that clients on behalf of end users are expected to set the attributes on outgoing calls and ignore on incoming calls while systems on behalf of call centers and PSAPs are expected to take the attributes into account when processing incoming calls.

Updated text to refer to RFC 5646 rather than the IANA language subtags registry directly. Moved discussion of existing 'lang' attribute out of "Proposed Solution" section and into own section now that it is not part of proposal. Updated text about existing 'lang' attribute. Added example use cases. Replaced proposed single 'hlang' attribute with 'hlang-send' and 'hlang-recv' per Harald's request/information that it was a misuse of SDP to use the same attribute for sending and receiving. Added section describing usage being advisory vs required and text in attribute section. Added section on SIP "hint" header (not yet nailed down between new and existing header). Added text discussing usage in policy-based routing function or use of SIP header "hint" if unable to do so. Added SHOULD that the value of the parameters stick to the largest granularity of language tags. Added text to Introduction to be try and be more clear about purpose of document and problem being solved. Many wording improvements and clarifications throughout the document. Filled in Security Considerations. Filled in IANA Considerations. Added to Acknowledgments those who participated in the Orlando ad-hoc discussion as well as those who participated in email discussion and side one-on-one discussions.

Updated text for (possible) new attribute "hlang" to reference RFC 5646 Added clarifying text for (possible) re-use of existing 'lang' attribute saying that the registration would be updated to reflect different semantics for multiple values for interactive versus non-interactive media. Added clarifying text for (possible) new attribute "hlang" to attempt to better describe the role of language tags in media in an offer and an answer.

Changed name of (possible) new attribute from 'humlang" to "hlang" Added discussion of silly state (language not appropriate for media type) Added Voice Carry Over example Added mention of multilingual people and multiple languages Minor text clarifications

Gunnar Hellstrom deserves special mention for his reviews and assistance.

Many thanks to Bernard Aboba, Harald Alvestrand, Flemming Andreasen, Francois Audet, Eric Burger, Keith Drage, Doug Ewell, Christian Groves, Andrew Hutton, Hadriel Kaplan, Ari Keranen, John Klensin, Paul Kyzivat, John Levine, Alexey Melnikov, James Polk, Pete Resnick, Peter Saint-Andre, and Dale Worley for reviews, corrections, suggestions, and participating in in-person and email discussions.