SDP Superimposition Grouping framework

This document defines semantics that allow for signaling a new SDP group "supim" for superimposed media in an SDP session. The "supim" attribute can be used by the application to relate all the fully or partly superimposed visual media streams enabling them to be added as an overlay on top of any one or more background visual media streams. The superimposition grouping semantics is helpful if the media stream data is separate and transported via different sessions. Media superimposition herein is defined to be a visual media stream (video/image/text) that is fully or partly superimposed on top of an already existing visual media stream such that the resulting foreground and background media can be displayed simultaneously. Superimposition can be recursive in that visual media that is superimposed against its background can, in turn, be the background of another superimposed visual media. The superimposed visual media displayed over a background media content may be anywhere between opaque and transparent. Examples of applications for video superimposition include real-time multi-party gaming, where these superimposed media may be used to provide additional details or stats about each player, or multi-party teleconferencing where visual media from users in the teleconference may be superimposed over a background media or over each other. This document describes new SDP group semantics for grouping the superimposition in an SDP session. An SDP session description consists of one or multiple media lines known as "m" lines which can be identified by a token carried in a "mid" attribute. The SDP session describes a session-level group-level attribute that groups different media lines using a defined group semantics. The semantics defined in this memo are to be used in conjunction with "The Session Description Protocol (SDP) Grouping Framework" . We have studied the existing specifications, including the CLUE framework and work in MPEG, and found that such work is not covering our intended application space; please refer to for details. The superimposition grouping as described below enables a compliant receiver/renderer implementation to know the relative relevance of the visual media as coded by the sender(s) and, in a compliant implementation, observed by the renderer through superimposition when needed.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

SDP is predominantly used for describing the format for multimedia communication sessions. Many SDP-based systems use open standards such as RTP for media transport and SIP for session setup and control. An SDP session may contain more than one media description, with each media description identified by "m"=line. Each line denotes a single media stream. If multiple visual media lines are present in a session, at present, rendering aspects, including their possible superimposition (foreground/ background), relationship at the rendering device is undefined. This memo introduces a mechanism in which certain rendering information becomes available. The rendering information herein is limited to the foreground/background relationship of each grouped media to other media streams through a layer order value, and optionally a transparency value. Where, spatially, the media is rendered is not covered by this memo, and is in many application scenarios a function of the user interface. An example is shown in , where three foreground media streams have been superimposed over a background media stream, with Media B being partly superimposed over Media C.

Of course, assuming sufficient screen real-estate, a renderer may not have to rely on superimposition mechanisms at all-when there is enough screen real-estate available, a valid display strategy may well be to show all media without overlapping and hence without superimposition. However, when the screen real-estate becomes insufficient, then the information provided by the mechanisms defined in this memo can be used to order (in the sense of foreground to background) the visual media according to a hierarchy chosen by the sender or a MANE (media-aware network element), and based on their application knowledge. When multiple superimposed streams are transmitted within a session, the receiver needs to be able to relate the media streams to each other. This is achieved by the SDP grouping framework by using the "group" attribute that groups different "m" lines in a session. By using a new superimpose group semantic defined in this memo, a group’s media streams can be uniquely identified across multiple SDP descriptions exchanged with different receivers, thereby identifying the streams in terms of their role in the session irrespective of their media type and transport protocol. These superimposed streams within the group may be multiplexed based on the guidelines defined in .

The "superimposition media stream identification" attribute, “supim”, is used to identify the relationship of superimposed media streams within a session description. In a superimposition group, the media lines MAY have different media formats. There is no defined behavior for the rendering of non- visual media being grouped in a superimposition group. It is assumed that all the media streams are that need to be time- synchronized are time-synchronized. Its formatting follows in the use of the 'mid' attribute to identify the media line to be included in the superimposition. It is used for grouping the foreground and the background media streams intended for the purpose of composition with foreground media to be superimposed over the background media stream. A media player that chooses to implement the extension and receives a session description that contains "m" lines grouped together using "supim" semantics is able to superimpose the foreground media streams on top of the background media stream in cases where there is overlap. For non-supporting devices, these media streams are treated as independent media streams. This document defines a standard semantics: Overlay. Semantics extensions follow the Standards Action policy [RFC8126].<-->

All group and mid attributes MUST follow the rules defined in . The "mid" attribute MUST be used for all "m" lines covering visual media within a session description for which a foreground/background relationship is to be defined. The foreground/background relationship of visual media within a session description that is not covered in a group is undefined. Multiple groups MUST not be used within one session. If the identification-tags associated with "a=group" lines do not map to any "m" lines, the identification-tags MUST be ignored.

This memo defines a new media-level attribute, "superimposition", with the following ABNF . The identification-tag is defined in .

/ "-" attribute = attribute =/ superimposition-attribute ]]> The transparency for the media stream is identified by its super-trans-val values in the super-trans attribute. The value MUST be an ASCII representation of an 8 bit signed integer with values between "-128" and "127", and linear weighting between the two extremes. A value of -128 means the media stream is opaque, and the highest value of 127 means it is transparent. Further details of interpretion is to be left open to the implementer. The layering order value for the media stream is identified by super-layer-val. It MUST be an integer value between 0 and n, where the value 0 represents the deepest background layer. For each k within 0..n, a reconstructed sample of the k-th media is superimposed (while perhaps applying an super-trans-val value) on the 0 to k-th reconstructed samples in the same spatial position. Each "m" line in a session MUST NOT contain more than one instance of super-opt attribute.

The following example shows a session description for superimposed media streams in an SDP session. The "group" line indicates that the "m" lines with tokens 1, 2 and 3 are grouped for the purpose of superimposition. In the example shown below, three media streams are being transmitted for superimposition. The background media stream along with the foreground media streams are grouped together using "supim". All media streams are videos with "superimposition" attribute. The media stream with layer order value 0 is intended for background.

The transparency value is used for composing the foreground with the background media . This value itself does not define the transparency of each pixel but is applied to each pixel within a frame and defines the factor by which the transparency of each pixel within a frame is to be increased or decreased. The "layer" value is relevant when two or more media streams are to be composed. When the transparency value of the foreground is -128, the composed image will be the foreground image, as it is being displayed as opaque. Similarly, if the transparency value for the foreground media is 127, the resulting image will be the background media, as the foreground media stream is being presented fully transparent, hence invisible. The details of the weighting of foreground and background sample values based on a given super-trans value is left to the implementation, beyond the abstract definition that value equal to -128 means opaque, and value equal to 127 means transparent, and the weighting is to be implemented such that it is visually linear for the values in between. We do not define a weighting formula in this specification as these formulae would depend on many factors such as the colorspace and the sampling structure of the media.

Edt. Note: maybe we remove this section later once there is a general understanding why the existing specifications in its current form is unsuitable. The CLUE framework is the IETF’s chosen technology for the applications requiring defining multiple “captures” (camera views), and their geo-spatial relationship to each. However, information pertaining to display/rendering is outside of CLUE’s scope. While many CLUE-capable receivers infer appropriate rendering strategies from the information offered by CLUE, the CLUE framework has generally assumed non-overlapped rendering of transmitted and reconstructed video streams from the multiple captures, often on different physical rendering devices. Insofar, we concluded that the CLUE framework neither supports the application we contemplate in this memo, nor would it be sensible to enhance the CLUE specifications with rendering-related mechanisms. There are certain technologies from standards bodies such as MPEG , often described as “scene descriptions”, that to a certain extent can address the applications we contemplate. We evaluated the technologies we are aware of and concluded that something different is required. We base our assumption on a) the complexity of these mechanisms, and b) their design as a metadata media stream, which in the IETF context would be conveyed in RTP sessions or similar, rather than a static or semi-static stream description that is best conveyed at session setup or renegotiation using SDP.

All security considerations as defined in apply: Using the "group" parameter with FID semantics, an entity that managed to modify the session descriptions exchanged between the participants to establish a multimedia session could force the participants to send a copy of the media to any destination of its choosing. Integrity mechanisms provided by protocols used to exchange session descriptions and media encryption can be used to prevent this attack. In SIP, Secure/Multipurpose Internet Mail Extensions (S/MIME) and Transport Layer Security (TLS) can be used to protect session description exchanges in an end-to-end and a hop-byhop fashion, respectively.

The following contact information shall be used for all registrations included here:

Stephan Wenger The IETF MMUSIC working group or its successor as designated by the IESG. ]]> This document defines a new SDP group semantics value for media superimposition for a SDP session. This attribute can be used by the application to group the foreground and the background media streams to be superimposed together in a session. Semantics values to be used with this framework should be registered by the IANA following the Standards Action policy . This document adds a new group semantics value to the sdp-paramters registry group defined in . IANA is requested to register the following semantics value in the "sdp-parameters" in the registry.

The "supim" attribute is used to group different media streams to be superimposed together with one background media stream and the rest foreground streams. Its format is defined in . IANA is requested to register the semantics value for SDP media-level attribute "superimposition" for "sdp-attributes(media-level only)". The registration procedure in applies.

SDP Attribute ("sdp-attributes(media level only)"):

The authors would like to thank Christer Holmberg and Paul Kyzivat for reviewing the draft and providing key ideas.