SDP Superimposition Grouping framework

Media superimposition can be defined to be a visual media (video/image/text) which is superimposed on top of an already existing visual media such that the resulting foreground and background media can be displayed simultaneously. Superimposition can be recursive in that a visual media that is superimposed against its background can, in turn, be the background of a another superimposed visual media. The superimposed visual media displayed over a background media content may vary in transparency. Examples of video superimposition include real-time multi-party gaming, where these superimposed media maybe used to provide additional details or stats about each player, or multi-party teleconferencing where visual media from users in the teleconference may be superimposed on a background media or over each other. An example is shown in figure below, where three foreground media has been superimposed over a background media with one foreground media being partly superimposed over another foreground media.

SDP is being predominantly used for describing the format for multimedia communication sessions. Many SDP-based systems use open standards such as RTP for media transport and and SIP for session setup and control. An SDP session may contain more than one media description with each media description identified by "m"=line. Each line denotes a single media stream. If multiple visual media lines are present in a session, at present, their spatial relationship at the rendering device is undefined. This memo introduces a mechanism in which certain rendering information becomes available. The rendering information herein is limited to the foreground/background relationship of each grouped media vis-à-vis each other, and optionally a transparency value. Where, spatially, the media is rendered is not covered by this memo and is in many application scenarios a function of the user interface. However, the superimposition grouping as described below enables a compliant receiver/renderer implementation to know the relative relevance of the visual media as coded by the sender(s) and, in a compliant implementation, observed by the renderer through superimposition. When multiple superimposed streams are transmitted within a session, the receiver needs to be able to relate the media streams to each other. This is achieved by the SDP grouping framework by using the "group" attribute that groups different "m" lines in a session. By using a new superimpose group semantic defined in this memo, a group’s media streams can be uniquely identified across multiple SDP descriptions exchanged with different receivers, thereby identifying the streams in terms of their role in the session irrespective of its media type and transport protocol. These superimposed streams within the group may be multiplexed based on the guidelines defined in . This document describes a new SDP group semantics for grouping the superimposition in an SDP session. An SDP session description consists of one or multiple media lines know as "m" lines which can be identified by a token carried in a "mid" attribute. The SDP session describes a session-level group level attribute that groups different media lines using a defined group semantics. The semantics defined in this memo is to be used in conjunction with "The Session Description Protocol (SDP) Grouping Framework".

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

The "superimposition media stream identification" attribute is used to identify the relationship of superimposed media streams within a session description. In a superimposition group, the media lines MAY have different media formats. Its formatting in SDP is described by the following Augmented Backus-Naur Form (ABNF) :

This documents defines a new group semantics "S" identification media attribute, which is used to identify super group media streams within a session description. It is used for grouping the foreground media streams to be superimposed on top of a background media stream together within a session. An application that receives a session description that contains "m" lines grouped together using "S" semantics MUST superimpose the corresponding media streams on top of the background media stream. The ordering of the "m" lines is significant: assuming the “m” lines to be counted from 0 to n, for each k within 0..n, a reconstructed sample of the k-th media is superimposed (while perhaps applying an alpha transparency value) on the 0 to k-th reconstructed samples in the same spatial position. This document defines a standard semantics: Overlay. Semantics extensions follow the Standards Action policy [RFC8126].<-->

All group and mid attributes MUST follow the rules defined in . The "mid" attribute MUST be used for all "m" lines covering visual media within a session description for which a foreground/background relationship is to be defined. The foreground/background relationship of visual media within a session description that is not covered in a group is undefined. No more than one group MUST be used within one session. If the identification-tags associated with "a=group" lines do not map to any "m" lines, it MUST be ignored.

This memo defines a new media-level attribute, "transparency", with the following ABNF . The identification-tag is defined in .

Alpha describes the transparency for the foreground media stream. It is identified by its transparency-tag values in the transparency-attribute. It could be an integer with values between 0 and 100. This is an informative value. Details of interpretion to be left open to the renderer, expect that a value of 0 means foreground media is opaque and value of 100 means that it is transparent.

The following example shows a session description for superimposed media stream in an SDP session. The "group" line indicates that the "m" lines with tokens 1 and 2 are grouped for the purpose of superimpositon and intended to be overlaid on top of a background video. In the example shown below, two superimposed media streams are being transmitted. Both media types are video with transparency attribute ("transparency"). The current focus of the draft is defining a group semantics for superimposed media stream. The relationship between the background and foreground media stream maybe defined in the future version of the draft.

Edt. Note: maybe we remove this section later once there is a general understanding why CLUE in its current form is unsuitable. The CLUE framework and its associated suite of I-Ds and RFCs describe a telepresence framework that, at the first glance seems to have a lot in common with the technology proposed herein. CLUE defines captures (camera ports), and their geo-spatial relationship to each other. A render can use this information to put the reconstructed samples of the streams from the various captures into a suitable arrangement such that visually pleasant rendering can be achieved. However, CLUE does not describe the relative relevance of the captures. For that reason, CLUE would need to be extended in a spirit very similar to the one described in this memo to achieve the desired functionality. CLUE has not seen wide deployment outside its intended key application (large room, multiple camera telepresence systems). It’s not reasonable to assume that small systems would willingly implement the overhead the (comparatively complex) CLUE protocols require when a simple SDP extension can serve the same purpose.

All security considerations as defined in apply: Using the "group" parameter with FID semantics, an entity that managed to modify the session descriptions exchanged between the participants to establish a multimedia session could force the participants to send a copy of the media to any destination of its choosing. Integrity mechanisms provided by protocols used to exchange session descriptions and media encryption can be used to prevent this attack. In SIP, Secure/Multipurpose Internet Mail Extensions (S/MIME) and Transport Layer Security (TLS) can be used to protect session description exchanges in an end-to-end and a hop-byhop fashion, respectively.

The following contact information shall be used for all registrations included here:

This document defines a new SDP group semantics for media superimposition for a SDP session. This attribute can be used by the application to group all the foreground media to be superimposed on a background media in a session. Semantics values to be used with this framework should be registered by the IANA following the Standards Action policy . This document adds a new group semantics and follows the registry group defined in . The following semantics needs to be registered by IANA in Semantics for the "group" SDP Attribute under SDP Parameters.

The "S" attribute is used to group different foreground media streams to be superimposed on a background media stream . Its format is defined in . The SDP media-level attribute "transparency" needs to be registered by IANA Semantics for "att-field (media-level only)". The registration procedure in applies.

SDP Attribute ("att-field (media level only)"): The IANA Considerations section of the RFC MUST include the following information, which appears in the IANA registry along with the RFC number of the publication. A brief description of the semantics. Token to be used within the "group" attribute. This token may be of any length, but SHOULD be no more than four characters long. Reference to a standards track RFC.