[Delta] Codec negotiation

[Peter Saint-Andre]

Justin Karneges wrote:
> On Friday 07 November 2008 08:44:49 Peter Saint-Andre wrote:
>> The SHOULD is there for consistency with SDP/SIP, but I'll check on that
>> in the SDP and SIP specs. I'd prefer a MUST here, which would mean "Bob
>> needs to choose from the payload-types that Alice offers and is not
>> allowed to counter-offer with other payload-types".
> 
> Without knowing what to do with these unoffered payload types, XMPP clients 
> are just going to throw them out (that's what I would do..).  We may as well 
> make this a MUST then, right?  Unoffered payload types coming from SIP can 
> just be thrown out by the gateway.
> 
> If we want the SHOULD, then the XEP should explain what special case this is 
> meant for.

Right. So I'd prefer to make this a MUST, but I would like to
double-check SIP/SDP usage first just to be 100% sure.

>> As far as I know, the codec negotiation is kind of black magic. For
>> example Alice might offer Codec X and Codec Y at 320x240 but the exact
>> resolution might be negotiated via the signalling channel (SIP or XMPP)
>> for Codec X whereas it might be negotiated via the data channel (RTP)
>> for Codec Y. And so on. This is an area that I don't know much about
>> because it's part of the black art of media negotiation. However I could
>> contact folks who know more than I do for clarification.
> 
> I don't know if 'negotiation' is the right word here, since RTP is supposed to 
> be one-way (unless somehow you can negotiate codecs with RTCP?).  But I could 
> see a scenario where not all parameters are known at the time of signaling, 
> and so even something as important as resolution wouldn't be known until you 
> started receiving RTP packets.
> 
> In practice, I'm not sure when resolution wouldn't be known.  I believe Theora 
> requires width and height parameters to be signaled.  Are there any video 
> codecs that can be negotiated via SDP with an undefined resolution?

I don't know. All I do know from people like Rob McQueen at Collabora is
that codec negotiation is all over the map -- sometimes it happens in
the signalling channel, sometimes there are these funky things in RTCP, etc.

> I suppose it may also be possible to lie about resolution, by reporting one 
> value in the signaling and doing something completely different in the RTP.  
> Or perhaps initially the resolution in the RTP matches the signaling, but is 
> later changed to something else, but only in the RTP.  Then it's not quite 
> a "lie" but more like a change of mind..

Right. I'd prefer to force explicit negotiation up front and stick to
that, but that may not be doable in the real world.

>>> It may be possible for each party to use a different resolution, if they
>>> both agree on it.  Like, if Alice offers both 320x240 and 352x288, I
>>> think it might be possible that Bob could accept both types.  Then either
>>> side could choose what to transmit with.
>> My understanding is that in SIP/SDP Bob can use anything that Alice
>> offers (e.g., they could switch to a new codec halfway through the
>> session if desired), but that's more of the black magic in play. It
>> seems clearer to me if the parties agree explicitly on the codecs and
>> their parameters in the signalling channel and then if they want to
>> change things they would modify the content definition. This stuff about
>> implicitly allowing changes mid-stream feels dangerous to me. But as I
>> said I need to check on this with people who know more than I do about
>> voice and video.
> 
> Yes, my understanding is that Alice and Bob agree on the same payload-info 
> set, and if they agree to more than one payload-info of the same media type 
> (e.g. three video payload-infos), then either party can transmit using any of 
> the agreed upon payload-infos.  If there is a desire to transmit anything 
> else, then they'd need to use content-add.

Correct.

> This does mean that Alice could transmit using one video payload for awhile 
> and then later switch to another, without having to do any signaling.  I can 
> see changing the payload being useful if, say, the network gets worse and you 
> want to downgrade quality.  It seems like you could do just as well by 
> agreeing on only one payload per media type at a time, and then use signaling 
> to change the one payload to something else, but the nature of SIP/SDP 
> systems seems to be that you should be able to handle multiple at once.

Right, that's my understanding too -- you don't want to wait for
signalling to complete before dynamically downgrading or upgrading
codecs to respond to changing network conditions.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/