Depacketizer for Track

[paullouisageneau]

It would be useful to introduce a media handler to depacketize the incoming RTP stream, with a jitter buffer to reorder packets and pace reassembled frames.

[Sean-Der]

@paullouisageneau I am going to start working on this.

Do you have any suggestions/rough sketch of an API you would like? I can get a PR together this weekend.

thank you!

[paullouisageneau]

Thanks! I've no very precise API in mind, I think something like a generic media handler implementing the jitter buffer part, an another one specialized per codec implementing the depacketization part would make sense.

[Sean-Der]

Hey @paullouisageneau I don't know the best way to do this. Can I get some guidance?

I don't see a way I can use RtcpReceivingSession because it is a MediaHandler.

Should I wrote something that allows a MediaHandler to be used as a MediaHandlerElement ?

Then I can contribute a JitterBuffer that is a MediaHandlerElement and chain them.

[paullouisageneau]

@Sean-Der I've merged #929 which refactors and drastically simplifies media handlers. It should solve this kind of issue, as now everything is a MediaHandlerand you can simply chain MediaHandler instances together. It keeps compatibility with user code as long as it doesn't implement a custom MediaHandler.

The methods are also simpler, taking a vector of messages that may be modified in-place by the handler.

[flying1314]

Hey @paullouisageneau I don't know the best way to do this. Can I get some guidance?

I don't see a way I can use RtcpReceivingSession because it is a MediaHandler.

Should I wrote something that allows a MediaHandler to be used as a MediaHandlerElement ?

Then I can contribute a JitterBuffer that is a MediaHandlerElement and chain them.

What is the current status of WHEP?@Sean-Der

[singpolyma]

https://github.com/paullouisageneau/libdatachannel/blob/master/include/rtc/rtpdepacketizer.hpp is mostly implementing this now, right? One think I'm curious about, once depacketized is it possible to still know what codec is in use so I know where to send it next?

[Sean-Der]

@singpolyma check out my code here Sean-Der/obs-studio@506ecb6

I accept + decode + render in OBS

[singpolyma]

@Sean-Der That looks very helpful, thanks! It does look like you're only advertising support for opus though, right? So you can just assume the packets have opus in them. Whereas if you advertise both PCMU and opus one needs some way to know what's actually being send. In the RTP packets we can look at the format number and compare to the rtpmap, but post-depacketize I'm not sure where that data is.

I'm also curious, you're using onFrame instead of onMessage is that a bit more low-level?

[singpolyma]

Answered some of my own questions and opened #1156 which I think is needed for codec detection.

[Sean-Der]

Sorry I missed this @singpolyma! I get so many GitHub notifications I tend to ignore them :/

I am on the discord instance, happy to chat anytime :)