How to convert YUV420 video format input and use in media element (WPF - C#)

[kondareddy]

We are integrating Agora in WPF (C#) - Agora SDK gives use raw video frames - which we need to render on WPF screens.

SDK Documentation Link.

https://docs-preprod.agora.io/en/video-legacy/API%20Reference/csharp/API/rtc_api_data_type.html#enum_videoframetype

We are searching for library which takes YUV420 format and give use stream, which can be used as media element source.

https://learn.microsoft.com/en-us/dotnet/desktop/wpf/graphics-multimedia/how-to-control-a-mediaelement-play-pause-stop-volume-and-speed?view=netframeworkdesktop-4.8

[brabebhin]

Hi,

Unfortunetely WPF media element is quite limited in this regard. Your only hope would be to implement a media source with old MF interfaces.

https://learn.microsoft.com/en-us/windows/win32/api/mfidl/nn-mfidl-imfmediasource

However, if using MediaElement is not a requirement, you can use the MediaPlayerElement wrapped control for desktop applications

https://learn.microsoft.com/en-us/windows/communitytoolkit/controls/wpf-winforms/mediaplayerelement

The MediaPlayerElement is much more extensible.
You can also skip the agora SDK and use this SDK- we can handle most audio and video streams out of the box with a rather simple API. My recommendation is to do so: using C# for media playerback typically doesn't ensure a smooth experience, due to GC delays. Our SDK is full native C++ (you can use it safely from desktop C# with a nuget package)

If the above is not good enough, you can use the winRT MediaPlayer in frame server mode, with a custom built MediaStreamSource of your own. You will need to handle rendering to a DirectX surface though

https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/play-audio-and-video-with-mediaplayer#use-mediaplayer-in-frame-server-mode

https://msdn.microsoft.com/library/windows/apps/windows.media.core.mediastreamsource

Windows might be able to render the YUV directly if you provide the proper encoding properties in the sample metadata.

However, I stromgly recommend you use ffmpeginteropx instead :)

[lukasf]

If you can use MediaPlayerElement (either by creating a WinUI app, or by hosting WinUI MediaPlayerElement in a WPF XamlIsland), then you should be able to easily render your YUV420 video using MediaStreamSource, without any additional frameworks (also without FFmpegInteropX).

You just need to create VideoEncodingProperties with MediaEncodingSubtypes.Iyuv, and create a MSS from that, somehow like this (minimal setup):

var descriptor = new VideoStreamDescriptor(VideoEncodingProperties.CreateUncompressed(MediaEncodingSubtypes.Iyuv, witdth, height));
var mss = new MediaStreamSource(descriptor);
mss.SampleRequested += ...
var mediaPlaybackItem = new MediaPlaybackItem(mss);

In the SampleRequested event handler, create a MediaStreamSample, fill the Sample's Buffer with the raw YUV420 data and set Timestamp, and set the sample to the eventArgs. Now you should be able to assign the mediaPlaybackItem to MediaPlayer.Source (if you want to use MediaPlayerElement with MediaPlayer, which is recommended), or directly to MediaPlayerElement.

Note that it is recommended to handle SampleRequested in native code, to avoid delay due to GC. So recommendation would be to do this in a C++/WinRT or C++/CX component, and use that from C#. But you can first try easily in C# to see if it generally works with your data. If you want to use it and want 100% stable frame rates, consider moving to native code.

Feel free to check our code for comparison, especially UncompressedVideoSampleProvider and FFmpegMediaSource.

[brabebhin]

There is no work around this. You need to decide if the GC freezes are a blocker for you. This is why we use c++ for the interop code.

[brabebhin]

The more complicated answer... The buffer doesn't get consumed immediately after you free it from the async queue. It gets consumed when the sample is rendered. But this is not the biggest problem.

The biggest problem is the way the interop works for c#. C# uses nondeterministic generational mark&sweep GC, winRT uses deterministic ref counting for memory management. In order to keep the object valid in c#, the interop layer will keep a reference alive in the winRT world, so that the winRT object doesn't self destruct. That reference isn't readily available to developers to free manually, so you have to wait for GC to collect it. The IBuffer interface doesn't provide IClosable interface, so you can't Dispose it manually, meaning the memory remains there until GC runs.

To make things even worse, the GC doesn't automatically know of the memory allocation in winRT world, so it won't know it's under memory pressure (you can add the memory pressure manually but that's error prone).

In c++ this is a non-issue, since the ref counted objects in winRT don't have hidden refs and they get to self destruct properly when they are no longer used.

[lukasf]

True. This is the reason why media engines are usually written in native code.

If the SDK allows you to provide the buffers for incoming frames, you might be able to improve things by using a buffer pool. Then instead of allocating a new byte array for every frame, you could get one from a pool (e.g. ArrayPool.Shared), and return it when the MediaStreamSample.Processed event comes. You'd need a lookup to map sample to buffer, so you know which one to return once a sample was processed.

But if this is not possible with the SDK you are using, then your only chance is to either live with GC issues, or do the sample handing in native code. Agora seems to have a native Windows SDK, so this would be possible. The rest of the app can stay in C#, only the raw data handling would have to be in a RuntimeComponent, which can then be used from the C# app, similar to how we do in our lib.

[brabebhin]

In my experience buffer pools don't help. When you marshal the IBuffer from c# to winRT, the data is actually copied by value between the two worlds. So even if you may be pooling the winRT memory, the managed memory still gets to create extra garbage, essentially a copy of the IBuffer.

The IBuffer interface and its interop with c# is simply not designed to handle large data transfers in real time.

[lukasf]

The IBuffer interface is a pure COM interface, so it will not be copied when handing it from C# to WinRT. I am not sure about the C# array to IBuffer conversion. When looking into the the AsBuffer() extension method, it looks a bit as if this will actually wrap the existing array without copy. But only part of the implementation can be seen in the disassembly, so I might be wrong here. Still, even if the buffer is copied from C# array to IBuffer, two pools could be used, one with IBuffers and one with byte arrays, and then you copy the memory yourself and re-use both the IBuffers and the arrays. Of course, this is more work. The best solution is always to do this kind of stuff in native code and avoid all these pitfalls.