C# Camera API for Getting Video Frame

If you have a USB camera, how can you build a simple C# camera application on Windows 10? There are three options:  WIA (Windows Imaging Acquisition), DirectShow and MediaCapture. After trying some sample code that downloaded from CodeProject and GitHub, I got the conclusion: 1. WIA is not good because it does not support my webcam.  2. DirectShow can work well, but there is no C# API provided by Microsoft. You need to create a wrapper for C++ API. 3. MediaCapture class that designed for UWP apps provides C# APIs which provide low-level control over the capture pipeline and enable advanced capture scenarios. In this article, I want to share how to create a simple C# webcam app in which I can handle every preview frame myself.

Getting Preview Frames via C# Camera API

Although the reference page of MediaCapture class is informative, to quickly learn APIs, I prefer to get started with UWP sample code that provided by Microsoft. Press Ctrl+F to search for camera, you will see following samples:

If you want to add custom effects or functionalities (OCR, barcode etc.) for the camera, a common way is to register a callback function for receiving every preview frame. After running all samples, I found the project CameraFrames is what I want.

How to build a UWP webcam app in C#

1. To use webcam, double-click package.appxmanifest to check the corresponding capability:
   
2. Create an image element for rendering preview frames:
   
3. Initialize FrameRenderer with the image element:

   _frameRenderer = new FrameRenderer(PreviewImage);

4. Initialize the media capture object:

                   // Create a new media capture object.
                   _mediaCapture = new MediaCapture();
   
                   var settings = new MediaCaptureInitializationSettings()
                   {
                       // Select the source we will be reading from.
                       SourceGroup = groupModel.SourceGroup,
   
                       // This media capture has exclusive control of the source.
                       SharingMode = MediaCaptureSharingMode.ExclusiveControl,
   
                       // Set to CPU to ensure frames always contain CPU SoftwareBitmap images,
                       // instead of preferring GPU D3DSurface images.
                       MemoryPreference = MediaCaptureMemoryPreference.Cpu,
   
                       // Capture only video. Audio device will not be initialized.
                       StreamingCaptureMode = StreamingCaptureMode.Video,
                   };
   
                   try
                   {
                       // Initialize MediaCapture with the specified group.
                       // This can raise an exception if the source no longer exists,
                       // or if the source could not be initialized.
                       await _mediaCapture.InitializeAsync(settings);
                       _logger.Log($"Successfully initialized MediaCapture for {groupModel.DisplayName}");
                   }
                   catch (Exception exception)
                   {
                       _logger.Log(exception.Message);
                       DisposeMediaCapture();
                   }

5. Use DeviceWatcher to list all connected devices:

           var deviceSelector = MediaFrameSourceGroup.GetDeviceSelector();
           _watcher = DeviceInformation.CreateWatcher(deviceSelector);
           _watcher.Added += Watcher_Added;
           _watcher.Removed += Watcher_Removed;
           _watcher.Updated += Watcher_Updated;
           _watcher.Start();
   
           private async void Watcher_Added(DeviceWatcher sender, DeviceInformation args)
           {
               await AddDeviceAsync(args.Id);
           }
   
           private async Task AddDeviceAsync(string id)
           {
               var group = await MediaFrameSourceGroup.FromIdAsync(id);
               if (group != null)
               {
                   await _dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
                       {
                           _sourceCollection.Add(new FrameSourceGroupModel(group));
                       });
               }
           }
   

6. Updates the current frame source to the one corresponding to the user’s selection:

   _mediaCapture.FrameSources.TryGetValue(info.SourceInfo.Id, out _source);

7. Initialize MediaFrameReader and register a callback function:

               if (_source != null)
               {
                   _reader = await _mediaCapture.CreateFrameReaderAsync(_source);
                   _reader.FrameArrived += Reader_FrameArrived;
               }

8. Start reading webcam frames:

   MediaFrameReaderStartStatus result = await _reader.StartAsync();

9. Receive all frames and render them on image element:

           private void Reader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
           {
               // TryAcquireLatestFrame will return the latest frame that has not yet been acquired.
               // This can return null if there is no such frame, or if the reader is not in the
               // "Started" state. The latter can occur if a FrameArrived event was in flight
               // when the reader was stopped.
               using (var frame = sender.TryAcquireLatestFrame())
               {
                   _frameRenderer.ProcessFrame(frame);
               }
           }
   
           public void ProcessFrame(MediaFrameReference frame)
           {
               var softwareBitmap = FrameRenderer.ConvertToDisplayableImage(frame?.VideoMediaFrame);
   
               if (softwareBitmap != null)
               {
                   // Swap the processed frame to _backBuffer and trigger UI thread to render it
                   softwareBitmap = Interlocked.Exchange(ref _backBuffer, softwareBitmap);
   
                   // UI thread always reset _backBuffer before using it.  Unused bitmap should be disposed.
                   softwareBitmap?.Dispose();
   
                   // Changes to xaml ImageElement must happen in UI thread through Dispatcher
                   var task = _imageElement.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
                       async () =>
                       {
                           // Don't let two copies of this task run at the same time.
                           if (_taskRunning)
                           {
                               return;
                           }
                           _taskRunning = true;
   
                           // Keep draining frames from the backbuffer until the backbuffer is empty.
                           SoftwareBitmap latestBitmap;
                           while ((latestBitmap = Interlocked.Exchange(ref _backBuffer, null)) != null)
                           {
                               var imageSource = (SoftwareBitmapSource)_imageElement.Source;
                               await imageSource.SetBitmapAsync(latestBitmap);
                               latestBitmap.Dispose();
                           }
   
                           _taskRunning = false;
                       });
               }
           }

Reference

Basic photo, video, and audio capture with MediaCapture

Source Code

https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/CameraFrames

The post C# Camera API for Getting Video Frame appeared first on Code Pool.